Section 10: The Server Died Mid-Move - Reliability, Failure & Recovery

Executive Summary: We climb a failure ladder from one game server to a whole region. The centrepiece is a game-server crash: its node lease (TTL 3 s, heartbeat 1 s) expires, placement re-homes its ~5.6K games across ap-south's 5 survivors (~1,100 each) and bumps owner_epoch from 1 to 2, and gateways resend held moves with both stamps. The game resumes in under 5 s with zero acked moves lost, and a zombie waking from a GC pause is fenced by the same conditional write that orders moves.

The interviewer picks up the eraser. "Your design works. Now I'm going to break it. Priya and Omar are mid-game on gs-aps-05, and that box just lost power."

You: "Ladder first, then that rung in depth: it's the one players feel."

Rung One: The Game Server Crash

Size it on our own game's cell. Each of ap-south's 6 nodes carries ~5.6K games, so one crash orphans 5.6K and each of the 5 survivors takes **1,100**: well under the 10K cap, which we set for exactly this moment. The busiest region spreads them thinner: ~510 each across 11.

Here is g_8Kd2. Priya's Re1 at ply 23 is durable and acked. Omar plays ply 24; his gateway stamps arrived_at and forwards it to gs-aps-05, which dies before writing it.

Walk the clock:

1. Detect. gs-aps-05 renews its etcd lease every 1 s; the last renewal was at most 1 s before the crash, so the 3 s lease expires 2–3 s after it.
2. Re-home. The Placement Service sees the expiry and rewrites each orphan's entry: g_8Kd2 → (gs-aps-03, epoch 2).
3. Load. gs-aps-03 reads one live_games item per game and makes a takeover write: SET owner_server = gs-aps-03, owner_epoch = 2 IF owner_epoch = 1. About 1,100 per survivor is trivial next to ap-south's ~5.6K writes/s.
4. Resend. Gateways subscribe to placement changes through the Placement Service (an etcd watch behind it); a stale cache still gets NOT_OWNER and re-resolves. Omar's gateway kept ply 24, as it keeps every forwarded move until its move event returns, and resends it to gs-aps-03.
5. Accept with the original stamps. The resend carries sent_at, arrived_at and Omar's sRTT, so the new actor charges him arrived_at − sent_at − credit from stamps taken before the crash. Priya's clock starts only when her gateway delivers ply 24. No clock froze; the outage falls between nobody's two stamps.

Key Insight: Timekeeping survived because it never lived on the machine that died. Both gateways kept their stamps and their flag deadlines through the crash, and the new actor re-arms its backstop timer from the record. Section 6's tier split is what makes this failover fair, not just fast.

The edge case: gs-aps-05 wrote ply 24 but died before pushing it. gs-aps-03 then loads ply 24, treats the resend as an identical retransmit, and replays the move event as the ack. Either the record has the move or the gateway does, and because we persist before we broadcast, nobody saw a move that isn't in live_games: zero acked moves lost.

Quick check before moving on: