Section 10: When the Referee Falls Over - Master Failover & Disaster Recovery
Executive Summary: This section kills the lock service's own master and follows the ny1 cell second by second: the master lease runs out within 10 s, an election takes about 1 s, recovery takes about 1 s, and a new master is serving at about 12 s typical, 30 s at p99. Clients ride through on 45 s of grace, the new master hands every recovered session a fresh 60 s lease, and 100,000 jittered session-resume RPCs, validating caches rather than flushing them, arrive at 40% of the master's ceiling instead of twice it. Then we climb the failure ladder to a whole cell (RPO 1 h) and end on the outages that actually happen: bad pushes, greedy tenants and dependency cycles.
Section 9 paused a client. The interviewer now goes after the thing every client depends on.
Interviewer: "The ny1 master's machine loses power. 100,000 sessions, 24,000 of them orders-db. Walk me from the crash to normal service, and tell me what orders-db sees."
You: "Orders-db should see two events, jeopardy and then safe, and nothing else. Its leader, master-c since Section 9's GC pause, keeps lock generation 42 throughout. Timeline first, then how the new master earns that."
The Timeline, Second by Second
Section 7 covered the election and the master lease, and Section 8 covered jeopardy and grace. Here they run together on one clock.
| Phase | Duration | What happens |
|---|---|---|
| Master lease expires | ≤ 10 s | Followers refuse to vote until the lease they promised has run out (D5). No one can be elected early, so two masters never overlap |
| Election | ~1 s | A randomized timeout fires, an up-to-date follower wins 3 of 5 votes, and the epoch goes up by one |
| Recovery | ~1 s | The database is already in RAM on every replica. The new master commits the uncommitted tail and rebuilds sessions and locks from replicated state |
| Serving | ~12 s typical, p99 ≤ 30 s | Any replica now names the new master (D13) |
On the client side, each library's local lease expires 0–12 s after the crash, so even the unluckiest client has at least 45 s of grace against a 30 s p99 failover. Orders-db's leader, following Section 8's contract, stops issuing tablet reassignments while in jeopardy. Its lock was never released, because lock state is replicated, so on the safe event it carries on with the same sequencer.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson