Section 10: Five Seconds Without a Leader - Reliability: Failover, Zone Loss & Operations

Executive Summary: This section breaks the healthy ensemble on purpose. A leader crash costs ≈ 5 s with no writes: 4 s to notice, 0.5 s to elect, 0.5 s to synchronize. Sessions survive because nothing expires while there is no leader and the new leader restarts their timers, and because the outage is shorter than the 10 s timeout, so clients never have to give their sessions up. Losing zone A leaves S3, S4 and S5 holding quorum, but read capacity falls to 120,000/s, exactly our peak, which moves the observer decision from "later" to "now". Slow disks, long JVM pauses and expiry storms are the failures that actually hurt. Underneath sit routine operations: snapshot purge, one-at-a-time restarts and four metrics that warn you early. All figures are our interview's assumptions.

Interviewer: "Your design works on a good day. Kill the leader. What happens, second by second?"

You: "S1 is the leader, the zxid is 0x5_0000a3f1, and Sam's sch-2 is connected to S3. Here's the clock."

The Leader Dies: A Five-Second Timeline

With a tickTime of 2 s and a syncLimit of 2 ticks, followers wait 4 s before declaring the leader gone. Section 8's election takes about 0.5 s: S2 and S3 tie on zxid 0x5_0000a3f1, and S3 wins on the higher server id (a tie-break detail of the fast-leader-election implementation, not a protocol rule). DIFF synchronization, which hands the lagging S4 and S5 the missing a3f1, takes another 0.5 s. (The session timeout itself must fall between 2 and 20 ticks, 4–40 s by default; our 10 s sits inside that.) Total: about 5 s with no writes, and by default no reads either, because a server without a leader stops serving clients. ZooKeeper later added an optional read-only mode for partitioned servers; treat it as an opt-in for clients that tolerate stale reads, not a default.

Sam's sch-2 sees Disconnected while S3 is LOOKING, then reconnects, and Section 7's rule applies: only a server at least as current as the client's last zxid accepts the session. In this scenario, with no client paused, sch-1's ephemeral n_0000000041 is never touched, and the session clock is why.

Key Insight: Sessions outlive the failover for two reasons. On the ensemble's side, with no leader nothing expires, and the new leader restarts every session's timer instead of charging the outage to the clients. On the clients' side, the outage is shorter than the timeout. A client that stays disconnected longer than its timeout must assume its session is gone (Section 9, and Curator's default behaviour). If failover were slower than the timeout, all 20,000 clients would abandon their sessions, re-create 200,000 ephemerals and re-run election inside every one of the 2,000 services. That reconnect stampede against a just-elected leader would expire real sessions too. The gap between those two numbers is the firewall.

Interviewer: "Five seconds is long. Drop syncLimit to one tick and fail over in three."

You: "Faster detection means more false detections: a 2-second GC pause or network blip on the leader would trigger an election that costs the same five seconds anyway. And the budget doesn't need it. 99.99% is about 52.6 minutes a year, room for roughly 630 five-second failovers. My flip trigger is failovers measurably eating that budget, and even then I'd fix what's killing leaders before shaving detection."

Quick check before moving on: