Section 10: Ten Seconds Without a Leader - Reliability: Broker Loss, Zone Loss & Operations

Executive Summary: Section 8 proved that an acknowledged write survives a broker's death. This section puts a clock on it. When broker 12 crashes, the controller notices after a session timeout of about 9 s, elects new leaders from the ISR in under 1 s, and the 200 partitions broker 12 led (1/30 of the cluster) go about 10 s without a leader while producers buffer and retry. A failed disk takes out only its own partitions. Losing zone B takes 10 brokers and leaves about 2,000 partitions without a leader for the same ≈ 10 s, but every partition keeps 2 of 3 replicas. Where both survivors were in sync, that is exactly min.insync.replicas, so acks=all writes continue. A second failure in another zone is where writes stop, by design. The rest is operations: moving data without starving clients, restarting without the 10 s gap, and the dashboard.

Interviewer: "You keep saying an acknowledged write survives a broker loss. Walk me through the ten seconds after broker 12's power supply dies."

You: "Second by second. Then a disk, then a zone, then an ordinary Tuesday."

Broker 12 Crashes: The Timeline

Detection dominates the window. Brokers heartbeat to the active controller, and in recent releases the controller fences a broker that has been silent for about 9 s. That default has moved between releases, so treat it as an assumption. Election and propagation take under a second: the controller picks a surviving ISR member for each of the 200 partitions and appends the change to its metadata log.

Interviewer: "Nine seconds is an eternity. Set the timeout to one second."

You: "Then every long GC pause or brief network blip moves 200 leaders, and moves them back when the broker reappears. A false positive costs more than the extra eight seconds, so I tune detection to the slowest healthy pause, not the fastest crash."

During the gap, partition 37 accepts no writes. Ari's producer buffers about 8 MB (10 s at the partition's peak 0.8 MB/s) and retries with backoff; its delivery timeout must comfortably exceed the failover window. ful-7 sees nothing new on partition 37, so fulfillment's 100 ms end-to-end p99 is missed on 1/30 of partitions for about 10 s.

Broker 3 loses nothing because it was in the ISR, so it holds every offset below Section 8's high watermark, through 903. Anything only broker 12 had was never acknowledged under acks=all; Ari's producer resends it, and Section 9's producer id and sequence numbers make the resend safe even if part of it landed.

Key Insight: A single crash costs 10 s on 1/30 of the partitions, about a third of a second of whole-cluster unavailability. A 99.99% target allows roughly 52 minutes a year. Single-broker failures barely touch it; correlated failures and bad operations spend it.

Afterwards, broker 12's 600 replicas (200 it led, 400 it followed) leave their partitions under-replicated at ISR 2. If it reboots, it truncates any divergent tail using leader epochs, catches up, and rejoins the ISR. If the hardware is gone, a replacement broker in zone B (keeping one replica per zone) copies its roughly 30 TB, 907 TB over 30 brokers, under a throttle. At a 250 MB/s throttle that's about a day and a half at ISR 2. The throttle trades exposure to a second failure against client latency.

Quick check before moving on: