Section 11: The Bad Week - Reliability, Failure & DR

Executive Summary: Section 10's challenges were single punches; this section is the sustained beating — a node dies at peak, a rack loses power, and leadership asks about losing the whole data center. We walk Mira's node-death timeline end to end (suspicion, hints, declaration, replacement, verification), add rack-awareness to the preference list, price a DC failover honestly in RPO and RTO, design backups for a store that never stops mutating, and finish with the four dashboards that turn every mechanism from Sections 5 through 9 into a number on a screen. The 401 lesson: reliability is not a component you add — it is the proof that the components you built compose into a system that survives its own operations.

It's Tuesday, 14:02, traffic at peak — roughly 1.2M reads/s and 300K writes/s — when node n17's kernel panics mid-write to cart:8842. The checkout team notices nothing, and this section explains why as a timeline, not a list. Interviewers love this question because it cannot be answered with a component name: it demands that you replay your own design under stress, in order, with numbers.

The Death of n17, Minute by Minute

T+0s — the crash. Writes in flight to n17 fail. Coordinators don't care: with N=3 and W=2, Section 6's quorum machinery acks every write off the two surviving replicas, and the sloppy quorum parks n17's copies as hints on n42. Reads don't flinch either — R=2 of 3 was always a race, and n17 just stopped entering it.

T+10s — suspicion. Gossip rounds run every second, and phi-accrual failure detection marks a silent node suspect in about ten seconds; the verdict propagates fleet-wide in about six gossip rounds — six seconds across 64 nodes. Note the word: suspect, not dead. Phi-accrual emits a rising confidence score, not a binary ruling, because a node in a 30-second GC pause looks exactly like a corpse.

T+10s to T+3h — hints accumulate. Every write that would have landed on n17 goes to a healthy node tagged "deliver to n17 on return." Hinted handoff covers outages up to three hours — long enough for a reboot, a GC pause, or a bad deploy to resolve itself with zero data movement.

The operator decision — wait or replace. Here candidates reveal whether they've operated anything. The tempting design — node suspect, trigger re-replication — is the alternative Section 6 rejected, and this is why: a transient network blip that makes five nodes look dead would launch five simultaneous multi-terabyte re-replication storms, converting a false alarm into a real outage by saturating the very network the cluster needs to recover. Suspicion only diverts traffic; replacement waits for a human (or a very patient automation: dead past the 3-hour hint window, hardware confirmed gone). The flip trigger: at ten times the fleet, human-in-the-loop stops scaling and you build automation with hysteresis — earned with false-positive data first.

Replacement — streaming, not copying. n17's replacement claims its 256 virtual-node tokens and streams its ~4.6TB share from dozens of peers in parallel — Section 5's virtual nodes paying their rent, because a plain ring would drain the full 4.6TB from one overloaded neighbor. Parallel streams finish in hours, not days, and no single source feels it.

Verification — trust, then verify. Parked hints replay, and Section 9's Merkle anti-entropy pass diffs the newcomer's vnode ranges against its replica peers, proving cheaply that nothing fell through the cracks — within days, unasked, on the continuous repair cadence.

Key Insight: Every layer in this timeline owns a different time horizon — quorums mask the failure in milliseconds, hints bridge it for hours, Merkle repair certifies recovery over days. A candidate who names which horizon each mechanism owns has internalized the design; one who says "replication handles it" has recited it.

Quick check before moving on: