Section 11: Climb the Ladder Before It Burns - Failure, Repair & Disaster Recovery
Executive Summary: This section takes the hardware away one rung at a time: a disk (five a day, rebuilt in about an hour, hidden behind degraded reads), a node, a rack, an availability zone (every stripe readable from its 10 surviving fragments, writes continuing at reduced margin, a third of the index re-homed in 10 seconds), a partition server, the control services, and the region, where the honest answer is asynchronous replication with an RPO of minutes. The bottom of the ladder is independent failure, already crushed to 10^-17 per year; the top is correlated failure, which is what actually spends an eleven-nines budget of about 100 objects a year. A recovery path that has never run is a hypothesis, so all of it is drilled on game days.
The interviewer puts the marker down. "Section 10 broke the design one tenant at a time. Now I take hardware away. Start small and keep going until you reach something you can't survive."
You: "I'll climb a ladder, and at each rung say what the design does, the number behind it, and what Priya sees. The bottom rungs are independent failures, which Section 7 priced at about 10^-17 per year across the whole fleet. The top rungs are correlated. Those spend the budget."
The Ladder, Rung by Rung
Here is the ladder as a flow from component to response to what the customer experiences.
| Rung | Scale | Response | Priya sees |
|---|---|---|---|
| Disk | 5/day, 100 TB/day to rebuild (N9) | Detected within 15 min; the 250-node cluster rebuilds at ~6 GB/s, one 20 TB disk in about an hour. Reads hedge to parity after 20 ms (D14) rather than wait for the dead disk | Nothing |
| Storage node | 36 disks | Same pipeline, about a day and a half at the cluster's 6 GB/s, stripes missing the most fragments first. A node never holds two fragments of one stripe | Nothing |
| Rack | Many nodes | At most one fragment per rack per stripe, so a rack is a disk from the stripe's point of view | Nothing |
| Partition server | ~40 partitions | Lease expires, the map reassigns the ranges, the new owner replays the log tail, under 10 s (N7). A gray server is moved proactively (Section 10) | A few retried requests |
| Partition map service | One service | Front-ends keep serving on the cached map, Section 8's static stability (D15); splits and moves pause | Nothing |
| Stripe manager | One per cluster | Open stripes keep accepting; new allocations stall, so each cluster keeps a pre-allocated pool of empty stripes | Nothing for the pool's lifetime |
Key Insight: Every rung below the AZ is the same repair pipeline at a different size. A system that handles five dead disks a day as routine has been tested five times a day.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson