Section 11: Twenty Seconds to Heal - Reliability: Failure Detection, Recovery & Disaster Recovery
Executive Summary: A 99.99% cell has 8.64 s of unavailability to spend each day, and Chubby alone takes about 4.1 s of it. Tablet servers die about five times a day. Each death costs ~20 s for 0.1% of the keyspace: 12 s for Chubby to prove the server dead, ~3 s to sort its 4 GB commit log in 64 parallel segments, and ~3 s for ~100 new owners to replay their slices. That adds up to **0.1 s/day, 1% of the budget**. A master failover takes ~20 s and stops no read or write. Two log-writer threads ride out slow GFS pipelines, sequence numbers remove the duplicates at recovery, and losing the whole cell falls to asynchronous cross-cell replication with an RPO of seconds.
Interviewer: "It's 10:00:00 and the tablet server holding com.cnn.www just lost power. Walk me through the next minute."
You: "It's twenty seconds, not a minute, and only for one tablet in a thousand. Before I start the clock, let me show where it sits in the budget, because that decides which failures deserve engineering."
The Budget Before the Clock
| Failure | Why it costs | Per day | Share of 8.64 s |
|---|---|---|---|
| Chubby unavailable | Local leases lapse, so every server stops serving (sessions expire only past 57 s) | ~4.1 s (the paper's 0.0047%) | ~47% |
| Tablet-server loss | ~5/day × ~20 s × 0.1% of keys | ~0.1 s | ~1% |
| Planned moves | Two minor compactions | < 1 s per move, for one tablet in a million | Negligible |
| Master failover | Off the data path | 0 s for assigned tablets | ~0% |
| Everything else | Bugs, bad deploys, corruption, partitions | ~4.4 s left | ~52% |
The Chubby row is an average over 14 clusters. The paper's worst cluster lost 0.0326% of server-hours, about 28 s a day, which is more than three times our entire budget.
Key Insight: Server deaths are frequent but cheap, and Chubby outages are rare but take down the whole cell. A candidate who spends the hour making tablet failover faster is polishing 1% of the budget. The two items that matter are keeping Chubby healthy and the ~4.4 s left for everything nobody planned for.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson