Section 10: Machines Are Weather - Reliability: Placement, Re-replication, Integrity & Garbage Collection
Executive Summary: With ~2 chunkserver failures a day, the fleet is always repairing itself somewhere. Four mechanisms keep three healthy replicas alive: rack-aware placement, prioritized re-replication (a machine restored in 30 minutes, a two-machine overlap ~1 min after the second death), 32-bit checksums per 64 KB block verified before any byte leaves a chunkserver, and lazy garbage collection that reclaims 1.4 PB raw a day for a standing cost of 3.4% of the fleet.
Interviewer: "Sections 8 and 9 assumed three good replicas. Two machines died this morning and a disk is quietly rotting. Who keeps those replicas alive?"
You: "The master, from heartbeats, in the background, never on a client's critical path. I'll follow a chunk's life: born, healed, checked, collected."
Where Replicas Are Born
Following the 2003 paper, the master places a new chunk's three replicas by three rules:
| Rule | Why | What it costs |
|---|---|---|
| Prefer chunkservers with below-average disk utilization | Equalizes fill over time | Nothing |
| Cap recent creations per chunkserver | A new chunk predicts heavy writes; an empty new machine would otherwise attract them all | New machines fill more slowly |
| Spread the 3 replicas across 3 racks | Survives a whole rack; reads draw on three racks' aggregate bandwidth | Two of three copies cross rack uplinks on every write (~52 GB/s raw at peak, per Section 3) |
Our chunk 0x7A3F lives on cs-0412 (rack 12), cs-0788 (rack 27) and cs-0153 (rack 05).
Rejected: one rack for all three makes writes cheap, but one switch then takes every replica down. 2× replication saves a third of the disk but leaves a chunk at one copy for the whole repair window after any single failure. Flip trigger: if the oversubscribed uplinks (~1.5:1) ever made cross-rack write traffic the bottleneck, I'd move to HDFS's default, two replicas in one rack and one elsewhere: it still survives a rack loss, but only 2 racks hold each chunk instead of 3. Sealed, cold data later moves to RS(6,3) erasure coding (Section 12).
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson