Section 10: When Forty Machines Vanish at Once - Reliability & Operations: Rack Loss, Master Loss & a Shared Cluster

Executive Summary: Section 8 survived one dead worker. Here the failures get bigger. When rack 10 loses its switch at minute 4, 40 machines vanish together: about 1,800 completed and 80 in-progress map tasks re-run as about half a wave on the remaining 3,920 map slots, 80 reduce tasks restart in map slots freed when the map phase ends, and rack-aware GFS placement keeps every split readable. The job lands under a minute late, inside the 15-min SLO. When the master dies, the job aborts and is resubmitted. Dying at minute 6 loses 6 min and the rerun finishes right at the SLO; the worst case loses about 8 min. Master death hits about 0.006% of jobs, and abort-and-resubmit is the right trade until jobs run for hours. Then the cluster view: ~1,000 jobs a day, one master each, a cluster scheduler, preemption made cheap by re-execution, and status pages and counters that let Riya run the job alone.

Interviewer: "One machine dying is easy. A top-of-rack switch fails and you lose forty at once. Walk me through it, minute by minute."

You: "Good, because correlated failure is where a design that only sounds fault-tolerant falls apart. Let's take rack 10. It holds w-0417, the machine that was running Riya's split 88,213."

Rack 10 Goes Dark at Minute 4

At minute 4 the inlinks job is about three minutes into its map phase when rack 10's switch dies. To the master, 40 machines just stop answering heartbeats. It can't tell crashed from cut off, and the design never needs to.

At minute 4 each rack-10 machine has finished about 45 map tasks (3 minutes × 2 slots ÷ 8 s), so ≈ 1,800 completed maps go back to idle because their output sits on unreachable local disks (Decision 3's bill, as Section 8 promised), plus 80 in progress. Unstarted tasks were never assigned to rack 10, so the surviving slots simply pull them as before. The re-runs are about half a wave on the remaining 3,920 map slots, and the cluster is 2% smaller for the rest of the map phase, which costs a few seconds. Had the rack died at the end of the map phase, all 40 × 78 = 3,120 would re-run on the critical path: one 8-s wave plus 15 s of detection.

GFS makes the re-runs possible. Split 88,213 had replicas on w-0417 (rack 10), w-1123 (rack 28) and w-1160 (rack 29), because rack-aware placement never puts all replicas on one rack. The master reschedules it on w-1123, still replica-local. Locality survives losing a whole rack.

The 80 restarted reducers are the bigger cost. Every surviving reduce slot already holds a reducer, so they start in map slots freed when the map phase ends and re-fetch 250 MB each, ≈ 20–40 s at the cross-rack share, before their normal 1.5 min of sorting and reducing. The map re-runs themselves cost seconds, because they ran in parallel mid-phase. The same loss during the reduce phase would add up to ≈ 1.5 min, still inside budget. The 2004 paper offers a sanity check: its sort benchmark with 200 worker processes killed mid-run took 933 s instead of 891 s, only 5% longer. That was killed processes on healthy machines, which the cluster scheduler restarted at once, so it's a lower bound on a rack loss, not a replica of one.

Interviewer: "You said the machines might still be alive behind the dead switch. What if the switch comes back and those zombies report in?"

You: "Section 8's atomic commits handle it. A zombie map's completion is for a task already reassigned; the first completion wins and the duplicate is ignored. A zombie reducer can only finish by renaming its temp file in GFS, deterministic code means both attempts hold the same bytes, and the atomic rename leaves exactly one part-NNNNN. No fencing tokens, no membership protocol. 'Dead' and 'unreachable' get the same treatment, safely, because every commit is idempotent."

Key Insight: Re-execution scales from one failure to forty with no new mechanism. Fine-grained tasks (Decision 5) spread the lost work cluster-wide, rack-aware replicas keep input readable, and atomic commits defuse zombies. A rack costs half a wave and a re-fetch, not a rerun.

Quick check before moving on: