Section 9: The Last Task Sets the Clock - Deep Dive: Stragglers, Skew & Bad Records

Executive Summary: A job ends when its last task ends, so the tail sets the job time. Three tail causes look identical on a status page. A slow machine (w-1555 at 1/4 speed: 8-s maps take 32 s, a reduce about 6 min) is fixed by backup tasks once a phase has no idle tasks left, for ≤ 3% extra compute, and the job ends near 9.5 min instead of ≈ 12.5. A hot key (100 M inlinks, 10 GB, 40× the average partition) is deterministic, so backups and a bigger R both fail; we salt it into 64 sub-keys of about 156 MB, which readers treat as one list. A bad record that crashes the parser is skipped after 2 failed attempts, and the job fails past 1,000 skips.

Interviewer: "Your status page shows the inlinks job at 99% for six minutes. What's going on?"

You: "Three different diseases share that symptom, and they need three different cures. Either a machine is slow, or one partition is enormous, or one record kills every attempt. The first move is diagnosis, because the cure for one does nothing for the others."

Diagnosis First: Is It the Machine or the Key?

The test is simple: does the slowness follow the machine or the work? Move a straggler's task to a healthy machine and it runs at normal speed. Move a hot key's task and it is just as slow, because the work is the problem.