Section 3: The Network Is the Budget - Scale Estimation & Workload Analysis

Executive Summary: Riya's nightly inlinks job reads 10 TB (1 B pages × 10 KB). Cut into 64 MB splits, that is M ≈ 156,000 map tasks, which run as 39 waves of 8 s on 4,000 map slots, so the map phase takes ≈ 5.2 min. The maps emit 1 TB of intermediate data, which hashes into R = 4,000 partitions of 250 MB, each small enough to sort in memory. The whole run is ≈ 8 min against a 15-min SLO. The real finding: maps want 32 GB/s of input, local disks supply 120 GB/s, and the network root only 25 GB/s, so locality is mandatory, not an optimization.

The interviewer taps Section 2's requirements: "Eight minutes nominal, fifteen with a failure. Show me that's possible. Then tell me what runs out first." The second sentence is the real question.

The Machines We Agreed On

You: "First, the assumptions. These are our interview's numbers, chosen to be plausible for 2004. They aren't Google's real figures, though the 2004 paper's benchmark cluster looked much the same: about 1,800 machines with 4 GB of RAM, two 160 GB disks, gigabit Ethernet and a two-level tree network."

You: "Look at the last two rows. Forty 1 Gbps machines share a 4 Gbps uplink: 10:1 oversubscribed. When everyone sends across racks at once, each machine gets 500 MB/s / 40 = 12.5 MB/s, a tenth of its NIC and a fifth of its own disks. And every machine runs a GFS chunkserver, so the input already sits on these disks."