Section 6: One Brain Per Job, Two Thousand Hands - High-Level Architecture
Executive Summary: Five kinds of box: a user program linked with the library, one master per job holding Section 5's ≈ 624 MB of task and region tables, workers with 2 map slots and 2 reduce slots on all 2,000 machines, a GFS chunkserver beside every worker, and local disks for the 1 TB of intermediate data. The organizing rule is that the master moves only metadata. Bytes flow from GFS to a map worker on the same machine, then to local disk, then over RPC to a reducer, then back to GFS. We walk Riya's split 88,213 from w-0417 to /gfs/inlinks/part-01729, then defend why each job gets its own master.
The interviewer taps Section 5's task table. "You've told me what the master remembers. Now put the machines on the board and walk one of Riya's records from the crawl to the output file."
You: "Five boxes, then the execution flow in order. One thing to watch: the master never touches a data byte. It decides where work runs and who to tell when it finishes, and that's all it does."
The Boxes on the Board
| Box | Count | Holds | Refuses to do |
|---|---|---|---|
| User program + library | One per job | Riya's ~40 lines: map, reduce, job spec |
Touch partitioning, retries or placement |
| Master | One per job | Task, worker and region tables (≈ 624 MB, mostly the M × R region table) | Carry a single data byte |
| Workers | 2,000 machines × (2 map + 2 reduce slots) = 4,000 map slots + 4,000 reduce slots | Running map or reduce tasks | Decide what to run next |
| GFS chunkservers | One on every worker machine | Input and output chunks, 3 replicas | Know about tasks |
| Local disks | 2 × 160 GB per machine | Intermediate regions (≈ 500 MB per machine for this job) | Replicate anything |
Row four is the most important structural choice on the board. Every worker is also a GFS chunkserver, so the 10 TB input already sits on the machines that will compute over it (Decision 4). We reject separate compute and storage clusters because Section 3 put map demand at 32 GB/s against a 25 GB/s root, so without locality the job is network-bound. That flips only on full-bisection, non-oversubscribed networks.
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson