Section 12: The Model Outlived the Machine - Follow-up Feature Evolution: From MapReduce to Hadoop, Spark & Dataflow
Executive Summary: Three feature requests show up once the framework is a success: chain my jobs, iterate my job, and answer me interactively. Our design absorbs the first badly, strains under the second, and refuses the third outright. The decade after the 2004 paper is the record of those refusals being answered. Hadoop rebuilt the same design in the open and flipped the reduce-side sort (Decision 7). YARN brought back the per-job master as a per-application master. FlumeJava loosened the restricted model from above (Decision 1). Spark kept re-execution but moved intermediate data into memory, guarded by lineage (Decisions 2 and 3). Dataflow and Apache Beam folded batch and streaming into one model. Each change fired a flip trigger we wrote down earlier, and that is the point of this section.
The interviewer leans back. "Say this ships and every team adopts it. Five years go by. What do they ask for that you can't give them, and what does the industry build instead?"
You: "Riya's team is the first to complain, so I'll use them. Their inlinks job is one step in a ranking pipeline, ranking wants PageRank, and their product manager wants a dashboard. That's three asks, and our design handles each one differently."
Three Asks: Absorb, Strain, Refuse
| Ask | What it costs in our design | Verdict |
|---|---|---|
| Chained jobs: inlinks, then anchor text, then ranking features | Each step is a separate job. It pays the ≈ 1 min startup, writes its full output to GFS with 3 replicas, and the next step reads it back. A 5–10 step pipeline, which is how the 2004 paper says Google's indexing was rebuilt, spends 5–10 minutes on startup alone | Absorbed, badly. It works, but every boundary materializes to GFS and a person has to wire the steps together |
| Iterative jobs: PageRank, 30 iterations | 30 separate jobs, so 30 minutes of startup before any work happens. The whole graph is re-read and re-written through GFS every iteration, even though nothing outside the loop ever reads those intermediate results | Strained. This is Decision 1's flip trigger: iterative pipelines are taking over |
| Interactive queries: "how many inlinks does this URL have?" | The ≈ 1 min startup by itself blows any interactive budget. The model scans everything and has no indexes | Refused. It's a different product |
Interviewer: "Couldn't you just make startup faster?"
You: "That fixes the symptom. The real cost of iteration is that Decision 3 and the GFS output step throw away state the next iteration needs. With a 2-second startup, PageRank would still write every iteration out three times and read it back again. The successors fixed the model, not the startup time."
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson