Section 1: One Program, Two Thousand Machines - Problem Understanding & Clarifications

Executive Summary: The prompt asks for "a framework", and that word can pull a candidate in three directions: a cluster manager, a general distributed-computing library, or a query engine. This section shows the first ten minutes of a better interview. Five clarifying questions shrink the prompt to something narrow and buildable: a programming model where the user writes two deterministic functions, and a runtime that runs them as batch jobs over files already sitting in the Google File System, on the same 2,000 machines that store those files. Each answer removes a whole category of machinery. The section ends with written assumptions, an engineer named Riya and their nightly inlinks job, and the sentence the rest of the design hangs on.

The interviewer reads you the prompt:

"Our engineers keep writing one-off distributed programs to crunch the web crawl, and each one reinvents partitioning, retries and failure handling — badly. Design a framework that lets any engineer run a computation over ten terabytes on our two-thousand-machine cluster in minutes, without knowing anything about distributed systems."

The trap is the word "framework". A candidate who hears "distributed computing" starts on message passing or a generic task graph, machinery nobody asked for. Look at the last clause instead: without knowing anything about distributed systems. That clause is the requirement that matters most. If the user must know nothing, the framework must own everything: splitting the input, placing work, moving data between machines, and surviving the machines that die along the way. The only way to own all of that is to limit what the user is allowed to write.

Key Insight: The product here is not a cluster and not a library of distributed primitives. It is a programming model plus a runtime. The model is narrow on purpose, because that restriction is what lets the runtime parallelize and recover automatically. Your job in the first ten minutes is to discover how narrow the interviewer will let it be.

Decomposing the Prompt

Before asking anything, map it out loud along four lines:

• What it is: a way to express a computation over a huge dataset, and a system that runs it on thousands of machines. Two products in one: the model and the runtime.
• Who uses it: ordinary engineers. Good programmers who have never debugged a network partition and shouldn't have to.
• Why it exists: every team currently re-implements partitioning, retries and failure handling, each one differently and badly.
• How it's used: batch jobs over large files in GFS: the crawl, logs, previous jobs' output. The user submits a job, walks away, and comes back to output files.