</> MAANG.io

Expert

Design landmark systems in depth, from Dynamo, GFS, Bigtable and Kafka to payments and ads pipelines.

0 of 232 solved 0%

Section 2: The Contract Is Two Functions - Requirements (Functional & Non-Functional)

Executive Summary: Section 1 reduced the prompt to one sentence: the user writes two functions, and the framework owns every failure. This section turns that sentence into a contract. Functionally: submit a job, get R sorted output files, plug in a combiner or partitioner, read counters, survive a poisoned record. Non-functionally: Riya's 10 TB inlinks job finishes in about 8 minutes, within 15 even if a machine dies, every job completes despite worker failures, and output matches a sequential run. Each number gets a mechanism, and each non-goal a reason.

The interviewer leans back. "Fine, two functions. Now tell me what the framework promises the person who writes them, in numbers I could hold you to."

You: "Riya writes map and reduce, about 40 lines. Everything they don't write is a requirement on me. So the functional list is short, and the obligations behind it are long."

What the user writes, and what that leaves for the framework

The user's side of the contract is tiny on purpose. Any responsibility the user doesn't hold lands on the framework as a requirement.

The user writes map and reduce in about 40 lines and the framework owns everything else: splitting input into M tasks, scheduling tasks next to their data, shuffling and sorting 1 TB by key, recovering from worker failures, finishing despite stragglers and bad records, and reporting counters and status; the shuffle yields R sorted output files in GFS and failure recovery guarantees output equal to a sequential run.

Key Insight: Every box under "Framework owns everything else" is something a one-off distributed program used to get wrong. The requirement isn't that these things can be done, but that the user never thinks about them.

Sign in to continue reading

The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.

Sign in to read the full lesson

Sign in to MAANG.io

Use your Google or Microsoft account — no password to remember.

Continue with Google Continue with Microsoft

Please accept the terms above to continue.