Section 1: Coordination Is the Part Everyone Gets Wrong - Problem Understanding & Clarifications

Executive Summary: This section covers the opening minutes of a prompt that sounds like an infrastructure wish list and is really a request for a tiny, very strict kernel. We break the prompt into what, who, why and how. Then we run a clarifying exchange in which each answer removes a whole class of system: bulk storage, a lock-only API, a master that serves every read, liveness from TCP connections, one global deployment. We write down what survives: a small replicated tree of metadata, writes in one total order, reads that may lag slightly, and sessions that detect dead clients. We contrast that bet once with the Distributed Lock Service module, and we meet Sam, whose three scheduler replicas recur in every section.

The interviewer reads the prompt aloud:

"Two thousand services at our company each reinvent leader election, configuration push and membership — and every home-grown version has broken during an outage. Design one coordination service they can all build on, and tell me what you'd refuse to put in it."

Two phrases carry the weight. Broken during an outage: these schemes work on a calm Tuesday and fail exactly when machines crash, pause or lose the network, which is the only time coordination matters. What you'd refuse: the interviewer wants a boundary, not a feature list. A candidate who answers "a distributed database with locks, queues and a config UI" has failed the second half before drawing a box.

Key Insight: Coordination code fails in the rare case, under failure, and it fails silently until then. Two thousand teams each writing their own election means two thousand untested edge cases around a paused process or a partitioned replica. The product here is not a feature. It is one implementation of the hard part, proven once and run by one team, so nobody else has to get it right.

Decomposing the Prompt

• What it does: hold small pieces of shared state, order every change to that state, and tell interested clients when it changes. The product is a coordination kernel: a few primitives that election, configuration and membership are built from. It is not each of those products separately.
• Who uses it: services, not people. Schedulers, storage masters, stream consumers, anything with replicas that must agree on who leads or what the config says. Sam's settlement-scheduler is typical.
• Why it exists: every home-grown election broke in an outage. The usual causes are heartbeats over a shared database, "lowest IP wins" scripts and config files polled from a bucket. One shared service replaces them all, and it must be the most boring, best-tested system in the company.
• How it's used: small metadata, read often, written rarely, with strong ordering on writes and fast detection of dead clients. Nobody stores a video in it. Everybody depends on it being right.

The decomposition already points at a shape. The server offers primitives. The clients build products from them.