Section 7: Five Machines, One Story - Deep Dive: Consensus, the Replicated Log & Master Leases

Executive Summary: Every guarantee here rests on the five replicas of a cell telling one story. We use a Raft-style replicated log, the same family as the Multi-Paxos in Google's 2006 Chubby paper. Terms act as epochs, and the election restriction means only an up-to-date candidate can win. An entry commits on 3 of 5 replicas in about 2–3 ms, with group-commit capacity of ≥ 10,000 writes/s against a 500/s peak. Five replicas survive 2 failures. A 10 s master lease lets the master answer reads from memory with no quorum round, at the cost of an assumption about clock rate. Snapshots bound the log, learners replace dead machines, and leadership transfer keeps planned upgrades under 1 s.

Interviewer: "You've drawn five boxes and called one of them master. Where do you want to go deeper?"

You: "Consensus first, because everything else assumes it. Sessions (Section 8, Sessions, KeepAlives & Consistent Caching) and sequencers (Section 9, Fencing, Sequencers & Lock-Delay) are only as good as the claim that there is exactly one master and one history."

The Log, the Term, the Majority

Each replica holds the same in-memory database and an append-only log on local SSD. State changes only when the master appends an entry (Acquire, SetContents, session create, ephemeral delete), and every replica applies committed entries in log order.

Time is divided into terms, increasing integers with at most one master each. The term is the epoch of Section 10 (Master Failover & Disaster Recovery): RPCs stamped with an old term are rejected. When master-c's SetContents for /ls/ny1/orders-db/master arrives, the master sends the entry to all four followers in parallel and commits once it is durable on a majority: itself plus two followers.

An in-DC round trip is 0.5 ms and an SSD fsync ~1 ms, with the master's fsync overlapping the send, so the median commit is 2–3 ms, well inside the p99 < 50 ms write SLO for a node nobody caches. Group commit batches every entry that arrives during an fsync into one disk write and one AppendEntries, giving ≥ 10,000 small writes/s; our 500 writes/s peak uses 5%.