Section 6: The Lie Hidden in "Written" - Replication & Tunable Consistency

Executive Summary: Section 5 ended with a preference list — three physical nodes that each hold a copy of cart:8842. This section decides what "written" and "read" mean across them. We derive N/R/W quorums from first principles, draw out the R+W>N overlap argument honestly — including the two things it does not guarantee, which is where strong candidates separate from paper reciters — and then reframe the whole apparatus as a product: consistency dials the platform sells to tenants at posted prices. Finally, node n17 dies mid-write during peak, and sloppy quorums with hinted handoff keep the checkout team's write flowing — at a cost we name out loud. Cross-datacenter quorums get a door, not a chapter: that bill comes due in Sections 11 and 12.

Your interviewer taps the preference list from Section 5. "So a put goes to three nodes. When do you tell the client it succeeded? When one copy lands? All three?" This is the fork in the road, and it is worth pausing to name the choice you are not making. You could elect a leader per shard — a Raft or Paxos group that serializes every write and hands you clean consistency. Rejected, with reasons: every write pays a trip to wherever the leader lives, and when a leader dies, writes to its shard pause until an election completes — and this platform promised 99.99% write availability. Primary-backup with async replication is rejected faster still: an acked write that exists only on a crashed primary is a lost write, and the durability contract says a confirmed write survives any single-node loss. So the design goes leaderless: any node can coordinate any request, and correctness comes from counting acknowledgments, not from electing a boss. The flip trigger is real, though — the day tenants demand transactions or strict per-key ordering, Raft-per-shard walks back in through Section 12's door.

Three Numbers That Define the Product

The entire replication contract is three integers:

• N — how many replicas hold each key. The preference list's length: N=3.
• W — how many replicas must durably acknowledge (WAL fsync, not just receipt) before a put returns success: W=2.
• R — how many replicas must answer before a get returns: R=2.

Why these defaults? W=2 means a confirmed write exists on two disks before the client hears "OK" — lose any single node a millisecond later and the write survives, which is precisely the durability promise and no more. And W=2 of 3 means one replica can be slow or dead without blocking writes. R=2 is the mirror image, and it carries a tail-latency gift the next section will cash in: the coordinator takes the second-fastest of three replicas, so one straggling node — mid-GC, mid-compaction — never sets your read p99. That is a large part of how the read budget stays under 10ms.

Now the famous inequality. With R + W > N — here, 2 + 2 > 3 — the set of replicas that acked the last write and the set answering your read must overlap in at least one node. It is a pigeonhole argument: two acks plus two answers is four slots across three nodes, so some node sits in both sets. Draw it in the room: writes landed on {n09, n17}; your read polls any two of {n09, n17, n23}; every possible pair intersects the write set. That overlapping replica hands the coordinator the newest acked version, versions are compared, and the freshest wins.

Key Insight: R+W>N guarantees exactly this — every read quorum contains at least one copy of the most recent acknowledged write. State it that precisely in the interview, because the precise claim is defensible and the sloppy one ("so reads are strongly consistent") is where candidates get dismantled.

Here is what the overlap does not buy, and saying so unprompted is a Staff-level signal:

• Not linearizability under concurrent writes. Two clients put cart:8842 through two coordinators at the same moment; both achieve W=2; the replicas now disagree about which write is "last." Quorum overlap ensures a read sees the conflict's winner-so-far — it does not decide the winner. That arbitration (and its honest casualties) is Section 9's whole job.
• Not clean failure semantics. A put that reached one replica and then timed out is neither committed nor erased — it may surface on a later read and even propagate. Quorums bound staleness; they do not give you transactional all-or-nothing. This is why Section 4's API made timeouts explicitly indeterminate and retries idempotent.

Quick check before moving on: