Section 9: Machines Die on Schedule - Deep Dive: Sloppy Quorum, Hinted Handoff & Anti-Entropy

Executive Summary: In a fleet of 66 nodes, failure is the normal state, so the design treats it as routine. Failure detection is local: B decides on its own that F has stopped answering, and nobody announces a global verdict. Membership changes only when an administrator says so. While F is down, its writes go to a stand-in that holds them as hints and hands them back when F returns. Our interview's numbers for a 30-minute outage at peak: 190 writes/s, ~344,000 hints ≈ 1.7 GB, returned in ~86 s. For drift that hints miss, read repair fixes keys people read, and per-partition Merkle trees (1,024 leaves, depth 10) fix the rest. When a whole data center goes dark, one-replica-per-DC placement keeps 2 of 3 home replicas for every key, so quorums stay strict. The price, stated honestly: during failures, R + W > N no longer guarantees that reads see the latest write.

Interviewer: "It's peak. Node F just lost its power supply. What happens to Dana's cart?"

You: "Nothing Dana can see. Let me walk the first minute, the next thirty, and F's return."

Who Decides F Is Dead? Nobody, Globally

The 2007 paper uses decentralized, local failure detection. Node B treats F as unreachable when F stops answering B's own messages. B then sends F's share of traffic to alternates and probes F every so often to see whether it's back. K may reach the same conclusion at a different moment, and that's fine: no decision depends on the nodes agreeing.

What B does not do is remove F from the ring. Membership changes only by explicit admin command (Decision 6). That's deliberate: a 30-minute power-supply swap is not a reason to re-home F's ~46 partition replicas.

Interviewer: "Why not detect the failure and re-replicate F's data right away?"

You: "Look at the arithmetic. Re-homing F means copying about 68 GB, F's ~46 partition files, at peak. Spread across 46 receivers that's only a minute or two of wall time, but it's 68 GB of churn, and when F comes back 30 minutes later every one of those partitions is either moved back or left lopsided: about 136 GB of traffic to cover an outage that 1.7 GB of hints handles. And a node that flaps every few minutes would trigger that storm repeatedly. Explicit membership has its own flip trigger, at thousands of nodes per instance, where full-membership gossip gets expensive and the paper itself says the design would need hierarchical extensions."