Section 10: When the Owner Dies - Reliability, Failure & DR

Executive Summary: This section kills each component in turn, judged by Section 1's rule: reactions are events we can lose, raised hands are state we must keep. An owner crash costs up to about ten seconds of applause and zero acknowledged hands, thanks to leases, epoch fencing and opId dedup stored in the document. A gateway crash becomes a jittered reconnect wave of 5K per second. A sick store makes hands fail closed while reactions carry on. A regional disaster costs at most 5 seconds of hand changes, and the client asks people to check their hand instead of re-raising it. We close with two SLI lists and why the resync rate is the canary.

The interviewer flips to a fresh page. "Your design works on a good day. Now start killing things. What does Maya see?"

The trap is "everything is replicated." Start with a promise instead: an acknowledged hand-state change must survive any single-node failure, and nothing about reactions needs to survive anything. Every failure below is checked against that one sentence.

The Owner Dies: Leases, Epochs and the Zombie

Each meeting has one owner, which holds a lease in the placement directory: TTL 10 s, renewed every 3 s. When the lease expires, the directory gives mtg-7f3a to a new owner and bumps the epoch from N to N+1. The new owner loads the hand document, claims it with a write conditional on both epoch N and the ver (or etag) it just loaded; if the claim fails, it reloads and tries again. Then it broadcasts a hand.snapshot stamped with the new epoch.

The dangerous case is the owner that only looked dead: after a long garbage-collection pause or a partition, it wakes up still believing it owns the meeting. That zombie is stopped by fencing: every write it makes is conditional on epoch = N, the document now says N+1, so the write fails and the zombie learns it has been deposed. Clients ignore any delta stamped with an older epoch than the one they last saw. Reaction frames carry the owner's epoch too, and gateways drop frames from an epoch older than the directory's current one.

That last pair of messages is where "zero acknowledged changes lost" is earned. Suppose the old owner wrote Maya's raise to the store and then crashed before sending hand.ack. Her client retries with the same opId. The new owner finds that opId in recentOps, which lives inside the hand document (last 256 ops or 5 minutes), and replies with an ack for the existing ver instead of raising her hand a second time. A dedup set in the old owner's memory would turn retries into duplicates on failover. Because we ack only after the durable write, any hand Maya was told about is in the store.

Reactions sent during the gap simply die, and a frame older than its 1 s TTL is dropped anyway. Nobody wants a clap from ten seconds ago.

The Pushback: "Ten seconds is a long time. Why not a 2-second lease?"

You: "Because a short lease mistakes slowness for death. Every false takeover forces a fresh snapshot to everyone and risks flapping. The budget can take the real ones: 99.99% of a 30-day month is about 259 seconds, so one 10-second takeover uses roughly 4% of it. If that ever stops being true, the fix is the one Section 5 already names: a 3-replica Raft group of owners per meeting shard, with a warm follower that takes over in about one election instead of one lease TTL."

Quick check before moving on: