Section 11: The Escalation Gauntlet - Interviewer Deep-Dive Challenges

Executive Summary: The interviewer tries seven ways to break the design. Three ask the ensemble to be something it isn't: a database, a linearizable read store, or one cluster stretched across regions. Two test whether we understand our own guarantees: the paused leader and the quorum arithmetic. One asks for 10× the load, and one asks us to defend the whole bet against the Distributed Lock Service design. Every answer gives the number, names the mechanism it hits, refuses or redirects, and names the trigger that would change our mind. All figures are our interview's assumptions from Section 3.

The interviewer closes the failover diagram. "Fine, five seconds. Now let me misuse it."

You: "For each one: what it hits, the number, and what I'd do instead."

Challenge 1: "Store Our 5 GB Service Catalog in It"

What they're testing: whether you know where the data physically lives.

You: "Every byte lives in memory on every server, which is Decision 5. Today the tree is about 600 MB inside an 8 GB heap. Add 5 GB and it's 5.6 GB, on all five servers. Each fuzzy snapshot grows to about 5.6 GB, so a follower that needs a SNAP sync has to pull that much before it can serve. A heap that full spends its time in garbage collection. A multi-second pause on the leader forces a five-second failover, and on a follower it strands that server's 4,000 clients until they time out and migrate. We've budgeted for pauses under 1 s. And the 1 MB per-znode cap would chop the catalog into thousands of fragments."

You: "So: put the catalog in a database or blob store. In ZooKeeper, keep a pointer of a few hundred bytes at /catalog/current, with a version and a watch. When it changes, services re-read the blob."

The senior line: "ZooKeeper should hold the fact that something changed, not the thing itself. If the coordination data really does pass a few GB, I split it into separate ensembles by use case."

Quick check before moving on: