Section 10: The Gauntlet - Interviewer Deep-Dive Challenges

Executive Summary: Nine sections in, the interviewer stops nodding and starts attacking. This section walks the six escalations every Dynamo-class design attracts — a hot key, an oversized value, a lying clock, a resurrected delete, a mid-peak partition, and a casual "now 10x it" — and demonstrates the pattern that survives all of them: right-size the problem with a number before reaching for a mechanism. Each challenge also carries a hidden probe, the real question beneath the stated one, and we name every probe out loud.

The interviewer has been patient. You derived the ring, defended the quorums, priced the LSM engine, and admitted what last-write-wins loses. Now the tone shifts from tell me your design to let me break it — the 401 interview working as intended. The Paper Reciter dies here, because recited mechanisms cannot be re-derived under pressure; the Single-Node Thinker dies here too, because five of the six attacks are distributed-systems attacks. The survivors share one habit: before answering how, they answer how big.

Key Insight: Every challenge below is a trap for the candidate who reaches for machinery first. The passing move is always the same two-step: quantify the blast radius, then pick the smallest mechanism that covers it. A mechanism proposed without a number is a guess wearing a lab coat.

Challenge 1: "One key is taking 100K reads per second."

It is the flags team's config:flags blob, and the arithmetic comes first: a single node sustains ~30K reads/s, and even with N=3 spreading the load, each replica absorbs ~33K/s — and that division already assumes the flags team reads at R=1, one replica per read; at the platform default R=2, every read touches two replicas and the burden doubles to ~67K/s each — pinned at its ceiling while the other 61 nodes idle. The fleet is fine; three machines are on fire. That framing rules out the reflexive fix — "add a global cache tier" — on sizing grounds: this hot key is ~8% of peak read load (a quarter of average reads) concentrated on 3 of 64 nodes. You do not re-architect the entire read path, and break read-your-writes for every tenant, to fix a problem that lives on three machines.

The right-sized mechanism stays local: the coordinator coalesces concurrent requests for the same key into one backend fetch, and holds a 1-second coordinator-local cache for keys it detects as hot. The honest cost is up to one second of staleness on that key — said out loud, because the flags team decides whether a one-second-old flag is acceptable (it is). Write-hot keys get the complementary treatment: key-splitting into k#0..k#7 shards merged on read — but only when the value is aggregatable (counters, sets — the Like Counter module's case), because shards are only useful if a read can merge them; a write-hot opaque blob cannot be split, and is a contention problem to serialize with CAS, not a splitting problem. If a tenant's entire workload becomes hot reads — feature flags at 10x — that tenant graduates to a purpose-built push tier; the platform default does not warp around its loudest customer.

What's probed: can you tell a local problem from a global one? This is the same insight the Like Counter module builds its whole design around — one hot row does not justify re-architecting the fleet; it justifies a mechanism scoped to the row.

Quick check before moving on: