Section 9: The Gauntlet at Midnight - Interviewer Deep-Dive Challenges

Executive Summary: The design is on the board, and now the interviewer goes looking for the seams. Eight rapid escalations, from split brain to 10× growth, each framed as what the interviewer says, what they're really asking, the model answer, and the senior line that earns the points. Almost every answer leans on three pieces you already built: the fencing epoch, the deterministic run_id, and per-tenant limits.

Notice that none of the answers adds a component. The failures were priced in when the boxes were drawn.

Challenge 1: Split Brain

"Your scheduler node is partitioned from etcd but can still reach the database. What happens?"

What they're really asking: do you trust a lease timer for safety?

Model answer: The node can't renew, so its lease expires within 10 s. It doesn't halt: it keeps firing its last-known partitions, because the lease clock was never what made firing safe. If another node takes a partition over, the new owner bumps owner_epoch. The stale node's next firing transaction checks owner_epoch = my_epoch inside the transaction, fails, rolls back, and the node drops that partition. If nobody took over (say etcd itself lost quorum), the stale node is still the only owner and keeps its partitions on time. If both nodes somehow insert the same run, the deterministic run_id hits the unique constraint and the second insert is a no-op.

The senior line: "Leases decide who should fire. The epoch inside the transaction decides who can. Two nodes that believe they own a partition are harmless, because only one of them can commit."

Quick check before moving on: