Section 11: An Auction Cannot Be Paused - Reliability, Failure & Disaster Recovery

Executive Summary: Most systems survive an outage by retrying later. An auction cannot: Sunday 19:00:00 arrives whether or not a shard is healthy, and a bid that was not accepted before it will never be accepted at all. This section derives every reliability choice from that one fact. We defend synchronous replication to a second availability zone before the 200 is sent (about 1 ms of the 2 ms accept transaction, and the one place where async would be unforgivable), walk a failure-mode table from a dead bid-service node to a lost region, and design disaster recovery honestly: a warm standby with seconds of RPO for the long tail, a standby-region log replica in the quorum only for designated hot items, a 15-minute RTO, and a published rule for auctions that end mid-outage. We close with a load-shedding order, two auction-specific alarms, a canary auction and a game day.

The Constraint That Writes the Rest of the Section

Interviewer: "Walk me through reliability. What is different about this system?"

You: "One thing, and everything follows from it. In a checkout system, a minute of downtime means a purchase happens a minute late. Here the deadline is part of the data. Auction #7731 ends at 19:00:00.000 on Sunday, and the cut-off is accepted_at < end_time, evaluated inside the accept transaction. If the bid path is unavailable from 18:59:40 to 19:00:10, every bid in that window is not delayed. It is dead. There is no 'later' for a 19:00:00 deadline."

That reframes availability: the 99.99 % target on the bid path, about 52 minutes a year, is a budget of minutes in which some auction's final seconds get stolen. It reframes durability harder still: an acknowledgement we cannot keep is worse than none, because the bidder stops acting the moment they read it.