Section 10: The Primary Dies at 10:06 - Reliability, Fairness & Abuse

Executive Summary: A ticketing system earns trust when something breaks with 20,000 shoppers inside and three million waiting. We start with the worst moment: the inventory primary for show E-7781 fails mid-on-sale. Synchronous replication keeps every acknowledged hold, the feedback loop pauses admission by itself, and idempotency keys make retries safe. Then the waiting room's own failures, a region loss, defensible fairness, and a rotating barcode that makes a resold screenshot worthless.

"It's 10:06. Jordan has two seats held, 20,000 people are shopping, and the database primary for this show just died. Go."

The Primary Fails Mid-On-Sale

You: "What does Jordan lose? Nothing. The hold on Section 114, Row K, seats 11–12 was acknowledged at 10:05:14. The inventory store acknowledges a commit only after a standby has written it durably. That's synchronous replication, and it's the reason a promoted standby holds every acknowledged hold. Jordan's rows are there, with their hold_id and the 10:13:14 expiry."

Interviewer: "Why not asynchronous? It's faster."

You: "Asynchronous replication acknowledges before the standby has the write. Promote it and the last holds it never received are gone: those seats read as available, the next fan holds seat 11, and two people believe they own it. The unique constraint on (show, seat) in tickets can't help: it lives in the same store, so if a lost write was a ticket row, the constraint never sees the first sale. So no seat write is acknowledged until two machines have it. The cost is one extra round trip per commit to a standby in another zone, which fits inside the 500 ms hold p99."

Interviewer: "And if the standby is the one that's down?"

You: "With a single synchronous standby, every commit would block. On normal days we run two in-region and commit when either one acknowledges:"

# Postgres-style, normal days: a commit waits for any one of two standbys
synchronous_commit = on
synchronous_standby_names = 'ANY 1 (standby_a, standby_b)'

"Failover promotes the standby with the highest flushed WAL position. With 'any one of two', the other may be behind, and promoting it would lose acknowledged holds. If both are gone, writes stop. We fail closed rather than quietly fall back to asynchronous and risk a double sale."

With no human in the loop:

1. Hold and checkout calls on E-7781 fail with 503 and Retry-After.
2. Section 8's watermark controller sees hold p99 and errors pass target, so it drops admission to zero. Nobody new enters a backend that can't serve them; the 3,000,000 waiting keep their places.
3. The standby with the highest flushed WAL position is promoted in tens of seconds. Holds are rows in the store, not cache entries, so they survive (Section 7 covers why Redis locks would not).
4. Admitted shoppers retry with the same Idempotency-Key. If a hold committed but its response was lost, the retry returns the original 201 and hold_id, because the key was written in the same transaction as the hold. A commit that never finished left nothing, so the retry is a fresh attempt.
5. Hold p99 falls back under 500 ms and the controller ramps admission toward 67 shoppers/s.

Holds keep their 8-minute TTL: tens of seconds doesn't justify mass-rewriting expires_at. Admission tokens have a fixed expiry, so the controller publishes the pause length with the watermark and the gateway adds it to each token's exp when checking it. Tokens are never modified in place, and nobody admitted just before the failure loses their turn. An in-flight checkout follows Section 7's handoff: if held → sold can't commit before the hold expires, the authorisation is voided.

Key Insight: The loop built for load control doubles as failure control. A down backend looks like an overloaded one, so admission pauses itself without a runbook.

Quick check before moving on: