Section 10: When the Failure Is Ours - Reliability, Multi-Tenancy & Failure Isolation

Executive Summary: Section 8 protected the system from other people's endpoints. This section covers our own failures. The defenses are headroom sized for two AZs, 10 cells that cap the blast radius at about 10% of tenants, two quotas per account, a priority ladder that decides who waits first, and a quarantine for poison messages. The rule under all of them: protect the publisher's clock, let the subscriber's clock slip, and never lose an acked message.

The interviewer moves on. "Nice work on the partner webhook. Now the failure is yours. An AZ goes dark at peak. What happens?"

Losing an Availability Zone

Start with Section 7's promise: an acked message is on 2 of 3 AZs, so losing one leaves a copy. Then walk the tiers:

• Message Log. The 30 brokers run 10 per AZ (3 per cell), so 20 survive against the 9 throughput needs. Partitions led from the dead AZ elect a new leader from a surviving replica, and their publishes stall for a few seconds. Every ack now needs both survivors, so tail latency rises. Lose a second AZ and there's no quorum: the Publish API returns 503 rather than ack a single copy (Section 7).
• Stateless tiers. Publish API, Fan-out Workers and Delivery Workers hold nothing a peer can't rebuild. Their partitions move to survivors.
• Work in flight. A dead worker's uncommitted tasks run again on the new owner. Some already reached the endpoint, so they're duplicates: at-least-once (D4), deduped by MessageId. Parked tasks live in the durable Parking Lot, so none are lost.

Interviewer: "Your Delivery Workers have 1.5× headroom. Lose an AZ at peak and you're at 100%. Isn't that a gap?"

You: "It's a choice. The Publish API gets 2× because its availability is the 99.99% number. Delivery at 100% means lag, and lag waits in the log, which holds 48 hours. The overload ladder below decides who waits. If a customer bought a first-attempt latency SLO that must hold during an AZ loss, I'd go to 2× on delivery too: 240 hosts instead of 180."

Quick check before moving on: