Section 11: Ads Fail Open, Money Fails Closed - Reliability, Multi-Region & DR
Executive Summary: Two promises have opposite failure directions. Serving fails open: if anything in the ads path breaks, the feed renders without ads, inside the deadline. Billing fails closed: no verified event, no charge, and an acknowledged billable event is never lost. Six active-active regions each hold a full index and a full user-feature copy, sized so any one can die (240K QPS of capacity against a 200K share). Budgets survive a region loss through per-region spend allotments and a pacing outage through a throttle table that ages toward conservative. Canaries and the calibration guard catch bad pushes.
Interviewer: "Section 10 assumed everything was up. Now break it. What fails, and who notices?"
You: "I'll sort failures by who feels them: Maya, Trailhead, or finance. Then I'll take a region down at Maya's 7:42 a.m. peak and walk it minute by minute."
The Failure-Modes Table
| Component | Failure | Detection | Blast radius | Response |
|---|---|---|---|---|
| Retrieval leaf shard | One shard's replicas down or slow | Fan-out misses deadline; health checks | 1/16 of candidates | Proceed with 15 of 16 shards (L1 PARTIAL_SHARDS); hedge at p95 ~8 ms, capped at 5% extra load |
| Inference fleet | Slow or short on capacity | Heavy-stage deadline misses, queue depth | Every request in the region | L2 REDUCED_CANDIDATES: heavy-score 300 instead of 500; requests still missing the deadline fall to L3 LIGHT_SCORES |
| User feature store | Slow reads | 8 ms context deadline missed | Personalization quality | Serve with partial context; L4 HOUSE_ADS if context is unavailable |
| Pacing loop | Throttle table stale | Table age past one 10 s cycle | Budget accuracy, not serving | Keep last table, decay toward conservative (diagram below) |
| Event log | Unavailable | Ingress append failures | Billing delay, pacing blind spot | No ack; client keeps the beacon and retries with backoff; consumer dedupe makes retries safe |
| Index push | Corrupt or wrong snapshot | Canary leaf recall and fill-rate checks | One canary replica | Halt; swap pointer back to previous snapshot |
| Model push | Miscalibrated model | Predicted/observed outside 1.00 ± 0.02 | 1% canary | Automatic rollback |
| Campaign DB | Down | CDC lag alarms | Freshness only | Serve on last index; edits pause |
| Region | Gone | Global load balancer health checks | ~1/6 of traffic | Move its users, feed and ads together, to five survivors; reassign spend allotments |
Key Insight: Almost every row degrades quality or freshness, not availability. Only the region row touches Maya's feed, and it costs seconds, because the front door is stateless and every region is self-sufficient (D12).
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson