Section 10: Most of It Is Rebuildable - Reliability, Failure & Disaster Recovery
Executive Summary: The interviewer stops asking how it works and starts asking how it breaks. The answer rests on one split: posts, the follow graph and the post-event log are durable, and everything else (feed caches, celebrity timelines, snapshots, rankings) is derived and can be rebuilt from them. We walk the failure map, then three real failures: a feed-cache shard holding 8M users, a fan-out backlog, and a whole region holding 167M daily users. Each ends the same way: serve a worse feed now, rebuild in the background, never lose an acknowledged post. A degradation ladder ties it together.
The interviewer caps the marker. "Good. Now break it. Start wherever you think it hurts most."
The Failure Map
Start by sorting the stores into two piles, because the pile decides the recovery.
| Component | Blast radius | Detection | Mitigation |
|---|---|---|---|
| Feed-cache shard (1 of 125) | 8M users lose their materialized feed | Replica heartbeat, error rate on feed:{user_id} reads |
Promote replica in seconds; if both die, rebuild on read |
| Fan-out workers | New posts arrive late in feeds | Fan-out lag p99 over 5 s | Newest-first, skip stale inserts, add workers |
| Post-event log | Publishes can't be acknowledged | Under-replicated partitions | 3-AZ replication; publish acks only when durable |
| Post cache | Hydration falls back to the post store | Hit ratio below 99% | Batch reads, L1 for hot keys, request coalescing |
| Ranking Service | No ranked order | Ranking p99 over 50 ms, errors | Chronological fallback from the same candidates |
| Celebrity timeline cache | One replica of ~20 | Health checks | Other replicas hold the full 80 MB |
| Whole region | ~167M DAU | Region health probes, SLO burn | Re-home users, degraded pull, rate-limited rebuild |
Key Insight: The durable pile is small: posts, the follow graph, the post-event log. Feed caches, celebrity timelines and snapshots are all functions of those three. That's why Section 5 let the feed cache live in RAM without durability, and why most failures here cost freshness or ranking quality, not data.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson