Section 11: Sunday Night, Something Breaks - Reliability, Failure & Disaster Recovery
Executive Summary: The interviewer stops asking how the marketplace works and starts asking how it fails. We split every component into source of truth (replicate it) and derived state (rebuild it), walk a failure-mode table, take apart the trickiest row (losing the detail cache), turn a 33-minute index rebuild into a zero-downtime alias swap, and price a region loss against an RPO of one minute and an RTO of thirty.
The interviewer leans in: "It's 7:45 on a Sunday. Maya posted the Trek five minutes ago, Daniel is about to search, and something in your diagram just died. You pick what."
A strong answer sorts the boxes first:
"Everything I drew is in one of two classes. Sources of truth are Postgres for listings, Cassandra for messages, and the object store for photos. Losing them loses something a user was promised, so they get synchronous or multi-copy replication. Derived state is the search index, the Redis cache, the CDN and the fallback lists. I can rebuild those from the truth, so I design them to degrade and refill rather than never fail."
Key Insight: That split decides where you spend: durability money on the three sources of truth, a fast rebuild and a graceful degraded mode for derived state. Backing up an index you can regenerate in half an hour is wasted effort.
The Failure-Mode Table
| Component fails | What still works | How it degrades | Recovery |
|---|---|---|---|
| Search node or shard | Everything; each shard has 3 copies | Surviving copies serve, slightly slower | Replicas re-allocate automatically |
| Search cluster | Posting, detail pages, messaging | Search walks the degradation ladder below | Repair, or rebuild into a new cluster (~33 min) |
| Kafka | Posting, search (existing index), detail, messaging | Outbox rows pile up in Postgres; new listings stay pending_review (moderation is fed by Kafka), so new supply pauses |
Relay drains in order; versioned writes make replays safe |
| Moderation ML | Synchronous rules (< 200 ms) | Trusted sellers fail open with post-hoc review; new accounts fail closed into the queue; hash-list matching never fails open | Queue drains when classifiers return |
| Postgres primary | Search, cached and replica reads, messaging | ~30–60 s without writes while one of two synchronous standbys (ANY 1) is promoted; losing a standby never blocks commits |
Clients retry with the same Idempotency-Key |
| Redis detail cache | Search, posting | ~21,000 QPS of misses aimed at replicas sized for ~2,100 | Coalesce, shed, warm up (below) |
| CDN provider | All APIs | Images break; origin is sized for ~10% of image traffic | Steer to a second CDN; origin shield collapses misses |
| Object store (regional) | CDN-cached images | New uploads fail; the app retries | Read from the cross-region replica bucket |
| Cassandra node or rack | Everything | Invisible: RF = 3, LOCAL_QUORUM |
Hinted handoff and repair |
| WebSocket gateway node | Persisted messages | Its ~50K connections drop | Jittered reconnect, then sync from Cassandra |
The Kafka row is the outbox paying off. A 30-minute outage at peak leaves roughly half a million outbox rows (~300 writes/s × 1,800 s), trivial for Postgres. Nothing is lost, only delayed. Detail pages stay right because the Listing service writes the cache on commit, not through Kafka. The moderation row reuses Section 9's fail-open/fail-closed split: if Maya is an established seller, the bike publishes and gets reviewed afterwards; a two-hour-old account waits, because scammers watch for the moment the classifier goes dark.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson