Section 10: Stale, Honest, Never Broken - Reliability, Integrity & Failure Recovery
Executive Summary: Everything this system produces is derived data, rebuilt on demand from the order log, so the reliability question isn't "how do we avoid losing data?" It's "what does Priya see while we rebuild?" The answer is a degradation ladder: fresh, then stale with an honest asOf, then the last good copy, and only at the very end a hidden badge. We walk every component's failure with its user-visible effect and recovery time, show how restarts avoid double counting, catch silent wrongness with the hourly audit, and then turn to the threat that's deliberate rather than accidental: sellers trying to buy the #1 badge.
The interviewer flips to a fresh page. "It's sale night, 20:05. Priya just clicked Top 10, and things start breaking. Walk me through it."
This is where you set the frame before you list any failures. Rankings are derived data. The order log in Kafka (7 days, 252 GB on disk) and the hourly Parquet lake are the truth, and every list and every rank can be recomputed from them. So the recovery point objective doesn't matter here: nothing in the ranking layer can be lost for good. What matters is how long the user-facing reads stay correct-looking and honest while some piece of the pipeline is down.
Key Insight: This design runs two separate reliability budgets, and mixing them up is the most common mistake in this section. The read path promises 99.99% availability (about 52.6 minutes a year), and it keeps that promise by serving the last good answer. The pipeline has a looser freshness objective: 99.9% of minutes with list age ≤ 2 min. That allows 0.1% × 525,600 minutes ≈ 526 minutes, or about 8.8 hours a year, of stale lists. The pipeline is allowed to fail. The page isn't.
The Failure Table
| Component | Failure | What Priya sees | Mitigation | Recovery |
|---|---|---|---|---|
| Order Service / Outbox / CDC | CDC stalls (normal lag < 5 s) | Checkout is unaffected. Top-10 lists stop advancing and asOf ages |
Outbox rows wait in the Orders DB. Alert when list age > 3 min | CDC drains the backlog on resume, with no loss |
| Window Aggregator or Top-K Merger (one Flink job; the merger is its final operator) | Task or job crash | Lists freeze at the last emitted minute | Restore buckets, counters, cap and per-line state from the last checkpoint (every 60 s) and re-read ≤ 60 s of events. Dedup + version-checked writes make the replay idempotent; the next emit republishes all 93 lists | About a minute. If state is corrupt, cold-replay 25 h from Kafka (24 h of windows plus cap history) in ≈ 5 min |
| One aggregator partition | Goes quiet (no events) | Nothing | An idle timeout: 10 s past a minute boundary with no events, a processing-time timer closes the minute for that partition | Automatic; one quiet partition never freezes the 93 lists |
| Top-K Store (Redis) | Primary down | Nothing at first | CDN stale-if-error. The origin keeps the last 93 lists (93 × 20 = 1,860 entries) in process |
Replica failover in seconds |
| Catalog Service | Hydration or filtering unavailable | The last hydrated list, a bit older | Serve the last filtered response. Never serve unfiltered candidates | When Catalog recovers |
| Rank Store | One replica lost | Nothing | The second replica serves (32 GB holds 2 versions × 2 replicas) | Re-sync the replica |
| Rank Store | Both replicas lost | Badges come from the PDP cache (TTL 5 min), then disappear | The PDP renders without the badge. Reload the last snapshot files from object storage | 40M records at 100,000 writes/s ≈ 7 min |
| Hourly Rank Job (Spark) | Run fails | The previous run's ranks, with an older asOf |
The previous snapshot stays live. Alert when rank age passes 75 min | The next run. One missed run means a worst case of 75 + 60 = 135 min |
| Hourly Rank Job (Spark) | Runs, but produces a bad snapshot | Nothing (it's blocked) | Validation gates (row count, top-100 sanity against the previous version, max rank movement) refuse the flip | Instant. If a bad version ever goes live, flip rank:current back to the retained version, possible until the next build starts (~45 min); after that, fail forward with a fixed run |
| Primary region | Region loss | Stale lists and badges served from the CDN and caches | Warm DR region | See below |
Two rows deserve the conversation the interviewer will start anyway.
Interviewer: "Catalog is down. You have the candidate IDs in Redis. Why not serve them unhydrated, or unfiltered?"
You: "Because the filter is part of correctness, not decoration. Those 20 candidates include items that may since have been delisted, gone out of stock or been restricted. A restricted product on ClipKart's homepage under a 'Top 10' banner is a worse incident than a list that's four minutes old. So the origin serves its last fully filtered response. If that copy gets too old, the button degrades. It never shows the raw list."
Interviewer: "And the Rank Store dying takes down product pages?"
You: "It mustn't. The PDP aggregator treats the badge as optional. It does the lookup with a tight timeout, and on failure or a cache miss it renders the page without the badge. A merchandising widget must never fail the page that sells the product, just as Section 6 made sure it can never fail checkout."
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson