Section 6: One Log, Two Clocks - High-Level Architecture
Executive Summary: This section assembles the components from Sections 4 and 5 into one architecture: a single order log, fed by a transactional outbox and CDC, consumed by two pipelines that run on different clocks. The speed path (Window Aggregator, Top-K Merger, Top-K Store, Top-10 API behind a CDN) refreshes 93 small lists every minute. The rank path (Lake Sink, Hourly Rank Job, versioned Rank Store) re-sorts 240M category memberships every hour. We walk Priya's helmet purchase through both, then defend the components we deliberately left out, because at this level the absences are scored as heavily as the boxes.
Drawing the Board
The interviewer slides the marker across: "Okay, you've got your APIs and your data model. Draw me the whole thing."
The temptation is to draw the busiest diagram you can. Resist it. The architecture should say, at a glance, the thing Section 1 discovered: two problems in one costume. So you narrate the shape before you draw a single box.
You: "Everything starts from one fact: an order line was paid for. That fact goes into one log. Two consumers read that log on two different clocks. The speed path cares about the last minute and only the top of each list. The rank path cares about the last hour and the full order of every category. They share an input and nothing else."
| Component | Job | The number that sizes it |
|---|---|---|
| Order Service + Outbox, CDC | Commit the order row and an outbox row in one transaction; CDC tails the outbox into Kafka | CDC lag normally < 5 s |
Kafka topic order-line-events |
The single ordered log both paths read | 64 partitions, keyed by productId; 234 events/s per partition at the 15,000 events/s design peak |
| Window Aggregator (Flink) | Dedup, per-buyer cap, per-line counted units, minute buckets, sliding 1h/4h/24h counters, local top-20 per list; emits a counted-units side output | ≈ 5.6 GB of state, ≈ 88 MB per partition |
| Top-K Merger | Final operator of the same Flink job: merge 64 local candidate lists into 93 global top-20 lists | 119,040 entries/minute |
| Top-K Store (Redis) | Hold the 93 current lists; a write lands only if its version (the last closed minute) is ≥ the stored one |
93 keys |
| Top-10 API + Catalog Service + CDN | Hydrate title, price and image; drop out-of-stock, delisted or restricted items; cache | 2,300/s at sale peak; ≈ 3 origin req/s with an origin shield |
| Lake Sink | Land the aggregator's counted units (post-dedup, post-cap, retractions as negative rows) as hourly_units, and the raw events beside them, as hourly Parquet |
≈ 3 GB/day compressed |
| Hourly Rank Job (Spark) | Decay scores, join memberships, sort every category, assign unique ranks | 240M rows, 5.8 GB, ~5–8 min |
| Rank Store | Two resident snapshot versions, each grouped into ~400,000 hashes, one atomic pointer | 32 GB RAM with replicas |
| Rank API, PDP aggregator, best-seller pages | Serve badges and top-100 lists | 116k/s PDP views at sale peak → ≈ 23k/s after the 5-min in-process cache |
| Audit job | Hourly: recompute the exact 24h Top-10 from raw lake events, alert on divergence; never served | One comparison per hour |
Three boxes deserve a sentence each because candidates usually forget them. The Integrity service is a producer, not a filter on the read path: when it flags an order as fraudulent, it emits a FRAUD_RETRACTED event into the same topic, keyed by productId, and the aggregator subtracts it exactly as it subtracts a cancellation. The counted-units side output is why the rank path never re-implements dedup or the per-buyer cap: the aggregator emits every unit it actually counted, and every retraction as a negative row tagged with its original event time, so the Lake Sink writes hourly_units from rules that run once. The Audit job is a deliberately independent check. It recomputes the 24h Top-10 with SQL over the raw lake events, compares it with what streaming published, and alerts if the two disagree. Its answer is never served.
Key Insight: Both paths read the same log, keyed the same way. That one choice gives you exact Top-K merges on the speed path, a replayable source of truth for the rank path, and a recovery story for both: a cold rebuild with no checkpoint replays 25 hours (24 h of windows plus cap history), about 62.5M events, in about 5 minutes.
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson