Section 6: Three Planes and a Stopwatch - High-Level Architecture
Executive Summary: The system splits into three planes with different clocks. The request path answers in 80 ms: a stateless front door, a user context service, 16 retrieval shards, a CPU light ranker, an accelerator heavy ranker and an auction. The control plane turns campaign edits into index deltas within 60 s, publishes throttle tables every 10 s and refreshes models hourly through a shadow-then-canary rollout. The feedback loop carries impressions and clicks into a replicated log that feeds spend, billing and training. The request path writes to no shared database; every piece of state it reads was pushed to it ahead of time. Maya's request crosses it in 38 ms at p50.
The interviewer slides the marker across. "You've got the API and the data model. Now draw it, and walk me through Maya's request."
You: "Three planes, because they run on three clocks. The request path has 80 milliseconds. The control plane has seconds to hours. The feedback loop has seconds for pacing and a day for billing. Most mistakes here come from letting a slow-clock concern onto the fast path."
The Request Path
The front door owns the request end to end, which is why the diagram is a hub and not a chain. It stamps the deadline, fetches user context once, scatters to all 16 shards, and calls each stage in turn with that stage's slice of the budget. When a stage runs out of time, the front door decides what happens next, so only one component ever needs to know how much time is left.
| Component | Job | State | Why it exists / rejected alternative |
|---|---|---|---|
| Ads front door | Deadline, fan-out, degradation, response assembly | Stateless | One owner for deadlines, auth and logging (D11). Rejected: each product surface calling rankers directly and reimplementing all of it |
| User context service | One read per request: profile, user embedding, frequency counters | Regional in-memory KV, ~10 TB per region | One fetch, reused by every stage (D5). Rejected: each stage fetching user data itself |
| Retrieval leaves | Targeting match, eligibility (including the pacing coin flip per candidate), int8 dot product, top 625 per shard | 16 shards × ~12.5 GB in RAM; ~5,800 leaf servers provisioned | Document-partitioned so every shard answers independently (D2). Rejected: term-partitioned index; a general search engine with GC pauses at p99 |
| Light ranker | 10,000 → 500 | Stateless, model loaded in memory; ~12,000 cores at peak | Cheap pass so the heavy model sees 500, not 10,000 (D1) |
| Ad feature cache | Ad features for the hot set | ~10 GB per ranker host, top 5M ads cover ~90% | Lets the front door send IDs, not features (D5). Rejected: remote fetch per candidate, a 500× fan-out |
| Heavy ranker | pCTR (and pCVR) for 500 candidates | ~400 GB of embedding tables per 8-accelerator host; 4,800 accelerators provisioned | Dynamically batched accelerator inference (D6). Rejected: CPU inference on cost; unbatched GPU calls on utilisation |
| Auction + re-rank | eCPM ranking, second price, re-rank constraints, exhausted-budget check | Throttle table refreshed every 10 s | Local budget decision (D9). Rejected: synchronous global budget counter (Section 10) |
Budgets are enforced in two places, each doing one job. The pacing coin flip runs at the leaf, per candidate, so a throttled campaign at participation 0.7 simply drops out of about 30% of retrievals. The auction only removes campaigns whose budget is already exhausted.
Interviewer: "Why is the light ranker a separate box? Fold it into the heavy ranker."
You: "Heavy-ranking all 10,000 candidates would take 80,000 accelerators instead of 4,000. The CPU light ranker is what closes that 20× gap. If the heavy model got about ten times cheaper per score, I'd widen the heavy stage. Section 9's ranking cascade covers the trade-off."
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson