Section 9: Spend the Expensive Model Where It Counts - Deep Dive: The Ranking Cascade
Executive Summary: Retrieval hands us 10,000 candidates, and the budget leaves 35 ms (5 ms light ranking, 10 ms hydration, 20 ms heavy ranking) to turn them into scores the auction can trust. Running the heavy model on all 10,000 would take 80,000 accelerators at peak; scoring only the light ranker's top 500 takes 4,000. Features are fetched once per request and travel as IDs. A per-host batcher keeps devices evenly loaded for at most 2 ms of latency, serve-time feature logging prevents skew, and a 1.00 ± 0.02 calibration guard decides whether a model ships, because pCTR is multiplied straight into money.
The interviewer taps the "10,000 candidates" arrow from Section 8. "Now what? You have a big model. Why not run it on all of these?"
You: "Because the arithmetic won't let me."
Why a Cascade at All
At peak the heavy ranker scores 1.2M × 500 = 600M candidate-scores/s. At 150K scores/s per accelerator, the sustainable rate at ~70% device utilisation so queueing delay stays small, that is 4,000 accelerators, or 4,800 provisioned for N+1 across six regions, packed 8 per host on 600 hosts. Heavy-ranking all 10,000 candidates would mean 12B scores/s and 80,000 accelerators, twenty times as many.
So the cascade has stages that get more expensive per candidate and see fewer candidates as they go:
| Light ranker | Heavy ranker | |
|---|---|---|
| Sees | 10,000 candidates | 500 candidates |
| Model | Small model, small feature set | Deep multi-task model, ~1,000 features, ~400 GB of sparse embedding tables |
| Hardware | CPU, ~1 µs per candidate per core | Accelerators, 150K scores/s each |
| Peak cost | ~12,000 cores | 4,000 accelerators (4,800 provisioned) |
| Budget | 5 ms | 20 ms |
| Job | Recall: don't drop the eventual winner | Precision: get the probability right |
The light ranker doesn't need to order the top 500 correctly; it only needs to keep the heavy ranker's eventual picks inside that 500. We measure it that way: how often the heavy model's top picks on a sampled full 10,000 survive the light cut.
Interviewer: "Why not one big model? Fewer moving parts."
You: "One big model on 10,000 candidates is the 80,000-accelerator plan. One small model on everything is cheap, but it would rank Trailhead's ad against other running brands on a fraction of the signal, losing revenue and relevance in every auction. The flip trigger is cost: if heavy scoring got about 10× cheaper per score, I'd widen the heavy stage to a few thousand candidates before I'd drop the light one."
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson