Section 6: Evidence In, Ranks Out - High-Level Architecture
Executive Summary: This section assembles Sections 4 and 5 into one architecture: a write path that verifies a run, appends it to the Kafka run-results log and updates both boards with one atomic script per Redis primary, returning Kai's ranks in the same response; three read paths that never touch a primary; and three background workers. We follow Kai's run r-7f3a91 through every box, then defend the components we left off on purpose, because at this level the absences are scored as heavily as the boxes.
Drawing the Board
The interviewer hands you the marker: "You've got the API and the data model. Draw me the whole thing."
Don't start with boxes; start with the shape, out loud, because that's what's being graded.
You: "One write path and three read paths, and they meet in exactly one place: the sorted sets. A run comes in as evidence, never as a score. We verify it, append it to a durable log, update both boards atomically, and hand the ranks back in the same response. Every other read goes to a replica or a one-second cache. And the log is the truth, not Redis. If we lost every Redis node tonight, we'd rebuild the boards from it in about five minutes."
| Component | Job | The number that sizes it |
|---|---|---|
| Game client | Plays the run, signs the summary, queues and retries with runId |
500M runs/day |
| Regional edge / API gateway | Terminates TLS near the player, authenticates the token, forwards to the one home region | p99 ≤ 300 ms worldwide for a submit |
| Run Validator | Signature check, plausibility bound, runId dedupe via the run_dedupe table (7-day TTL) |
≤ 2,000 points per second of run time |
| Score Service | Appends to Kafka, then runs the board applier: one atomic Lua script per board primary, in parallel | ≤ 50 ms server time; 17,400 runs/s peak, 35,000 at surge |
| Redis board primaries | One board per standalone primary (not Redis Cluster): its sorted set plus its pending and shadow side keys on the same node | 34,800 ops/s at peak, 70,000 at surge, against a 100,000 budget; 15 GB all-time, 16 GB weekly (current + previous) |
| Redis replicas | Serve every read except the submit's own rank | ≈ 50,400 ops/s on the weekly board, ≈ 25,200 per replica |
Kafka run-results |
The durable truth, partitioned by playerId | 100 GB/day; 7-day retention × RF 3 = 2.1 TB |
| Best-Score Writer | Consumes the log, filters through a per-partition in-memory best cache, then conditional "only if greater" writes to player_best |
≈ 4,350 conditional writes/s at peak |
| Leaderboard Read API | Top-100 from an in-process cache, around-me, friends | 22,600 top-100 requests/s → 400 ZREVRANGE/s |
| Profile service / cache | Display names and avatars for hydration | 100 rows per top-100 refresh |
| Anti-Cheat Reviewer | Replays provisional runs, then promotes or removes them | ≤ 15 min per provisional run |
| Season Closer | Monday 00:00 UTC: wait out the grace, freeze, snapshot, pay rewards | 5-minute grace; top 10,000 frozen |
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson