Section 10: The Gauntlet - Interviewer Deep-Dive Challenges

Executive Summary: With three deep dives behind it, the design now faces deliberate escalation. We run seven challenges: the election-night news property, a CFO who wants exact unique users, session IDs logged as a dimension, a bad deploy that corrupts six hours of device data, a regional Kafka cluster dying at peak, "isn't Lambda architecture dead?", and forged ghost hits. Each answer names what the interviewer is testing, gives the mechanism, the number and the trade-off, and points back to the decision that already handled it.

The interviewer leans back. "Let's break it." Every question below has an answer that sounds impressive and is wrong, and a plainer one that is right because the design anticipated it. The Staff move is to find the decision you already made, quote its number, and say where it stops working.

Challenge 1: Election Night

"P-90001 goes to 250K hits/s, and three hundred people in the newsroom have the real-time card open. What melts?"

What they're testing: whether you know where the hot key actually lives, on the write side and on the read side.

You: "On ingest, nothing. 250K hits/s is about 7% of the 3.5M/s global peak, and the key hash(property_id, client_id) spreads it across all 2,048 partitions at about 122 hits/s each, instead of 62 MB/s on one partition. Real-time has no hot key either: each streaming worker builds a partial per-minute sketch from its own partitions, and partials merge by register-wise max. The busiest property costs the real-time store the same 30 sketches as Trailhead, roughly 360 KB."

"The reads are the risk. Three hundred dashboards polling every 15 seconds is 20 QPS of the same question. The result cache is keyed by (query, data version), and a real-time entry lives 15 seconds, one poll interval, so all three hundred share one computation per poll without adding staleness beyond the 60-second freshness promise. Real-time flips from polling to server push past 10% of query QPS: 350 QPS at peak, or about 5,250 dashboards open at once. The newsroom's 14-month explorations draw from P-90001's own token bucket, weighted by rows scanned. Their queries queue. Dana's don't notice."

Quick check before moving on: