Section 9: Breaking Things on Purpose - Interviewer Deep-Dive Challenges

Executive Summary: With the design drawn and both deep dives done, the interviewer starts attacking it. This section runs five escalations: one link going viral, losing the whole Redis cluster, losing a region, ten times the traffic, and a product request for 6-character codes. Every answer follows the same two steps. First, right-size: use a number to say whether the threat is real at our scale. Second, show the range: name the mechanism that takes over when the threat becomes real, and the trigger that tells you it's time.

You've defended the read path in Section 7 and the code minting in Section 8. Now the interviewer leans back and starts breaking things. Answer every attack with new machinery and you look easily pushed; wave them away and you look unable to scale. The safe route starts with a number.

Key Insight: Answer every escalation twice. Right-size: does this problem exist at our measured scale? Say so, with the number. Range: name the mechanism for the scale where it does exist, plus the trigger that would bring it in.

Challenge 1: Maya's Link Goes Viral

"A celebrity reposts Maya's link. Ten million clicks in ten minutes. What happens?"

Right-size. 10 M clicks over 600 s is ≈ 16.7 K rps on a single key: aZ3kP9q. The fleet is sized for 350 K rps at peak, so the total volume isn't the danger. Concentration is. Every request hashes to the same code, which means the same Redis shard and the same KV partition. One Redis node handles roughly 100 K ops/s, so it would survive alone, but it never has to: the in-process LRU (60 s TTL, from Section 7) catches the key first. Each of the ~50 redirect servers misses once, fetches from Redis, and serves the next minute from its own memory. Redis sees about 50 reads a minute for the hottest link on the internet. The KV store sees close to none.

Range. Suppose a single key goes past 200 K rps worldwide. The next step is to cache that specific 302 at the edge with Cache-Control: public, s-maxage=5 (browsers still revalidate), well inside the 60 s disable bound. This is a deliberate exception to the 302-with-max-age=0 default, not a reversal of it, and it has a cost: those clicks never reach our servers, so per-click analytics for that link have to be rebuilt from CDN logs. The trigger: one key holding a large fraction of the fleet's capacity, which is why the exception stays per key.

Quick check before moving on: