Section 10: Now Break It - Interviewer Deep-Dive Challenges

Executive Summary: The interviewer tries to break the design eight times: Lumen's timestamp prefix at 50K PUT/s against a 3,500 writes/s partition, a 5 TB object in 10,000 parts, 10 billion lifecycle deletes, a billion-key LIST and the count trap, one tenant pulling 1 Tbps (18% of the 5.6 Tbps peak), a bit-flipping NIC, a slow-but-alive partition server, and "make it cheaper." Every strong answer is mechanism, number, trade-off. Almost every mechanism already exists in Sections 7 to 9; reusing them under fire is the signal.

The interviewer caps the marker. "I believe the happy path. Let's see what it does under stress."

You: "For each one: the weak answer, then mechanism, number, and cost."

Challenge 1: The Hot Prefix

Interviewer: "It's 9 a.m. Lumen's cameras all write to 2026/10/31/… at once, 50K PUT/s into one key range."

The weak answer is "tell Lumen to randomize their prefixes": the pre-2018 workaround, blaming the customer for the index's shape.

You: "Those keys land in one range partition with a design limit of 3,500 writes/s (N6). First, the front-end's per-partition token bucket (D9) returns 503 Slow Down with jittered retries for everything over budget. Within about a minute the auto-splitter sees sustained overload and splits the range at a midpoint key, repeating until every child is under its ceiling, roughly fifteen children for 50K/s. If Priya ends up with camera-first keys, it is because her transcoder lists per camera, a data-model reason; the throttle and the split must work for the keys she has. The trade-off is that first minute: real 503s her SDK must retry, and a latency bump at each map change. A global rate limiter was rejected: it throttles every tenant for one tenant's prefix."

The split-and-throttle loop:

Quick check before moving on: