Section 9: Pulling on Every Thread - Interviewer Deep-Dive Challenges

Executive Summary: The design is done. Section 7 chose a Snowflake-style 64-bit ID, and Section 8 showed how clock rollbacks, sequence overflow and duplicate worker IDs are closed. Now the interviewer attacks it with five escalations: ten times the traffic, a clock that jumps back five seconds, the regional etcd cluster going dark, a product request for strictly ordered chat IDs, and the day the timestamp runs out. Every answer uses the same two steps. Right-size: use a number from the ledger to say whether the threat is real at our scale. Range: name the mechanism that takes over when it is real, and the trigger that brings it in.

The interviewer caps the marker and says, "Good. Now let's break it." This is where the interviewer learns whether your design was derived or memorized. If you answer every attack with new machinery, you look easy to push around. If you wave every attack away, you look like you can't scale. The safe path starts with a number.

Key Insight: An ID generator has one property it can never trade away, uniqueness, and several it can bend: latency, availability and ordering. Every challenge below is really a question about which of these you give up under pressure. The answer is always the same ladder from Section 2: refuse before you duplicate.

Challenge 1: Ten Times the Traffic

"The company grows 10×. Peak is 10 million IDs a second. What breaks?"

Right-size. Not the bits, and that surprises people. One worker's 12-bit sequence allows 4,096 IDs per millisecond, or 4,096,000 IDs/s in theory. The largest region at 10× needs 4 M IDs/s, which fits inside the sequence space of a single worker. What breaks is the fleet. Each node serves 25,000 requests/s usable (half of a 50,000 req/s ceiling). If every request carries one ID, the largest region needs 4,000,000 ÷ 25,000 = 160 nodes. The region only has 7 worker bits, which is 128 slots. You can't run 160 nodes on 128 worker IDs without two nodes sharing one, and two nodes sharing a worker ID is exactly the duplicate Section 8 spent its whole length preventing.

Range. Climb the ladder one rung at a time, and stop at the first rung that works:

Rung 1, batching. The NextIds(count ≤ 1,000) call and the client library's 100-ID prefetch buffer already exist (Section 4). With an average batch of 10, the largest region drops from 4 M RPC/s to 400,000 RPC/s, which means 16 nodes, the same fleet as today. The worker bits never noticed the growth. Batching doesn't hurt ordering, because the client library throws away any buffered ID older than 1 s, so the ~1 s k-sorting promise still holds.

Rung 2, a spare region ID. Five of the eight region IDs are in use. A region that outgrows 128 workers can be given a second region ID and another 128 slots. Nothing about the layout changes, and Decode still works. But it isn't free: by written policy the three spare codes (5, 6 and 7) are reserved for new regions first, so every code lent to service growth is a region we can no longer open without a re-layout.

Rung 3, re-layout. This is the last resort, and the interviewer wants to hear why.

Interviewer: Just take a bit from the sequence. You're using 0.6 % of it.

You: True, an 11-bit sequence still gives 2,048 IDs per millisecond. But changing the layout is a migration, not a config change. During a rollout, an old-layout node and a new-layout node read the same 22 low bits differently, so in the same millisecond they can produce the same 64-bit value. Every node in the region has to switch at one agreed future millisecond, and Decode has to choose the layout by timestamp from then on. I'd only do it once batching and spare region IDs are both used up.

Quick check before moving on: