Section 8: Where the Five Seconds Go - Deep Dive: Distribution & the Kill Switch
Executive Summary: Section 7 ended with a pure function waiting for fresh input. This section is the pipe that delivers it. We take the 14:07:00 kill and spend its five-second budget hop by hop: 50 ms to commit, 0.35 s for the publisher's priority lane to reach all 40 relays, 1–2 s of fan-out over 100,000 streams, under 10 ms inside each SDK, 99% converged by 14:07:04. Then we defend deltas over snapshots (D4), the kill's own lane (D5), push with a poll floor (D7), sinceVersion resume on one monotonic version sequence, and how the on-call watches the fleet converge instead of trusting a 202.
The interviewer circles the arrow between the relays and the SDKs. "You promised 99% of a hundred thousand instances in four seconds. I want every millisecond accounted for." This is the lane Section 2's five-second kill requirement said we'd have to build, and the budget Section 6 drew with a stopwatch. Now we defend it line by line.
The Budget, Line by Line
| Hop | Wall clock | Cost | Why it costs that |
|---|---|---|---|
| On-call presses kill | 14:07:00.00 | — | Pre-authorized, reason required (D5) |
DB commit: flag_state.killed=true + audit row |
14:07:00.05 | 50 ms | Two rows, one transaction, no compile |
| Publisher emits priority kill delta to all 40 relays | 14:07:00.40 | 0.35 s | ~2 KB payload, signed, skips the compile queue |
| Relays fan out tick + delta over 100,000 streams | +1–2 s | dominant | 2,500 writes per relay onto open sockets |
| Each SDK verifies, swaps pointer, writes LKG | +<10 ms | per instance | Signature check + one atomic store |
| 99% of instances on killVersion 9031 | 14:07:04 | — | p99 ≤ 5 s met with margin |
The shape of this table is the argument. Almost nothing is spent computing, because a kill is designed to be tiny: the delta is roughly {flag: "smart-reply-v2", killed: true, killVersion: 9031}. Section 3 said the whole budget would be spent on fan-out and propagation, and here it is — the only hop measured in seconds is the one that touches 100,000 sockets.
Interviewer: "Why does the kill get to skip your compile queue? A normal edit takes longer."
You: "Because a kill doesn't need compiling. A normal edit — say the 25% step, flag version v41 to v42, environment version 8812 to 8813 — has to recompile every affected service's snapshot, validate it, and sign it. A kill changes one pre-declared override that already sits at the top of every compiled rule order (D5). The publisher keeps a priority lane for exactly this shape of change: nothing to compile means nothing to wait for. :unkill goes back through the normal lane, because turning a feature on deserves the full validation path."
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson