Section 9: The Machine That Knows When to Stop - Deep Dive: Progressive Rollouts & Health Gates

Executive Summary: Section 4's rollout plan was a statement of intent: Ring 0 for a day, then 1% → 5% → 25% → 50% → 100%, with bake times, approvals and gates. This section builds its executor: a controller that is a state machine over rollout_plans, advancing by writing ordinary flag versions, and a health gate that compares treatment against control over the same window and pauses the plan — never kills it — when treatment degrades. Day 3's 13:55 → 14:00 → 14:07 timeline is the proof.

Section 8 settled delivery: any change travels as a version-addressed delta and reaches 99% of connected servers at p99 ≤ 5 s, with kills on a priority lane. The interviewer moves the marker to the one box with no human inside. "Your pipe is fast. But fast delivery of a bad decision is just a faster incident. Who decides that smart-reply-v2 goes from 5% to 25% — and what stops it before anyone notices?"

The Controller: A State Machine With No Side Channel

You: "Section 4's plan is a row in rollout_plans, and the controller is a loop over that table: four states — ACTIVE, PAUSED, DONE, KILLED — plus a current_step pointer. Every transition is one of three events: bake and gate say advance, the gate breaches and pauses, or a human approves, resumes or kills."

The crucial choice is what "advance" means. To move smart-reply-v2 from 5% to 25%, the controller issues the same PUT /v1/flags/smart-reply-v2/environments/prod/rules with If-Match that a human would, producing flag version v42 and environment snapshot 8813. Section 6 called the principle "no side channel"; Section 4 promised the controller executes the plan resource; this is where both cash out (D8).

Interviewer: "The controller is a trusted internal service. Why make it go through the API instead of flipping flag_state directly?"

You: "Because then there are two ways to change production and only one is versioned, audited and concurrency-checked. Through the API, every step gets an immutable version, an audit row answering 'who moved this to 25%' — the controller, executing this plan — and the same 412 protection: if a human edited the flag a second earlier, the controller re-reads instead of clobbering them. The step also rides the pipe Section 8 just defended, so there's one propagation behavior, not two. The cost is a few hundred milliseconds on a clock that ticks in hours."