Section 7: Keep Going or Stop - Deep Dive: Rollout Orchestration, Health Gates & Rollback

Executive Summary: The interviewer hands you the wheel, and you go to the one decision this system makes thousands of times a day: advance the rollout or halt it. Three parts carry it: a 5-second control loop with one lease per rollout that advances only when the batch has converged, the gate is green and the capacity floor allows it; a health gate that compares the canary with an equal-size control group, honest about what 2 instances can tell you; and a rollback that is just another rollout, to the previous digest already on disk, done in 15 minutes or less at p95. Sam's r-1432 exercises all three.

Thirty-five minutes in, the interviewer puts the marker down. "You've drawn the boxes. Pick somewhere to go deep." Artifact distribution is tempting, because it has the big numbers. But slow bytes make a deploy late; a wrong keep-going decision turns it into an outage.

You: "The place this design bears weight is the decision to keep going or stop. Every one of 5,000 rollouts a day faces it at every batch, and the 5% change failure rate says 250 times a day the answer is stop. I'll cover the loop that decides, the gate that informs it, and how rollback works."

The Control Loop

Orchestrator workers are stateless. Each rollout row in the state store carries a lease owner and a version number, and the worker holding the lease runs one iteration every 5 s:

Three properties make the loop safe to crash:

• One lease per rollout, not one global leader. One leader for 1,300 in-flight rollouts is a bottleneck with a fleet-wide blast radius. Per-rollout leases mean a dead worker's rollouts expire and get picked up while every other rollout carries on.
• Every write is conditional on the rollout's version. A worker that stalls (a long GC pause, a partition) and wakes up on an expired lease writes with an old version, and the strongly consistent store rejects it. Two workers can't both advance one rollout.
• Steps are idempotent, keyed by (rollout, stage, batch). Writing batch 4 of stage 3 twice yields the same rows and generation, so a new owner resumes from the persisted stage and batch without starting a batch twice. The write is conditional: it sets a slot's generation to N+1 only if the slot's desired state is still at N, so a replay by a new lease holder is a no-op.

"Converged" is Section 5's rule: every slot in the batch reports generation_seen at or above the desired generation and passes readiness. Hosts that stopped heartbeating are left out of batches (Section 8).

Key Insight: The capacity check is the step most candidates skip. Section 3's budget is 21% under 25%: a lost zone plus one batch still fits inside every service's headroom, so a rollout can keep going through a zone outage. What it must not do is stack a third loss on top. Before each batch, the loop counts instances already out of service for any reason (unhealthy, draining, on a dead host) and adds the batch; if the sum would cross the floor, it waits a tick. Waiting costs minutes. Crossing the floor costs the service.

Pause rides the same loop: a human pause or the global deploy guard sets a flag the loop reads first, so no new batch starts after one tick (5 s at most). In-flight batches finish rather than leave instances half-drained.

Interviewer: "The container scheduler already does rolling updates. Why build this?"

You: "Section 6 takes that one. In short, the scheduler is our actuator inside a cluster; staging, canary analysis and the guard live here."

Quick check before moving on: