Section 10: The Day the Deployer Breaks - Reliability, Safety & Security: Deploying the Deployer

Executive Summary: A deploy system can touch every machine in the company, so the interviewer asks what happens when it fails or is turned against you. Fail-static agents keep the fleet serving through any control-plane outage, which is why 99.9% is enough. Leases, a strongly consistent store and regional isolation keep each failure small. A break-glass path and a bootstrap rule mean the deployer never needs itself to recover. Signatures verified at the host mean a compromised registry or orchestrator still can't run code CI never built.

The interviewer turns the marker around. "Your deployer is down. Or someone owns it. What happens to the 100,000 hosts?"

You: "Two different questions. Down should mean 'nothing breaks, you just can't deploy.' Compromised should mean 'it can only deploy what CI signed.' First one first."

Fail-Static: Down Means Frozen, Not Broken

Every host agent keeps its last desired state on local disk. If the watch stream drops, or the whole regional control plane disappears, the agent keeps running exactly that. It does not stop instances or revert anything, and a host that reboots comes back to the same state from disk and its image cache, pulling nothing. Only a re-imaged or replaced host, whose disk is gone, has to fetch again; that is Section 8's zone-recovery burst.

The rejected alternative is an agent that stops or self-reverts when it loses contact. After a network partition, every agent in the region would act at once, turning a control-plane blip into a region-wide restart storm. That choice doesn't flip, and the deploy system stays out of the request path.

Interviewer: "At this company, deployment is critical infrastructure. Why only three nines?"

You: "Because of what an outage costs. 99.9% allows about 43 minutes a month, during which nobody can start a deploy and serving is unaffected. At the peak rate of 400 starts an hour, a 43-minute outage delays about 287 deploy starts (400 × 43 ÷ 60), and because the newest queued release supersedes older ones, the backlog drains quickly. Five nines would buy a multi-region consensus control plane the fleet doesn't need."

Interviewer: "Not being able to deploy is fine. Not being able to roll back isn't."

You: "Agreed, and that's the flip trigger: if rollback depended only on the control plane, three nines would be too low. So rollback has a second path that skips the orchestrator: break-glass."

Critical Insight: Fail-static is not fail-safe. It freezes whatever was running, including half a bad release. That is why the reliability story needs a way to move hosts without the normal machinery.

Quick check before moving on: