Section 2: Three Hours Out, Fifteen Minutes Back - Requirements (Functional & Non-Functional)
Executive Summary: This section turns Section 1's answers into a contract you can test: nine numbered functional requirements, walked through with Sam's release r-1432, and four safety invariants the system enforces whoever pushes the button. Then the numbers: merge to canary in 30 minutes, canary to 100% in about 3.2 hours on the standard lane and 33 minutes on the emergency lane, rollback in 15 minutes at p95, pause in 5 seconds, and a control plane that needs only three nines because it fails static. The pushback worth rehearsing is the asymmetry: why getting out is so much faster than getting in.
The clarifications are done, and the interviewer leans back. "So what does this thing actually promise?" The weak answer is "it deploys code safely and quickly." Nobody can test "safely," and "quickly" hides the real tension: every minute of waiting buys detection and delays a fix. So every requirement below is testable, and every number is one an on-call engineer could be paged on.
The Functional Contract
Strip the prompt to its verbs: register, plan, roll out, watch, stop, undo, record. Numbering lets Section 4's endpoints and Section 5's tables trace back to a promise.
| # | Requirement | What it really commits us to |
|---|---|---|
| FR1 | Register releases | CI hands us a tested artifact addressed by digest, with provenance and a signature. Immutable: the same digest is promoted through every stage. |
| FR2 | Rollout plans with sane defaults | Stages, batch size, bakes and gates. Most teams get the standard plan; stateful or slow-starting services override it. |
| FR3 | Control verbs | Start, pause, resume and roll back any rollout, from the UI or the API. |
| FR4 | Automated health gating and automatic rollback | Metrics judge each stage, not a person watching dashboards. A failed gate halts and rolls back without waiting for a human. |
| FR5 | Emergency lane | A faster plan for hotfixes, requiring a second approver. It is still canaried, still batched and still gated. |
| FR6 | Global deploy guard and freezes | One switch pauses every rollout, globally or by region or service, when fleet-wide health degrades or a freeze is declared. |
| FR7 | Per-host convergence and status | Each host converges to its desired version and reports what it actually runs, slot by slot. |
| FR8 | Full audit trail | Who started, approved, paused or rolled back what, when, and with what evidence. |
| FR9 | Notifications | The release owner hears about stage progress, gate failures and rollbacks, with the evidence attached. |
Now walk Sam through it. At 14:32 UTC on Tuesday, Sam merges commit a1b2c3d to checkout-api. CI spends 20 minutes building and testing, then registers release r-1432 with digest sha256:9f4e… (FR1). Sam never writes a plan; checkout-api gets the standard one (FR2). At 14:52 the canary starts in zone A1: 2 instances on r-1432 beside 2 control instances on the previous release r-1101. Every host involved pulls, verifies and reports (FR7).
Stages 2 and 3 pass, and around 16:44 the rollout enters region B, where a new call to fraud-scorer hits a slower regional endpoint. Checkout p99 on new-version instances climbs from 300 ms to 520 ms, past the 450 ms threshold. At 16:48 the gate halts the rollout and rolls back every stage (FR4). Sam gets the gate's evidence (FR9), and every step lands in the audit log (FR8). Nobody touched FR3, FR5 or FR6 that afternoon. They exist for the afternoons that aren't normal.
Key Insight: "The same digest is promoted through every stage" rules out two popular mistakes. Rebuilding per environment means what you tested isn't what you shipped. A mutable tag like latest lets two hosts run different code under one name. This choice has no flip trigger: it holds at ten hosts and at a hundred thousand.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson