Section 12: Grading the Rollout - Final Summary & Candidate Scorecard
Executive Summary: This closing section turns the module into a grading instrument. We replay the interview as the signal each phase recorded, then give a 100-point scorecard naming what wins and what loses points on this problem: the SSH loop, the per-environment rebuild, the revert-and-rebuild rollback, the canary compared against yesterday. Score bands map totals to readiness, Sam's Tuesday is replayed once more, and a short catalog shows where this question hides.
Go back to minute five. Two answers sound competent and score badly: a script that loops over 100,000 hosts restarting them, and "use the scheduler's rolling update." Both move bits. Neither says what happens when the release is bad, a zone is down mid-deploy, or the deploy system itself is broken. The interviewer was scoring whether you treated the change as the likely cause of the next outage and built around stopping it early and undoing it fast.
The Arc, Replayed as Signal
| Section | What you did | Signal recorded |
|---|---|---|
| 1. Problem Understanding & Clarifications | Asked who owns CI, containers or hosts, stateless or stateful, speed versus safety | Scope cut with reasons |
| 2. Requirements (Functional & Non-Functional) | Rollback ≤15 min p95, pause ≤5 s, control plane 99.9% with fail-static | Safety as numbers and invariants |
| 3. Scale Estimation & Workload Analysis | 5,000 deploys/day, 3.2 h per rollout, 1,300 in flight; 11% + 10% = 21% < 25% | The load is tiny; the budget is blast radius |
| 4. API Design (with Pushback) | Digest-addressed releases from CI; 202 rollouts; If-Match on resume |
Async, idempotent contracts |
| 5. Data Modeling | Desired versus actual state with a generation number; strongly consistent store | Knows how a batch proves convergence |
| 6. High-Level Architecture | Per-rollout leases, host agents, zone caches, health analyzer, deploy guard | Pull-based, outside the request path |
| 7. Deep Dive: Rollout Orchestration, Health Gates & Rollback | Canary against a same-zone control, two bad 1-minute windows, roll forward to N-1 | Mechanism on the decision that matters |
| 8. Deep Dive: Artifact Distribution & Host Convergence | Zone caches at 25 GB/s, N-1 on disk; zone recovery in 15 min, not 74 | Sized the burst, not the average |
| 9. Interviewer Deep-Dive Challenges | Expand and contract; config as a release; guard pause in ≤5 s; 33-min emergency lane | A rule for each failure |
| 10. Reliability, Safety & Security: Deploying the Deployer | Fail-static, bootstrap path, break-glass, signed artifacts | Planned for its own failure |
| 11. Follow-up Feature Evolution | Traffic-weighted canaries, dark deploys, P2P for 20 GB models | Features land on old decisions |
Key Insight: Read the right-hand column and almost every row is the same idea at a different depth: make the bad release small, visible and cheap to undo. The 10% batch, the 1% canary, the never-two-regions rule, N-1 kept on disk and the global guard are each a different answer to "how much damage can one change do before we stop it?"
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson