Section 6: A Brain That Writes, a Fleet That Converges - High-Level Architecture
Executive Summary: The architecture splits into two planes that never share a hot path. The control plane is small and careful: a release API, stateless orchestrator workers holding one lease per rollout, a strongly consistent state store, a health analyzer and a global deploy guard. The data plane is large and dumb on purpose: 100,000 host agents that pull their desired state, fetch bytes through zone caches, verify signatures and converge. This section draws both planes, walks Sam's r-1432 from registration to stage 4, and answers the question every interviewer asks here: why not just use the scheduler's rolling update?
Twenty-five minutes in, the interviewer slides the marker across. "Draw it."
You: "I'll draw two pictures. First the control plane, which decides what should run where. Then the byte path and the health path, which is how the fleet gets there and how we learn whether it worked."
The Control Plane
Each box has one job, a rejected alternative and, where one exists, a flip trigger.
| Component | Job | Rejected alternative | Flip trigger |
|---|---|---|---|
| Release API | Accepts a built, signed artifact from CI and records it by digest, once | Rebuilding per environment; mutable tags like latest |
None: the same digest goes through every stage |
| Rollout API | Writes control verbs to the rollout row with compare-and-set | Calling the lease-holding worker directly (workers come and go; the row stays) | None |
| Rollout orchestrator | Stateless workers; each rollout is held by one lease; a 5 s loop advances batches and stages | One global leader for all rollouts (a bottleneck with a fleet-wide blast radius); no leases (two workers both start the same batch) | None |
| Global store | Releases, rollout rows and their leases, in one small strongly consistent cluster spread across regions | Rollout rows in a regional store, orphaned when that region fails | None |
| Regional state store | Desired and actual state per host slot, strongly consistent within the region | An eventually consistent store, where an agent can see desired state go backwards | None |
| Desired-state distribution | Regional fan-out of desired state over long-lived watch streams | The orchestrator pushing commands host by host | A fleet of tens of hosts, where a script is honest |
| Host agents | Watch, fetch, verify signature, drain, restart, report | SSH loops from a central runner | Same as above |
| Health analyzer | Queries the monitoring platform, compares canary with control, issues gate verdicts | Comparing with yesterday's baseline; a human watching dashboards | Low-traffic services that need longer bakes or synthetic traffic (Section 9) |
| Deploy guard | Global, regional or per-service freeze; pauses all rollouts when fleet health degrades | Per-service autonomy with no global view | None |
Pull versus push for the agents is Section 4's argument and I won't replay it. Its consequence is visible here: the orchestrator holds zero connections to hosts. The per-region distribution tier holds the 100,000 watch streams and absorbs 10,000 heartbeats a second.
Key Insight: The orchestrator writes intent; it never executes. Every arrow from the brain to the fleet goes through a store, and so does every arrow from a human to the brain: a pause is a write to the rollout row, which the lease holder reads on its next tick, so it lands within 5 s. That gives you three things for free: a crashed worker loses nothing (the next lease holder reads the rollout and resumes), a host that was offline catches up by reading its current desired state, and the audit trail is the same data the system runs on.
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson