Section 6: One Click, a Hundred Thousand Machines - High-Level Architecture
Executive Summary: This section draws the whole platform on one board. It has three planes: a control plane where flags are edited, versioned and rolled out, a distribution plane that turns each committed change into signed snapshots and streams it to 100,000 server instances, and an evaluation plane that lives inside every service process and at the edge for clients. We walk Dana's compose request through it, trace the 14:07 kill from click to the 99% mark four seconds later, and defend the boxes a weaker design would have drawn.
The interviewer hands you the marker: "Show me the boxes, and how one change gets to a hundred thousand machines." That second clause is the real question. Section 3 showed reads never leave the process, so the architecture is mostly about moving a 2 KB change very fast.
The Board: Three Planes
Start with the split, out loud: "Three planes, each with its own availability target. The control plane, where humans and the rollout controller write, is 99.9%. The distribution plane is 99.99%. The evaluation plane is a library in your process with no remote dependency, so its availability is your service's."
Walk the components in the order a change travels:
| Component | Job | Key number |
|---|---|---|
| Management UI and API | Edits rules with If-Match, starts rollouts, presses kill |
p99 ≤ 500 ms |
| Control-plane DB | Relational source of truth: immutable flag_versions, flag_state, audit log |
5,000 changes/day |
| Rollout controller | Advances plans step by step, enforces bake times, pauses on gate breach | min bake 1 h per step |
| Snapshot publisher | Compiles each affected service's snapshot, signs it, emits deltas | ≤ 2 MB per snapshot |
| Snapshot store and CDN | Holds signed full snapshots for boot, catch-up and the 60 s poll | ~60 MB for every flag |
| Regional relays | Hold the long-lived streams and fan out ticks and deltas | 4 per region, 2,500 streams each |
| Server SDK | Evaluates from an in-memory snapshot, persists last-known-good (LKG) to disk | ~1 µs, p99 ≤ 50 µs |
| Edge evaluation service | Evaluates client flags and returns values, not rules | p99 ≤ 50 ms, ~35,000 req/s peak |
| Telemetry collector and usage store | Rolls SDK aggregates into (flag, service, variation, minute) | ~17 GB/day, 90-day retention |
The last edge closes the loop: variation-tagged metrics and usage rollups feed the rollout controller, which is how a rollout stops itself (Section 9).
Interviewer: "Why is the rollout controller writing to the database instead of talking straight to the publisher?"
You: "So a rollout step is just another flag change, with a version, an audit row and the same propagation as a human edit. A side channel would mean two ways to change production, only one of them audited."
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson