Section 11: Draw the Blast Radius Before the Boxes - Reliability, Failure Domains & DR
Executive Summary: Section 10 broke one component at a time; this section zooms out to failure domains. A 99.95% data plane leaves 21.6 minutes a month, less than one region-wide bad deploy takes to detect and roll back. So the platform is nested domains (sandbox, host, AZ slice, cell, region), each sized so its failure is absorbed or confined. We cover cells and shuffle sharding, an AZ loss absorbed by pre-bought headroom, static stability with the control plane down, dependency failures, and the honest async RPO when a region is lost.
The interviewer leans back: "You survive a dead placement node. What does a bad day for the whole region look like?"
You: "I'll start from the budget, because it decides everything else. Invoke is 99.95% a month. That's 21.6 minutes of error budget. The control plane gets 99.9%, so 43.2 minutes. A bad worker-agent release pushed to all ~3,150 hosts at once can't be detected and rolled back in 21 minutes. So no change, ours or a tenant's, may reach the whole region at once. Everything else follows."
Nested Failure Domains
| Domain | What a failure takes out | What contains it |
|---|---|---|
| Sandbox | One invoke | The microVM wall (Section 7); the caller sees a function error |
| Host | Its in-flight invokes and warm sandboxes | Spread rule (D14): no function keeps much of its pool on one host |
| AZ | A third of every cell's hosts | Fleet sized for 2 of 3 AZs (N10); async queue replicated across 3 AZs |
| Cell | Roughly one-sixth of accounts | Cell-by-cell deploys; per-cell placement, pollers and queue partitions |
| Region | Everything in it | Independent regions; customers deploy to more than one |
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson