Section 11: The Platform That Must Outlive Its Own Outage - Reliability, Safety & Security
Executive Summary: This section answers Section 10's closing demand: show the failure modes and the permissions. We walk the failure matrix plane by plane — control database, publisher, relays, snapshot store, edge, a whole region — and show that in every row the fleet keeps serving. We face the honest gap: a kill pressed during a full distribution outage waits. Then we contain the scariest incident class, the platform's own bad change, with publish validation, golden contexts, signatures, staged propagation and a version pin, and we close with who may press which button.
"Show me the failure modes and the permissions." The demand has three parts: what breaks, how far a bad change can spread, and who is allowed to act. You open with the stance that organizes all three: this platform is designed so that its own death is boring.
The Failure Matrix: Every Plane Can Die, Evaluation Doesn't
The planes fail independently, and you take them in order of how much anyone would notice.
| Plane down | What stops | What keeps working | Recovery |
|---|---|---|---|
| Control DB | All writes — edits and, honestly, new kills | All 100,000 SDKs evaluate from in-memory snapshots; new pods boot from the snapshot store (D4) | Continuous backup: RPO a few seconds, RTO minutes via regional replica failover |
| Publisher | Changes and kills queue after commit | Everything already published keeps serving | Restart, drain the queue in version order |
| Relays degraded | Push latency | Lose 1 of 4 in a region → ~3,333 streams each, inside the 10,000 capacity; lose all 4 → SDKs reconnect cross-region, worst case the 60 s poll floor | Stateless; replace and let jittered reconnects (0–10 s) refill |
| Snapshot store/CDN | Poll fallback, cold boots | Streams still push deltas; a restarted pod uses its last-known-good, a brand-new pod uses code defaults (D10) | Multi-region blob store; CDN origin failover |
| Edge evaluation | Fresh client fetches | Web and mobile serve cached values through their ttlSeconds: 300; 99.99% multi-region target |
Anycast away from the dead location |
| Full region | That region's relays and edge | SDKs reconnect to another region's relays with sinceVersion; snapshots are replicated; control plane fails over |
Nothing in a region is the only copy of anything |
The control-DB row carries the boldest claim: even total loss of the database is survivable at runtime. Snapshots sit, compiled and signed, in the snapshot store and on 100,000 disks, and keep serving for however long the restore takes. What you lose is the ability to change anything — at 5,000 changes a day, a bad afternoon, not an outage. Section 2's 99.9% control-plane target versus 99.99% for distribution is this row written as an SLO.
Key Insight: Every row of the matrix ends the same way: the fleet serves the last answer it was given. Fail-static isn't one mitigation among many — it's the posture. The request path never needs the platform to be up; it needs the platform to have been right once.
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson