Section 11: The Gatekeeper's Own Bad Day - Reliability, Failure & DR
Executive Summary: This section walks the failure matrix for every plane Section 6 drew — an edge box restarting, the distribution plane dark, the mirror database down — and gives full weight to the row that breaks cache intuition: losing the mirror's data, which the vendor's 150-rps keyhole turns from a refill into a disaster (D5). We rank the degraded modes by harm, show how the worst two are prevented rather than survived, cover the security and compliance brief, and close with the machinery that watches the watcher: canary probes through every PoP every minute, and the honest answer to "what's your worst incident?"
"You block traffic at five million requests a second, partly because legal says so. Show me the failure matrix — and what you tell legal." The right opening stance: because every decision is made in-process from a local artifact (D1), almost nothing here can fail in a way users feel. What can fail is freshness, policy agility, and — in one scenario — the system's memory.
The Failure Matrix: The Edge Survives Everything Behind It
Take the components in the order Section 6 drew them, answering two questions each: what still works, and how do we get back?
| Component down | What stops | What keeps working | Recovery |
|---|---|---|---|
| Edge box restart | That box, briefly | The rest of the PoP; every other PoP | Fetch the current signed snapshot, or load last-known-good from disk — then Section 10's boot invariant: the vendor class may fail open, but the box takes no traffic until the legal set is loaded. Cold start is a readiness check, not a race |
| Distribution plane down | Snapshot propagation — so overrides and policy changes freeze. That's the honest cost | Every PoP serves its last applied snapshot, fail-static, still ≤ 2 µs. No request is touched | Redeploy; PoPs reconcile by version. Alarm at snapshot staleness over 2× normal cadence — a frozen fleet must never be a quiet fleet |
| Mirror DB down | Compiles stop; the crawl scheduler pauses — its queue is an index on verdicts (D5) |
Edges are untouched; they owe the mirror nothing at request time | Synchronous replica failover, RPO ≈ 0, RTO minutes. The scheduler resumes mid-queue — the queue is the durable state |
| Mirror data loss | The system's memory — see below; this row is the section | Enforcement continues from the last compiled snapshot; the legal class is unaffected (D2) | Point-in-time restore. Re-earning verdicts through the vendor is the one thing we cannot do quickly |
| SentinelDB down | Re-validation and first-look checks | Everything — Section 10, Challenge 1: verdicts age, coverage decays, requests never notice (D1) | Circuit breaker resets; the scheduler drains its backlog by priority |
| Bad GeoIP dataset | Country-rule accuracy for one weekly refresh | The deny set already released | The refresh rides the same compiler gates as every list change — ±20% cap, canary bake, dataset_version pinned per snapshot — so rollback is the same < 60 s version pin |
Now the special row, said slowly. "Restore from backup" sounds routine until you do the arithmetic. The mirror holds ~8M verdicts against a ~10M checks/day budget — so re-validating from scratch sounds like about a day. Except that budget isn't idle: first-look validation of brand-new IPs claims ~6M checks/day of it, and those checks bound D4's fail-open exposure window. Starve them and every never-seen abusive IP gets a longer free ride; realistically a cold rebuild is days of degraded coverage with the exposure metric climbing. A normal cache refills from its origin in minutes; this one refills through a 150-rps keyhole. The vendor holds the data but sells it back too slowly to count — the mirror is the real archive, which is why replicas, point-in-time backups and rehearsed restores are first-class here, a sentence most cache designs never have to say (D5).
Key Insight: Every row but one ends in "edges unaffected" — not six mitigations but one posture: the only thing that ever reaches an edge is a signed, versioned snapshot, and an edge that stops receiving them serves the last one. Failures behind the distribution plane cost freshness and agility, never availability. The row that breaks the pattern is the one where the rate limit, not the architecture, sets the recovery time.
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson