Section 7: Silence Is the Worst Failure - Reliability, Multi-Region Deployment & SRE Architecture
Executive Summary: Everything so far assumed the machines stay up. This section drops that assumption. We define reliability for a detector as three promises (no acknowledged event lost, fresh detections in 99.9% of minutes, liveness proven), survive a zone, score nine failure scenarios, state plainly what a region loss costs, and make the Canary Injector the instrument behind the freshness SLO. A crashed detector looks exactly like a quiet day, so the system has to prove it is alive.
The interviewer leans back. "Fine. Now a node dies. Then a zone. Then West US 2."
When a web service fails, users complain within seconds. When a detection service fails, nothing happens. No alerts arrive, which to Contoso's SOC looks like a calm Monday while 203.0.113.45 works through alice's password. So before discussing any failure, you write down what "reliable" has to mean here.
Three Promises, Each With a Number
| Promise | Number | What keeps it |
|---|---|---|
| Durability | An event acknowledged by the Ingest Edge is never lost | Zone-redundant hubs; normalized hub retained 3 days |
| Freshness | 99.9% of minutes meet p99 ≤ 30 s → error budget 43.8 min/month per stamp | 2× stamp headroom, eager firing, effectively-once output |
| Liveness | Proven every minute, per stamp | Canary Injector |
Durability is never traded. Freshness gets spent, and the error budget says how much: a failure that delays detections draws down the 43.8 minutes, while one that loses acknowledged events breaks the contract.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson