Section 10: Designing for the Bad Day - Reliability, Security & Disaster Recovery
Executive Summary: Every box from Sections 4 to 9 fails eventually, so we say what each failure costs: lost data, late data, or nothing. A backpressure chain from storage to the agent's WAL means acknowledged data is never lost, and unacknowledged data is lost only when a WAL fills, P2 first, every drop counted. A region fails over to its paired stamp in at most 15 minutes with RPO 0. Tenant isolation is enforced at every layer, the P0 archive is immutable, and the platform is treated as the first thing an attacker wants to blind.
The interviewer puts the marker down. "It works on a good day. Now break it."
Don't wait to be told where. Walk the pipeline in order and say, for each part, what is gone for good.
What Breaks, and What It Costs
| Component | Failure | What customers notice | Mitigation | Lost or delayed |
|---|---|---|---|---|
| Agent host | Crashes or loses network | That host goes silent | WAL on disk; replays on resume with jittered backoff | Nothing acked. Destroyed disk: only its un-acked tail |
| Front door | A node, or a whole zone (3 nodes) | Nothing | 9 nodes, 3 per zone; losing a zone leaves 6 = the 30k req/s design | Nothing; un-acked requests are retried |
| Event Hubs | Zone loss | Brief throttling at most | Zone-redundant namespace; producers retry | Nothing acked (ack waits for the durable write) |
| ADX writer or ADX cluster | Lag, or cluster down | Recent data missing from hot queries | Queue holds 72 h; lake unaffected; per-table freshness shown | Hot freshness delayed |
| Lake writer | Down | Nothing in hot queries | Queue holds 72 h; ADX unaffected | Lake and archive commits delayed past the 15 min p99 |
| ADX data | Corrupt or missing extents | Wrong results for a time range | Rebuild the affected days from the lake, the system of record | That range, until it's rebuilt |
| Agent rollout | Bad version crashes or floods | A ring goes silent or noisy | Rings (canary 1% first); auto-halt on silence and seq-gap metrics | Contained to one ring |
| Poison batch | Writer can't parse it | Nothing | Dead-letter to blob with a reason; the partition keeps moving | That batch, until replayed |
| Region | Stamp unreachable | About 15 min of ingest errors, then slower history | Paired stamp (below) | RPO 0 for acked data; hot history rebuilt over hours |
The last column says "delayed" or "nothing acked" almost everywhere. That's what the ack point bought: it waits for the zone-redundant queue, and ADX and the lake read that queue independently, so losing one store never stalls the other.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson