Section 11: The Watcher's Own Bad Day - Reliability, Failure & DR
Executive Summary: This section pays the debt Section 10's challenges left open: show the failure matrix. We walk every component — ingest gateway, durable queue, evaluator, the database that is the honest single point, the notifier and its external edges, the providers' own rate limits — and show what still works and how each recovers. We rank the degraded modes by harm, map each to the design choice that bounds it, draw the honest DR line for an internal tool, cover the data-loss nuance and the security brief, and give D11 its full treatment — ending with the question every monitoring system must answer: what is your SEV1?
"You page my on-calls. Show me what happens when you break." The right opening stance: almost every component here can die and the outcome is lateness, not loss — and the one outcome lateness can't excuse, dying silently, is bounded by machinery that lives outside the system entirely.
The Failure Matrix: Everything Converges, One Thing Pauses
Take the components in the order Section 6 drew them, answering two questions each: what still works, and how do we get back?
| Component down | What stops | What keeps working | Recovery |
|---|---|---|---|
| Ingest gateway | Webhook intake at the edge | Some providers retry briefly, most — GitHub included — don't; the real backstop is the reconciler's 5-minute poll, which rebuilds every missed run through the same adapters (D1) | Stateless; redeploy behind the load balancer. The run store converges — detection latency degrades from 30 s toward ~5–10 min, nothing is permanently lost |
| Durable queue | The buffer between 202 and processing | Gateway sheds; whatever the few retrying providers don't redeliver, reconciliation closes the gap on | Managed queue with its own replication; this is why it's a boring bought queue, not a Kafka cluster we operate (D3) |
| Evaluator | Health verdicts | Events buffer in the queue; raw payloads are already persisted | Restart and replay — evaluation is a deterministic, transactional function of stored state (D4), so catch-up yields exactly the verdicts real-time would have |
| Database | The honest single point: detection, dedupe, escalation, dashboards | The gateway still 202s and the queue buffers, so ingestion never lies to providers; the external pager's dead-man's switch covers the paging gap (D11) | Synchronous replica failover: RTO minutes, RPO ≈ 0. Drain the queue; the sweep and the evaluator are pure functions of the tables, so they resume mid-stride |
| Notifier / Slack / pager | Message delivery | Every notification intent is an outbox row committed with the state change — durable, retried with idempotency keys. The pager is a bought edge with its own SLA | Outbox drains on recovery in order; nothing to reconstruct |
| Provider APIs rate-limited | Reconciliation freshness | Webhooks still flow; the freshness alarm distinguishes "we're being limited" from "they've gone silent" | Reconciler backs off per provider; at ~1.7 req/s total we sit far inside every provider's budget, so this is a transient, not a design flaw |
The ingest-gateway row deserves a sentence said slowly, because it is the deep reason D1 exists. Webhook-only designs treat the edge as sacred: drop a webhook, lose an outage. Here the webhook path is merely the fast path. The reconciler polls every provider from a stored checkpoint into the same adapters, and because every event upserts by (provider, external_run_id, attempt, event_type), the run store converges on identical rows whichever source delivered them. An edge outage degrades a 30-second detection into a ten-minute one — noise, for escalation steps measured in hours (+2 h to SEV3, +8 h to SEV2).
Key Insight: Almost every row ends in "converges" or "resumes" because state lives in two reconstructible places — the providers' APIs and one transactional database — and every process between them is a replayable pure function of that state. The matrix isn't six mitigations; it's one posture, bought with D1, D3 and D4 back in Sections 4–7.
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson