Section 6: One Webhook, Forty-Two Seconds - High-Level Architecture
Executive Summary: This section draws the whole machine on one board. Events flow through nine components in a straight line — ingest gateway, adapters, run store and health evaluator, incident service, sweeper, correlator, notifier, dashboards — with three sidecars (reconciler, watchdog, canary) guarding the gaps. We trace the 09:31:00 webhook for run #8,403 becoming incident INC-4312 at 09:31:42, watch the 11:31 sweep escalate it to SEV3, and then defend the boxes a weaker design would have drawn: no Kafka, no CEP engine, no time-series database, no bespoke pager.
The interviewer hands you the marker: "Show me the boxes — and show me how a failed run becomes a page." The second clause is the design. Section 3 proved this is a 23-events-per-second system, so the architecture isn't about moving data; it's about never losing a semantic step between a webhook you can't trust and a human you're about to wake up.
The Board, in Event Order
Draw the components in the order a single event travels, because that's the order the interviewer will interrogate them in.
| Component | Job | Key number |
|---|---|---|
| Ingest gateway | Verify webhook signature, persist raw payload, ACK | 202 in < 100 ms |
| Provider adapters | Normalize 4 providers into canonical RunEvents (D2) | ~2M events/day in, one model out |
| Run store + health evaluator | Upsert runs idempotently; evaluate rules transactionally (D4) | health transition ≤ 30 s p99 |
| Incident service | Create/attach/reopen incidents, enforce dedupe (D6) | ~50 incidents/day |
| Sweeper | Scan open incidents, apply escalation policies (D7) | 60 s tick, ≤ a few hundred rows |
| Correlator | Group new incidents by failure signature (D9) | umbrella at ≥ 50 pipelines / 10 min |
| Notifier | Drain the outbox to chat and the paging provider | event → first notification ≤ 60 s p99 |
| Dashboards / query API | Health and incident reads for owner teams | 300 teams |
| Reconciler (sidecar) | Poll each provider since a checkpoint (D1) | every 5 min, ~1.7 req/s total |
| Watchdog (sidecar) | Stuck runs; per-provider freshness | timeout + 10 min; silence alarm at 15 min |
| Canary (sidecar) | Synthetic failure through the whole path (D11) | hourly, expects incident within SLO |
Two structural points to make out loud. First, the reconciler is not a second ingestion system: it feeds the same adapters that webhooks do, and because every event upserts by (provider, external_run_id, attempt, event_type), the two sources converge on identical rows (D1). One code path, two freshness tiers. Second, everything between the adapters and the notifier is reading and writing one relational database (D3) — the run store, health rows, incidents, policies, and the transactional outbox are tables, not services.
Interviewer: "Why does the sweeper go through the incident service instead of paging directly?"
You: "Because every state change — created, escalated, paged, acked — must land in the incident timeline and the outbox in one transaction. A component that pages out-of-band is a component whose actions you can't audit or replay. The sweeper decides; the incident service records; the notifier delivers."
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson