Section 13: The Page You Didn't Send - Final Summary & Candidate Scorecard

Executive Summary: This closing section compresses twelve sections into the artifacts you can rehearse: a sixty-second recap built on one chain of custody and three clocks; the decisions ledger, D1 through D12, each with its rejected alternative and flip trigger; the three traps from the module's opening and the arithmetic that closes each; a scorecard for mid-level, Senior and Staff performances on this exact question; the variants interviewers reach for next; and how to practice. Nothing here is new — everything is the design you already built, folded small enough to carry into the room.

The interviewer's last request is "Summarize it for me," and it is scored: a design you cannot compress is a design you do not own. Everything below is that compression.

The Sixty-Second Recap

The system is one chain of custody for a fact nobody hands you reliably. Webhooks you can't trust — they carry no delivery guarantee: duplicates, reordering, and roughly 0.1–1% silent loss — arrive at an ingest gateway that ACKs fast and never does work inline, while a 5-minute reconciliation poll converges whatever they dropped (D1). Adapters normalize four providers into canonical run events (D2) feeding transactional detection: a deterministic evaluator that reads a 20-run window, a 30-day per-pipeline baseline and the current health row inside one transaction, and abstains rather than lie on sparse data (D4, D10). Health transitions propose; the incident service disposes, and a partial unique index makes incident creation an exactly-once effect on at-least-once delivery — deduped incidents by constraint, not code discipline (D5, D6). The severity clock measures degradation, not attention: a 60-second sweeper walks open incidents up SEV4 → SEV3 at +2 h → SEV2 at +8 h while health stays bad; ack pauses paging, never the clock (D7, D8). A correlator folds storms into one page — 1,100 pipelines failing in ten minutes becomes one umbrella incident, children linked and suppressed (D9) — and the monitor watches itself from outside: an hourly canary, per-provider freshness alarms at 15 minutes, a dead-man's switch living in the external pager (D11).

That chain runs on three clocks. Detection ticks in seconds — terminal event to health transition in ≤ 30 s p99, to incident plus first notification in ≤ 60 s. Escalation ticks in hours — the +2 h and +8 h policy steps a sweep applies. Baselines tick in weeks — the 30-day per-pipeline failure rate, held in 13 months of run history, that makes "unhealthy" mean worse than your own normal.

And the Tuesday story, in three sentences. At 09:31, run #8,403 is the third consecutive failure of checkout-service-deploy on main, and INC-4312 is created within seconds of it — DEGRADED committed at 09:31:05, the SEV4 incident inserted right behind it, and the notice posted to #orion-ci at 09:31:42 with the three run links and the common failing test. At 11:31, still red, the sweeper promotes it to SEV3 and pages Orion's on-call; the 11:34 ack pauses the repaging and nothing else. At 16:20, the third consecutive green run auto-resolves it and writes the number that matters to the timeline: 6 hours 49 minutes of degradation.

Quick check before moving on: