Section 2: Sharp Enough to Catch, Quiet Enough to Trust - Requirements (Functional & Non-Functional)
Executive Summary: This section converts Section 1's assumptions into a contract: functional requirements the system must demo against checkout-service-deploy, and non-functional numbers a test can check, such as incident creation in ≤ 60 s p99 and a 1,100-pipeline storm folded into roughly one page. We name what stays out of scope and why, survive two pushbacks — "isn't this just Datadog alerts?" and "why doesn't ack stop escalation?" — and frame the tension that disciplines every later section: sensitivity versus trust.
The clarifications are done, and the interviewer leans back: "So what exactly are we committing to?" Two audiences hide in that question. The 300 owner teams want to be told early and told once when their pipeline is genuinely broken. The platform team wants a system whose pages are believed — because the moment a team mutes the channel, the monitor is worse than nothing. Every requirement, then, must be a promise a test could fail. "Detect degradation quickly" can't fail. "A terminal run event flips pipeline health within 30 seconds at p99" can.
The Functional Contract
You tie each requirement to the running example so the interviewer can picture Tuesday's demo:
| Requirement | What it means | The checkout-service-deploy demo |
|---|---|---|
| Ingest from 4 providers | GitHub Actions, Jenkins, GitLab, internal runner, normalized to one canonical run model, idempotent under duplicates (D1, D2) | Run #8,403's webhook arrives twice; the store records it once |
| Per-pipeline health state | HEALTHY / DEGRADED / UNHEALTHY, derived and recomputable, separate from incidents (D5) | Three failures on main flip health to DEGRADED at 09:31 |
| Three core detection signals | Consecutive failures (N = 3 on the default branch), failure rate over the last 20 runs exceeding the 30-day baseline + 25 points (minimum 10 runs, else abstain — D10), and a stuck-run watchdog | #8,401–#8,403 trip the consecutive rule; nightly-data-archive at 1 run/day is carried by the other two |
| Auto-created, deduped incidents | Exactly one open incident per pipeline and signal class, routed by ownership (D6) | INC-4312 created once, posted to #orion-ci with the three run links |
| Escalation policy engine | Default ladder SEV4 → SEV3 at +2 h → SEV2 at +8 h; release-blocking tag opens at SEV2 immediately |
Still failing at 11:31, INC-4312 walks to SEV3 and pages Orion's on-call |
| Ack / resolve lifecycle | Ack pauses paging only (D8); 3 consecutive greens auto-resolve; a re-degradation within 30 min reopens the same incident | Ack at 11:34; three greens by 16:20 close it with a 6 h 49 m duration |
| Storm umbrellas | Correlated mass failures collapse into one umbrella incident; children linked, their paging suppressed (D9) | The 14:02 runner-pool-eu outage becomes INC-4388, not 1,100 pages |
| Audit & timeline | Every transition, page, ack and resolve is an append-only timeline entry | "Who acked INC-4312, and when did it hit SEV3?" |
| Dashboards & API | Health and incident queries for 300 teams; per-pipeline threshold overrides | Orion raises N from 3 to 5 for a known-flaky suite |
The first row the interviewer attacks isn't a row at all — it's the table's existence.
Interviewer: "Stop. Isn't this just Datadog alerts on a failure-count metric? Why are we designing anything?"
You: "Generic metric alerting has no run semantics. It can't say 'three consecutive failures on the default branch, where a canceled run doesn't count and a retry counts only its final attempt.' It has no per-pipeline baseline, so a 20%-flaky integration suite and an always-green deploy pipeline would share one threshold. It can't dedupe into a durable incident with an ownership route, pause on ack, damp a flap, or fold a runner-pool outage into one umbrella. What I will buy rather than build is the paging edge — the last mile of phones and rotations is a solved product. Everything upstream of that page is run semantics, and that's ours."
Key Insight: The build-vs-buy line lands exactly where domain semantics end. Buy the delivery of pages; build everything that decides whether a page deserves to exist.
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson