Section 9: One Page for a Thousand Fires - Deep Dive: Storm Correlation & Noise Control

Executive Summary: Section 8 perfected the lifecycle of one incident; this section defends the system on the day 1,100 of them try to open at once. The correlator (D9) fingerprints every new incident candidate with a cheap failure signature — provider, runner pool, failing stage, error class — groups candidates in a 10-minute sliding window, and past 50 distinct pipelines folds them into one umbrella incident, INC-4388, suppressing child paging while every child still records. We replay the 14:02 runner-pool-eu outage decision by decision, show why the morning's INC-4312 stays separate, resolve the umbrella honestly, and then zoom out: correlation is one instrument in a noise-control budget — Section 2's <5% false-page bar — that the design spends everywhere, because alert fatigue is how monitoring systems die.

The interviewer writes a time on the board. "14:02. Your EU runner pool loses its network path. Eleven hundred pipelines fail inside ten minutes. Section 8 says each one deserves an incident and a notification. Do 1,100 phones ring?"

If they do, the design is dead — not technically, reputationally. By 14:15 every team in the company knows the pipeline monitor cried wolf a thousand times about one broken runner pool, and the muting begins. The requirement from Section 2 was exact: 1,100 pipelines failing in 10 minutes produce approximately one page. This section is the machinery that cashes that promise.

The Failure Signature: Cheap on Purpose

Correlation starts with a fingerprint. When the health evaluator proposes an incident (D5), the correlator computes a failure signature from fields we already have on the canonical run: the provider, the runner pool or label the run was scheduled on, the failing stage, and an error class derived from the exit code and the first lines of output — runner-unreachable, provision-timeout, test-assertion, oom.

Notice what the signature is not: it is not log mining. We do not parse stack traces, diff failure messages, or embed anything. Four cheap, already-normalized dimensions (D2 bought us this — every provider's run looks the same by now) are enough to separate "the EU pool is down" from "Orion's checkout test is broken," and they cost microseconds per candidate. Root-cause precision belongs to humans reading the umbrella; the signature only has to answer one question: are these failures plausibly the same event?

Quick check before moving on: