Section 7: The Machine That Refuses to Guess - Deep Dive: The Health Detection Engine

Executive Summary: This section opens the evaluator from Section 6's architecture and shows the decision it makes a hundred thousand times a day. The engine is a pure, replayable function (D4): terminal run event plus last-20 window, 30-day baseline, current health row, and the clock in — verdict out. Three rules carry it: consecutive failures at N=3 with final-attempt semantics (where the false-page arithmetic lives), rate-vs-baseline at +25 percentage points with a hard abstention floor (D10), and a stuck-run watchdog that catches what webhooks never reported. The theme throughout: on sparse data, the honest answers are "degraded," "healthy," and "I don't know yet" — and a detector that can't say the third will lie to you with the first two.

The interviewer circles the evaluator box. "This is where your system earns its keep. Run #8,403 just failed. Walk me through exactly how that becomes DEGRADED — and convince me it won't page Orion for noise."

A Deterministic, Replayable Decision

You: "The evaluator is a pure function wrapped in one transaction. Inputs: the new terminal run, the pipeline's last 20 runs, its 30-day baseline, its current health row, and the clock. Output: one verdict — stay, DEGRADED, UNHEALTHY, or recovered — plus the health transition and an outbox record, written atomically (D4)."

Determinism is the property everything downstream leans on. Re-feed the same canonical events and you get the same transitions — which is how we debug a 3 a.m. page the next morning ("replay Tuesday, show me the verdict at 09:31"), and how rule changes get tested against 13 months of history before they ship. The rejected alternative was in-memory streaming windows: state that dies on restart and can never be replayed. A rules-engine/CEP product was rejected too — generality we'd pay for and never use. There is no model, no learned weight, no sampled randomness anywhere in the path. Every verdict can be reconstructed from rows in the database.