Section 12: Consumers, Not Forks - Follow-up Feature Evolution

Executive Summary: This section answers the interviewer's closing move: six months later, the org wants the monitor to fix things, leadership wants DORA metrics, the business wants to sell it as SaaS, and someone asks where the ML is. Every evolution is a new consumer of existing primitives (D12): auto-remediation consumes health transitions behind propose-then-act policy gates; DORA analytics reads a CDC replica of the run store, whose 13-month normalized history is the real asset; multi-tenant SaaS is the day D3's and D7's flip triggers finally fire, while the canonical model, dedupe constraint and severity clock survive untouched; ML advises while rules keep paging authority, because the rules generate the labels the ML needs.

The whiteboard is full and the design just survived its own worst day in Section 11. The interviewer leans back for the classic closing move: "Six months later: teams are tired of pressing 'rerun' themselves, the VP wants DORA metrics, the company decides to sell this as a product, and a new hire asks why a modern system pages off hand-written rules. What changes?" The question audits your earlier self: Section 1 scoped remediation out, Section 3 promised D3's flip triggers would matter someday, Section 7 deferred ML with a one-line boundary. Later is now.

Auto-Remediation: The Monitor Learns to Act

You: "Nothing changes in detection. A health transition is already a durable, replayable record — the evaluator writes it transactionally with an outbox entry (D4). Remediation is a new consumer of that stream (D12), not new logic inside it."

Three actions, deliberately bounded, in rising order of ambition:

Bounded auto-rerun. Section 7's retry-pass tagging already marks runs flaky-suspect — a failure whose signature has historically passed on retry. When a transition's triggering run carries that signature, the remediator may request exactly one rerun, and never on a pipeline tagged release-blocking. The bounds are the design: unbounded reruns turn the monitor into a flake-laundering machine, and a release-blocking pipeline is precisely where a human must look before anything retries on their behalf.

Quarantine proposals. When the same flaky-suspect test signature recurs across pipelines, the remediator opens a quarantine proposal — a ticket with the evidence attached — never an automatic quarantine. A quarantined test is a disabled safety check, and disabling safety checks is a decision teams make with the data in front of them.

Culprit bisect. Three consecutive failures (the D6 signal class behind INC-4312) define a suspect range: the commits between the last green run and #8,401. The remediator schedules bisect runs across that range — they cost real CI runner compute, but a handful of targeted runs is cheap against a day of blocked deploys — and annotates the incident with the culprit commit.

The governing rule for all three is propose-then-act with team policy gates: each team's escalation policy grows a remediation block declaring which actions may auto-execute and which only propose. Orion might allow the auto-rerun and require approval for bisect; a payments team might allow nothing. Every remediation event — attempted, succeeded, failed, proposed — is appended to incident_events, the timeline Section 5 built and Section 8 writes to. Had #8,403's failure signature matched a known flake, the remediation policy could have fired an auto-rerun at 09:32 and recorded it there — for INC-4312 it correctly stayed silent, because nothing about this failure looked like flake. Either way the on-call learns something: a remediation that acted and failed is itself diagnostic signal, and so is one that declined to act.

Interviewer: "Why cap the rerun at one? If it's flaky, retry until green."

You: "Because 'retry until green' is how organizations stop knowing their flake rate. One rerun distinguishes flake from breakage; five manufacture green. The rate rule still counts flaky-suspect failures (D10's spirit: flake is real cost), so the signal survives the convenience."

Quick check before moving on: