Section 8: The Clock Nobody Can Argue With - Deep Dive: Incident Lifecycle & Escalation

Executive Summary: Section 7 ended with a verdict and deliberately no opinion about what happens next; this section is that opinion, mechanized. A health transition meets a dedupe decision arbitrated by Section 5's partial unique index — under real concurrency, where two inserts race and the loser attaches (D6). The surviving incident carries a severity clock anchored at the moment degradation began, advanced by a 60-second sweeper that applies policy steps idempotently (D7), paused by nothing — ack pauses repaging, never the clock, and severity never walks back down (D8). Resolution comes only from three measured greens, with a 30-minute flap cool-down that reopens rather than re-creates. We run INC-4312's entire Tuesday — 09:31:42 to 16:20, 6 h 49 m — through every one of these mechanisms.

The interviewer takes Section 7's handoff and makes it concrete: "Your evaluator flipped checkout-service-deploy to DEGRADED at 09:31:05. A verdict is not a page. Who decides what happens to a DEGRADED — and how do you stop that decision from being made twice?"

You: "The incident service decides, and it decides inside a transaction (D5: a health transition proposes, the incident service disposes). And the 'twice' problem is exactly where I want to start, because it's where naive designs quietly double-page people."

The Dedupe Decision, Under Real Concurrency

The decision tree looks innocent: is there an open incident for (pipeline, signal_class)? Attach. Was one resolved less than 30 minutes ago? Reopen it. Otherwise, insert a new one. The trap is that this tree is evaluated concurrently. Webhook-driven and reconciliation-driven evaluations (D1) can commit the same DEGRADED transition seconds apart; two evaluator instances can process retried deliveries of the same event in the same instant. Both read "no open incident." Both proceed to insert.

This is why the check in the tree is advisory and Section 5's partial unique index — one open incident per (pipeline_id, signal_class) — is the real arbiter (D6). Both transactions insert; the database commits exactly one. The loser catches the unique violation, re-reads, finds the winner's row, and attaches: its evidence becomes an incident_events timeline entry on INC-4312 instead of a second incident. No lock to hold, no dedupe cache to invalidate, no coordination beyond the constraint that was already there. Exactly-once incident creation as an effect layered on at-least-once everything.

Quick check before moving on: