Section 10: Six Ways to Break the Monitor - Interviewer Deep-Dive Challenges

Executive Summary: The deep dives are done; now the interviewer attacks. Six challenges, each aimed at a promise an earlier section made: fresh data, honest detection, trusted pages, a monitor that notices its own death, exactly-one incident, and a correlator that fails safe. Each gets the same shape: what they're really asking, the answer with its mechanism and number, and the senior line. The through-line is the Section 2 tension — sensitivity versus trust — because every one of these attacks is really an attack on credibility.

The interviewer leans back. "Nice machine. Let's see what it does when the inputs lie, the data is sparse, and the humans stop believing you." A Senior candidate defends every component. A Staff candidate concedes what's physics, names the mechanism that bounds the damage, and gives the number.

Challenge 1: "GitHub webhooks go down for two hours."

What they're really asking: Did you actually design for untrustworthy webhooks, or did you just say "at-least-once" and move on?

You: "Nothing is lost, and we know within 15 minutes." Webhooks were never the source of truth — D1 pairs them with a reconciliation poll every 5 minutes against a per-provider checkpoint, so each cycle closes the gap within 10 minutes. The per-provider freshness alarm (D11) notices business-hours silence at 15 minutes, so the on-call learns GitHub is dark before GitHub does. The honest cost is latency: detection degrades from 30 seconds to roughly 5 minutes, poll-cadence instead of push-fresh. When webhooks return, the backlog replays harmlessly: every event upserts by (provider, external_run_id, attempt, event_type), the evaluator is a deterministic state machine (D4), and incident creation is idempotent under the partial unique index (D6) — so if run #8,403's failure arrives twice, once by late webhook and once by poll, INC-4312 is still one row.

The senior line: "Webhooks are an optimization on top of reconciliation, not the other way around. Losing them costs latency, never truth."

Quick check before moving on: