The Watcher That Must Earn Its Pages: Designing a CI/CD Pipeline Health Monitoring System

It's 09:31 on a Tuesday. A checkout service's deploy pipeline has just failed for the third consecutive time on main. Forty-two seconds later, the incident is in the owning team's channel: severity four, routed to the team that owns the pipeline, with the three failing runs linked and the common failing test named. Nobody filed it. At 11:31, with the pipeline still red, the incident promotes itself to severity three and pages the on-call. At 16:20, after the third green run in a row, it closes itself and writes down that the degradation lasted six hours and forty-nine minutes. The system that did all of this handles about twenty-three events a second — and that gap between how small the data is and how much judgment the design needs is exactly why interviewers ask it.

Why Interviewers Love This Question

"Monitor 2,000 pipelines" sounds like a weekend of scripting against a CI API. The interview is designed to find out whether you can see the three traps under the surface:

• The Kafka Reflex: The candidate hears "2,000+ pipelines across the organization" and sketches Kafka, Flink and a time-series database. The arithmetic says 23 events per second. Signal: reaches for machinery before doing the napkin.
• The First-Failure Page: The candidate pages on every red run. At typical flake rates that's a false page somewhere in the org every few minutes, and within a month every team has muted the system. Signal: no feel for alert economics.
• The Ack-Stops-the-Clock Trap: The candidate lets acknowledging an incident freeze its severity. Incidents rot at low severity for days while the pipeline stays broken. Signal: confuses attention with recovery.

The passing performance turns each trap into a design decision:

• One relational database, defended with arithmetic: transactions power idempotent ingest and incident dedupe; the streaming stack is named, rejected, and given an explicit flip trigger.
• Rules that know when to abstain: consecutive failures, rate against a per-pipeline baseline, and a stuck-run watchdog — with "insufficient data" never reported as healthy.
• A severity clock that measures degradation, not attention: acknowledgment pauses paging, never the clock; resolution comes only from measured recovery.
• Storms folded to one page: a runner-pool outage that fails 1,100 pipelines in ten minutes becomes one umbrella incident, not a 3 a.m. page storm.