Section 10: Who Pages You When the Pager Breaks - Reliability: Monitoring the Monitor

Executive Summary: Every team relies on this platform to tell them something is broken, so it can't be allowed to break quietly. We start with independence (don't share fate with what you watch), walk each component's failure and what absorbs it, then face the dangerous case: a platform that is up but blind, where missing data looks like good news. Stale-data alerting, cross-region watchers and an external dead man's switch close the loop, and drills prove the numbers. The lesson: an alerting system's worst failure is silence, so silence has to page someone.

The interviewer leans back. "It's 02:14. Dana's latency rule should fire, but that region's Kafka cluster has been down since 02:04. What does Dana get?"

You: "Nothing, unless we designed for it. No fresh samples reach the ingesters, so the p99 series stops updating. A naive evaluator computes p99 > 800 ms over an empty window, gets false, and leaves the rule Inactive. The pager didn't fail loudly. It went quiet. That's the failure I want to design against."

Principle: Don't Share Fate With What You Watch

Ingest and alerting target 99.99% (about 4.3 minutes a month), query 99.9% (about 43). That only holds if the usual causes of a production outage don't also take out the monitoring.

• Separate Kafka. A product team's runaway producer can't fill our roughly 78 TB buffer (52 TB for logs, 26 TB for metrics), and our log storm can't slow their orders.
• Separate deploy pipeline, so one bad shared release can't break both.
• No dependency on the production service mesh or DNS where avoidable. Agents use static gateway lists; evaluators reach notifiers directly. When the mesh is what broke, signals about it still need a way out.
• Deploys staggered by cell. One outage longer than 4.3 minutes spends the alerting path's whole monthly budget, so a bad release may only ever blind one cell.
• An external paging provider, on someone else's infrastructure.

Interviewer: "Why not have the platform alert on itself?"

You: "For degradation, it should: ingest lag and dropped batches go through normal rules. For total loss it's circular. If the evaluators or notifiers are down, the alert saying so passes through them. That signal has to come from outside."