Section 11: Neither Spam Nor Silence - Reliability, Failure Modes & DR
Executive Summary: This section breaks the design one component at a time: a failure matrix, a degradation ladder that sheds the long tail before hot products or delivery, the signals that show which way the system is failing, and a warm-standby DR plan (RPO ≤ 1 min, RTO ≤ 30 min). The thread: a price-alert system fails in two directions, spam and silence, and every guard must block one without causing the other.
The interviewer turns a page: "Fine, it works on a good day. Now break it. What fails, what does the user notice, and how do you get back?"
The Two Failure Directions
You: "This system hurts a user in two ways. Noisy: a duplicate push, or a $0.01 bad read announced to 500,000 people. Silent: the Prime Day $278.00 comes and goes and Maya never hears. Retry harder and you get duplicates; retry less and you lose notifications."
| Direction | Cause | Guard |
|---|---|---|
| Spam | Redelivered intents, crash after 201, crash mid-band |
Deterministic notification id, conditional-put ledger, Topic header, client dedupe |
| Spam | Bad API read or momentary flicker, spoofed hints | Mass fan-out guard (D13); hints never notify (D1); a live misprice is real and is notified with an "as of" time (Section 10) |
| Silence | Intent lost between evaluator and sender | Transactional outbox in the evaluation transaction (D6) |
| Silence | Stuck deliveries | Ledger terminal states + reconciliation sweeper |
| Silence | Stale prices | Tiered freshness SLOs + degradation ladder |
Key Insight: The two correctness SLOs, duplicates < 0.01% and every intent reaches a terminal state, are the spam guard and the silence guard written as numbers. A failure plan that only measures availability misses both.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson