Section 11: The System That Forgets Fast Enough to Heal - Reliability, Failure & Mobile Realities

Executive Summary: This section breaks the system on purpose. It starts from the insight that makes the design unusually robust — a lost location update is obsolete within thirty seconds, so the hot path needs no retries — states the guarantee precisely as at most once, promptly, and contrasts it with the durable-queue reflex. Then the reconnect storm that follows a session server's death, broker shard loss, the mobile realities of battery and offline buffering, the server-driven cadence as an incident lever, regional failure, and the four dashboards that say whether it all holds.

The interviewer has watched you defend privacy in a healthy system. Now they do what production does: "A session server just died. A broker shard lost its leader. Half the fleet is on a train entering a tunnel." This is where the reflexes imported from payments and messaging — durable logs, retries, acknowledgements, exactly-once — turn out to be the wrong ones, and the candidate who reaches for them is the Realtime Maximalist by another name.

Self-Healing by Cadence

Start with the sentence the rest of the section hangs from: every phone re-sends its location every thirty seconds anyway. A dropped update is therefore not data loss; it is a value that was going to be superseded within thirty seconds regardless, and retrying it would deliver a coordinate the system already considers stale.

So state the guarantee in the interviewer's vocabulary, because the precision is worth real points: the hot path is at most once, promptly. An update reaches a friend's channel zero or one times, never twice, and if it arrives at all it arrives inside the five-second arrival budget. No at-least-once, no dedup table, no replay. The repair mechanism is not a retry — it is the clock.

The rejected alternative is the one almost everybody reaches for: a durable queue in front of the fan-out, with retries and acknowledgements on the live path. Price it honestly. You add broker persistence, consumer offsets and redelivery logic to protect a payload with a thirty-second shelf life, the retry lands after the data it carries has been replaced, and you pay in tail latency and operational surface for a guarantee the product cannot perceive. A durable log belongs only in the analytics lane, never between Maya's phone and Raj's screen.

Key Insight: This is why the location and notification path is allowed three nines while the friend-graph and bootstrap read path is held to four. An availability target states what a failure costs, not how serious you are — and a failed update here costs one skipped tick out of two per minute, invisible against a freshness SLO measured in seconds. The flip trigger is a feature that makes a single update authoritative, such as a geofence alert that must fire exactly once; then that one path earns durability.

Quick check before moving on: