Section 10: Comments Can Fail; Video Cannot - Reliability, Failure & Disaster Recovery

Executive Summary: This section breaks the design on purpose. We separate the small durable core from disposable soft state, walk every component through a failure table, trace a gateway crash, do the reconnect math for losing a region, and close with a degradation ladder that never touches the video.

The interviewer caps the marker. "Good. Now let's break it. Pick a box and kill it."

Before killing anything, state the rule that decides every recovery: video playback never depends on comments. Every comment component sits beside the video pipeline, never in its path. Then the promises: delivery at 99.95% (21.6 minutes per 30-day month), posting at 99.9% (43.2 minutes), and one with no budget at all: an acked comment is never lost.

Two Things Must Survive; Everything Else Can Die

You: "Only two components hold state I'm not allowed to lose. The Comment Log holds every acked comment: replication factor 3, min in-sync replicas 2, acks=all. The Comment Store holds the history. Everything else is soft state: gateway subscriptions, the 60 s ring buffers, router leases, the reverse friend index, the router's last 1,000 comments. All of it can be rebuilt from the log or from a reconnect."

Interviewer: "Why not replicate the gateway's connection state to a hot standby?"

You: "That's 50K connections' cursors and lane state, replicated on every comment, to save a reconnect costing one handshake plus a ring-buffer read. I'd revisit it only if every reconnect meant a full backfill."

Key Insight: Reliability here comes from how little must survive. Keep the durable core small, and every other failure turns into "reconnect and resume" rather than "restore from backup".