Section 10: Pull the Plug on Every Box - Reliability, Failure & Disaster Recovery

Executive Summary: The interviewer stops asking how the design works and starts asking how it dies. The candidate's answer rests on one structural fact: Kafka is the durable record, and everything downstream (the bucket store, the live tier, the archive) is a derived view that can be rebuilt by replaying it. From it come a failure table, a four-minute worst-case live-tier rebuild, a plan for losing a zone or region, and a degrade order that spends the 43.8-minute monthly error budget on exports and history before "right now".

"Good. Now break it. Pick any box on your diagram and kill it."

Rather than reciting "we add a replica" box by box, the candidate starts with the property that makes most failures cheap.

The one property: the log is the truth, everything else is a view

"An event counts as accepted once Kafka acknowledges it. The topic event-lifecycle runs with replication factor 3, acks=all and min.insync.replicas=2, and each broker's broker.rack set to its AZ, so an acknowledged write is on at least two brokers in two AZs before the outbox marks it sent. That's where 'zero loss of acknowledged events' is enforced. Retention is 7 days, and the Parquet archive covers everything older."

"And the indexer?"

"It holds no state that matters. Every write is an upsert keyed by event_id and stamped USING TIMESTAMP version, so applying a message twice, or out of order, ends in the same rows. If I can replay, I can rebuild any table downstream of Kafka."

Key Insight: Reliability here comes from rebuildability, not from making every box unbreakable. The idempotent, version-stamped indexer from Section 8 is the same mechanism that makes disaster recovery a replay instead of a restore.

Quick check before moving on: