Section 10: Rebuild the Index, Never the Truth - Reliability, Failure & Disaster Recovery
Executive Summary: Cassandra and Kafka are the truth. Elasticsearch, the is_current flags, the pointer cache and Iceberg are derived. That split decides every recovery. A Cassandra AZ can die and LOCAL_QUORUM carries on, flips included. A lost search cluster comes back from an hourly snapshot in 2.8 h plus a short replay, not an 18.5 h reindex. A dead operation service pauses flips while readers see whole old runs. A lost region costs at most 5 minutes of writes. A four-rung degradation ladder ties it together.
The interviewer leans back. "You've built it. Now break it. What fails, and what does Priya see when it does?"
Two Piles: Truth and Derived
You: "One sort first. Cassandra holds annotations, operations and scope pointers. Kafka holds every acknowledged write for 3 days. Everything else can be rebuilt from those two. Truth needs replication. Derived data needs a fast rebuild path."
| Failure | Blast radius | What keeps working | Recovery |
|---|---|---|---|
| Cassandra node or AZ (8 of 24 nodes) | One replica per token range | RF 3 over 3 AZs; LOCAL_QUORUM needs 2 of 3 |
Hints, then repair |
| One ES data node | Its primary shards | Replicas promoted in seconds | Re-replicate |
| An ES AZ (5 of 15 nodes) | One copy of about two-thirds of the shards | The other copy, in another AZ | Run yellow until it returns |
| Whole ES cluster | All search; flips paused | Per-entity reads and rollbacks from Cassandra | Snapshot + replay, ~3 h |
| Kafka broker or AZ | Some partition leaders | RF 3, min ISR 2, acks=all |
Leader election |
| Store writers behind | Bulk freshness | Reads; interactive edits | 429 to bulk producers |
| Index drift | Missing or stale docs | Pointer + post-filter | Reindex the op |
| Operation service | No STARTs, no flips | Every read, on old runs | Restart; leases paused |
| Primary region | Everything | Async replica DC elsewhere | RPO 5 min; reads 30 min, search ~4 h |
Two rows need a sentence. After a Cassandra AZ loss, 16 nodes serve and coordinators store hints for the missing replicas. Section 3 kept disks at 50% full, which leaves room for hints and the repair that follows. Flips still work, because the LWT's Paxos round needs a quorum, and 2 of 3 is one.
The ES AZ loss has a trap. Each data node holds 1.5 TB, 75% of a ~2 TB disk, and the lost AZ held one copy of about two-thirds of the shards. Re-creating them on the 10 survivors pushes each toward 20 TB / 10 = 2 TB on a ~2 TB disk. Forced awareness won't stop that with three zones and two copies, so we use delayed allocation (index.unassigned.node_left.delayed_timeout) for short outages and cluster.routing.allocation.enable: primaries for long ones, with the disk watermarks as the backstop. The cluster runs yellow until the AZ returns.
Key Insight: The pointer lives in the truth pile. Rollbacks and per-entity reads run on Cassandra alone, so losing every Elasticsearch node costs catalog-wide search, never correctness. New flips do wait: the commit check counts in ES, so sealed runs queue until the index is back, and their 2-hour FAILED timer pauses, just as leases pause when the operation service is down.
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson