Section 10: The Topology Outlives Its Machines - Reliability: Worker, Machine & Nimbus Failure

Executive Summary: Sections 8 and 9 promised that unfinished trees replay and count state returns from a 30-second checkpoint plus Kafka replay. Here those promises meet real failures. A worker crash costs about 5–10 s to restart the JVM, but if it held a count executor, its doomed tuple trees throttle ingest until they time out at 30 s. A machine loss (sup-07, workers 25–28) is worse than its 8% of executors suggests: about 12% of posts' trees get stuck, ingest throttles to a trickle for 60–90 s until Nimbus moves the workers into spare slots, and ≈ 36–54 M posts of Kafka backlog then take ≈ 3–4.5 min to drain. A Nimbus loss costs no throughput, only the ability to change anything. Duplicates stay inside ±1% throughout, and nothing is lost. We close with how Jordan operates the topology and the four metrics that show trouble first.

Interviewer: "You've told me what the system promises. Now break it. Pull a machine at peak and walk me through the next ninety seconds."

You: "First I'll split failures by who notices and who repairs, since that sets each recovery time. Then I'll pull sup-07."

Three Failure Domains, Three Repairers

Section 6 kept the control plane off the data path (Decision 6), so failures separate cleanly. Each layer has its own watcher:

• A worker JVM is watched by its local supervisor, which reads the worker's local heartbeat and restarts it on the same slot.
• A supervisor machine is watched by Nimbus, through heartbeats stored in ZooKeeper.
• Nimbus is watched by nobody in the data path. Tuples never pass through it.

The 2014 paper describes Nimbus and the supervisors as fail-fast and stateless (state in ZooKeeper or on local disk), so a process monitor simply restarts a dead daemon. The detection times below are our interview's assumptions; the relevant timeouts vary by release and setting.