Apache Storm
maang.io System Design Series
Apache Storm pioneered real-time stream processing: you build a topology — a DAG of spouts (sources) and bolts (processing) — and tuples flow through it. Its clever acker tracks each tuple's tree with XOR checksums to guarantee at-least-once processing. Studying Storm shows why later systems like Flink adopted event time and exactly-once, and where Storm still fits.
This deep dive comes in three parts:
- Foundations & Core Concepts — topologies, spouts/bolts, and stream groupings.
- Internals — tuple-tree acking, Nimbus/supervisors/workers, and Trident, with worked examples.
- Interview — Staff/Principal Q&A: reliability semantics and Storm vs Flink/Spark.
Apache Storm — Part 2: Internals & Reliability
maang.io System Design Series
What is this?
Back to our parcel-sorting hub. In Part 1 we watched packages ride the belts from docks (spouts) to workstations (bolts). Now we go into the back office and ask the questions that keep an operations manager up at night: Who assigns staff to belts? What happens when a workstation catches fire? And — the big one — how do we prove that every single package that entered the building actually made it to a chute, and re-send the ones that fell off the belt?
Storm's answer to that last question is the single most elegant idea in the whole system: a tracking desk that follows every package and all its descendants using one 64-bit number and the XOR operation — constant memory, no matter how many packages, no matter how deep the belt. Get this one mechanism and you understand Storm.
The one-line idea: Storm guarantees at-least-once processing by tracking each source tuple's entire tuple tree with a dedicated acker task that keeps a single running XOR checksum — when the checksum returns to zero, every tuple in the tree has been fully processed; if a timeout hits first, the spout replays the source tuple.
We'll build up to that. First, who runs the hub.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson