Section 12: Every Successor Fixed a Flip Trigger - Follow-up Feature Evolution: From Storm to Heron, Flink & Kafka Streams

Executive Summary: The 2014 paper's Storm is the design we built: record-at-a-time topologies, a stateless Nimbus, mixed executors in shared worker JVMs, and at-least-once through the XOR acker. Everything after (Storm's later releases, Heron, Flink, Kafka Streams) pulled one or more of our flip triggers. Heron reversed Decisions 5, 6 and 7 for debuggability and flow control. Flink reversed Decisions 3, 4 and 9 by making exactly-once state, event time and disk-backed state part of the engine. Kafka Streams made Decision 8's replayable log the whole platform. The reasoning carries over; the mechanisms changed.

Interviewer: "You've designed Storm as the 2014 paper describes it. The Heron paper, a year later, describes Twitter replacing it. Was the design wrong?"

You: "No. Its workload grew past its trade-offs. Every successor fixed a flip trigger from our ledger, so I'll go through them that way and keep versions loose."

Storm Grew Its Own Answers

Storm itself absorbed several flip triggers. Treat the release grouping as approximate:

• Trident, a micro-batch API with transactional and opaque-transactional state. This is Decision 3's flip trigger (billing-grade counts) handled inside Storm, at the cost of batch latency. Section 9 priced it.
• Netty transport for Section 6's worker-to-worker data path.
• Nimbus HA with leader election. Decision 6's flip trigger: in Section 10's sup-07 timeline, a dead Nimbus would have left those 4 workers unassigned.
• Automatic backpressure, which throttles spouts when a bolt's queue fills, instead of relying only on max.spout.pending (Decision 5).
• A resource-aware scheduler and isolation scheduling for shared clusters (Decision 11, Section 11's "fifty teams").
• Windowing and stateful-bolt checkpointing APIs: built-in sliding and tumbling windows (which buffer the window's tuples rather than keeping bucketed counts) and checkpointed state: the library versions of what Jordan hand-built in Section 9.
• A separate heartbeat service (Pacemaker), so worker heartbeats stop loading ZooKeeper, the pressure Section 10 flagged.
• A 2.x core rewritten largely in Java.

Key Insight: Storm never dropped the acker or the record-at-a-time model. Every addition was opt-in beside the original core. That kept old topologies running, and it kept the original costs too: 2.4 M acker messages a second and mixed tasks inside one JVM.