Section 9: Counting Once When Everything Happens Twice - Deep Dive: State, Windows & Exactly-Once

Executive Summary: Section 8's at-least-once promise runs out in the count executors' memory. Each holds its share of the five-minute window as a ring of 60 five-second buckets per hashtag, about 84 MB, and loses it when its worker dies. Our design (Decision 4) checkpoints the rings every 30 s with the Kafka offsets they cover, then replays the posts between the checkpoint and the crash to the one executor that lost state; the normal acker replay and Kafka's backlog cover however long it stayed down. That gets close to exact, not all the way: core Storm can't say which posts a snapshot contains, so the seam leaves an error of the order of Section 8's 0.18%. Trident makes state updates exactly-once with transaction-id'd batches, paid for in batch latency. Trending keeps at-least-once. Billing wouldn't.

Interviewer: "Your count bolt keeps the window in a hash map. Walk me through it, then kill it."

You: "Layout first, because the layout decides what a crash costs."

The Window Is a Ring, Not a List

Count executor 9 owns every hashtag that hashes to it, #worldcup included. Per key: 60 four-byte counters plus the key, about 270 B, plus a running total so reading a window never sums 60 numbers. With 5 M distinct hashtags that's 1.35 GB cluster-wide, ≈ 84 MB per count executor.

Time moves on a tick tuple every 5 s (configurable in recent releases). On each tick the executor first emits its local top-1,000 candidates as (tag, windowTotal) tuples, fields-grouped on tag so each tag lands in exactly one partial ranker (16 × 1,000 per 5 s, about 3,200 tuples/s). Then it advances the ring index modulo 60, subtracts that slot's old count from each running total, zeroes it as the new current bucket and evicts keys whose total hit zero. Each partial ranker keeps its top 100 for the global ranker. One thread, so ticks and increments never race and need no lock.

Buckets are stamped in processing time (Decision 9). Rejected: a timestamp per increment (~360 M per window) and event-time windows. Flip trigger: late or out-of-order data that must land in the right window, which needs event time and watermarks (Section 11).

Interviewer: "So your five-minute window is really 'the last sixty ticks'?"

You: "Close. A published total is the last 59 complete buckets plus the current partial one, so the last 295 to 300 s depending on where in the bucket you read, and the API says so. That's a definition, not an error. The ±1% target is about counting errors, duplicates or loss, and the edge is the same for every tag, so it never reorders the top 100."