Section 3: Count the Threads, Not the Machines - Scale Estimation & Workload Analysis

Executive Summary: A peak of 600,000 posts/s at 1 KB is 600 MB/s from a 64-partition Kafka topic. With 2 hashtags per post, that becomes 1.2 M hashtag tuples/s. The unit of capacity is the executor, a thread holding about one core. Sized at 75% of each stage's per-executor rate, the topology needs 16 spouts, 40 parse bolts, 16 count bolts, 16 partial rankers, 1 global ranker and 12 ackers: 101, call it 100 executors. Parsing JSON binds, at 40 of them. They run as 48 workers in the 60 slots of 12 machines, keep enough cores (132 ≥ 100) and spare slots (12) to absorb the loss of one machine, and use about 9% of the network. Window state is 1.35 GB cluster-wide (84 MB per count executor), and at most 320,000 posts are in flight, tracked by the ackers in 6.4 MB, which bounds crash duplicates at about 0.18% of the window. Every number here is our interview's assumption, not Twitter's or anyone else's real figures.

The interviewer points at Section 2's numbers. "Size it. And tell me which stage you run out of first." Dividing bytes per second by bytes per machine uses the wrong unit. Decision 1 makes this a record-at-a-time topology: every tuple is handled by one thread running one bolt's execute. Capacity is a count of threads per stage; machines are just where the threads live.

Input: Posts, Then Hashtags

You: "On average, 200,000 posts/s. I'll plan for a 3× peak, 600,000 posts/s, because trending matters most during the events that cause spikes. At 1 KB each that's 600 MB/s coming in from the Kafka topic posts. It has 64 partitions and 7 days of retention, and the retention is what lets us replay (Decision 8; the Apache Kafka module covers the log itself)."

You: "The count stage doesn't count posts, though. It counts hashtags. Each post carries 2 on average, so the parse stage turns 600,000 posts into 1.2 M hashtag tuples/s. A hashtag tuple is about 100 B: the tag, the post id, a timestamp and a tuple id. That makes 120 MB/s from parse to count. Jordan's p-77120 is one post in and two tuples out, #worldcup and #goal."

Interviewer: "So everything downstream of parse doubles."

You: "The tuple rate does. The bytes shrink fivefold: a 1 KB post becomes two 100-byte tuples. That's why the network won't be the problem."