Section 1: Answers While the Data Is Still Moving - Problem Understanding & Clarifications

Executive Summary: "Count some hashtags" is really a request for a computation that never finishes: it runs over an unbounded stream and keeps running while machines fail under it. We break the prompt into what, who, why and how, run a clarifying exchange in which each answer rules out a class of design, and compare batch, micro-batch and record-at-a-time processing. What survives is a long-running topology of spouts and bolts reading a replayable log. We meet Jordan, who owns it, and post p-77120, which travels through every section that follows.

The interviewer reads the prompt aloud:

"Six hundred thousand posts a second arrive at peak. Product wants a trending-hashtags panel that reflects the last five minutes and refreshes every five seconds, and it must keep counting when machines die. Design the system that computes it — and tell me what it guarantees about each post."

Three phrases carry the prompt. The last five minutes, every five seconds: we compute as the data arrives, not after it lands somewhere. Keep counting when machines die: the computation is long-lived and outlives its machines. What it guarantees about each post: the interviewer wants a processing guarantee by name. A candidate who draws a cron job over a warehouse has failed the first phrase; one who answers "exactly-once" without its cost has failed the third.

Key Insight: This is a standing query that the data flows through, not a query over stored data. A batch job asks "what happened?" once the data is at rest. A stream processor answers "what is happening?" while the data is still moving, and never finishes. Every later mechanism (groupings, the acker, checkpoints) exists because the job never ends and its machines eventually fail.

Decomposing the Prompt

• What it does: parses each post, extracts its hashtags, counts them in a sliding window, ranks the top 100 and publishes the result, continuously. The product is a platform for standing computations; trending is its first job.
• Who uses it: data engineers like Jordan (they/them), who write and deploy the trending-hashtags topology and get paged when it lags; and the app, which reads the finished panel from a fast store and never talks to the processor.
• Why it exists: a batch pipeline (the shape the MapReduce module designs) answers correctly but hours late, after the trend is over. Freshness is the product.
• How it's used: deployed once, then left running for weeks, rebalanced and redeployed without stopping the stream.

The 2014 paper describes Storm as used widely at Twitter for real-time analytics, a workload of this shape. The numbers in this module are our interview's assumptions, not Twitter's figures.