Section 2: A Guarantee per Tuple - Requirements (Functional & Non-Functional)

Executive Summary: Section 1 settled on a long-running, record-at-a-time topology over a replayable Kafka log. This section writes its contract. Jordan defines topologies of spouts, bolts, groupings and parallelism, runs and rebalances them indefinitely, and picks a processing guarantee per topology. The system must take 600,000 posts/s at peak, reflect a post in a count within p99 ≤ 2 s, refresh a top-100 panel every 5 s within ±1%, survive a machine loss without losing a post (then catch up within minutes), and keep processing when the master dies. It refuses event-time correctness, default exactly-once into external systems, and SQL.

The interviewer taps the topology sketch from Section 1: Kafka, spout, parse, count, rank, Redis. "Fine, that's a shape. Now tell me what it promises. Especially about each post."

You: "Two halves. What the platform lets Jordan do, since this is a shared cluster, not one job. Then what it promises, each with a number and a mechanism. The guarantee per tuple is the heart of it."

Functional requirements: what Jordan can do

The platform is the product, and trending-hashtags is its first tenant.

• Define a topology. A directed graph of spouts (sources; ours reads Kafka topic posts and emits each post as a tuple) and bolts (parse, count, rank). Jordan sets each component's parallelism (how many threads run it) and, on every edge, a stream grouping that picks the receiving task: shuffle, fields (by a key such as the hashtag), global, and a few others.
• Run it continuously. A topology runs until someone kills it; the input never ends.
• Rebalance and redeploy. Change a running topology's worker or executor counts in place; kill and redeploy for a code change.
• Choose the processing guarantee per topology (and, as Section 4 shows, per tuple):
  • at-most-once: no tracking; a crash silently loses what was in flight;
  • at-least-once: every post is fully processed or replayed, so some may be counted twice;
  • exactly-once state updates: through Trident, the higher-level micro-batch API, at a latency and throughput cost.
• Windowing. Sliding and tumbling windows over processing time. Ours is a 5-minute window that slides every 5 s: each published total covers the last 59 complete 5-s buckets plus the current partial one, so the last 295–300 s.
• Metrics. Throughput, spout-to-fully-processed latency, failed tuples, per-bolt capacity and Kafka lag: "is it keeping up?" is every page's first question.

Interviewer: "Why make the guarantee selectable? Just always do exactly-once."

You: "Because each step up has a price. At-most-once costs nothing. At-least-once costs acker traffic and replay, about 2.4 million acker messages a second at our peak (Section 3). Exactly-once costs batching, so a sub-second path waits for batches to commit. Trending tolerates ±1%, so it takes the middle step (Decision 3: at-least-once via the acker). The rejected alternatives are at-most-once, which loses posts silently on every crash, and exactly-once everywhere, which charges every topology Trident's cost. The flip trigger is billing-grade counts. If this pipeline billed advertisers, it would move to transactional exactly-once, and Section 9 shows how."

Quick check before moving on: