Section 5: A Tuple, Its Id and Its Ancestry - Data Modeling
Executive Summary: The data model is mostly in motion: no tables, no per-post rows, just five shapes. The tuple is a named list of values on a declared stream. Each anchored tuple gets a random 64-bit id and records which spout tuples it descends from. The acker entry tracks a whole tuple tree in about 20 bytes. The window state in each count executor maps each hashtag to a ring of 60 × 5-s buckets, about 84 MB per executor. The checkpoint record pairs that state with the Kafka offsets it covers, and Kafka itself is the only durable copy of the posts.
The interviewer taps Section 4's sequence diagram. "Your spout 'emits a tuple' and your parse bolt 'anchors' two more. What are those things in memory? And what has to survive a crash?"
You: "The durable copy of every post is already in Kafka, on posts, for 7 days (Decision 8). So the topology never stores a post. It holds tuples in flight, the bookkeeping that proves they finished, and the window counts. Only the counts are worth checkpointing."
The Tuple: A Named List on a Declared Stream
A tuple is an ordered list of values. Field names come from the stream schema that the emitting component declared at build time, so they aren't stored in each tuple.
| Stream | Declared fields | Example for p-77120 | Size |
|---|---|---|---|
spout → parse (default) |
postId, text, ts |
p-77120, "Final whistle! #WorldCup #Goal", 20:13:58 |
≈ 1 KB |
parse → count (default) |
tag, postId, ts |
#worldcup, p-77120, 20:13:58 |
≈ 100 B with its tuple id |
count → partial ranker (candidates) |
tag, windowTotal |
#worldcup, 45,000,000 |
top 1,000 per count task per 5 s (≈ 3,200 tuples/s in all) |
partial → global ranker (rankings) |
rankings |
top 100 (tag, count) pairs | one per 5 s per ranker |
With its values, a tuple carries its source component, source task and stream id. A bolt can declare several named streams, and subscribers pick one by id. Groupings look fields up by name: fieldsGrouping(new Fields("tag")) hashes the tag value mod the count bolt's 16 tasks, which is how #worldcup lands on count executor 9.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson