Section 5: A File Named by Its First Offset - Data Modeling
Executive Summary: The data model has three layers, and none of them is a table. The unit of storage is the record batch: one header (base offset, producer id and epoch, base sequence, timestamps, codec) over compressed records that carry only small deltas. Batches go into segment files named by their first offset, each with a sparse index (one entry per ~4 KB) and a time index, so any offset is a binary search, an index probe and a scan of a few kilobytes away. Retention deletes whole segments older than 7 days, compaction keeps the last record per key, and committed consumer offsets are just records in a compacted internal topic.
The interviewer points at Section 4's reply to Ari's producer: (partition 37, offset 1,284,553,901). "That number has to mean something on a disk. What's actually there?"
You: "A directory of files. No row per message, no per-consumer table. Section 4 made the offset the only thing a consumer needs, so the data model has two jobs: turn an offset into a byte position quickly, and throw old bytes away cheaply."
The Record Batch: The Unit Everything Moves In
Ari's producer doesn't send order-58213 alone. Decision 9 batches it with everything else headed for partition 37 during the 5 ms linger, compresses the batch and ships it. The broker stores that batch as received: the bytes on disk are the bytes later sent to consumers. Apache Kafka's current (v2) format (formats have changed across releases):
| Level | Field | Why it's there |
|---|---|---|
| Batch | base offset | offset of the first record; the leader stamps it on append |
| Batch | batch length | a reader can skip the whole batch without decompressing it |
| Batch | partition leader epoch | which leader wrote it; Section 8's truncation uses it |
| Batch | producer id, producer epoch, base sequence | duplicate detection for the idempotent producer (Decision 8, Section 9) |
| Batch | first and max timestamp | time index and retention work without opening records |
| Batch | attributes | compression codec (lz4/zstd), transactional flag, control-batch flag for commit markers |
| Record | offset delta, timestamp delta | small varints relative to the header |
| Record | key, value, headers | order-58213, the event, tracing headers |
Interviewer: "Why deltas? Put the full offset on every record."
You: "Size is the small reason: a delta takes a byte or two where a full offset takes eight, on every record of about 1 KB. The big reason is that the producer doesn't know the offsets. It sends deltas 0, 1, 2 and so on, and the leader writes one base offset into the header. That means the leader can assign offsets to a whole compressed batch by writing one header field, without recompressing it."
As in the 2011 paper, a message is addressed by its logical offset in its partition's log, not by a message id, so there's no global id space or id index. (partition 37, offset 1,284,553,901) is unique; the key order-58213 is not, because one order emits many events.
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson