Section 7: Skip What You Can Prove Is Absent - Deep Dive: The Log Pipeline, Chunks & Tiered Search

Executive Summary: The interviewer hands you the wheel, and you take the log path, because it carries 1 GB/s and most of the platform's cost. Four parts: how an ingester turns a Kafka stream into chunks without losing lines on a crash; how a query gets fast by skipping data (label and time pruning, per-chunk bloom filters, a parallel column scan); how three storage tiers map onto the latency SLOs; and why this beats a full inverted index by 9–18× on storage. The lesson: at this write volume you don't make search fast by indexing more. You make it fast by proving which chunks can't match.

About twenty-five minutes in, the interviewer caps the marker. "Where do you want to go deeper?" Alerting feels urgent, but the money sits on the log side: 2M lines/s on average, 6M at peak, every line kept for 30 days.

You: "I'd like to take the log path end to end: how a line gets from Kafka into a chunk, how a search avoids reading most of the data, and where each age of data lives. Then I'll defend not using a full-text index."

Part One: From Kafka to a Chunk

Gateways key the logs topic by stream, the hash of the label set (service, env, region). Every line of a stream lands on one partition, so one ingester owns that stream and builds its chunks in order without coordination. Partitioning by host would fan every service query out to every node; by time, one partition would run hot.

Each ingester keeps one open chunk per stream in memory, appending lines into columns (timestamp, level, message, attributes). A chunk is cut at 64 MB raw or 5 minutes, whichever comes first. On cut, the ingester:

1. compresses each column with zstd (about 10× overall),
2. adds the chunk's tokens to a bloom filter in the bloom index,
3. uploads the object under a deterministic key (stream ID plus the chunk's first Kafka offset),
4. writes a catalog row: {stream_id, start, end, last_offset, object_key, level_counts, bloom_ref},
5. and only then commits the Kafka offset.

Interviewer: "A chunk can stay open for 5 minutes. Your freshness SLO is 30 seconds. Those don't fit."

You: "They do, because the open chunk is queryable. Queriers ask the ingesters for anything newer than the last flush and storage for the rest, then merge. A line is searchable seconds after the agent flushes. The cut controls object size and count, not freshness. 64 MB keeps object-store reads large enough to amortize request latency and small enough to spread a scan across cores; 5 minutes bounds open-chunk memory and how much Kafka an ingester replays after a crash."

Interviewer: "And when an ingester dies holding 4 minutes of open chunks?"

You: "Nothing acknowledged is lost. Kafka keeps it for 24 hours at RF 3. A partition carries many streams, so the committed offset sits just before the oldest unflushed line across all of them, and some streams have already flushed past it. On restart the ingester replays its local WAL, resumes Kafka after the WAL's last record, and skips each stream's lines at or below the last_offset in that stream's catalog row. Re-cut chunks never overlap flushed ones, and a crash between upload and catalog write just re-uploads the same chunk to the same key. Agent retries are the other duplicate source, and the gateway drops those by batch_id, a hash of host, file and the batch's start and end offsets. At-least-once delivery, deduplicated storage."

A stream that outgrows its partition gets split with a shard label (Section 9).

Quick check before moving on: