Section 6: Three Paths and a Buffer - High-Level Architecture

Executive Summary: The platform has three paths: a write path that turns 2M log lines and 10M metric samples a second into durable chunks and blocks, a read path that answers engineers and dashboards, and an alert path that reads the freshest data and pages within 60 s. Three choices shape all of it: Kafka as a 24-hour buffer between ingest and storage, object storage as the source of truth, and regional cells that keep data in the region that produced it. Dana's 02:14 log query walks the read path end to end.

The interviewer slides the marker across. "You've sized it and modeled it. Draw it."

You: "Three paths, because this is three products sharing one collection fabric. Write path first: it carries 1 GB/s and most of the cost."

The Write Path

• Host agent. One per host (a DaemonSet), tailing files and stdout and scraping local /metrics endpoints. It batches, compresses with zstd (5×: 1 GB/s raw becomes 200 MB/s) and pushes. Its 1 GB disk buffer covers about 14 h of upstream outage at average rate.
• Ingest gateway. Stateless and regional. It authenticates the agent, enforces the per-service quota (3× the trailing 7-day p95, floor 1,000 lines/s), validates the batch, writes to Kafka with acks=all, and only then returns Section 4's 202.
• Kafka. Separate topics for logs and metrics, 24 h retention, RF 3, min.insync.replicas 2: about 52 TB for logs plus about 26 TB for metrics, roughly 78 TB of disk. Logs are partitioned by stream hash and metrics by series ID, so one log stream always lands on one ingester and one series on the same three.
• Log ingesters. Each appends lines into an in-memory chunk per stream, builds that chunk's token bloom filter and writes a WAL. A chunk is cut at 64 MB raw or 5 minutes, flushed to object storage and registered in the catalog. Until then, the ingester serves queries for it.
• Metric ingesters. Hold the last 2 h of samples in a head block replicated RF 3 (about 900 GB of RAM across the fleet), then flush 2 h blocks. The replicas are for availability, not durability, since Kafka already covers that; Section 8 explains why.
• Compactor. Merges blocks, dedupes the three metric replicas into one stored copy, writes the 1-minute and 1-hour rollups, and deletes data past retention.
• Stream processor. Reads the logs topic and emits counts per service and level every 10 s into the metrics topic. Log-based alerts never touch the log store.

Key Insight: The freshness SLO decides who serves recent data. Logs must be searchable within 30 s (p99), but a chunk can stay open for 5 minutes, which is 10× that budget. So the open chunk in ingester memory has to be queryable. Metrics likewise: 10 s freshness means the head block answers.

Quick check before moving on: