Apache Kafka
maang.io System Design Series
Apache Kafka is not a queue but a distributed commit log: topics split into ordered, replicated partitions of immutable, offset-addressed records. Producers append, consumer groups read at their own pace, and retention keeps messages replayable. Master partitions, the in-sync-replica set, log segments, and exactly-once semantics, and you understand the backbone of modern event-driven and streaming systems.
This deep dive comes in four parts:
- Foundations & Core Concepts — topics, partitions, offsets, and consumer groups.
- Internals — brokers, leaders/ISR, the high-watermark, log segments, and retention, with worked numbers.
- Advanced Internals — the idempotent producer, transactions/exactly-once, and KRaft.
- Interview — Staff/Principal Q&A: ordering, delivery guarantees, and capacity reasoning.
Apache Kafka — Part 2: The Log, The Write Path & Consumers
maang.io System Design Series
What is this?
Back to the kitchen pass. In Part 1 we saw the rails — one ordered ticket rail per partition — from across the room. Now we lean over the counter and watch what the expediter actually does with a ticket, and how the rail is physically built so that clipping a ticket on takes a fraction of a second even when a thousand tickets a second are pouring in.
The secret is embarrassingly simple, and it's the same secret behind the LSM-tree you met in the Cassandra topic and the Write-Ahead Log sibling: only ever append to the end, sequentially, and never seek back to modify. A spinning disk or SSD is slow at random access (jumping around to find and overwrite a byte) but astonishingly fast at sequential writes — often hundreds of MB/s. Kafka refuses to do anything but the fast thing.
The one-line idea: A Kafka partition is literally a file (well, a series of files) that Kafka only ever appends to. Producers write to the tail; consumers read forward with an offset; the operating system's page cache does most of the work. Turning "messaging" into "append to a file and let the OS handle caching" is why one broker can push a firehose that would melt a database.
Let's follow a record from producer.send() all the way onto disk, then trace a consumer reading it back out.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson