Apache Kafka
maang.io System Design Series
Apache Kafka is not a queue but a distributed commit log: topics split into ordered, replicated partitions of immutable, offset-addressed records. Producers append, consumer groups read at their own pace, and retention keeps messages replayable. Master partitions, the in-sync-replica set, log segments, and exactly-once semantics, and you understand the backbone of modern event-driven and streaming systems.
This deep dive comes in four parts:
- Foundations & Core Concepts — topics, partitions, offsets, and consumer groups.
- Internals — brokers, leaders/ISR, the high-watermark, log segments, and retention, with worked numbers.
- Advanced Internals — the idempotent producer, transactions/exactly-once, and KRaft.
- Interview — Staff/Principal Q&A: ordering, delivery guarantees, and capacity reasoning.
Apache Kafka — Part 3: Replication, Exactly-Once & KRaft
maang.io System Design Series
What is this?
One ticket rail is a single point of failure — if that section of the pass catches fire, every order on it is gone. So a serious kitchen keeps a runner photocopying each rail onto a backup rail at the next station, in real time. If the primary burns, the backup is promoted and service continues. But there's a rule the expediter enforces: never call a dish "ready to serve" until the backup rail also has that ticket. Otherwise a fire could destroy an order the dining room already thinks is confirmed.
That discipline — copies kept in sync, and a "safe to serve" line that only advances once the copies have caught up — is Kafka's replication, its in-sync replica (ISR) set, and its high watermark. This chapter is the distributed-systems heart of Kafka: how it survives broker failure without losing acknowledged data, how it can promise exactly-once, and how it recently tore out its own dependency on ZooKeeper.
The one-line idea: Every partition has one leader and several follower replicas. A write is only "committed" (safe, readable) once the leader and all currently in-sync followers have it — that boundary is the high watermark. Kafka's whole durability story is a careful dance between how many copies you demand and how much availability you're willing to give up to get them.
This is where Kafka meets the CAP theorem (the 101 chapter) and the Replication sibling topic head-on. Let's go.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson