Section 8: Committed Means Everyone In Sync Has It - Deep Dive: Replication — ISR, High Watermark & Leader Election

Executive Summary: Section 7 skipped the per-message fsync and bet durability on copies held by other machines. This section pays for that bet. Followers fetch from the leader like consumers. The leader keeps the in-sync replica set (ISR), the replicas caught up within 30 s (the default in recent releases). The high watermark (HW) marks how far every ISR member has replicated, and consumers read only below it. Under acks=all the leader acknowledges a write once every current ISR member has it, and accepts it only while the ISR has at least min.insync.replicas = 2 members. When broker 12 dies, broker 3 takes over partition 37, every acknowledged offset survives, and unacknowledged records are truncated and retried. This is Decision 3: it gives up the majority quorum's indifference to slow replicas in exchange for a cheaper replication factor.

Interviewer: "Your write path says 'followers replicate, then ack.' Replicate how, and ack when, exactly?"

You: "Followers pull, the leader tracks positions, and 'committed' has a precise definition. Then I'll kill broker 12."

Followers Are Just Consumers With a Job

Partition 37 has three replicas, one per zone: broker 12 leads, brokers 3 and 25 follow. A follower isn't pushed data. It sends the leader a fetch request with its next offset, the same request a consumer sends, and appends what comes back. A follower asking for offset N is telling the leader it holds everything before N, so the leader learns every follower's log end offset (LEO) for free. Replication also inherits Section 7's batching and zero-copy reads.

If a follower hasn't caught up to the leader's log end for replica.lag.time.max.ms (30 s in recent releases; the default has changed over time), the leader shrinks the ISR and drops it. When its log end reaches the HW again, the leader expands the ISR to readmit it; a dead broker is removed by the controller. Either way the change is recorded by the KRaft controller quorum (Decision 6) in the metadata log, because the controller elects the next leader from that recorded set. In the ZooKeeper era the ISR lived in ZooKeeper.

Quick check before moving on: