Section 12: Every Addition Was a Flip Trigger - Follow-up Feature Evolution: From the 2011 Log to KRaft, Tiered Storage & Queues

Executive Summary: The 2011 paper described a log with no replication, ZooKeeper holding consumer offsets, and at-least-once delivery. Everything Kafka added afterwards (replication with an ISR, compaction, offsets in a topic, idempotence and transactions, KRaft, tiered storage, a broker-side group protocol, share groups) is a flip trigger from our decisions ledger that real users pulled. The core idea survived all of them: an append-only, partitioned log read by offset. Pulsar, Redpanda and object-storage logs re-solve the same problem, and the useful comparison is which of our decisions each one reverses.

Interviewer: "Your design is close to Kafka as it runs today. How much of it was in the original paper?"

You: "Less than people assume. The paper had the log, partitions, pull, offsets and the page cache, but not replication. Almost everything since is one of our decisions with its flip trigger pulled. I'll keep versions loose; the pressure behind each addition matters more than its release."

What the 2011 Design Left Open

The paper's system carried LinkedIn's activity data and operational metrics to online consumers and offline Hadoop loads. Its bets are our Decisions 1, 4 and 5: partitioned topics of roughly 1 GB segment files, logical offsets, consumers pulling and acknowledging by offset, no per-consumer broker state, time-based retention, the page cache, and sendfile.

Three things were deliberately thin:

1. One copy per partition. A dead broker made its partitions unavailable; a dead disk lost them.
2. ZooKeeper did all coordination: broker and consumer registries, rebalance triggers, consumed offsets.
3. At-least-once was the only promise. Retries could duplicate; a crashed consumer replayed from its last stored offset.

Fine for metrics and click logs, not for Ari's orders topic. Each gap became a release line.