Section 12: The Kernel Everyone Embedded, Then Replaced - Follow-up Feature Evolution: From ZooKeeper to etcd & KRaft
Executive Summary: The design from the 2010 paper did not stay frozen. ZooKeeper itself added observers, multi-op, dynamic reconfiguration, persistent and recursive watches, and the Apache Curator recipe library, most of them flip triggers from our decisions ledger that real users hit. etcd re-solved the same problem with Raft, leases, MVCC revisions and streaming watches, and Kubernetes built on it. Kafka moved its metadata off ZooKeeper into an internal Raft quorum called KRaft. Every successor kept the core idea, a small, totally ordered, replicated log of coordination state. What changed was the client contract and where that log lives.
Interviewer: "It's 2026. Would you still stand up a ZooKeeper ensemble for these 2,000 services, or is this a history lesson?"
You: "Both, and the history is the useful part. Almost everything that came after ZooKeeper is one of our decisions with its flip trigger pulled. Let me go through which triggers fired, and for whom."
What ZooKeeper Added to Itself
The 2010 paper describes seven calls, one-time watches and a fixed set of voters. I'll keep version numbers loose; the pressure behind each addition matters more than its release.
| Addition | What it does | The ledger entry it answers |
|---|---|---|
| Observers | Non-voting members that receive committed transactions and serve local reads | Decision 7: reads above 70% of capacity, or a remote site wanting local reads. Each observer adds about 40,000 reads/s and costs no write latency |
| Multi-op | An atomic batch of operations, such as check a version, create one node and delete another, that all commit or all fail | Section 4 deferred it. It replaces fragile multi-step recipes that can leave a half-finished state behind when a client crashes partway through |
| Dynamic reconfiguration | Changes ensemble membership, for example moving S5 to a new zone, through a committed reconfiguration operation, with no rolling restart | Section 10's operations burden: changing members used to mean editing static config files and restarting carefully |
| Persistent and recursive watches | A watch that stays registered after it fires, optionally covering a whole subtree | Decision 3's flip trigger: re-registration churn. Added in the ZooKeeper 3.6 line |
| Apache Curator | A client library of vetted recipes: leader latch, locks, barriers, service discovery | Decision 1's flip trigger: teams kept getting recipes wrong |
Notice what is missing from that list: nobody added server-side blocking locks, and nobody made default reads linearizable. The core bets survived, and the additions sit around them.
Interviewer: "Persistent watches sound like they undo Decision 3. Were you wrong to choose one-time watches?"
You: "No. One-time watches were the right starting contract: cheap, never blocking a writer, and correct even when events are missed, because the client re-reads current state. Their cost appears when a client mirrors a large subtree and must re-register after every event. Persistent watches remove that churn and keep the re-read. At 25 watches per session, the original contract is fine."
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson