Section 6: Every Server Answers, One Server Decides - High-Level Architecture

Executive Summary: The ensemble is five servers, one leader and four followers, placed 2 / 2 / 1 across three zones. Every server holds the full ≈ 600 MB tree in memory and answers reads from it, so reads scale with servers: ≈ 40,000 reads/s each, 200,000/s in total. Every write is forwarded to the leader, whose request processor turns it into an idempotent transaction that Zab commits once 3 of 5 servers have it logged. Then every server applies it, and the client's own server replies and fires its local watches. Typical cost is ≈ 4–5 ms (p99 ≤ 20 ms). Two rules: no read on the leader's critical path, and no write committed except through the leader.

The whiteboard has Sam's subtree from Section 5 on it: the election znodes and /config/settlement/batch-size at version 7. The interviewer points at the empty space beside it.

Interviewer: "Now draw the machines. Where does a request actually go?"

You: "It depends on whether it changes anything. Every server answers reads, and only one server decides writes. ZooKeeper's architecture comes down to that split. I'll draw the boxes first, then walk one read and one write through them."

The Ensemble on the Board

There are five voting servers in our design, S1 and S2 in zone A, S3 and S4 in zone B and S5 in zone C (Decision 6). Quorum is 3. The ensemble survives 2 failed servers or the loss of any one zone, and Section 10 walks through exactly that. Right now S1 is the leader and the other four are followers. Leadership is a role, not a machine: any of the five can win the next election (Section 8).

Every server, leader included, runs the same three-stage pipeline the 2010 paper describes. A request processor prepares writes. Atomic broadcast, which is Zab, agrees on their order. A replicated in-memory database holds the tree. Each server also writes to its own transaction log on a dedicated SSD, where an fsync costs about 1 ms (Decision 11).

Clients get a list of all five servers, connect to one, and open a session there. Sam's sch-2 is on S3; each server carries about 4,000 sessions and ≈ 1,200 pings/s. If S3 dies, sch-2 moves to another server and keeps the same session, as long as it arrives inside the 10 s timeout. Session creation and expiry are state changes that go through the leader like any write, so a session belongs to the ensemble, not to one TCP connection (Section 9).