Section 7: Fresh Enough, Ordered Always - Deep Dive: Consistency — Linearizable Writes, FIFO Clients & sync()

Executive Summary: ZooKeeper doesn't promise that every read is current. It promises linearizable writes (one total order, stamped by zxid) and FIFO client order (each session's requests run in the order sent, async ones included). Reads come from the local server's memory and reflect an earlier prefix of the write history, usually only milliseconds behind the leader, with no time bound promised. That's the price of 200,000 reads/s of capacity instead of one leader's 40,000. This section shows the one bug staleness really causes (a hidden channel between clients), fixes it with sync() for one round trip through the leader, and covers the two rules that keep it honest: watch events arrive before the data that triggered them, and a client never reconnects to a server older than its last zxid.

Interviewer: "Section 6 said any server answers reads. So your coordination service hands out stale data. Why should anyone trust it?"

You: "Let me state exactly what's guaranteed, show the one case where staleness bites, and give the tool for it. If most reads need that tool, the design is wrong, and I'll say what changes."

The Contract, Stated Precisely

1. Linearizable writes. Every create, delete and setData takes effect at a single point between request and reply. All writes form one total order, which is the zxid order from Section 5. Sam's setData on batch-size commits as zxid 0x5_0000a3f1, and every server applies it at that same position, after 0x5_0000a3f0 and before whatever comes next.
2. FIFO client order. A session's requests execute in the order the client issued them. This holds for the async API too, so sch-2 can pipeline twenty requests and they still apply in sequence.
3. Local reads. getData, exists and getChildren are answered from the memory of the server the session is connected to. Each read reflects some prefix of the zxid order, and that prefix may be shorter than the leader's.

The 2010 paper calls guarantees 1 and 2 together A-linearizability. FIFO order has a corollary: you always read your own writes. If sch-2 does setData then getData, the read executes after the write, because S3 applies the write before replying.

Key Insight: The contract is global for writes and per session for order, never per read. What's missing is any promise about what other clients did that you haven't heard about yet. That gap is the entire consistency story.

How stale can a local read be? Usually only by the commits still in flight to that follower: a few milliseconds. The protocol, though, promises a prefix of the history, not a bound. A follower that is connected but slow (a GC pause, an apply backlog) can fall further behind. A follower cut off from the leader keeps answering until it notices, which takes up to syncLimit (4 s), and then stops serving unless clients opted into read-only mode. Design for "some earlier prefix", and treat milliseconds as the common case, not a promise.