Section 9: Built From Five Primitives - Deep Dive: Sessions, Watches & Recipes

Executive Summary: Everything Sam's team actually wants, from membership to configuration push to leader election, is built in the client library out of five server primitives: znodes, versions, the ephemeral flag, the sequential flag, and one-time watches. Sessions turn "this client is alive" into server state. A 10 s timeout and idle pings every ≈ 3.3 s cost about 1,200 pings/s per server. Watches are cheap, but a single config write can still fan out to 20,000 notifications and 200 MB of re-reads. The herd-free election recipe cuts a lock release from 20 MB of traffic to 1 notification and 1 small read. What ZooKeeper cannot do: a leader paused for 15 s wakes still believing it leads, and only a fencing token checked by the downstream resource stops it.

Interviewer: "You keep saying locks are a recipe. Show me one, and tell me what breaks."

You: "Sessions first, because every recipe depends on them. Then watches, recipes, and the failure recipes can't prevent alone."

Sessions: Liveness as Server State

When sch-2 connects to S3 it opens a session with a 10 s timeout (Decision 8). When the connection is idle, the client library pings after about a third of the timeout, ≈ 3.3 s. Across 20,000 sessions that's at most 6,000 pings/s, or ≈ 1,200 per server. In our design the leader owns session expiry. If no server hears from a session within 10 s, the leader commits a close-session transaction. That transaction deletes every ephemeral znode the session owns and fires the watches on them. Expiry goes through Zab like any other write, so every server agrees on its order.

The client sees three states, and each one asks the application to do something different:

While disconnected, no watch events arrive. On reconnect, the client library sends its watches and its last zxid to the new server, which fires any watch whose znode changed in the gap. One hole remains: an exists watch on a znode that was created and deleted during the disconnection is never triggered. So after a reconnect, the recipes re-read the state they depend on instead of trusting silence.

Interviewer: "Why not just use the TCP connection to decide liveness?"

You: "Because a connection drops during a server failover the session should survive. With no leader, nothing expires, and the new leader restarts every session's timer. Section 10's failover takes about 5 s, so the client's disconnect is also shorter than its 10 s timeout and it never has to give the session up. Flip trigger: if GC pauses or jitter push clients near 10 s, we raise the timeout and accept slower failure detection."

Critical Insight: A partitioned client learns it expired only when it reconnects. So a client that has been Disconnected for longer than the timeout has to assume it already lost its ephemerals, which is what well-built client libraries do by default. That makes "Disconnected" a reason to stop leading, not something to wait out.

Quick check before moving on: