Section 2: Two Guarantees and a Heartbeat - Requirements (Functional & Non-Functional)

Executive Summary: Section 1 shrank the prompt to a small, strongly ordered coordination kernel. This section writes its contract. Functionally, it is a tree of small znodes with seven non-blocking calls, version checks, ephemeral and sequential flags, one-time watches and sessions. Correctness rests on two ordering guarantees, linearizable writes and FIFO client order, plus a heartbeat (the session) that turns a dead client into a deleted znode. Non-functionally, our interview assumes read p99 ≤ 5 ms, write p99 ≤ 20 ms, 99.99% availability, a failover of about 5 s that sessions survive, the loss of any one zone, and 20,000 sessions driving 120,000 reads/s and 2,400 writes/s at peak. It refuses bulk storage, general transactions and locks inside the server.

The interviewer nods at the decomposition from Section 1. "Fine. Now commit to something. What does it do, what does it promise, and what won't it do?"

You: "Three lists: the primitives, which I can demo with Sam's settlement-scheduler; the guarantees, stated precisely, because for a coordination service the ordering rules are the product; then the numbers and the refusals."

Functional requirements: primitives, not features

Every item shows up in Sam's replicas sch-1, sch-2 and sch-3, which must elect one batch runner and share one config value.

• A hierarchical namespace of znodes, addressed like file paths: /services/settlement/election/, /config/settlement/batch-size. Each znode holds a little data and can have children.
• create, with two flags. Ephemeral ties the znode to the creating session, so it disappears when the session ends. Sequential appends a monotonically increasing counter, which is how Sam's replicas get n_0000000041, n_0000000042 and n_0000000043.
• delete and setData with an expected version. Sam's setData(..., version 7) succeeds only if nobody else changed the znode first. A version of −1 skips the check.
• exists, getData and getChildren, each able to leave a watch.
• Watches: one-time triggers that say "this changed" without carrying the new data. A client always sees the notification before it can see the state that caused it.
• sync, the escape hatch for a read that must be fresh.
• Sessions: one per client process, kept alive by heartbeats. Ending a session deletes its ephemerals and fires the watches on them.
• Synchronous and asynchronous forms of every call, so a client can keep many requests in flight without giving up its ordering.

What's missing is deliberate.

Interviewer: "Section 1's clients want locks and leader election. I don't see lock() or elect() in your list."

You: "They're recipes, not calls (Decision 1). Sam's election is three ephemeral sequential znodes. The lowest number leads, and each other replica watches the one just ahead of it. A lock is the same shape. Every call I listed completes without waiting on another client. That's what 'wait-free' means here, and it's the central bet of the 2010 paper."

Quick check before moving on: