Section 2: Correct First, Then Available - Requirements (Functional & Non-Functional)

Executive Summary: Section 1 established that clients use this service to elect leaders, store small config, find each other and track membership. This section turns that into a contract. Functionally, the service is a namespace of small files and directories, each of which can be locked, with sessions, events, sequencers, ACLs and a client cache. Non-functionally, it promises cached reads under 1 ms, uncached reads at p99 under 10 ms, writes to uncached nodes at p99 under 50 ms (under 100 ms when up to 1,000 sessions cache the node), 99.99% availability (4.3 minutes a month), a master failover of p99 ≤ 30 s that sessions survive, and acknowledged writes that survive two replica losses, all for 100,000 sessions per cell. The most important requirement has no latency figure. It is a precise statement of what "correct" means for a lock, including the one guarantee this service cannot make by itself.

The interviewer leans back. "Good, you know who uses it. Now tell me what you're promising, and define 'correct' tightly enough that I could catch you breaking it."

You: "Three lists: what it does, which I can demo; what it guarantees, which I can measure; and what it refuses to be, which I can defend. Correctness goes ahead of the latency numbers, because for a lock service correctness is the product."

Functional requirements: the demo list

Each item shows up in one running example: orders-db in cell ny1, whose three candidate masters compete for /ls/ny1/orders-db/master.

• A hierarchical namespace of nodes: files and directories, addressed like /ls/ny1/orders-db/master. A node is either permanent or ephemeral. An ephemeral node belongs to a session and is deleted when that session ends. That gives group membership for free, because a dead tablet server's node disappears on its own.
• Whole-file reads and writes, capped at 256 KB. There are no partial reads, appends or renames. Writes support compare-and-set on the content generation, so two writers can't silently overwrite each other.
• Locks on any node: exclusive (writer) and shared (reader). Clients can call blocking Acquire or non-blocking TryAcquire, and Release. Locks are advisory (D11): they block other lockers, not readers or administrators.
• Sessions: each client process gets one session, and the library keeps it alive. When the session dies, its locks are released and its ephemeral nodes are deleted.
• Events a client subscribes to per handle: contents modified, child added or removed, lock acquired, master failover, handle invalid. The 24,000 orders-db readers learn about a new master through the contents-modified event, not by polling.
• Sequencers: an opaque token naming the lock, its mode and its lock generation, which a downstream server can check. master-b attaches /ls/ny1/orders-db/master:EXCLUSIVE:41 to its commands.
• ACLs: read, write and change-ACL permissions, stored as files themselves and inherited on create. Identity comes from mTLS.
• A client-side cache of file contents and metadata, kept strictly consistent (D9). It is a requirement, not an optimization. Section 3 shows the master can't survive without it.

Interviewer: "A file system? I asked for a lock service."

You: "Every lock needs a name, an ACL and a place to publish who won. A namespace gives me all three in one shape engineers already understand, and per-team directories make the ACLs inherit. Flat lock names would push all three problems onto every client. If you'd prefer an etcd-style flat key space with prefix watches, that's an acceptable alternative shape (D7). What matters is that the lock and its data live together."

Quick check before moving on: