Section 9: Holding the Lock Is Not Permission - Deep Dive: Fencing, Sequencers & Lock-Delay
Executive Summary: The lock service can guarantee one exclusive holder in its own state. It cannot stop a holder that froze, lost its session while frozen, and woke up still believing it leads. This section replays orders-db's 40 s GC pause: the session expires at t = 12 s, master-c takes the lock at generation 42, and at t = 40 s master-b's command stamped 41 is rejected by a tablet server. The fix is a sequencer, the lock's name, mode and generation, checked by every downstream server against the highest generation it has seen or against CheckSequencer. Servers that cannot check get lock-delay: after an expiry, the lock stays unavailable for a client-chosen 0–60 s (default 30 s). That is a heuristic, and it stretches application failover from 12 s to 42 s. With sequencers, lock-delay is 0.
The interviewer underlines the last safety property from Section 2. "Section 8 gave you healthy sessions and a jeopardy event. So a leader that holds the lock is safe to act. Agreed?"
You: "No, and this is where most lock designs quietly break. A session tells you the lock was yours when the library last heard from the master. It says nothing about the moment your write lands. Let me show you with orders-db."
The 40-Second Pause
master-b won /ls/ny1/orders-db/master at generation 41, published 10.4.7.22:7000, and stamps /ls/ny1/orders-db/master:EXCLUSIVE:41 on every command it sends to the 4,000 tablet servers. Then the JVM stops the world.
| Time | What happens |
|---|---|
| t = 0 | master-b enters a 40 s stop-the-world GC pause. Its threads, KeepAlive thread included, freeze. |
| t = 12 s | No KeepAlive arrives, so the session lease (12 s) expires at the lock-service master. master-b's lock is released and its ephemeral nodes are deleted. |
| t ≈ 12 s | orders-db checks sequencers, so its lock-delay is 0. master-c's blocked Acquire returns; the lock generation becomes 42. |
| t ≈ 12–14 s | master-c writes its address; 24,000 cached copies are invalidated in ~1–2 s (Section 8). Tablet servers start receiving gen-42 commands. |
| t = 40 s | master-b wakes mid-function and sends "reassign tablet 1187" stamped 41. The tablet server has seen 42 and rejects it. |
| t = 40 s+ | master-b's library sends its next KeepAlive, gets SESSION_EXPIRED, invalidates every handle. master-b exits. |
The same incident as a conversation between the four parties.
Critical Insight: Every part of the lock service behaved correctly. The lease expired on time, the lock had one holder at every instant according to the replicated state, and master-c was legitimate. The thing that saves orders-db is the tablet server, because only the resource sees the order in which commands actually arrive.
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson