Section 8: One Heartbeat Carries Everything - Deep Dive: Sessions, KeepAlives & Consistent Caching
Executive Summary: Section 7 gave the cell one consistent story; this section is how 100,000 clients hear it without drowning the master. A session holds a 12 s lease renewed by a long-poll KeepAlive, about 8,300 per second, and that same RPC carries events and cache invalidations. The client keeps a conservative local lease, and when it runs out the library enters jeopardy, disables its cache and waits out a 45 s grace period. The cache is write-through and strictly consistent: the master invalidates every cached copy before it commits a write. For the orders-db master file cached by 24,000 sessions, that write takes about 1 to 2 s, and it is correct.
The interviewer taps the client library box from Section 6. "You said this is the hardest part. Convince me."
You: "The consensus log of Section 7 is a solved problem. What isn't solved is keeping 100,000 processes in agreement with that log, cheaply. Three mechanisms do it, and they all ride on one RPC: the session lease, the client's view of that lease, and cache invalidation."
The Session Lease and the Long-Poll
Every client process has one session, and all its handles and locks hang off it. The session holds a 12 s lease. While the lease is valid, the master promises not to expire the session, so its locks stay held and its ephemeral nodes stay alive.
The renewal is a KeepAlive, and the trick is that it is a long-poll. The client sends one, and the master holds it, replying shortly before the lease would end. The reply grants a fresh 12 s, and the client immediately sends the next. There is always exactly one KeepAlive in flight per session: 100,000 / 12 ≈ 8,300 KeepAlives/s, which is most of the master's ~10,000 RPC/s steady state but only a fraction of its 50,000/s ceiling.
Interviewer: "Why a long-poll? Just ping every few seconds."
You: "A ping has to be answered to count, and then the master needs a second channel to push invalidations. Holding the request gives me both in one RPC: the reply is the renewal, an early reply is the push, and the next request is the ack. And the lease length stays a load knob: under overload the master lengthens it, up to 60 s, which cuts KeepAlive traffic five-fold at the price of slower dead-client detection. Chubby's master did the same. If growth rather than a spike is the cause, the fix is proxies past 150,000 sessions (Section 12), not a permanently longer lease."
| Liveness design | Cost at 100,000 clients | Verdict |
|---|---|---|
| Session lease 12 s, KeepAlive long-poll | ~8,300 RPC/s; one lease covers all handles | Chosen (D8) |
| 1 s heartbeats | 100,000 RPC/s = 2× ceiling | Rejected |
| Per-lock TTLs (Redis style) | A renewal per lock, not per client | Rejected: liveness belongs to the process |
| No leases | A dead holder keeps its lock forever | Rejected |
Here is the part that earns the name of this section. When the master has something to say, it doesn't wait for the lease to run down: it replies early, and the reply carries events and invalidations.
Key Insight: The next KeepAlive is the acknowledgement. No push channel, no separate ack RPC, no extra connection: renewal, notification and acknowledgement share one request the client was going to send anyway.
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson