Section 3: Reads Scale, Writes Don't - Scale Estimation & Workload Analysis

Executive Summary: 2,000 services × 10 instances = 20,000 sessions. Each session reads about twice a second and writes about once every 25 seconds, so normal load is 40,000 reads/s and 800 writes/s, rising to 120,000 and 2,400 at a 3× peak. That is 50:1, inside the 2:1 to 100:1 range the 2010 paper targets. Five servers serve 200,000 reads/s from local memory, so peak reads use 60% of capacity. The quorum caps writes at about 20,000/s, so peak writes use 12%. The whole tree is about 600 MB on every server, against an 8 GB heap. The arithmetic makes three decisions: reads are served locally (Decision 2), read growth is met with observers rather than voters (Decisions 6 and 7), and the fsynced log gets its own disk (Decision 11). Every number here is our interview's assumption, not any company's real figures.

The interviewer points at Section 2's requirements list. "Put numbers on it. And tell me which number you'd lose sleep over." The trap is to size this like a database. A coordination kernel stores very little; what decides the design is the shape of the traffic, and which half of it the architecture can actually scale.

Sessions and Heartbeats

You: "Start with clients. There are 2,000 services and I'll assume about 10 instances each. Every instance holds one session, so that's 20,000 sessions. Sam's settlement-scheduler contributes three of them: sch-1, sch-2 and sch-3."

You: "The session timeout is 10 s. An idle client pings after about a third of that, so roughly every 3.3 s. That gives 20,000 / 3.3 ≈ 6,000 pings/s across the ensemble, or about 1,200 per server over five servers. That's a ceiling, because any real request also counts as a heartbeat, so a busy client doesn't send pings at all."

Interviewer: "Why not a 2-second timeout? You'd detect a dead leader replica faster."

You: "Pings would go up fivefold, but the real cost is worse. A 2 s timeout is shorter than the ensemble's own failover of about 5 s (Section 10). The new leader restarts session timers, but every client would still be disconnected for longer than its timeout, assume its session was lost and give up its leadership, and thousands would then race to reconnect inside a 2 s window. With a 2 s tickTime the server wouldn't even grant it: the minimum negotiable timeout is two ticks, 4 s. At 10 s, sessions outlive a failover. The flip trigger for Decision 8 points the other way: if GC pauses or jitter get close to 10 s, I raise the timeout and accept slower failure detection."