Section 7: Keeping One Brain Honest - Deep Dive: The Single Master

Executive Summary: The single master is the first thing an interviewer attacks, so we defend it with four mechanisms. First, an operation log, group-committed to local SSD and 2 remote replicas before any reply (1–2 ms). Second, checkpoints at every **1 GB** of log, so a restart costs 25 s. Third, shadow masters that tail the log and already hold the chunk-location map: after a Chubby-driven failover they serve metadata again by ~20 s, and reads never stop. Fourth, pathname read-write locks that let thousands of namespace operations run concurrently without deadlock. Against **12,000 RPC/s** at peak and a ~100,000/s budget, the master is neither a bottleneck nor a durability risk. It stops being enough when metadata passes ~192 GB, roughly 2.2 years out.

Interviewer: "One master. That's a single point of failure and a bottleneck. Convince me otherwise, or shard it."

You: "Two separate claims, and sharding is the wrong fix for both today. Section 3 showed ~12,000 RPC/s at peak against ~100,000/s, with no data byte through the master, so 'bottleneck' is off the table for about two years. 'Single point of failure' is the real attack, and I'll answer it with four mechanisms."

The Operation Log: The Only Truth

All metadata lives in RAM (~80 GB). RAM doesn't survive a crash, so every mutation of the namespace, a file's chunk list or a chunk's version number becomes a record in the operation log. Following the 2003 paper, the log is the only persistent copy of metadata, and it is also the logical timeline: its order defines the order of concurrent operations. Two creates racing for the same name are serialized by the write lock on that name; the log then records the winner's create.

The rule that makes this durable: the master replies to a client only after the record is flushed to its local SSD and to two remote log replicas. Until then the change is invisible to every client.

At peak that is ~2,300 mutating records/s × ~200 B ≈ 460 KB/s. Group commit folds every record arriving during one flush into a single disk write and network send, so the ~1–2 ms is paid once per batch, not per record. That fits inside the p99 < 10 ms metadata SLO. Reads like FindChunks skip the log entirely.

Interviewer: "Why not a Paxos group of masters, like your lock service?"

You: "Because I only need consensus once, at election time, not on every metadata write. Chubby already runs consensus, so I borrow it to pick the master and keep the log a plain synchronous replication stream. If ~20 s of metadata unavailability becomes unacceptable for interactive users, I move to many small replicated metadata shards. That's the Colossus direction in Section 12."

Quick check before moving on: