Section 12: Growing Past One Brain - Follow-up Feature Evolution: From GFS to Colossus

Executive Summary: The single master is the right design for about 2.2 years, and this section is about what comes after. Five evolutions arrive, each at a named trigger: several masters over static namespace partitions as a stopgap; metadata moved into a sharded, replicated table, which is how Colossus works; Reed-Solomon RS(6,3) for sealed, cold files, saving 27 PB raw (about 225 chunkservers); smaller chunks and faster media for latency-sensitive data; and failover measured per shard instead of per cluster. HDFS walked the same road in public. Through all of it this design keeps the client library, data that bypasses metadata servers, version-numbered replicas, lazy garbage collection and checksums.

The interviewer flips back to the decisions ledger. "Three years from now this cluster holds three times the metadata and Gmail wants to live on it. What breaks first?"

You: "The master, in two different ways. Its memory runs out because of file count, and its latency and failover time don't suit interactive users. We don't have to guess at this, because GFS's own operators described both. In the 2009 ACM Queue interview 'GFS: Evolution on Fast-forward', Sean Quinlan named file counts and the single master as the limits, said latency-sensitive applications like Gmail were a poor fit for a design built for batch throughput, and described a multi-cell, multiple-master arrangement. So my roadmap follows the order those limits actually showed up in."

The Trigger Ladder