Section 14: Same Skeleton, Different Bets - Designing HDFS: Where It Diverged

Executive Summary: HDFS reimplements the GFS paper, so Sections 5–10 are already its skeleton. The interviewer probes the four places it chose differently. A file has one writer, so the crawl becomes 500 files. HA comes from a quorum of 3 JournalNodes. The namespace lives inside a Java heap: 84 GB for this cluster, the same small-file wall (18 TB with GC headroom), plus GC pauses. And placement and storage are tuned for compute that runs on the DataNodes' own machines.

The Move When the Prompt Says "Design HDFS"

Draw this module's core with HDFS names: the master becomes the NameNode, chunkservers become DataNodes, and 64 MB chunks become 128 MB blocks. In the room you still draw the four boxes. For the full mapping, see Section 12's "HDFS: The Same Road, in Public" table.

Interviewer: "So it's just GFS?"

You: "The skeleton is: one in-memory metadata server, data that never passes through it, replicas across racks, background re-replication. HDFS chose differently in four places: one writer per file, a quorum journal, a namespace bounded by a heap, and placement and storage tuned for co-located compute. I'll take them in order, using the crawl."