Section 7: Five Disks a Day - Deep Dive: Durability & Erasure Coding

Executive Summary: The data layer stores every 640 KB of an object as 10 data and 5 parity pieces, RS(10,5), spread 5/5/5 across three availability zones on distinct racks, nodes and disks, for 1.5× raw overhead instead of replication's 3×. Any 5 of the 15 fragments can vanish, including a whole AZ. Against independent disk failure the arithmetic gives about 10^-17 per year against an eleven-nines budget of 10^-11. That margin is not the point: the budget is spent by correlated failure, so the section's real content is the defences. End-to-end CRC32C checksums, a 30-day scrub at ~6 MB/s per disk, vendor diversity inside every stripe, a repair pipeline that detects within 15 minutes and rebuilds a 20 TB disk in about an hour, and a garbage collector that verifies before it deletes, then waits.

The interviewer taps the bottom layer of Section 6's diagram. "Ninety thousand spinning disks, and you promised me eleven nines. Walk me through how that is not a lie."

You: "A 2% annual failure rate across 90,000 disks is about five failures a day, roughly 100 TB of rebuild work every day, forever (N9). Durability is not a property of the hardware; it is a steady-state job, and the design is the pipeline that does that job faster than the disks undo it."

The Stripe: 640 KB at a Time

Every byte lands in a stripe: 15 fragments of 64 MB holding 640 MB of logical data, 10 data fragments and 5 parity (N13). Encoding does not wait for 640 MB. The encoder works on 640 KB groups: 10 data pieces of 64 KB go in, 5 parity pieces of 64 KB come out, and piece i of every group appends to fragment i. A streaming writer buffers under 1 MB, a range read fetches only the groups covering its bytes, and each 64 KB piece carries its own CRC32C. Priya's part 3 of clip-00123.mp4, 48 MB, is 75 such groups written inline to the same 15 disks; her 40 KB thumbnail rides the 3× journal and is packed into a stripe within about ten minutes (D4, Section 9's subject).

The placement rules matter as much as the code:

• 5 fragments per AZ, so losing any one AZ leaves exactly 10, enough to read.
• Distinct racks and nodes within an AZ, never two fragments of a stripe on one node: a rack or a 36-disk node costs a stripe at most one fragment.
• Mixed disk vendors and firmware revisions across the 15 disks, so a bad firmware batch costs one or two fragments, not six.

One group of Priya's part 3 becoming part of a stripe.