Section 9: Two Doors In, One Road Out - Deep Dive: The Data Path

Executive Summary: Section 8 decided who owns a key; this section follows the bytes, which never touch the index server. Writes split at 4 MB: smaller objects go 3x into a journal, one replica per AZ, and are packed into RS(10,5) stripes within ~10 minutes; larger objects and every multipart part are erasure-coded inline, one 640 KB group at a time, onto 15 fragments across three AZs. Multipart Complete writes a manifest and copies nothing. A range GET maps bytes to groups to 64 KB pieces, reads the 10 data fragments in parallel and hedges a straggler with parity after 20 ms; one stream runs at 100 MB/s and parallel ranges scale to multi-GB/s. Every piece carries a CRC32C, bytes are encrypted before coding, and a DELETE is a tombstone: the compactor reaches a ~30%-dead stripe within hours, its fragments are freed 24 h after retirement, and disk space returns a day or more after the DELETE.

The interviewer taps the data layer box from Section 6. "Now show me the bytes. Priya uploads a 40 KB thumbnail and a 480 MB clip. Walk both into the disks, read them back, then delete them."

You: "Different doors in, the same road out, and the deciding question is round trips. A 40 KB object cannot afford fifteen of them. A 48 MB part cannot afford to be written twice."

Two Doors In: The 4 MB Split

For the thumbnail clip-00123/0001.jpg, the front-end appends 40 KB to three journal extents, one per AZ, and waits for all three; with an AZ down it accepts 2 of 3 and a background catch-up fills the third later (Section 11). Journal extents are shared, thousands of small objects packed into each, so the thumbnail is an offset and a length inside an extent. Section 8's index commit records that triple and Priya sees 200 OK: one index round trip plus three parallel appends, well inside the 200 ms SLO (N7).

Within about 10 minutes (N12) the encoder drains sealed journal extents into 640 KB groups, encodes each as 10 data and 5 parity pieces of 64 KB (N13), appends them to a stripe's 15 fragments, and swaps the object's extent list in the index to point at the stripe. Journal bytes whose index record never committed are simply skipped when the encoder drains the extent. Only then are the journal bytes freed, so the journal stays small: 100 GB/s of peak ingest held for 600 seconds at 3x is ~180 TB, invisible against 1.8 EB.

Part 3 of clip-00123.mp4 takes the other door. The front-end streams the 48 MB through a one-group buffer: every 640 KB it produces 15 pieces, appends each to its fragment on one of 15 disks, 5 per AZ, and reuses the buffer once every append is acked. A 48 MB part is 75 groups, so each fragment receives 4.8 MB. The stripe seals at 640 MB of logical data (Section 7's state machine).

The split, as the bytes see it.

Interviewer: "Why two paths? Pick one."

You: "Each is wrong for the other size." The table is the argument (D4):

The threshold moves down only if small objects ever dominate ingest bytes, which N2 says they do not, and never up past 5 MB, the part minimum (N10), because every part must stay on the inline path.

Quick check before moving on: