Section 8: Every Request Waits Here - Deep Dive: The Metadata Index

Executive Summary: All 5 million peak requests a second touch the index before they touch a byte, so this is where latency, consistency and hot-prefix pain collect. The design is ~40,000 range partitions of ~50 GB over (bucket_id, key) on ~1,000 partition servers, each an LSM-tree whose commit log and SSTables live, 3× replicated, in the data layer. One owner per partition makes read-after-write free: a PUT commits when a log quorum has the append, and only the owner serves reads, inside 2 ms p99. A map service grants leases; a dead owner is replaced in ≤ 10 s by replaying the log tail. The average partition sees ~125 req/s against a limit of 3,500 writes/s + 5,500 reads/s, so the real job is skew: auto-splitting turns Lumen's timestamp-prefixed keys from an outage into a non-event.

The interviewer circles the middle box of Section 6's diagram. "Ten trillion keys, five million lookups a second, strong consistency. Walk me through this layer until I believe it."

You: "Three decisions carry it: how the keyspace is cut, who may write to a piece of it, and what happens when that someone dies."

Cutting the Keyspace: Ranges, Not Hashes

The index is partitioned by contiguous ranges of (bucket_id, key) (D2). Section 5's 200 B record makes 10 trillion entries 2 PB of logical index, which at 3 replicas and ~2× LSM space amplification is **12 PB of NVMe** (N6). Cut into 50 GB ranges, that is **40,000 partitions**, about 40 per server.

Interviewer: "Hash partitioning spreads load perfectly. Why take on the skew that ranges invite?"

You: "Because LIST is a first-class verb. A prefix listing is a range scan in key order; a hash destroys the order, so every LIST becomes a scatter-gather over 40,000 partitions plus a merge sort. Hash wins only with no ordered listing, and that is not an object store. The price of ranges is skew, and I pay it with splitting, never re-hashing."

Quick check before moving on: