Section 9: Don't Touch the Disk - Deep Dive: Making Reads Fast

Executive Summary: Maya's read has a 10 ms p99, and one GFS seek costs ~10 ms, so the p99 is really a miss-rate target: fewer than 1 read in 100 may touch disk. Five per-group knobs get there. Locality groups give each group its own SSTables, so the indexer reads 100 TB instead of 400 TB and recent: lives in RAM. Compression is per group and per block, and favours speed. A scan cache and a block cache hold only immutable data, so no write invalidates them. Bloom filters skip SSTables that can't hold the row/column pair. Block size sets how many bytes a miss drags in. Then Maya's latency budget, and why neither "everything in RAM" nor memcached is the answer.

Interviewer: "Section 8 left you with a read that merges the memtable and every SSTable in the tablet. Maya's p99 is 10 ms. How many disk reads can you afford?"

You: "Roughly none. An RPC is 0.5 ms and a GFS seek is ~10 ms, so one seek on the critical path spends the whole p99. At 1 M Profiles reads/s peak, about 1,000 per server, the 99th percentile lets at most 10 of them per server per second touch disk. Everything in this section is about answering the other 990 from memory."

Locality Groups: Separate Files for Separate Readers

You: "A client groups column families into locality groups, and the tablet server writes a separate SSTable for each locality group in each tablet. A read or scan that names only some families opens only those groups' files."

You: "Webtable puts contents: in one group and anchor: plus language: in another. The indexer reads only the metadata group, so at Section 3's 14 GB/s it finishes in 2 hours instead of 8."

You: "A group can also be declared in-memory. Its SSTables are loaded into RAM lazily, the first time they're touched, and then read with no disk access. Profiles' recent: is one, about 7 GB per server. METADATA's location family is another, which is why Section 7's lookups cost an RPC, not a seek. The files stay on GFS, so a tablet that moves simply loads them again."

Interviewer: "Why not one group per family? Maximum separation."

You: "Families that are read together should share files. A cached-page view that wants contents: and language: would open twice the SSTables and, on a miss, pay twice the seeks. So I group by access pattern, not by schema. The flip trigger is a new reader that scans one family alone at scale. That family earns its own group."