Scuba (Real-time Analytics)
maang.io System Design Series
Scuba is Facebook's in-memory database for real-time, ad-hoc analytics — the tool engineers reach for to debug a live incident. It holds recent data fully in RAM, sharded across many leaf servers, and answers arbitrary aggregations in sub-seconds via a scatter-gather tree, using sampling to bound cost. It trades exactness and history for speed and freshness — the opposite bargain from a warehouse.
This deep dive comes in three parts:
- Foundations & Core Concepts — in-memory columnar storage and the fresh-vs-exact trade.
- Internals — the scatter-gather aggregator tree, sampling, and sizing, with worked numbers.
- Interview — Staff/Principal Q&A: real-time analytics design and sampling trade-offs.
Scuba (Real-time Analytics) — Part 2: Internals & Implementation
maang.io System Design Series
What is this?
Back to the election-night tally room. In Part 1 we watched it from the anchor's chair — ask a question, get a number in a second. Now we walk onto the floor and see how the sausage is made: how each clerk actually stores their pile of ballots so they can count it fast, how new ballots get handed out, how the floor managers organize the fan-out so nobody drowns, and — the clever bit — how they cheat with sampling to always make air-time.
Three ideas do all the work, and they're the opposite of what a disk-based database like Cassandra does. Cassandra's whole game (see its internals chapter) is avoiding full scans — bloom filters, indexes, sorted files — because disk seeks are murder. Scuba throws all of that away. Its bet: if the data is in RAM and you have enough machines, a brute-force parallel scan is faster than maintaining any index.
The one-line idea: Scuba has no indexes. It stores recent data as compressed columns in RAM, sharded randomly across hundreds of leaves, and answers every query by brute-force scanning the relevant time range on every leaf in parallel — then bounds the cost with sampling. Speed comes from RAM + parallelism + column compression, not from clever lookups.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson