Scuba (Real-time Analytics) — Part 1: Foundations & Core Concepts

maang.io System Design Series

What is this?

Picture the newsroom on election night. Somewhere behind the anchor is a giant tally room: hundreds of clerks, each handed a random shuffle of the ballots coming in right now. The anchor turns and asks, "How many votes for Candidate X in the last hour, broken down by district?" Nobody runs to a central archive. Instead the question ripples out to every clerk, each one counts their own pile in a few seconds, and the partial tallies flow back up to a floor manager who adds them together. If a handful of clerks are still counting when air-time hits, you don't wait — you take what you have and scale it up: if 90% of the clerks reported, multiply by 1 / 0.9 and put a number on screen with a confidence note.

That tally room is Scuba. Facebook built it to answer exactly this kind of question — slice-and-dice this firehose of recent events, right now, in under a second — for engineers staring at a latency spike, a crashing app, or a dip in ad revenue at 2am. It holds all its data in RAM, spreads it across hundreds of machines (the "clerks"), fans each query out through a tree of aggregators, and leans on sampling to keep every answer fast. The bargain it strikes is the whole point of this topic.

The one-line idea: Scuba trades exact + historical + slow for approximate + recent + instant. It keeps only the last little while of data, entirely in memory, and answers ad-hoc questions by brute-force scanning that data in parallel across many machines — accepting a sampled, slightly-approximate answer so the answer always arrives in about a second.

We'll follow that election-night tally room all the way down to the wire.

1. Why Scuba exists — what was painful before

Before Scuba, an engineer debugging a live incident had two bad options:

The data warehouse (batch OLAP). Rich, exact, historical — but you write a query, it schedules, it scans terabytes off disk, and you get your answer in minutes to hours. Useless when a site is on fire now and you need to ask twenty exploratory questions in a row. (This is the world of Azure Synapse Analytics and other column-store warehouses — the OLAP contrast we'll keep coming back to.)
Pre-built dashboards with fixed metrics. Fast, but they only answer the questions someone thought of yesterday. The instant you want to GROUP BY a dimension nobody pre-aggregated — "is this spike only on Android, only in Europe, only on app version 4.2?" — you're stuck.

Debugging is inherently ad-hoc: each answer suggests the next question. What engineers needed was the interactivity of a dashboard with the freedom of arbitrary SQL, over live data, at human-conversation speed. Scuba's insight: if you (a) keep only recent data, (b) hold it all in RAM, and (c) accept approximate answers via sampling, you can make ad-hoc slice-and-dice feel instant. Give up exactness and history; buy back interactivity and freshness.

💡 Scuba is an observability / debugging database, not a system of record. It is deliberately the opposite of a data warehouse — and the two are complementary, not competitors. Most real stacks run both: Scuba for "what is happening right now," a warehouse for "what exactly happened last quarter."

2. Core concepts & vocabulary (the mental model)

Keep the tally room in your head and the vocabulary falls out naturally:

Table — a stream of events, like web_errors or ad_impressions. A row is one event: a bag of (column, value) pairs. The schema is flexible — any row can introduce a new column, and Scuba just starts tracking it. No migrations, no ALTER TABLE. Every row carries an automatic time column.
Leaf server ("clerk") — one of many machines, each holding a random subset of a table's rows entirely in memory. Because the subset is random, any set of leaves is a representative sample of the whole (this property is the engine behind sampling — remember it).
Aggregator ("floor manager") — Scuba fans a query out and merges results through a tree of aggregators, so no single machine drowns collecting from thousands of leaves.
Scatter-gather — the query pattern: scatter the question to all the leaves, each computes a local partial answer, then gather and merge the partials back up the tree. (You'll meet this same pattern in Elasticsearch and any sharded search system.)
Sampling & scale-up — Scuba may compute an answer from only a fraction of the rows (or a fraction of the leaves) and then multiply the result back up. A COUNT of 900 from a 1-in-10 sample becomes a reported ~9,000. This is what bounds cost and latency.
Retention — data ages out. Each table keeps its rows until they hit an age limit or a memory budget, whichever comes first; the oldest rows are dropped to make room. Scuba is a sliding window over the recent past — days, maybe, not years.

That's the entire mental model: random shards in RAM + a scatter-gather aggregation tree + sampling + a sliding time window.

3. How you actually use it

Two sides: getting data in, and asking questions.

Logging a row in — applications don't talk to Scuba directly; they log events into Scribe (Facebook's log-aggregation bus — there's a sibling topic on it), and Scuba tails those categories. A logged row is just a flat object:

{
  "time": 1720051200,
  "event": "http_error",
  "status": 503,
  "service": "checkout",
  "region": "eu-west",
  "app_version": "4.2.1",
  "latency_ms": 812
}

Notice there's no pre-declared schema. If tomorrow you start logging a device_type field, Scuba picks it up and you can immediately group by it.

Asking a question — Scuba is driven mostly through a point-and-click UI, but the query it builds is SQL-shaped. Conceptually:

SELECT service, COUNT(*) AS errors, AVG(latency_ms) AS p_avg
FROM   web_errors
WHERE  time > now() - 3600          -- last hour only; this is a recent-data store
  AND  status >= 500
GROUP BY service
ORDER BY errors DESC;

You get columns, counts, averages, percentiles, and time-series — sliced by any dimension you logged — back in about a second, with a small completeness indicator telling you what fraction of the data actually answered (more on that in Part 2). Ask the next question, and the next. That tight loop is the product.

4. When to reach for it — and when not

Reach for Scuba when: you're debugging or monitoring; the questions are exploratory and change minute to minute; the data is recent and you can tolerate a ~1% sampling wobble; you value an answer now over the exact answer eventually.

Reach for something else when:

You need exact figures — billing, financial reporting, anything audited. Sampling makes Scuba's numbers estimates. → warehouse.
You need deep history — quarter-over-quarter trends, year-old data. Scuba already dropped it. → warehouse / Object Storage + OLAP.
You need joins across tables or complex relational queries. Scuba is a single-table, scan-and-aggregate engine. → SQL / OLAP.
You need a system of record with durability guarantees. Scuba is in-RAM and approximate. → a real database (Cassandra, DynamoDB, Postgres).

⚠️ The classic misuse is treating Scuba as a source of truth. The moment someone quotes a Scuba number in a revenue report, they've forgotten it's a sampled estimate over a sliding window. Right tool, wrong contract.

5. Key takeaways

Scuba is Facebook's in-memory, real-time, ad-hoc analytics/observability store — built so engineers can slice-and-dice a firehose of recent events in under a second while debugging live systems.
Its defining trade is approximate + recent + fast over exact + historical + slow. It keeps a sliding window of data entirely in RAM, sharded randomly across many leaf servers.
Queries are scatter-gather over a tree of aggregators: fan the question out to every leaf, each scans its own slice, merge the partials back up.
Sampling is the load-bearing trick — answer from a fraction of the data (or a fraction of the leaves) and scale the result back up — which is what bounds latency and cost.
It complements a data warehouse, it doesn't replace one: Scuba for "what's happening right now," an OLAP store like Azure Synapse for exact, historical, joined analytics. Never quote a Scuba number where exactness is the contract.

Next, Part 2 opens up the tally room: the in-memory columnar storage inside a leaf, the ingest and query paths end to end, exactly how sampling and scale-up work, and the failure modes a Staff engineer has to reason about.

Intermediate