Scribe (Log Aggregation)
maang.io System Design Series
Scribe is Facebook's canonical log-aggregation design: a lightweight agent on every host receives log messages by category and forwards them up a hierarchy to central aggregators, buffering to local disk when downstream is unavailable. That store-and-forward resilience is the pattern behind every large-scale log firehose — and studying it clarifies what modern equivalents (Kafka, Fluentd) are really doing.
This deep dive comes in three parts:
- Foundations & Core Concepts — the per-host agent, categories, and store-and-forward.
- Internals — the aggregation hierarchy, disk buffering, fault tolerance, and successors, with worked examples.
- Interview — Staff/Principal Q&A: designing reliable log collection at scale.
Scribe (Log Aggregation) — Part 3: System Design & Interview Mastery
maang.io System Design Series
What is this?
You now know Scribe from the category label down to the disk buffer. This chapter turns that into interview points. Almost every large system-design prompt — "design a metrics system," "design a news feed," "design an ad click aggregator," "design a distributed logging platform" — has a moment where mountains of events need to move from thousands of producers to some central brain. The candidate who calmly says "I'll put a log-aggregation tier here, with local agents that store-and-forward and buffer through outages" — and can then defend the delivery guarantees — signals real infrastructure maturity.
We'll do two things: see where the log-aggregation pattern fits in real designs (with a reference pipeline), then run a mock interview — each question with what they're really asking, a model answer, and the senior line that reads as Staff/Principal.
The one-line idea: interviewers use log aggregation to test whether you can move a firehose of events off the critical path — decoupling producers from a fragile backend with local agents and buffering — and then reason honestly about the guarantees you did and didn't buy.
Keep the mailroom from Parts 1–2 in your back pocket; it's a great way to explain the design out loud under pressure.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson