Scribe (Log Aggregation)
maang.io System Design Series
Scribe is Facebook's canonical log-aggregation design: a lightweight agent on every host receives log messages by category and forwards them up a hierarchy to central aggregators, buffering to local disk when downstream is unavailable. That store-and-forward resilience is the pattern behind every large-scale log firehose — and studying it clarifies what modern equivalents (Kafka, Fluentd) are really doing.
This deep dive comes in three parts:
- Foundations & Core Concepts — the per-host agent, categories, and store-and-forward.
- Internals — the aggregation hierarchy, disk buffering, fault tolerance, and successors, with worked examples.
- Interview — Staff/Principal Q&A: designing reliable log collection at scale.