Scribe (Log Aggregation) — Part 1: Foundations & Core Concepts

maang.io System Design Series

---

What is this?

Picture a giant company with ten thousand office buildings scattered across the world, and every desk in every building is constantly writing memos that headquarters needs to read. If each worker tried to personally courier every memo to HQ, the roads would melt and HQ's front desk would drown. So the company builds a smarter system. Every building has its own mailroom. Workers just drop memos in the local mailroom's slot — a two-second walk. The mailroom sorts the memos into bins by category (payroll, engineering, legal), bundles them up, and ships each bundle to a regional sorting center. The regional centers bundle again and forward to headquarters, where the memos finally land in the archive.

And here's the clever part: if the road to the regional center is washed out — flood, blizzard, a truck breaks down — the local mailroom doesn't panic and it doesn't throw the memos away. It just stacks them in a back room and keeps accepting new memos from workers. The moment the road reopens, it ships the backlog and catches up. Workers never even notice the outage.

That mailroom network is Scribe. Facebook built it around 2008 to move an ocean of log messages — every click, every page render, every error — off tens of thousands of web servers and into the data warehouse, without any single server ever having to know or care where those logs ultimately went.

The one-line idea: Scribe puts a local log-collecting agent on every host that accepts messages instantly, then stores-and-forwards them up a hierarchy toward central storage — buffering to local disk whenever the next hop is down, so a producer's write never blocks and logs survive downstream failures.

This is the canonical log-aggregation design. Learn Scribe and you understand the skeleton inside Fluentd, Logstash, the Kafka-based pipelines, and every "ship your logs somewhere central" system you'll ever design. Throughout these three chapters we'll keep coming back to the mailroom to keep the machinery concrete.

---

1. Why Scribe exists — what was painful before

Imagine Facebook circa 2008 without Scribe. You have thousands of web servers, and you want all their logs — for analytics, for debugging, for feeding the Hadoop warehouse. The naive options are all bad:

• Each app writes directly to the central store (HDFS, a database). Now every one of ten thousand servers holds an open connection to the warehouse and blocks on it. When the warehouse hiccups or a network link flaps, your user-facing web servers stall waiting on a logging write. You've coupled the health of your logging backend to the health of your product. 🚫 That's the cardinal sin — logging must never be able to take down the thing it's logging.
• Each server writes to local files, and you scrape them later with a batch job. Simpler, but now you have ten thousand piles of files to find, tail, rotate, and ship, each with its own failure modes, and your data is always hours stale.
• Everyone POSTs to one central log service. That service is now a single funnel that must absorb the combined write rate of the entire fleet, and if it falls over, every producer either blocks or drops data on the floor.

The painful common thread: producers were directly coupled to a fragile, faraway, shared backend. What you actually want is for a web server to hand off its log line to something trivially close and always available — and let that something else worry about the long, unreliable journey to the warehouse.

That "something close" is the local Scribe agent. It runs on the same host, so handing it a message is a fast local call that essentially never fails. Everything hard about the journey — batching, routing, retrying, buffering through outages — becomes Scribe's job, not the application's. This is the same decoupling instinct behind the Asynchronous Messaging and Publish–Subscribe Pattern topics in this tier: put a buffer between producer and consumer so neither one's speed or health is chained to the other's.

---

2. Core concepts & vocabulary

Four ideas carry the whole design. Get these and the internals in Part 2 will feel obvious.

2.1 The local agent (a Scribe server on every host)

Scribe runs as a lightweight daemon (written in C++) on every machine in the fleet. Applications don't ship logs across the network themselves — they send each message to the Scribe daemon on localhost. Because it's a same-host call, it's fast and it almost never blocks the application. The local agent absorbs the message and takes ownership of getting it where it needs to go.

2.2 The category (the routing abstraction)

Every log message is just a pair:

(category, message)

• category is a short string — like "web_access", "payment_errors", "newsfeed_impressions". It's the only thing Scribe looks at to decide what to do with the message. It's the "which bin in the mailroom" label.
• message is an opaque blob of bytes. Scribe does not parse it, validate it, or care what's inside — JSON, a raw string, a serialized struct, whatever. It just moves the payload.

This tiny abstraction is why Scribe is so general. Adding a new kind of log doesn't require a code change or a schema — you just start sending a new category, and configuration decides where that category flows. Categories are the routing key; the payload is none of Scribe's business.

2.3 Store-and-forward through a hierarchy

Scribe agents are wired into a tree. Leaf agents (one per host) forward to a middle tier of aggregators, which forward to a central tier that finally writes to durable storage (HDFS, files, the warehouse). At every hop, an agent applies the same rule: try to forward downstream; if downstream is unavailable, buffer locally and retry. Nobody drops data just because the next hop is temporarily down.

The tree exists for the same reason the mailroom has regional centers: fan-in. Ten thousand hosts talking directly to a handful of HDFS writers would be a connection and load nightmare. Each tier funnels and batches, so the number of connections the central store sees is tiny compared to the number of original producers.

2.4 Thrift (the wire protocol)

Scribe speaks Thrift — Facebook's cross-language RPC-and-serialization framework (the same lineage as gRPC/Protocol Buffers). Thrift matters here for two practical reasons: it's compact and fast on the wire (important when you're moving millions of messages a second), and it's polyglot — a PHP web tier, a Java service, and a C++ backend can all log to the same Scribe agent because Thrift generates client stubs for every language. Scribe's interface is literally one Thrift method, which we'll look at next.

---

3. How you actually use it

From an application's point of view, using Scribe is almost boringly simple — which is the whole point. The Scribe service is defined in Thrift roughly like this:

enum ResultCode { OK = 0, TRY_LATER = 1 }

struct LogEntry {
  1: string category,   // the routing key
  2: string message     // opaque payload — Scribe never looks inside
}

service scribe {
  ResultCode Log(1: list<LogEntry> messages)   // batch of messages
}

Your app builds a small batch of LogEntry records and calls Log(...) against the local Scribe agent:

# Illustrative — send a batch of log lines to the on-host Scribe agent.
entries = [
    LogEntry(category="web_access",    message='{"path":"/home","ms":42}'),
    LogEntry(category="payment_error", message='{"code":"DECLINED"}'),
]
result = scribe_client.Log(entries)     # talking to localhost, not the warehouse
# result == OK        -> accepted, Scribe owns delivery from here
# result == TRY_LATER -> Scribe is backed up; back off and retry

Two things to notice, because they define Scribe's contract:

1. You send to localhost, not the warehouse. You never open a connection to HDFS or the central tier. Your only dependency is the agent on your own box.
2. The only two answers are OK and TRY_LATER. OK means "I've accepted it, delivery is now my problem." TRY_LATER means "I'm overloaded or my buffers are full — back off and resend." There's no PERMANENTLY_FAILED. This is Scribe's honest, humble contract: best-effort delivery, not a durable transactional queue. We'll dig into exactly what that means (and where messages can still be lost) in Part 2.

On the other side of the pipeline, configuration — keyed by category — decides where messages go. A central agent might be configured so that everything in web_access gets written to an HDFS path, bucketed by hour:

# Illustrative Scribe config fragment (central writer).
<store>
  category=web_access
  type=file
  fs_type=hdfs
  file_path=/logs/web_access
  rotate_period=hourly
</store>

You'll almost never write raw config as an app engineer — but seeing it makes the model click: the app just names a category; config, elsewhere, decides that category's destiny.

---

4. When to reach for it — and when not

Scribe (and the pattern it embodies) is the right tool when:

• You have many producers (a big fleet) and want their logs/events funneled to central storage — a warehouse, a data lake, an analytics system.
• Producers must be decoupled from the backend: a slow or down warehouse must never block or crash an app.
• You can live with best-effort, high-throughput delivery — occasional loss or duplication under extreme stress is acceptable, as it usually is for logs, metrics, and analytics events.
• You mostly need a firehose to batch/warehouse systems (Hadoop, Hive), not a system of record.

Reach for something else when:

You need…	Reach for…
Durable, replayable, ordered streams with strong delivery guarantees	Apache Kafka — a replicated, disk-durable commit log (see the Kafka topic)
Rich parsing / filtering / enrichment of log content in the pipeline	Logstash or Fluentd (structured processing, hundreds of plugins)
Consumers to replay history or many independent consumer groups	Kafka again — Scribe forwards-and-forgets; it isn't a retained log
Ultra-low-loss internal transport at Facebook scale	LogDevice — Facebook's own durable, distributed log store that succeeded this era

💡 Rule of thumb: if the words "we can afford to lose the occasional log line, but we can never afford logging to slow down the app" describe your problem, you want the Scribe pattern. The moment you hear "we must not lose a single event" or "consumers need to replay," you've outgrown fire-and-forget aggregation and you want a durable log like Kafka.

Historically, Scribe itself is now a retired design — Facebook moved to newer internal systems and the open-source project stopped being maintained around the mid-2010s. But you study it for the same reason you study any classic: it's the clearest, most stripped-down expression of log aggregation, and its ideas — local agent, category routing, store-and-forward buffering, hierarchical fan-in — are alive in every modern pipeline you'll build.

---

Key takeaways

1. Scribe is the canonical log-aggregation system: a lightweight local agent on every host accepts (category, message) pairs instantly and takes ownership of delivering them to central storage, so applications are never coupled to a fragile, faraway backend.
2. The category is the whole routing abstraction — a string that decides where a message flows; the message is opaque bytes Scribe never inspects. New log types need no code or schema change, just a new category.
3. Agents form a store-and-forward hierarchy (leaf → aggregator → central writer). At every hop, "forward if you can, buffer to local disk if you can't" — this is the fault tolerance that lets producers keep writing through downstream outages.
4. It speaks Thrift (compact, polyglot) and offers a deliberately humble contract: OK / TRY_LATER, best-effort delivery — great for logs and analytics, wrong for anything needing durability, ordering, or replay.
5. Reach for it (or its heirs Fluentd / Logstash) for high-volume fire-and-forget firehoses; reach for Kafka when you need a durable, replayable, ordered log instead.

Next, Part 2 opens the agent up: the store types that make routing and buffering work, the exact write path a message takes, how the disk-buffer catches an outage, and the honest failure modes — including precisely where a "best-effort" system still drops data.