Logstash — Part 2: Internals & Pipeline Execution

maang.io System Design Series

What is this?

Back to our central mail-sorting facility. In Part 1 we watched a letter travel dock → sorting line → delivery truck. Now go behind the scenes and watch how the facility actually runs when it's busy: how many workers stand at the line, how mail waits in the holding room when the line is full, what happens when a delivery truck breaks down, and what stops the whole place from collapsing when trucks arrive faster than the workers can sort.

That last question — what happens when data arrives faster than it can be processed — is the single most important thing to understand about Logstash internals. Everything below is really about that: worker parallelism (how fast the line runs), the queue (the holding room), backpressure (the "stop accepting mail" signal), and durability (the ledger that survives a power cut).

The one-line idea: a Logstash pipeline is a set of worker threads that pull batches of events off an internal queue and run each batch through the filters and outputs. The queue — in-memory (fast, lossy) or persistent (disk-backed, durable) — is the fulcrum on which durability, buffering, and backpressure all balance.