Logstash
maang.io System Design Series
Logstash is the transformation stage of the ELK stack: a pipeline of input → filter → output that ingests logs and events, parses them (grok), enriches them (geoip, dates), and ships them onward — usually to Elasticsearch. Understanding its queueing, backpressure, and its lighter-weight Beats companions is key to designing a log pipeline that doesn't drop data under load.
This deep dive comes in three parts:
- Foundations & Core Concepts — the input/filter/output pipeline and grok parsing.
- Internals — persistent queues, backpressure, and the Beats→Logstash→Elasticsearch topology, with worked examples.
- Interview — Staff/Principal Q&A: designing a durable log pipeline.
Logstash — Part 3: System Design & Interview Mastery
maang.io System Design Series
What is this?
You now know Logstash from the input plugin down to the persistent queue's checkpoint file. This chapter is where that knowledge earns the offer. In a system-design interview, "logging" or "a centralized log/observability pipeline" comes up constantly as a sub-problem of a bigger design — and the interviewer isn't looking for you to name-drop "ELK." They want to watch you place the ingest tier correctly, defend the durability and backpressure trade-offs, and size it under load.
We'll do two things: first see where Logstash fits in real designs (with an architecture that survives failure), then run a mock interview — for each question, what they're really asking, a model answer, and the senior line that signals you've operated this for real.
The one-line idea: interviewers use Logstash (and the ELK pipeline) to test whether you understand ingest as its own hard problem — decoupling producers from a slow/failing sink, choosing durability vs throughput, and handling backpressure — not just "logs go into a search box."
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson