Logstash
maang.io System Design Series
Logstash is the transformation stage of the ELK stack: a pipeline of input → filter → output that ingests logs and events, parses them (grok), enriches them (geoip, dates), and ships them onward — usually to Elasticsearch. Understanding its queueing, backpressure, and its lighter-weight Beats companions is key to designing a log pipeline that doesn't drop data under load.
This deep dive comes in three parts:
- Foundations & Core Concepts — the input/filter/output pipeline and grok parsing.
- Internals — persistent queues, backpressure, and the Beats→Logstash→Elasticsearch topology, with worked examples.
- Interview — Staff/Principal Q&A: designing a durable log pipeline.