Distributed Logging and Monitoring System Design

The prompt sounds like plumbing: "Design the logging and monitoring platform for a company running about 5,000 microservices on 50,000 hosts across three regions. Engineers need to search logs from the last few minutes back to 30 days, watch dashboards of service metrics, and get paged within a minute when something breaks."

Most candidates have an answer ready before the interviewer finishes: ship everything to Elasticsearch, put Grafana on top, wire up some alerts. It works for a startup. At this size it falls apart under a few questions. What does it cost to fully index 86 TB of raw logs a day? How do you find one request ID across every service in the last 15 minutes without scanning 900 GB? What happens to the bill when one deploy adds a user_id label to a counter and creates 50 million new time series? And when the whole company is on fire at 02:14, what keeps the system that pages you from burning with it?

The Trap Built Into a Familiar Question

Two failures show up again and again:

• The Shopping List: "Fluentd, Kafka, Elasticsearch, Prometheus, Grafana." Every box is real, none is sized, and nothing explains why. The interviewer asks what a 30-day regex costs and the answer is silence. Signal: knows the tools, hasn't run the system.
• The One-Product Answer: treats "logging and monitoring" as one thing and designs log search in depth while metrics and alerting get a single box. They are three products with different users, read patterns and failure costs, sharing one collection fabric. Signal: didn't decompose the prompt.

The passing performance splits the problem into logs, metrics and alerts, then lets the arithmetic drive each one. It notices that the system is written far more than it is read, so it keeps ingest cheap and makes reads clever: compressed chunks on object storage, pruned by labels, time and bloom filters, instead of a full index on every line. It treats cardinality as the real cost driver of metrics. It sees that machines, not humans, issue most of the queries, because alert evaluation is about 80% of metric reads. And it answers the question most candidates never get to: who monitors the monitor?