Section 11: Five Features, No Rewrite - Follow-up Feature Evolution
Executive Summary: The interviewer asks what year two looks like. You answer with five features (tracing, SLO burn-rate alerts, a compliance archive with PII redaction, cost showback, anomaly detection) and, for each, what changes, what stays, and which earlier decision made it cheap. The pattern to show: a good platform grows by adding consumers of the same Kafka topics, chunk format and rule evaluators.
Interviewer: "Say this ships. A year from now, what do teams ask for, and what does it cost you to say yes?"
You: "If a request needs a new storage engine or collection path, that's a warning sign. If it's a new Kafka consumer or a new recording rule, the design is doing its job. Five examples."
1. Distributed Tracing
Section 1 deferred tracing until the dominant question becomes "why is this request slow across 20 hops?" Dana's 02:14 incident is close: it took two searches and a copied request ID to connect checkout-api, payments-gateway and fraud-scorer.
What doesn't change. Section 5 already made trace_id a field on every log line, so blooms can find it. The agent, Kafka, object storage and the chunk-plus-bloom format are all reused.
What changes. A spans topic is added next to logs and metrics. Spans are partitioned by trace_id, not by stream hash (a planned exception to the stream-hash partitioning used for logs), so every span of a trace reaches the same consumer. A tail-sampling processor buffers each trace for a decision window (say 30 s after its first span), then keeps every trace with an error, every slow trace and a small baseline sample. Kept spans go into chunks with trace_id as a bloom token, so a trace lookup reuses Section 7's prune-then-scan path.
| Sampling choice | Verdict |
|---|---|
| Head sampling (decide at the first span) | Rejected as the only mechanism: cheap, but it drops the rare failing trace you wanted |
| Tail sampling (decide after a fixed window) | Chosen. Keeps errors and slow traces, at the cost of buffering spans briefly |
| Keep 100% of spans | Rejected. Span volume grows with fan-out, faster than logs |
Flip trigger: if the tail-sampling buffer becomes the bottleneck, push head sampling down to the agent for high-volume, low-value services.
The glue is exemplars: a latency histogram bucket stores the trace_id of one request that landed in it.
Dana clicks one dot on the latency panel and lands on the slow card-network-b span, then its log lines.
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson