Section 5: The Label That Ate the Index - Data Modeling

Executive Summary: This section opens the batches from Section 4 and decides how each signal is stored. A log line becomes a row in a compressed columnar chunk belonging to one stream, defined by a few low-cardinality labels. A metric sample joins a series identified by its name plus sorted labels and found through an inverted label index. An alert is a rule plus a small state machine in a strongly consistent store. One rule ties them together: labels are what you index, fields are what you scan. Dana's req-7f3a9c shows why.

The interviewer taps the API slide: "Fine, the batches arrive. What's actually inside one, and what do you store?"

You: "Three shapes: a log event, a metric sample and an alert. But one decision sets the cost of everything else, so I'll state it first: what is a label and what is a field."

The Log Event

Every line the agent ships is structured JSON, about 500 B including metadata:

{
  "ts": 1793067240123456789,
  "service": "payments-gateway", "env": "prod", "region": "eu-west",
  "host": "h-20731", "pod": "payments-gateway-7d9f-k2x4p",
  "level": "ERROR",
  "trace_id": "4bf92f3577b34da6", "request_id": "req-7f3a9c",
  "message": "upstream=card-network-b timeout after 2000ms",
  "attributes": { "http.route": "/v1/charge" }
}

The timestamp is in nanoseconds because at 2M lines/s, 2,000 lines land in every millisecond fleet-wide, and order inside a stream must be stable. attributes is an open map, so teams add keys without a schema change.