Section 4: Accepted Is Not Searchable - API Design (with Pushback)

Executive Summary: This section turns Section 3's numbers into contracts. Agents push zstd batches that are acknowledged with 202 Accepted the moment Kafka holds them, not when they become searchable. Over-quota services get a 429 that tells the agent what to shed rather than a silent drop. Queries must name a label selector and a time range and are priced before they run. Alert rules are versioned with If-Match. Then the interviewer pushes on four choices: push over pull, 202 over 200, why every batch carries an idempotency key, and why nobody gets to write raw SQL.

The interviewer looks at the board, where Section 3 left 2M lines/s in and about 100 log queries/s out. "Okay. What does an agent actually send, and what does Dana actually type?"

You: "Eight endpoints across three products, each shaped by a number we already have."

POST /v1/logs:batch                  agent pushes a log batch           -> 202 | 429
POST /v1/metrics:write               agent pushes samples               -> 202 (partial)
POST /v1/logs:query                  search, streamed results
GET  /v1/logs:tail                   live tail over WebSocket or SSE
POST /v1/metrics:query_range         dashboard and ad-hoc metric queries
PUT  /v1/alert-rules/{id}            create or replace a rule, If-Match versioned
POST /v1/silences                    mute matching alerts until a deadline
POST /v1/exports                     async bulk export for analysts

Ingest: Durable Before Searchable

Each host runs one agent. It tails files and stdout, scrapes the local /metrics endpoints, and pushes. A log batch closes at 1 MB or 1 s, whichever comes first. A typical host emits 20 KB/s raw, so the timer almost always wins, and the fleet sends about 50,000 hosts × 1 batch/s = 50,000 requests/s to the regional gateways.

POST /v1/logs:batch
Content-Type: application/x-protobuf      (OTLP logs)
Content-Encoding: zstd
Idempotency-Key: <batch_id = hash(host, file, start offset, end offset)>

The gateway authenticates the agent, validates the batch, checks the service's quota, drops a batch_id it has already seen, and produces to the logs topic with acks=all, RF 3 and min.insync.replicas=2. Only after Kafka acknowledges does the agent get a 202.

Agents hash to a gateway by host, so a retry lands on the gateway that saw the original and a short-TTL seen-set catches it. Quotas are enforced as a per-gateway share of the service's limit, refreshed every few seconds from a central counter, so neither check needs a shared lookup on the hot path.

The 429 is an instruction, not a loss. Its body names what to shed, following the overload policy: a service's quota is 3× its trailing 7-day p95 rate, floor 1,000 lines/s; past it, DEBUG goes first, then INFO is sampled, and WARN and above, and audit, are never sampled. Anything not yet sent waits in the agent's 1 GB disk buffer, about 14 h at the average rate.

Metrics use POST /v1/metrics:write, compatible with Prometheus remote-write and OTLP. Agents flush within the 5 s slot that Section 3's detection budget reserves for them. The response is a partial accept: samples for existing series are always taken, and new series past the service's 200,000 active series limit are rejected by name in the response body, with the owning team alerted.

Quick check before moving on: