Section 6: Everything Before "202" - Deep Dive: Data Ingestion

Executive Summary: Section 5's agent has flushed a batch, and the front door has 200 ms to decide what to do with it. We defend the ingestion API (202 not 200, no tenantId in the URL, one batch per flush, plain HTTP not streams), then walk the pipeline: authenticate, meter, decompress safely, validate lightly, and answer only once Event Hubs holds every part durably. We also cover 429, throttled 202s and 503, the node math, and why the door keeps no dedup table.

The interviewer taps the Ingestion Front Door box. "Three hundred thousand batches a second, nine hundred thousand at peak. Show me the API, then what this box does before it says yes."

The Contract on the Whiteboard

You start with the request, because every later decision hangs off it:

POST /v1/batches
Authorization: Bearer <token>        (agents: mTLS + token; tenant derived from credential)
Content-Encoding: zstd
Content-Type: application/x-ndjson   (each line carries "table": "ProcessEvents")
X-Batch-Id: a7f3…:boot-12:seq-88412

→ 202 Accepted {"batchId": "...", "accepted": {"P1": 20}, "throttled": {}, "rejects": []}
→ 202 under shedding: "throttled": {"P2": {"count": n, "retryAfterSec": 30}}
→ 400 (whole batch malformed) · 401/403 · 413 (> 1 MB compressed)
→ 429 + Retry-After (tenant over quota) · 503 + Retry-After (can't take even P0)

That is Contoso's batch from vm-web-07, flushed at 09:14:10: 20 events, including the w3wp.exe → powershell.exe -enc JAB… process creation, in about 4 KB of zstd. Each line names its table, and the schema registry maps ProcessEvents to P1, so a producer can't promote its own data to P0. The 413 limit sits just under Event Hubs' 1 MB maximum event size, leaving room for headers.