Section 4: Three Doors, One Truth - API & Integration Design (with Pushback)

Executive Summary: Section 3 proved the load is laptop-sized, so every API decision here buys correctness, not capacity. This section designs the system's three surfaces: an ingest endpoint that verifies a webhook's signature, enqueues the raw payload, and answers 202 in under 100 ms; a provider adapter interface whose 5-minute reconciliation poll makes webhook loss survivable; and consumer APIs where incidents get :ack and :resolve verbs, escalation policies get monotonicity validation, and dashboards get a health query. One canonical RunEvent and one idempotency key hold it all together. Three rounds of pushback test the choices — including the one teams will actually fight you on.

The interviewer taps the whiteboard: "Twenty-three events a second. Show me the contracts." The instinct is to draw one API. Resist it. This system has three kinds of caller with three opposite demands: providers that fire webhooks and punish slow receivers, a reconciler that must ask providers what the webhooks never said, and 300 owner teams who need to ack, resolve, tune and query. Three doors — all opening onto one canonical model (D2).

Surface One: Ingest — Accept Fast, Trust Nothing

Every provider posts to the same shape of endpoint:

POST /v1/ingest/github
X-Hub-Signature-256: sha256=8f1c...
{ ...raw GitHub Actions workflow_run payload... }
-> 202 Accepted            (p99 < 100 ms)

The handler does exactly three things: verify the provider's HMAC signature, append the raw payload to a durable queue, and return 202. Nothing else — no parsing, no database writes, no detection. An adapter consumes the queue and normalizes each payload into the one canonical event everything downstream understands (D2). Here is run #8,403, the one that will flip checkout-service-deploy to DEGRADED at 09:31:

{
  "provider": "github-actions",
  "external_run_id": "8403",
  "attempt": 1,
  "event_type": "run.finished",
  "pipeline": "checkout-service-deploy",
  "branch": "main",
  "commitSha": "9f3c2e1",
  "trigger": "push",
  "conclusion": "failure",
  "started_at": "Tue 09:24:10",
  "finished_at": "Tue 09:31:00",
  "stages": [
    { "name": "build", "conclusion": "success" },
    { "name": "test", "conclusion": "failure", "failed_step": "checkout-totals.spec" },
    { "name": "deploy", "conclusion": "skipped" }
  ]
}

The first four fields are the contract's keystone: the idempotency key (provider, external_run_id, attempt, event_type). Every canonical event is upserted by that key, so when GitHub delivers the same webhook twice — and at a 0.1–1% duplicate rate across 2M events a day, it will, thousands of times daily — the second delivery lands on the same row and changes nothing. Note what the key does not contain: a sequence number. Webhooks arrive unordered, and we don't fight it. Detection reads the pipeline's current state — the last-20 window and health row — inside one transaction (D4), so a run.started arriving after its own run.finished is merely an upsert into history, not a corrupted stream. Out-of-order tolerance isn't a recovery mechanism here; it's the default physics of the design.

Interviewer: "You're handling 23 events a second. Why the 202-and-queue theater? Process the webhook inline and save a component."

You: "Because the 202 isn't about our load — it's about their patience. Providers give webhook receivers a short window, a few seconds, and GitHub will outright disable a hook that times out repeatedly. The moment our database has a slow day, inline processing turns one slow query into a disabled webhook, and a disabled webhook is silent data loss at the front door. Reconciliation would eventually backfill the runs (D1), but detection latency degrades from 30 seconds to minutes — during exactly the kind of incident where minutes matter. Accepting fast and doing the work async means ingestion availability is decoupled from everything behind it. At 200 events a second peak, the queue is almost embarrassing — but it's buying failure isolation, not throughput."

Quick check before moving on: