Section 6: Five Million Alerts, Two Hundred Thousand Incidents - Alerting & Integration Layer

Executive Summary: Section 5's Detection Engine emits about 5 million raw detections a day (≈ 58 a second), far more than a SOC can read. This section takes them from the detections hub to an analyst's pager. An Alert Service drops duplicates, applies suppressions, and correlates alerts that share an entity into incidents, a 25:1 reduction to 200,000 a day. Cosmos DB's change feed acts as the outbox, and per-sink Service Bus queues retry for 24 hours before dead-lettering. Contoso's INC-20419 escalates from Medium to Critical with one page, not two.

The interviewer points at the arrow leaving the Detection Engine. "Fifty-eight detections a second across 20,000 tenants. What does Contoso's SOC actually see?" About ten incidents a day, each telling a story (who, from where, doing what) instead of listing rule firings. Getting there is half data reduction and half plumbing for sinks that fail, and the plumbing is where designs quietly lose alerts.

The Pipeline, Box by Box

You walk it left to right.

The detections hub is partitioned by tenantId. That reverses Section 5's choice on purpose. The Detection Engine rejected tenant keys because the largest tenant is a 100k events/s hot key. Here the largest tenant produces about 5% of alerts, which is 250k a day or roughly 3 a second. At that rate a tenant key is cold, and it gives correlation what it needs: every alert for one tenant lands on one Alert Service worker, in order.

Dedupe is a conditional create. The alertId is deterministic, hash(tenantId, ruleId, ruleVersion, entityKey, windowBucket), so a checkpoint replay in the Detection Engine reproduces the same id. The Alert Service creates the alert document with that id as its Cosmos DB item id. A 409 Conflict means the alert already exists, so the worker drops it. This is the step that makes Section 5's effectively-once promise true. That only works because the day in the partition key comes from the alert's window bucket, not the clock: Cosmos DB checks id uniqueness per logical partition, so every copy of an alert must target the same one.

Suppression comes next. Tenants write suppression rules, such as "impossible_travel for users on our VPN egress ASN", and each one must have an expiry and a reason. Suppressed alerts are still stored, in an incident with status Suppressed, but never reach a sink. Without an expiry, a temporary exception becomes a permanent blind spot.

Correlation turns alerts into incidents. Alerts that share an entity (user, IP, host, resource or service principal) within 1 hour join the same incident. Each worker keeps an in-memory index from entity to open incidentId for the tenants it owns, rebuilt on start from the incidents updated in the last hour.

Interviewer: "Why not dedupe and correlate inside Flink and delete this service?"
You: "Dedupe already happens there, per key. Correlation can't. Section 5 keys state by (tenantId, entityType, entityValue), so alice's brute force and an IP's password spray live on different keys, often in different task slots. Correlation is the join across keys, so it needs the whole tenant in view, and at 58 a second that's a small service with one owner per tenant."

Interviewer: "At 58 a second you could query Cosmos for matching incidents on every alert. Why keep an in-memory index?"
You: "Ownership, not speed. Two workers querying at once could each find no incident for alice and create two. One owner per tenant means one writer, and the incident's etag still guards writes during rebalancing. The index is a cache; Cosmos is the truth."

Two refinements keep correlation honest. High-prevalence entities, such as the tenant's own corporate NAT egress IP, never count as shared, or every alert in the company would merge. And an alert matching two open incidents joins the most recently updated one and links the other; merging is an analyst's call.

The two-document write can resume after a crash. The worker creates the alert, with its incidentId set, then updates the incident under an etag check, and only after both writes checkpoints its hub offset. If it crashes between the writes, the replayed alert hits a 409. Instead of dropping it blindly, the worker checks the incident's alertIds and finishes the update if needed.

The Alert Store is Cosmos DB with hierarchical partition key [tenantId, day] and a 365-day TTL: 4 KB alerts, 20 GB a day, about 7.3 TB a year, and about 1 GB for the largest tenant's day, well under the 20 GB logical-partition limit. Azure SQL lost because write scale-out and multi-region are harder for this shape; Azure Data Explorer lost because batched ingestion adds latency and analysts need point updates. If customers want alert analytics, we export a copy to ADX or Fabric.

Quick check before moving on: