Section 11: The Third Consumer - Follow-up: Real-Time Detection & Correlation

Executive Summary: The interviewer's follow-up is always "now catch attacks in real time." The platform already has the hook. Detection becomes a third consumer group on the P0 and P1 hubs, with a queue read lag of 5 s or less at p99. It forks at the queue rather than at storage, so ingestion doesn't change. We keep scheduled queries alongside streaming. We sketch the engine and hand its internals to the Azure Real-Time Threat Detection System topic. Then we list what this platform owes that engine and walk Contoso's attack from sign-in to incident in seconds.

The interviewer leans back. "Customers say a scheduled query every five minutes is too slow. They want the alert while the attacker is still typing. What do you change?"

The strong answer starts with what you don't change.

Fork at the Queue, Not at Storage

You: "Nothing in ingestion changes. Section 7 already gave the queue three consumer groups: the ADX writer, the lake writer and a detection hot path. Detection is a new reader of the same durable log, not a new pipeline."

Interviewer: "Why not detect off ADX? It's already indexed."

You: "That puts storage freshness inside the alert latency: a P1 event takes up to 2 minutes at p95 to become queryable in ADX, but it's readable at the queue within seconds of the 202. It would also tie detection to ADX health, so a writer 40 minutes behind would blind detection too. Consumers of one log fail independently. I'd also reject detection inline in the front door: it would add time before the ack, and a bad rule could take ingestion down with it."

The hot path subscribes to P0 and P1 only. P2 (network flows, verbose diagnostics) is half the volume and the first class we shed, so a rule built on it can't promise complete input. The flip trigger: a detection family that needs flows, such as beaconing, subscribes to the P2 hub and is labelled best-effort.

Key Insight: The alert edge feeds back into the front door. The engine writes alerts the way any producer writes telemetry: through the ingestion API, as P0, into an Alerts table. Alerts is the one low-volume table on ADX streaming ingestion, so an alert is searchable within seconds of being raised. The loop means the engine reads its own P0 alerts, so rules exclude Alerts and Incidents as input, except correlation, which reads them deliberately. An analyst investigates the alert and the raw events behind it with the same KQL, under the same tenant isolation and retention. No second store, no second query language.

Quick check before moving on: