Section 5: Three Million Rules, One Job - Detection Engine Internals & Execution Flow

Executive Summary: Section 4 — Data Modeling & Rule Design turned a rule into data; this section makes 3M of them run. In each stamp, one Flink job reads the normalized-events hub, a Router sends each event once to every entity its tenant's rules group by, and keyed Rule Evaluators keep windows, baselines and partial sequences in RocksDB. Rules that can only become more true fire the moment they hold, so Contoso's brute force pages on the 10th failure, not at window close. Watermarks are kept for "A not followed by B" rules and state cleanup. We then handle duplicates, late events and hot keys, and get effectively-once alerts from 30-second checkpoints without the checkpoint interval entering the latency budget.

The interviewer circles the Detection Engine box. "That box holds 3M (tenant, rule) instances and sees 3M events a second at peak. Show me how it fires in seconds."

You start with the arithmetic. Indexed by (tenantId, sourceType), an event meets about 20 candidate rules: 60M rule evaluations a second fleet-wide, 5M/s per stamp. A job per rule or per tenant would mean millions of jobs, so each stamp runs one Detection Engine job over every tenant's rules, with the rules arriving as data. The stage gets 5 s of the 22-second latency budget.

The Job Topology

The Rule Service publishes compiled, versioned rule sets on the rule-sets stream, and Flink broadcasts that stream: every parallel Router and Rule Evaluator task keeps a full copy of the stamp's rule sets in broadcast state, indexed by (tenantId, sourceType). Broadcast state is replicated, so every task must apply each update deterministically. A swap is atomic per task: the current record finishes under the old version, the next one sees the new. Flink makes no ordering promise between the broadcast and data inputs, so two tasks can briefly be one version apart. They converge in seconds, inside the 60 s activation SLO, and every alert records its ruleVersion.

For each event, the Router looks up the candidate rules, collects the distinct groupBy entities those rules need, and emits one keyed record per entity, listing the rule IDs to run. Contoso's failed sign-in from 203.0.113.45 is a candidate for brute_force_user and impossible_travel (grouped by actor.userId; travel's filter then drops the failure) and password_spray_ip (grouped by actor.ipAddress). That gives two records: (t-4821, user, oid:4f1da9e2) and (t-4821, ip, 203.0.113.45). Fan-out averages 2.5, which means 625k keyed records/s per stamp (7.5M fleet-wide).

Interviewer: "Why not key by tenant? Then one evaluator sees everything a tenant does."
You: "The largest tenant runs at 100k events/s. Keyed by tenant, that's one key on one task slot, a hot spot that no amount of parallelism can split. Keying by rule fails the other way: each event gets copied about 20 times. Detection state belongs to an entity. A brute-force count is about one user and a spray count is about one IP, so we key on the entity and each event travels 2.5 times on average."

source_silent groups by (tenantId, sourceType), and forwarding every event to it would bring the per-tenant hot key back. Instead, each Router task sends a periodic heartbeat per (tenant, sourceType) it has seen, and each heartbeat resets a processing-time timer; silence has no event time. Two guards keep that timer honest. Silence is judged against the Ingest Edge's last-arrival time for the (tenant, sourceType), not our own processing progress; and while the stamp is itself behind (consumer lag over 60 s, a failing canary, or the first minutes after a restore or failover), overdue absence timers are re-armed for 15 minutes from now instead of firing. Otherwise one stamp stall would raise source_silent for every tenant on it.

Quick check before moving on: