Section 5: Three Million Machines We Don't Control - Deep Dive: Data Collection

Executive Summary: Every event this platform stores starts on a machine we don't own. Push beats pull, which survives only as connectors. The agent is a small pipeline: collectors, central collection rules, a write-ahead log (WAL) on disk (1 GB, or 10 GB on domain controllers), a batcher (10 s or 1 MB, zstd) and a sender that backs off with jitter and keeps acked batches for 1 hour. Every batch carries an ID (agentId:bootEpoch:seq), every drop is counted, the tenant comes from the credential, and agent builds ship in rings.

The interviewer taps the leftmost box on Section 4's architecture diagram. "Everything to the right assumes the data shows up. Convince me it does."

Split the producers in two. The Azure service emitters (Entra sign-ins, Activity Log, diagnostics, Defender signals) send 4M events/s. We run that code, on our own networks, and it honours the same producer contract as the agents: a durable local buffer and the 1-hour replay window. The 3M agents on VMs, AKS nodes and hosts send 6M events/s, and they're the hard half: customer machines whose network, disk and reboots we don't control, and which an attacker may already be on.

Push, Not Pull

Interviewer: "Prometheus scrapes. Why not have the platform pull?"
You: "Pulling every 10 seconds is 300k pulls a second, a registry of 3M targets and coordination so exactly one puller owns each. Most hosts sit behind firewalls or NAT that block inbound connections. The real problem is buffering. If the puller is down for an hour, something on the host has to hold that hour of data. Once the source buffers, retries and batches, you've built an agent anyway."

Push needs only outbound HTTPS to one stamp endpoint, learned from the tenant registry at enrollment, behind Azure Front Door and WAF. There's no target registry, because the credential says who's calling.

The flip trigger is a source that can't run our code, such as a SaaS audit API. For those, pull connectors poll the API and push into the same ingestion API, so there's still one way in, one auth model and one quota system. Push's cost is that the front door must survive the crowd, so the agent must take backpressure gracefully.