Section 12: Every Thread Pulled - Interviewer Challenges & Final Scorecard

Executive Summary: Contoso's encoded PowerShell has been traced from vm-web-07 to an analyst's screen, and now the interviewer pulls threads. Eight challenges follow, each answered with the mechanism that already handles it, the number that bounds it, and the senior line. Then the interview is scored on a 100-point card, and we map where this question hides in real loops.

The Gauntlet

Challenge 1: "One tenant suddenly sends 10× its quota"

Interviewer: "Contoso's contract is 10,000 events/s. They start sending 100,000."

What they're really asking: Is the noisy neighbour bounded by a mechanism, or by luck?

You: "Past its 20,000 burst, Contoso gets 429 with Retry-After from the front door's per-tenant token bucket. The excess never reaches the queue; Contoso's agents hold it in their WAL and retry with jitter. If a WAL fills, P2 drops first, and every drop is counted. A real incident gets a quota raise. If it's the new normal and planned peak passes 250k events/s, Contoso moves to dedicated stamps."

The senior line: "A 429 is the tenant's problem; a throttled 202 or a 503 is mine."

Challenge 2: "ADX in one stamp is down for two hours"

What they're really asking: Do you know what is lost, what is late, and what is untouched?

You: "Nothing acknowledged is lost: the queue holds 72 h. The lake writer and the detection hot path read their own consumer groups, so they carry on. Hot search on that stamp falls back to lake search jobs that take minutes. For that stamp's tenants, two hours is about a quarter of their 8.76-hour yearly query budget; fleet-wide it's 1/60 of that. On recovery the writer drains about 1.2 TB raw of backlog through design headroom. The step people miss: scheduled rules failed meanwhile, but they window on ADX ingestion_time(), so later runs see every row as the backlog lands. Alerts arrive late, not never."

The senior line: "In a dual-consume design an outage costs freshness, not data, as long as rules window on when rows land."

Challenge 3: "Cut storage cost 40%"

What they're really asking: Do you know where the money is, or only where the bytes are?

You: "Archive holds the most bytes, 37.8 PB of 72.5, but it's the cheapest per TB. ADX holds only 3.2 PB, at about ten times the lake's cost per TB. So I cut by class and age, after query telemetry shows which tables and days anyone reads. P2 is half of all volume: filter its verbose fields at the source and keep the P2 lake for 90 days, not 365. That removes 0.5 × 31.5 PB × 275/365 ≈ 11.9 PB of the 31.5 PB lake. Lake days 31–365 are read only by search jobs, so they move to the Cool or Cold access tier. P2 tables get a shorter SSD cache. On typical list prices, these together reach about 40% of lake and hot spend. P0 keeps its 7 years."

The senior line: "Cost follows class and age: cut the bulk tail, never the evidence."

Challenge 4: "Guarantee exactly-once"

What they're really asking: Do you know what the guarantee costs and where it can't reach?

You: "The HTTP hop rules it out: an agent whose 202 was lost has to resend. So we run at-least-once, and every writer commit is idempotent, keyed by (hub, partition, offset range). In the lake that's Delta's txnAppId and txnVersion, and in ADX it's ingest-by tags with ingestIfNotExists. Those stop consumer replays, not producer resends: a resent batch lands at new offsets as duplicate rows. They share a BatchId and Seq, so queries and lake compaction collapse them. I won't keep a front-door dedupe table: a consistent lookup on every request's hot path adds latency and a failure mode, the keys don't follow a tenant when failover sends the replay to the paired stamp, and the duplicates are harmless and removable."

The senior line: "The pipe is at-least-once; I buy effectively-once per consumer with offset-keyed commits plus BatchId dedupe where a consumer needs it."

Challenge 5: "A producer ships a schema change that breaks parsing"

What they're really asking: Does one team's deploy become everyone's outage?

You: "The front door checks only the envelope, so the batch still gets its 202. Writers parse against versioned table definitions from the schema registry; unknown fields fall into AdditionalFields, mismatches into the tenant's ingestion-errors table, poison batches into the dead-letter store. The raw bytes stay in the queue for 72 h, so we fix the parser and reprocess."

The senior line: "A producer's schema change is a new version in the registry, not an outage."

Challenge 6: "Events arrive timestamped 2019, or next year"

What they're really asking: Do you separate event time from arrival time, and know what a lying clock does to partitions?

You: "Each row carries TimeGenerated, the time the producer claims, and IngestTime, which we stamp ourselves. The lake partitions by date= from TimeGenerated. A 2019 date lands in a partition retention already dropped; a future one escapes every sweep. So at ingestion we clamp events more than 1 h in the future, or older than the lake's 365-day retention, to ingest time, flag them, and keep the original in AdditionalFields.OriginalTimeGenerated. Some are attackers timestomping, so the flag is evidence. A domain controller replaying a 6-hour WAL lands in its true date partition."

The senior line: "Partition on the time you can trust, and keep the time you were told."

Challenge 7: "Prove to an auditor the logs weren't altered"

What they're really asking: Can you prove integrity to someone who doesn't trust you?

You: "The P0 archive is WORM under a locked time-based retention policy, so not even we can rewrite it. Every lake commit gets a signed manifest hash-chained to the one before, which the auditor can recompute. Batch IDs carry boot epochs and sequence numbers, so a missing batch shows up as a seq gap. Deletes need a separate privileged role and a delay, and each request is itself P0 audit data."

The senior line: "Immutability stops the change; hash chains prove it to someone who doesn't trust us."

Challenge 8: "A customer wants their data out in 30 days"

What they're really asking: Did you design for exit, or only for ingest?

You: "First: does 'out' mean a copy, or gone? A copy comes from the lake, the system of record. A year of Contoso in the lake is about 15.8 TB, plus about 19 TB of P0 archive (0.2 × 15.8 TB × 6 years): about 35 TB, roughly 13 MB/s over 30 days. Archive rehydration can take ~15 h, so it starts on day one. Gone means crypto-shredding a CMK tenant's key, effective once the 72 h queue retention passes, or a batched copy-on-write rewrite plus an ADX purge for a shared one. WORM against deletion is a legal-hold call."

The senior line: "If you can't get a tenant out, your isolation only exists on paper."

Key Insight: None of the eight answers adds a component.

Quick check before moving on:

---