Section 9: Breaking the Watchtower - Interviewer Deep-Dive Challenges

Executive Summary: With both deep dives on the board, the interviewer stops asking how things work and starts asking how they fail. Seven escalations follow, from a log storm to a credit card number in a log line. Each is answered by a decision you already made, with a number and a flip point.

Size the threat with a number, name the mechanism that already covers it, and say where the decision flips. Redesigning on the spot reads as panic.

Challenge 1: The Log Storm

"A bad deploy makes payments-gateway log 50× normal, right in the middle of an incident."

What they're really asking: Can one team's mistake take down everyone's observability, at the exact moment everyone needs it?

The model answer: payments-gateway runs at 20,000 lines/s, so 50× is 1,000,000 lines/s: half the fleet's 2M average, from one service. The defense is per-service and ordered:

The lines Dana needs at 02:14 (upstream=card-network-b timeout after 2000ms) are WARN or ERROR, so they survive. The buffer is finite: a host at 50× emits 20 KB/s × 50 = 1 MB/s and fills 1 GB in about 17 minutes, so the agent sheds before it buffers. If the storm itself is ERROR lines, sampling never touches them. The agent collapses identical messages into one line with a repeat count, and past a hard ceiling WARN and above wait in that host's buffer instead of being dropped, so the cost lands on the noisy service's hosts, not the shared pipeline. I rejected a global first-come drop (the noisy service starves everyone) and unlimited autoscaling (cost explodes, the pipeline still lags).

The senior line: "Quotas are per service and shedding goes by severity, so a storm costs its owner DEBUG lines and costs nobody else anything."

Quick check before moving on: