Section 11: The Most Dangerous Service in the Company - Safety, Security & Reliability

Executive Summary: The service built to delete data on request can also delete the wrong data at scale, on schedule, with a tidy audit trail. This section treats the Privacy Request Orchestrator as a threat in its own right and covers four layers: bounding the blast radius of a wrong deletion, blocking destructive account takeover, stopping exports from becoming an exfiltration channel, and making sure an accepted request is never lost and a one-day outage does no legal harm. Every control comes with its number, the alternative it beat, and what would change it.

After Section 10's seven escalations, the interviewer flips the question: "You've convinced me it deletes everything. What stops it from deleting everything?"

"A buggy service usually drops requests or serves stale data. A buggy deletion pipeline does irreversible damage with good intentions, fanned out to 1,200 holders at 634 tasks per second. So I treat this as the most dangerous service in the company and build controls so that no single layer has to be perfect."

Bounding the Blast Radius of a Wrong Deletion

"The bug I fear most isn't exotic: a handler that receives an empty identifier set. Some ORMs and query builders turn an empty IN-list into no WHERE clause at all, and a nil filter quietly dropped from a map-based query does the same, so the DELETE runs unfiltered. One subject's erasure becomes a whole table's."

The Privacy Agent SDK from Section 6 sits between the event and each handler, so the guards live there and all 1,200 teams inherit them:

• Identifier validation. The SDK won't call the handler if the identifier set is empty, any value is null, or a value doesn't match its registered format, and every handler asserts a non-empty, exact-match predicate before it deletes. The task goes FAILED with a reason; it never guesses.
• Dry-run, then canary. A new or changed handler ships in dry-run mode, counting the rows it would delete. After a week of plausible counts it runs live against Section 9's synthetic canary subjects, and only then against real users.
• The per-service anomaly threshold. Each privacy.yaml declares an upper bound. If a task reports more than 1M rows for one subject, the SDK stops before committing and the task goes to a human.
• The global breaker. If erasure intake in any hour exceeds 10× the trailing 7-day hourly average, fan-out pauses for approval. Intake keeps accepting, and requests wait in GRACE or FANNING_OUT.
• No data-store credentials. The orchestrator publishes events and records acks; it can't connect to a single holder's database, because each service deletes its own data. Compromised or buggy, the worst it can do is send bad events, and each must still pass every service's own guards.

The interviewer pushes: "A 10× breaker will trip during a real privacy scandal. Section 3 planned for exactly that peak."

"It will, and that's the point. A real 10× surge and a runaway bug look identical for the first hour; a person tells them apart in minutes from the news, the source mix and the verification rate. A pause costs hours against a 6-day buffer. A missed bug costs data I can't get back. I rejected auto-resume after a cooldown, because it resumes the bug too. I'd allow auto-resume for a source only once its verified-request rate is trusted, such as in-session requests with 2FA."

Critical Insight: The 14-day grace period does two jobs. Section 2 justified it as regret protection, but it is also the system's biggest safety margin: a bug in intake, identity resolution or the breaker surfaces while every affected account is merely disabled, and disabling is reversible.

Quick check before moving on: