Section 8: Where DELETE Doesn't Exist - Deep Dive: Erasing Data That Resists Erasure
Executive Summary: Section 7 got Lena's erasure task to every one of the 1,200 holders. This section follows it into storage that was never built to forget anything: caches, search indexes, append-only logs, columnar data lakes, application logs, backups and immutable archives. We sort storage into classes, give each one a mechanism and a time-to-gone, and defend two decisions with arithmetic: batched deletion vectors for the lake, and backup expiry plus crypto-shredding for everything that can't delete in place. The rule underneath all of it is that "gone" means beyond use within the deadline. It does not mean every byte was overwritten that afternoon.
The interviewer taps the architecture diagram. "Fine. Your agent SDK calls a handler and the handler deletes some rows. But Lena's photos are also in a nightly backup, a Kafka topic, a Parquet table with ten thousand files and a log line from 2019. Which of those does the handler delete?"
I resist answering store by store. "Not all storage is the same problem. I'd sort it by how it forgets, then match each class to a mechanism that finishes inside Lena's 10-day service SLA, or argue for an exception where it can't."
Classify First, Then Erase
| Storage class | Examples | Erasure mechanism | Time to "gone" |
|---|---|---|---|
| OLTP source of truth | Mail, photos metadata, profiles | Delete by key, in a transaction | Minutes |
| Derived copies | Caches, search indexes, feature stores | Cache invalidate + TTL; delete by doc ID then refresh; CDC from the source of truth | Seconds to hours |
| Append-only event logs | Kafka topics | Time retention (dispatch log: 14 days); compacted topics get a null-value tombstone per key, with max.compaction.lag.ms bounding the cleaner; long-retention topics use per-user encryption |
Up to retention or the compaction lag, or immediate via shred |
| Data lake / warehouse | Iceberg, Delta, Hudi tables | Daily batched row-level deletes; touched files rewritten every 3 days; snapshot retention cut to 2 days | ≤ 1 day logical, ≤ 6 days physical |
| Application logs | Service logs | PII scrubbed at ingestion; 30-day retention | ≤ 30 days, and little to find |
| Operational backups | Database snapshots | 35-day expiry + tombstone replay on restore | ≤ 35 days; a restore replays tombstones first |
| Long-retention archives, immutable stores | Archival blobs, WORM buckets, event-sourced stores | Crypto-shredding | Immediate once every key copy is gone |
| Third-party processors | Email vendor, support desk | Platform connectors call the vendor's deletion API (Art. 28 processors must assist; Art. 19 recipients are notified) | Contractual SLA set inside the 10-day service SLA |
| ML models | Ranking, recommendation | Out of scope here (Section 12) | Retrain cadence |
The first two rows are cheap: they're what the Privacy Agent SDK handler does, and derived copies follow because the source of truth emits a change event. The trouble starts at row three.
On event logs, I'm precise about one thing because interviewers check it. "Kafka compaction only applies to compacted topics. It keeps the latest record per key, and a null-value record (a tombstone) for Lena's key marks it for removal: the cleaner drops her earlier values on a later pass, and the tombstone itself stays for delete.retention.ms (24 hours by default) so consumers see it, then goes too. Compaction timing isn't guaranteed, since it depends on segment rolling and the dirty ratio, so we set segment.ms and max.compaction.lag.ms to bound it. Topics without compaction just age out by time. Our own dispatch log carries only claim checks, a request ID and an internal reference and never an email or phone number, so its 14-day retention holds no personal data. A team running an event-sourced topic with years of retention can't rewrite history, so its events about Lena are encrypted under her key, which puts them in the crypto-shredding row."
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson