Section 9: Eight Ways to Break a Tag - Interviewer Deep-Dive Challenges

Executive Summary: The design is on the board and both deep dives have held. Now the interviewer escalates on purpose: delete a tag on 1.2M items, merge two tags, import 50M assignments, swap the store for Elasticsearch, give each product a labels field, take Bitbucket's hydration down, feed it Turkish and German text, spam it. Every answer has the same shape. The part users see happens in O(1), inside the request. The O(n) part runs on a lane that can't hurt anyone. And whatever reaches a screen still passes through the owning product. This section runs all eight challenges and gives each its mechanism, its number and the trade-off you accepted to get it.

The interviewer has watched you build the funnel in Section 8, and now they stop collaborating and start pushing. Each question below is built to catch one of two reflexes: doing O(n) work inside a request, or showing a user something its owner never approved. If you name the reflex before they spring it, the challenge turns into a chance to show what you know.

Key Insight: Split every admin operation into a logical change (one row, effective at once, visible to every reader) and a physical change (millions of rows, rate-limited, resumable). Users only ever wait on the first.

Challenge One: "Delete frontend. Now."

"Meridian's frontend tag is on 1.2M items across 800 containers. An admin wants it gone."

You: "Gone for users right now, and gone from disk in about twenty minutes. Those are two operations, and only the first runs inside the request."

DELETE /v1/tags/{tagId} performs one UPDATE on the tag row: state = 'deleted', and normalized_name is rewritten to the sentinel chr(1) || 'deleted:' || tag_id. The sentinel frees the name while keeping Section 5's UNIQUE (tenant_id, normalized_name) and Section 7's ON CONFLICT target valid, and no real name can collide with it because tag names can't contain control characters. The forward cache holds tag ids only, and every read resolves those ids through the tag-metadata cache ((tenant_id, tag_id) → display name, state, alias_of), dropping anything marked deleted. The delete publishes an invalidation on Meridian's pub/sub channel, so frontend drops off board cards, the tag page and suggestions within seconds, with the cache's 60 s TTL as the backstop if a message goes missing. The forward cache may still list the old id until its 10 min TTL expires, but resolution filters it out, so those entries are harmless. Freeing the name matters too. If Ines types frontend again tomorrow, the new tag gets a fresh tag_id and never inherits the old rows still waiting to be purged. Then the endpoint returns 202 with a job id.

Rows to purge:     1,200,000 assignments
Bulk lane rate:    1,000 rows/s (Meridian's token bucket)
Purge time:        1,200,000 / 1,000 = 1,200 s = 20 min
Count rows:        800 (one per container), zeroed by the job
Events:            1 TagDeleted, not 1.2M per-item removals
                   (1.2M x ~400 B = ~480 MB kept off tag-events)

Interviewer: "Why not one DELETE FROM tag_assignment WHERE tag_id = ...?"
You: "A 1.2M-row delete is one long transaction on Meridian's dedicated shard. Replicas fall behind the ~1 s lag that tag pages are built on, and the shard's interactive writes stall behind it. In batches of a thousand a second, each transaction stays short, and a crash costs one batch, not the whole delete."

Interviewer: "And a PUT that resolved frontend's id a millisecond before your UPDATE?"
You: "It can't slip a row in behind the purge. We close that race with a lock rather than a grace-period sweep. Get-or-create runs before the assignment transaction, so inside it the writer re-reads the tag row FOR SHARE and checks the state. The admin's UPDATE waits on those share locks. So every in-flight writer either commits first, and the purge's scan finds its row, or reads deleted afterward. An add by id then gets a 404, and an add by name re-runs get-or-create, which creates a fresh tag because the name is free."

The trade-off is in the event contract. TagDeleted is a tag-level event keyed by tag_id, so the product search indexes have to understand it and drop that id from their own index. It has no ordering against per-item events, so consumers ignore any per-item event for a tag id they already hold as deleted. A consumer that only understands per-item events would go stale. We accept that, because otherwise we'd emit a million events to describe a single decision.

Quick check before moving on: