Section 10: Show Less, Never More - Security, Tenant Isolation & Reliability

Executive Summary: This section answers the interviewer's two closing threats: who could abuse this system, and what happens when its parts fail. Tenant isolation is built into the keys, with Postgres row-level security as a backstop. Every channel through which a tag can leak (the page, counts, suggestions, error codes, tag metadata by id, the cursor, events and logs) gets a named guard. Every failure in the matrix degrades in the same direction, so the user sees less and never more. We close with regional cells pinned to residency and a disaster-recovery story with real RPO and RTO numbers.

The interviewer has watched Section 9's gauntlet land. Now they change the question: "Forget performance. Who could use this service to see something they shouldn't, and what happens when half of it is on fire?" Most candidates treat those as two topics, a security slide and a reliability slide. They are one topic. A tagging service that owns neither the content nor the permissions has exactly one way to fail badly, and that is showing someone an item, a count or a tag name they had no right to. So every design choice in this section passes the same test: when in doubt, show less, never more.

Tenant Isolation by Construction

The service runs 64 shards, one of them dedicated to Meridian. Bluefin shares one of the other 63 with its neighbors: 250,000 tenants on the 63 shared shards ≈ 4,000 tenants per shard. Isolation therefore can't rely on careful engineers. It has to be structural.

The first layer comes from Section 4: the tenant comes from the token, never from the path or the body. The gateway resolves the token to (tenant, user), and no endpoint accepts a tenant parameter. The second comes from Section 5: tenant_id leads every key, so every index lookup is scoped to one tenant before it reads a single tag. That makes the tenant-scoped query the fast one, but it doesn't make a forgotten predicate fail. Postgres 18's B-tree skip scan can use a non-leading column, and a small test dataset hides a slow scan anyway.

The third layer is the backstop for the bug neither of those catches:

ALTER TABLE tag_assignment ENABLE ROW LEVEL SECURITY;
ALTER TABLE tag_assignment FORCE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation ON tag_assignment
  USING      (tenant_id = current_setting('app.tenant_id')::bigint)
  WITH CHECK (tenant_id = current_setting('app.tenant_id')::bigint);

-- Every request transaction starts with:
SELECT set_config('app.tenant_id', $1, true);   -- $1 from the token, never the request

The same policy sits on tag, tag_container_count and outbox. The third argument, true, scopes the setting to the transaction, like SET LOCAL (which can't take a bind parameter), so a pooled connection can't carry Bluefin's tenant into the next request. If the setting is missing or empty, the cast fails, so the query errors out instead of matching another tenant's rows. The app's database role is neither a superuser nor BYPASSRLS, because both skip policies.

Interviewer: "Isn't row-level security redundant if every query already filters on tenant_id?"

You: "On a good day, yes. But we'll have dozens of queries across the API, the bulk workers and the content-events consumer, written by different people over years. Relying on a WHERE clause alone means one forgotten predicate is a cross-tenant breach. The policy turns that bug into an error. It costs one equality check on a column that already leads the index, and that's cheap insurance against the worst incident this service can have."

The tenant directory is the only structure that spans tenants, and it holds only tenant → shard. No tag names, counts or content refs live there, and there's no global index. Section 6 deliberately left one out.

Quick check before moving on: