Section 7: Where the Weight Lands - Deep Dive (Candidate-Driven)

Executive Summary: The interviewer hands you the wheel, and you pick four load-bearing walls. The durability point: Publish acks after a quorum write to 2 of 3 AZs, inside a 100 ms p99. Large-topic fan-out: the 8M-subscriber flash sale becomes 8,000 page tasks expanded in parallel, with the payload passed by reference. Filter policies: evaluated before a task exists. Subscription Cache staleness: push invalidation under 1 s, a 30 s TTL backstop, and tombstones. Each wall comes with its mechanism, its number, and what you gave up to get it.

"Where would you like to go deeper?" This question is scored. Your answer shows what you think would break first in production. So name your four walls, and say what you're saving: "The endpoints we don't control deserve their own dive. I'll take that next, in Section 8. First, the four places inside our own pipeline where a wrong choice costs us."

Wall One: The Durability Point

The Publish API returns 202 {messageId} only after the message is on the Message Log in 2 of 3 AZs (D1). That's the promise. An acked message survives losing any one AZ, and it will be fanned out.

The Message Log is a quorum-replicated log: Raft-style, or Kafka with acks=all and min.insync.replicas=2 (D2). One catch with Kafka: acks=all waits for every in-sync replica, so a slow third replica adds latency until it drops out of the ISR.

Here's the p99 budget of 100 ms:

That's why ShopFast's order #84721 is acked in about 15 ms, near the 20 ms p50.

Interviewer: "What if only one AZ is healthy?"

You: "Then there's no quorum, and the Publish API returns 503 rather than ack a single copy. One copy is one disk failure away from losing a message we promised to keep, and the publisher may delete its copy the moment we ack. A 503 is honest: the publisher still holds the message and retries. I'm choosing durability over availability, and that outage spends error budget, which is the right currency for it."

Interviewer: "The publisher times out at 100 ms and retries. Now what?"

You: "If the first write landed, we have two copies with two different messageIds. Subscribers see a duplicate. I accept that. It's one of the three duplicate sources under at-least-once (D4), and subscribers dedupe on messageId anyway."

Quick check before moving on: