Section 12: Growing at the Seams - Follow-up Feature Evolution

Executive Summary: With ten minutes left, the interviewer asks for the features customers want next. Each one attaches at an existing seam. FIFO topics flip the partition key to (topicId, messageGroupId), deliver each group serially, and add a 5-minute dedup window. Archive and replay copies opted-in topics to object storage and feeds replays back into the Delivery Queues. Large payloads use a claim check. Cross-region replicates opted-in topics asynchronously and delivers from the subscription's home region. Delivery status logs are opt-in, sampled, and written off the hot path. The API, the Message Log and the fan-out model stay as they are.

"Customers love it. Now they want ordering, replay, bigger messages and a second region. How much of this do you rebuild?"

The answer the panel wants is "none of it": show where each feature plugs in and what it leaves alone.

FIFO Topics: Order Per Group, Not Globally

ShopFast's payments team hit the problem from Section 9: OrderCancelled arrived before OrderPlaced. A FIFO topic fixes that for anyone who opts in.

• Create: POST /v1/topics {name: "order-events-fifo", type: "fifo"}.
• Publish adds two fields: messageGroupId (ShopFast uses the order id, so every event for #84721 is one group) and messageDeduplicationId.
• Partitioning: D3 flips for FIFO topics only. They key by hash(topicId, messageGroupId), so one group always lands on one partition, in publish order. Standard topics keep hash(messageId).
• Serial per group: a group's next message isn't delivered until the previous one is acked. Different groups run in parallel.
• Dedup: the Publish API drops a repeat messageDeduplicationId seen in the last 5 minutes (D4's flip). At ShopFast's 500 publishes/s that's 500 writes a second and 500 × 300 s = 150,000 keys, against 150,000 writes a second for every topic, the dedup store Section 4 refused to build.
• Subscribers: queue protocol only, and the topic stays within one page (≤ 1,000 subscriptions), so expansion is inline and in order.

Interviewer: "Why only queue subscribers? The payments team wants a webhook."

You: "Because a group blocks behind its head message. If the head is retrying against a webhook, the group waits through the whole backoff, about 17 minutes, and during Section 8's 40-minute partner outage every group on that subscription would freeze. A queue we run answers in milliseconds. The payments team can subscribe a FIFO queue and read it at their own pace. If customers push hard for FIFO over HTTP, the flip is per-group parking: one blocked group parks without stalling the others."

The cost is throughput per group. With serial queue sends of about 10 ms (illustrative), one group tops out around 100 messages/s. Parallelism comes from many groups, which is the point of keying by order id rather than by topic.

Quick check before moving on: