Message Queues & Pub/Sub — Part 3: Idempotency, the Outbox Pattern, and Broker Comparisons

maang.io System Design Series

What is this?

Imagine the kitchen's busiest night. A cook grabs a ticket, makes the dish — and just as they plate it, the rail system glitches and re-clips the same ticket back up. A second cook, not knowing, makes the same dish again. Now you've got two identical risottos and one angry accountant. The fix isn't to pray the rail never glitches (Part 2 told us it will — that's at-least-once). The fix is to write the ticket number on a whiteboard the moment you start cooking, and have every cook check the whiteboard first: "table 4's risotto? Already done. Skip it."

That whiteboard is idempotency, and it's the single most important practical skill in production messaging. Part 2 left us with an uncomfortable bargain: at-least-once delivery means duplicates will happen, and true exactly-once is a myth. This part is the resolution — how real systems make duplicate-prone delivery behave as if each message were processed exactly once, and how they reliably get events into the broker in the first place without losing them or lying to themselves.

The one-line idea: you can't stop duplicates from being delivered, so you make your consumer not care — an idempotent consumer plus a transactional outbox turns messy at-least-once delivery into business-level "exactly-once," and that combination is what production event-driven systems actually run on.

Then we'll arm you with the comparison interviewers always ask for: Kafka vs RabbitMQ vs SQS/SNS — when each one is the right answer.