Message Queues & Pub/Sub — Part 1: Foundations of Asynchronous Messaging
maang.io System Design Series
What is this?
Walk into a busy restaurant kitchen and watch what doesn't happen. The waiter does not stand at the chef's elbow, repeating the order, waiting motionless until the plate is done, refusing to take another table's order in the meantime. That would be absurd — one slow risotto and the entire dining room grinds to a halt.
Instead, the waiter scribbles the order on a ticket, clips it to a rail above the pass, and walks away. The ticket sits there. Whenever a cook is free, they pull the next ticket off the rail and start cooking. The waiter and the cook never have to be ready at the same instant. The rail absorbs the difference between how fast orders come in and how fast food goes out.
That ticket rail is a message queue. The waiter is a producer, the cook is a consumer, and the rail is the broker sitting patiently in between. This one shift — from "wait for the answer" to "drop it off and move on" — is the entire idea of asynchronous messaging, and it's the only genuinely asynchronous pattern in this whole tier. Every other communication topic you've met so far — REST, RPC, gRPC, WebSockets — is fundamentally synchronous: the caller asks and then waits for a reply, the two sides locked together in time. A message queue breaks that lock.
The one-line idea: a message queue lets a producer hand off work and walk away — decoupling sender from receiver in time — so the two never have to be fast, available, or even awake at the same moment.
This part is about why you'd want that decoupling, the three-actor model that makes it work, and the single most important distinction in the whole topic: a queue (one ticket, one cook) versus a topic (one announcement, every cook hears it).
1. The thing that makes messaging different: time
Let's be precise about what "asynchronous" actually buys you, because the word gets thrown around loosely.
In a synchronous call — the world of every prior chapter — the caller's thread blocks. Service A calls Service B over gRPC and sits there, holding a connection open, until B answers. A and B are temporally coupled: they must both be up, both be reachable, and B must be fast enough that A's wait is tolerable. If B is down, A's call fails right now. If B is slow, A is slow too. They rise and fall together.
A message queue inserts a durable buffer — the broker — between the two sides and dissolves that temporal coupling. The producer writes a message to the broker and immediately gets on with its life. The consumer reads it whenever it's ready: a millisecond later, or ten minutes later if it was busy or restarting. Neither side waits on the other.
That single change cascades into four concrete superpowers. Hold onto these — we'll keep coming back to them.
- Decoupling. The producer doesn't even need to know who the consumer is, how many there are, or where they live. It speaks only to the broker. You can add, remove, or redeploy consumers without the producer ever noticing.
- Buffering & spike-smoothing. The rail holds tickets when orders pour in faster than the kitchen can cook. Traffic spikes hit the queue, not your fragile downstream service.
- Resilience. If the consumer crashes, messages pile up safely in the broker and get processed when it comes back. Nothing is lost; the work just waits.
- Retries. Because the broker holds the message until the consumer confirms success, a failed attempt can simply be redelivered. The work survives a bad night.
💡 The mental test for "do I want a queue here?": ask "does the caller actually need the answer right now to continue?" If yes, call synchronously. If the caller just needs the work to eventually happen, hand it to a queue and walk away.
2. The three actors: producer → broker → consumer
Every messaging system, no matter how fancy, is these three roles. Get them crisp and everything else slots into place.
- Producer (a.k.a. publisher). Creates a message — a small, self-contained envelope of data, usually JSON, protobuf, or Avro — and sends it to the broker. It does not wait for the work to be done; it waits only for the broker to say "got it, it's safely stored." That confirmation is fast and cheap.
- Broker (the message queue / message bus itself — RabbitMQ, Kafka, Amazon SQS, Google Pub/Sub). The heart of the system. It stores messages durably and routes them to the right consumers. Its whole job is to be the reliable middleman so the two ends never touch.
- Consumer (a.k.a. subscriber / worker). Reads messages from the broker, does the actual work — sends the email, charges the card, resizes the image — and then acknowledges ("ack") that it's done so the broker can forget the message. That ack is the linchpin of reliability; we'll dissect it thoroughly in Part 2.
The message itself is worth a closer look. A good one is small (kilobytes, not megabytes — you send a reference to the 50 MB video, not the video), self-describing, and carries metadata: a unique ID, a timestamp, a type, maybe a routing key. Keep payloads lean. The broker is a post office, not a warehouse — it moves envelopes, it shouldn't be schlepping furniture.
3. The big fork: queue (point-to-point) vs topic (pub/sub)
Here is the distinction that the rest of this topic — and half your interview answers — pivots on. There are two fundamentally different delivery shapes, and confusing them is the most common conceptual mistake engineers make.
3.1 Queue — point-to-point, one consumer wins each message
A queue is the ticket rail. Many cooks may be standing at the pass, but each ticket is pulled by exactly one of them. The order "table 4: one risotto" is cooked once, not once per cook. The message is a unit of work, and work should happen exactly once.
This is competing consumers: you put N workers on one queue and they split the load between them. Add workers, go faster — this is how you scale work horizontally. The classic use is a task / job queue: "resize this image," "send this email," "process this payment." You want each job done once, by whichever worker is free.
3.2 Topic — publish/subscribe, every subscriber gets a copy
A topic is a different beast entirely. Picture the kitchen's overhead announcement: "We're out of salmon!" That isn't a task for one cook — every station needs to hear it. The grill, the sauté, the expo — each gets its own copy and reacts in its own way.
This is fan-out. One published event — say, OrderPlaced — is delivered to every interested subscriber, each of which does something independent: the email service sends a confirmation, analytics updates a dashboard, the audit log records it, recommendations refresh. The publisher has no idea any of them exist. Tomorrow you bolt on a fraud-detection subscriber and the publisher's code never changes. That is the decoupling superpower at its most dramatic.
⚠️ Don't blur these. A queue load-balances one job across competing workers (do-it-once). A topic broadcasts one event to many independent subscribers (everyone-gets-a-copy). "Should this message be handled once, or heard by everyone?" is the question that picks your shape.
3.3 The combined pattern you'll actually deploy
Real systems often nest these: a topic fans an event out to several consumer groups, and within each group a pool of competing workers shares the load. One copy per group, then split inside the group. Kafka's consumer groups and SNS-fronting-SQS both do exactly this — broadcast between teams, load-balance within a team. We'll see the machinery for it in Part 2.
4. A worked example: the spike that takes down checkout
Abstractions get real with numbers, so let's watch a queue earn its keep.
Your e-commerce site normally takes 100 orders/second. Each order kicks off a chain of slow work: charge the card (300 ms), reserve inventory (150 ms), email a receipt (~400 ms), update analytics. Done synchronously, the customer's "Place Order" click hangs for nearly a second while all of that completes inline — and every one of those steps is a downstream service that can be slow or down.
Now a flash sale hits and traffic jumps to 2,000 orders/second for ninety seconds.
Synchronous design: the payment service can handle maybe 500 charges/second. At 2,000/s it's swamped 4× over. Requests queue up inside it, time out, threads exhaust, and the failures ripple backward into checkout — which now hangs and fails too. One overloaded service has taken down the whole purchase flow. This is the cascading failure interviewers love to probe (the Rate Limiting and Load Balancer chapters are the other half of defending against it).
Queue-based design: checkout does the bare minimum synchronously — validate the cart, write an OrderPlaced message to the broker, return "Order received! 🎉" in ~20 ms. That's it. The slow work — payment, inventory, email — runs as consumers pulling from the queue at their own sustainable pace.
During the spike, ~180,000 messages pile up in the queue. The payment workers chew through them at 500/s and fully catch up a few minutes after the burst ends. No request was dropped. No service fell over. The customer got an instant response, and the receipt email arrived two minutes later — which is completely fine, because nobody expects a receipt instantly. The queue converted a catastrophic spike into a tolerable backlog. That trade — instant acknowledgment now, eventual completion later — is the bargain at the center of async messaging.
5. When a queue is the wrong tool
A great engineer knows the limits of their favorite hammer. Queues are not free, and reaching for one reflexively is its own footgun.
- 🚫 When the caller genuinely needs the answer to proceed. "Is this username available?", "what's this user's current balance?", "did the payment succeed so I can show the confirmation page?" — these are questions, and a question needs an answer now. Don't fire a message into the void and poll for a reply; just make a synchronous gRPC or REST call. Forcing request/response through a queue is painful and slow.
- 🚫 When you can't tolerate eventual consistency. Async means the work happens later. For a heartbeat that's nothing, but if your UX or business logic assumes the effect is already visible the instant the producer returns, a queue will burn you. (The CAP Theorem chapter is the deep dive on why "later" is sometimes unacceptable.)
- 🚧 The operational tax is real. A broker is one more distributed system to deploy, monitor, secure, and reason about. It can be down. It can fill up. It introduces a brand-new class of bugs — duplicate deliveries, out-of-order messages, poison messages, consumer lag — that simply don't exist in a plain function call. We'll spend Parts 2 and 3 taming exactly those.
- 🚧 Debugging gets harder. A synchronous call has one neat stack trace. An async flow scatters across producer logs, broker metrics, and consumer logs, stitched together only by a correlation ID you'd better have remembered to add. The post office is reliable, but you can no longer just watch the letter travel.
💡 Rule of thumb: reach for a queue when the work is a fire-and-forget side effect (send, log, sync, notify, process-eventually) and the caller doesn't need the result to continue. Keep it synchronous when the work is a question whose answer blocks the next step.
6. Where this sits in the bigger picture
Notice the through-line: messaging is the temporal-decoupling layer of a distributed system, and it pairs with the patterns you've already met. An API Gateway routes the synchronous front door; behind it, services often talk to each other asynchronously over a broker so a slow downstream can't stall the request path. A Load Balancer spreads synchronous traffic across instances; a queue with competing consumers does the spiritual equivalent for background work. And the broker's durability story leans directly on the storage and consistency ideas from the Databases and CAP Theorem chapters.
Part 2 cracks the broker open: how it stores messages durably, what an "ack" really guarantees, the three delivery semantics (at-most-once, at-least-once, and the slippery "exactly-once"), message ordering, and the two great architectural lineages — the log model (Kafka) versus the traditional broker (RabbitMQ/SQS). That's where the ticket rail gets its engineering.
Key takeaways
- A message queue decouples producer from consumer in time — the producer hands off work and walks away — making it the one asynchronous pattern in a tier otherwise full of synchronous request/response (REST, RPC, gRPC, WebSockets).
- Everything reduces to three actors: a producer that sends, a broker that durably stores and routes, and a consumer that processes and acks. Keep messages small — send references, not payloads.
- The defining fork is queue vs topic: a queue load-balances one job across competing workers (handled once); a topic fans one event out to every subscriber (everyone gets a copy). "Once, or for everyone?" picks the shape.
- Queues earn their keep by absorbing spikes, decoupling services, surviving consumer crashes, and enabling retries — trading instant acknowledgment now for eventual completion later.
- They're the wrong tool when the caller needs the answer to proceed, when you can't tolerate eventual consistency, or when the operational and debugging tax outweighs the gain. Async is a deliberate choice, not a default.