Publish–Subscribe Pattern — Part 1: Foundations & Core Concepts

maang.io System Design Series

---

What is this?

Picture a radio station. A DJ sits in a studio and broadcasts on 98.5 FM. She has no idea who is listening — one car, ten thousand cars, or nobody at all. She just talks into the microphone. Anyone who tunes their radio to 98.5 hears her, instantly and simultaneously; anyone tuned elsewhere hears nothing. The DJ never phones each listener. She never keeps a list of them. She publishes to a frequency, and the airwaves fan her voice out to everyone who cares.

That is the entire publish–subscribe ("pub/sub") pattern in one image. The DJ is a publisher. The frequency 98.5 is a topic. Each tuned-in radio is a subscriber. And the broadcast tower doing the fan-out — copying one signal to every listener — is the broker. Nobody on either side knows or names anybody on the other side. They only agree on the frequency.

We'll ride this radio analogy through all three chapters, because it captures the four things that make pub/sub special: the publisher is decoupled from subscribers, delivery is one-to-many (fan-out), subscribers opt in by topic, and — crucially — if your radio is off when the song plays, you miss it unless you've arranged to record it. Hold that last thought; "durable vs ephemeral" is where half the real-world bugs live.

The one-line idea: In publish–subscribe, a producer sends a message to a named topic — not to a recipient — and a broker fans that one message out to every subscriber of the topic. The producer never knows who (or how many) will receive it. Decoupling by topic is the whole point.

---

1. Why it exists — the pain before pub/sub

Imagine you're building an e-commerce checkout. When an order is placed, five things must happen: charge the card, reserve inventory, send a confirmation email, update the analytics warehouse, and notify the shipping service. The naïve design has the checkout service call all five directly:

This works — until it doesn't. Every new consumer of "an order happened" means editing and redeploying the checkout service. The checkout is now coupled to five downstream services: it knows all their addresses, it blocks on their latency, and if Analytics is down, the whole checkout can stall or fail. Add a sixth consumer (fraud scoring) next quarter and you're back in the checkout code again. This is tight coupling, and it's exactly the pain the Asynchronous Messaging topic describes — pub/sub is the pattern that dissolves it.

Flip it around. The checkout publishes one message to a topic called order.placed and moves on. It has no idea who's listening. Payments, Inventory, Email, Analytics, and Shipping each subscribe to order.placed on their own schedule. Adding fraud scoring next quarter means fraud scoring subscribes — the checkout code never changes. The publisher's job shrank from "orchestrate five downstream calls" to "announce that an event happened."

That inversion — from "I call my dependents" to "I announce, they listen" — is why pub/sub underpins nearly every event-driven architecture, from microservice choreography to real-time notifications to the data pipelines feeding CQRS read models.

---

2. Core concepts & vocabulary

Let's name the mental model precisely. You'll use these words in every interview.

• Publisher (producer): whoever creates a message and sends it to a topic. Knows the topic name; knows nothing about subscribers. (The DJ.)
• Subscriber (consumer): whoever registers interest in a topic and receives its messages. Knows the topic; knows nothing about the publisher. (The listener.)
• Topic: a named channel — the rendezvous point. Publishers write to it, subscribers read from it. This is the only thing both sides share. (The frequency 98.5.)
• Broker (message broker / event bus): the infrastructure in the middle that accepts published messages and delivers a copy to every subscriber of the topic. It holds the subscription list and does the fan-out. (The broadcast tower.)
• Message (event): the unit of data — a small, self-describing record like {orderId: 913, total: 42.00}.
• Subscription: the standing relationship between a subscriber and a topic. This is a real object the broker tracks, and its properties (durable? filtered? push or pull?) decide everything about how delivery behaves.

The defining move is one message → many independent deliveries. Contrast that with a phone call (one-to-one) or the direct-call checkout above. Pub/sub is a broadcast, mediated by a broker so the publisher and subscribers never touch.

2.1 Pub/sub vs the point-to-point queue

Here's the distinction interviewers probe most, so nail it now. A point-to-point queue (a "work queue") also decouples sender from receiver through a broker — but with competing consumers: a message goes to the queue, and exactly one worker pulls it off and processes it. Ten workers on the queue means the load is shared; each message is handled once.

• Queue = load balancing. One message, one consumer. Use it to distribute work — e.g. a pool of image-resize workers sharing a backlog.
• Pub/sub = broadcasting. One message, every subscriber. Use it to notify many interested parties of the same event.

💡 The rule of thumb: if you're asking "which worker should do this job?" you want a queue. If you're announcing "this thing happened, whoever cares should react" you want pub/sub. Same broker infrastructure, opposite delivery semantics. And as we'll see in Part 2, log-based systems like Apache Kafka cleverly give you both at once.

---

3. The two dials every subscription has

Two properties of a subscription decide its whole personality. Get these straight and pub/sub stops being mysterious.

3.1 Push vs pull delivery

• Push: the broker sends messages to the subscriber as they arrive (calls an endpoint, holds a socket open). Low latency, but the broker sets the pace — a slow subscriber can be overwhelmed unless there's flow control. This is the live radio broadcast: the signal arrives whether you're ready or not. Redis Pub/Sub, Amazon SNS (HTTP/Lambda targets), and MQTT push.
• Pull: the subscriber asks for messages when it's ready (polls, or long-polls). The consumer controls its own rate — great backpressure, slightly more latency. This is a podcast: same content, but you download episodes on your schedule. Kafka consumers pull; SQS is polled.

The trade is latency vs control. Push is fast but the consumer can drown; pull lets a slow consumer self-pace at the cost of a poll delay. Part 2 shows why this choice ripples all the way into the broker's data structures.

3.2 Durable vs ephemeral subscriptions

This is the one that bites people. Back to the radio: if your set is switched off during the 3 p.m. news, that broadcast is gone — the airwaves don't remember. That's an ephemeral subscription: you only receive messages published while you're connected. Disconnect, and anything sent in the gap is lost forever. Redis Pub/Sub is the canonical example — fire-and-forget, no storage.

A durable subscription is like a DVR that keeps recording your channel while you're on vacation. The broker remembers where you left off and holds messages for you until you come back and acknowledge them. Reconnect after an hour and you replay everything you missed, in order. Kafka consumer groups (via offsets) and durable topic subscriptions in classic brokers work this way.

⚠️ The single most common pub/sub production surprise: a team uses Redis Pub/Sub (ephemeral) for something that must never be lost — payment events, say — and discovers that every subscriber restart, deploy, or network blip silently drops messages. Ephemeral pub/sub is perfect for live dashboards and presence ("who's online"), where a missed update self-heals on the next one. It is wrong for anything you must not lose. Match durability to how much a lost message hurts. The Redis Pub/Sub topic goes deep on exactly this footgun.

---

4. How you actually use it

The API is deliberately tiny — that's the beauty. Here's the shape in three flavors, so you recognize it anywhere.

Publishing an event (broker-agnostic pseudocode):

// Publisher — announce, don't orchestrate
broker.publish(
  topic = "order.placed",
  message = { orderId: 913, userId: 27, total: 42.00 }
)
// ...and we're done. No idea who receives it.

Subscribing (a durable, filtered subscription):

// Subscriber — opt in by topic, react to each message
broker.subscribe(
  topic        = "order.placed",
  subscription = "email-service",   // named + durable → survives restarts
  filter       = "total > 0",       // content-based (see §5)
  onMessage    = (msg) => {
      sendConfirmationEmail(msg.userId, msg.orderId)
      msg.ack()                     // tell the broker "done, don't resend"
  }
)

Two details that matter even in this toy snippet. The subscription has a name (email-service) — that's what makes it durable; the broker tracks progress per named subscription, so Email and Analytics each get their own independent copy and their own place-marker. And the handler calls ack() — an explicit acknowledgment that says "I've safely processed this; you can advance my marker." No ack (a crash mid-processing) means the broker redelivers later. That ack is the seed of at-least-once delivery, which we'll unpack in Part 2.

---

5. Filtering — topic-based vs content-based

Subscribers rarely want everything. Two ways to narrow the firehose:

• Topic-based routing: you subscribe to a whole named channel. Fine-grained control comes from a topic hierarchy with wildcards — e.g. MQTT's sport/football/# or an AMQP topic exchange's sport.*.scores. You pick which stations to tune to. Cheap and fast: the broker routes by name alone.
• Content-based routing: you subscribe to a topic plus a predicate on the message's attributes — "only orders where total > 100 and region = EU." A smart car radio that only un-mutes for traffic alerts. Amazon SNS filter policies and JMS message selectors do this. More expressive, but the broker must evaluate the predicate on every message per subscriber, which costs CPU at high fan-out.

💡 The design tension: topic-based routing pushes the "what do I care about?" decision to publish time (you must name topics well up front), while content-based routing defers it to subscribe time (subscribers filter a broad stream). Staff-level move: use a sensibly granular topic hierarchy for the coarse cut, and content filters only for the last-mile predicate — don't make the broker evaluate expensive filters on a firehose you could have split into topics.

---

6. When to reach for it — and when not

Reach for pub/sub when:

• One event has many independent reactions — the checkout example. Fan-out to consumers who evolve independently.
• You want to decouple producers from consumers so either side can be added, removed, or scaled without touching the other. This is the backbone of event-driven microservices and CQRS.
• Real-time notifications / streaming — live scores, presence, price ticks, activity feeds.
• Buffering load spikes — the broker absorbs a burst so slow consumers aren't crushed (see the Circuit Breaker and async-messaging patterns).

Reach for something else when:

Situation	Better fit
Exactly one worker should handle each job	Point-to-point queue (work queue)
You need a synchronous reply (request/response)	Direct RPC / HTTP, or request-reply over the broker
One producer → one specific consumer, private callback	Webhooks
Strong multi-record transactional guarantees across services	Distributed transactions / a shared DB (see Distributed Transactions)
You just need to cache computed data for fast reads	A cache (see the caching chapter)

🚫 The classic misuse: reaching for pub/sub when you actually need a synchronous answer. Pub/sub is fire-and-forget by nature — "I announced it." If your caller must wait for a result before proceeding, bolting request/response onto pub/sub fights the pattern. Use it for notifications and events, not for "give me back an answer now."

---

Key takeaways

1. Pub/sub decouples by topic: a publisher sends one message to a named topic; the broker fans it out to every subscriber. Neither side knows the other — that's the whole value.
2. Fan-out vs work-sharing: pub/sub delivers each message to all subscribers (broadcast); a point-to-point queue delivers each message to one competing consumer (load balancing). Know which one the problem needs.
3. Two dials define a subscription — push vs pull (latency vs consumer control) and durable vs ephemeral (kept-and-replayed vs live-only). Ephemeral pub/sub like Redis Pub/Sub silently drops messages you weren't connected for — never use it for data you can't lose.
4. Filtering is topic-based (route by named channel, cheap) or content-based (predicate on attributes, expressive but CPU-heavy at fan-out). Use topics for the coarse cut, content filters for the last mile.
5. Reach for pub/sub when one event has many independent reactions and you want producer/consumer decoupling; reach elsewhere for one-worker-per-job (queue), synchronous replies (RPC), or a single private callback (Webhooks).

Next, Part 2 opens up the broker: the data structures that hold subscriptions, the publish (write) and deliver (read) paths, how copy-based brokers differ from log-based ones like Kafka, delivery guarantees (at-most / at-least / exactly-once), ordering, and the failure modes a Staff engineer must anticipate.