Webhooks — Part 2: Internals & Reliable Delivery

maang.io System Design Series

What is this?

Back to the restaurant buzzer. In Part 1 the buzzer was magic — the kitchen presses a button, your buzzer lights up, done. But now imagine you're the restaurant, handing out ten thousand buzzers at once, and some of them are out of range, some have dead batteries, some diners wandered to the parking lot. How do you make sure everyone eventually gets buzzed, exactly the news they asked for, without a malicious diner spoofing a buzz to steal someone else's order? That's the engineering. A webhook is trivial to describe and genuinely hard to deliver reliably at scale.

The core difficulty is that the network between producer and consumer is hostile and unreliable. The consumer's server might be down, slow, deploying, rate-limited, or briefly unreachable. HTTP POSTs time out and get lost. So a real webhook system is not "make an HTTP call" — it's a small distributed, at-least-once delivery system with a durable queue, retry logic, and cryptographic authentication bolted on. This chapter builds that system.

The one-line idea: Reliable webhook delivery is a durable queue + worker pipeline: the producer commits every event to a queue first, workers attempt HTTP delivery, and failed attempts are retried with exponential backoff — which makes delivery at-least-once, which is exactly why the consumer must be idempotent and every payload must be signed.

We'll build it in the order the data actually flows: how a producer emits and delivers, why retries make duplicates inevitable, how the consumer de-duplicates, and how both sides authenticate each other.