Section 8: Every Message Arrives Twice - Webhooks, Idempotency & Async Processing
Executive Summary: Sections 4 through 7 described each money movement as if every message arrived exactly once. None of them do. This section closes the three sources of duplicates: client retries, PSP webhook redelivery and internal consumer replays. Client Idempotency-Keys commit with the state they guard, in short transactions that never stay open across a PSP call, and outbound PSP keys are derived from our own operation ids. Webhooks are verified, stored and acknowledged within 200 ms, then treated as hints that trigger a fetch and a monotonic state transition. A transactional outbox feeds Kafka consumers that dedupe by event id. The result is exactly-once effects on top of at-least-once delivery.
The interviewer flips back through your earlier diagrams. "Every arrow on these slides is a network call. What happens when one of them fires twice?" For a payment system, that question decides whether Maya gets charged once.
Three Sources of Duplicates
No network delivers a message exactly once. When a sender can't tell "lost" from "slow", it retries, and the receiver sees the message again. So we don't try to prevent duplicates; we make the second arrival harmless.
| Source | Why it repeats | Defense | Dedupe key |
|---|---|---|---|
| Client retries | Maya's app loses the response in a tunnel and resends | Idempotency record committed with the state it guards (D10) | (caller, Idempotency-Key) |
| PSP webhook redelivery | The PSP saw no 2xx in time, or it re-sends on its own | Unique raw-event row, then a monotonic apply (D9) | PSP event_id |
| Internal consumer replays | Kafka is at-least-once; a consumer crashes before committing its offset | Processed-events table or versioned upserts (D8) | Outbox event_id |
Our own calls out to the PSP are the mirror image. There we're the client, and the PSP is the one deduping.
Key Insight: "Exactly once" is at-least-once delivery plus an idempotent apply. Each receiver must answer one question atomically with its side effect: "have I already done this?" If the answer is stored anywhere other than the transaction that does the work, a crash between the two writes reopens the gap.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson