Section 8: When Someone Else's Server Goes Dark - Deep Dive: Delivering to Endpoints You Don't Control
Executive Summary: Every webhook we call belongs to someone else, and some will hang. Four mechanisms contain that: a 5 s timeout with strict failure classification, Delivery Queues partitioned by endpointKey so one worker owns each endpoint's 1,000 in-flight cap, a durable Retry Scheduler with jittered backoff, and a per-endpoint breaker plus Parking Lot whose tasks spend no retries and return on a ramp. ShopFast's 40-minute partner outage parks 1.2M tasks, drains in about 13 minutes, sends nothing to the DLQ, and the fraud webhook never notices.
The interviewer leans in. "ShopFast's analytics partner just stopped answering. What happens?"
Healthy HTTP is 180,000/s at about 100 ms: 18,000 in flight fleet-wide. One endpoint carrying 18,000/s that times out at 5 s holds 90,000 in flight alone, 5× the healthy fleet. In a shared pool it takes the slots the fraud webhook needs. That's head-of-line blocking: a noisy neighbor makes someone else's deliveries late. Section 2's NFR-Isolation forbids it.
Mechanism 1: Timeouts and Failure Classification
Every attempt runs on async I/O with a 5 s timeout, so 1,000 hanging sockets cost memory, not threads. Each outcome lands in one bucket:
| Outcome | Examples | Action | Spends a retry? |
|---|---|---|---|
| Success | 2xx within 5 s | Ack, done | n/a |
| Retryable | timeout, connection error, 5xx, 408, 429 (honor Retry-After) |
Retry Scheduler | Yes |
| Permanent | any other 4xx (400, 403, 404, 410) | Subscription's DLQ now (or drop and count) | No |
| Dead token | mobile push "token unregistered" | Disable the subscription | No |
| Breaker open | endpoint judged down | Parking Lot | No |
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson