Section 9: Growth Has a Speed Limit - Deep Dive: Scaling, Concurrency & Async Invocation

Executive Summary: Section 8 made one sandbox appear in ~110 ms. This section decides how many may appear, and for whom. Each function grows by at most +1,000 concurrency every 10 s. Concurrency is counted where leases are granted, against an account limit (default 1,000) split into reserved carve-outs and a shared pool, and enforced across 64 placement partitions with budget slices. Async invocation is a durable per-function queue with at-least-once delivery and a stable eventId. Function errors get 2 retries, throttles are retried up to the 6 h max age, and then the event goes to the on-failure destination. For event-source mappings, queues scale out and streams keep order per shard.

The interviewer moves to a fresh patch of whiteboard: "One function at a time works. Now it's 09:00. Tidewater's flash sale fires, a thousand other functions in the cell are busy, and somebody's photo backlog is draining. Who gets the sandboxes?"

The Speed Limit on Growth

You: "Every function can add 1,000 concurrent executions every 10 seconds, about 100 new sandboxes a second. It doesn't matter how much traffic shows up. Tidewater's checkout-api runs at 300 concurrent before the sale and needs 3,000 at 20,000 req/s. Starting from 300, it reaches 1,300 after the first step, 2,300 after the second, and enough after the third. That's about 27–30 seconds. Sync requests that arrive faster than the ramp get a 429 with Retry-After."

"Why throttle growth at all? You told me the cell creates sandboxes easily."

"It does, about 3.3 a second per host (Section 3). The limit protects three other things:

1. The cell's headroom. Without a ramp, one runaway function (a retry storm, a bot, a recursion bug) can drain the cell's pre-booted microVMs in a second, and every other tenant's next cold start fails.
2. The placement partition. It carries ~16K acquires/s at peak. Tens of thousands of creations in one tick would turn it into a scheduler queue.
3. The customer's own dependencies. A database sized for 300 sandboxes' connection pools isn't ready for 3,000 at once. The ramp gives it, and the customer's alarms, time to respond."

The rejected alternative is unbounded instant scale-out, which turns every bug into a cell-wide event. For a spike you know is coming, the answer is to pay for capacity in advance:

Provisioned sandboxes skip both the ramp and Tidewater's ~200 ms of init. The account limit still applies, which is why Tidewater raised it from 1,000 to 5,000.