Section 8: A Millisecond or a Hundred - Deep Dive: Cold Starts & Sandbox Placement

Executive Summary: Every invoke asks two questions: is there an idle sandbox for this exact function version, and if not, where do we build one and how fast? A placement service split into 64 in-memory partitions answers the first in about 1 ms with a lease, and survives its own crashes by rebuilding from worker reports. The second is answered by optimistic host selection and a cold path done mostly ahead of time: a pre-booted microVM pool, tiered content-addressed code caches, lazy image loading and opt-in snapshot restore, for ~110 ms of platform overhead instead of ~230 ms. An adaptive 5–15 minute keep-alive decides how much of the 430 TB of warm-idle memory we pay for. The lesson to keep: a cold start is a short list of stages, and each one should be finished before the request arrives or not be needed at all.

The interviewer points at the placement box from Section 6. "You said the lease is the only coordination on the hot path. Convince me it can run a million times a second, and then tell me what happens when it comes back empty."

You: "Every invoke asks two questions. Is there an idle sandbox for this exact version? Section 7's packing rule makes a v41 sandbox useless to v42. And if not, which host builds one, and how much of that work was done before anyone asked? The first is a data-structure problem. The second is a latency budget."

Question One: Is There a Warm Sandbox?

Placement is 64 partitions region-wide, 10–11 per cell, and a version hashes to one of its home cell's partitions (D3). For each version it owns, a partition keeps two structures in memory: a stack of idle (host, sandbox_id, az) entries and a table of outstanding leases. That's the whole of Section 5's soft sandbox record, **200 B × 1.26M ≈ 250 MB** across the region (N26).

• Acquire. Pop an idle entry, preferring the front-end's AZ and then the most recently used sandbox, whose caches are warmest. Check function and account concurrency in the same step (D8, Section 9's subject). Record a lease expiring at the function timeout plus a grace period. Return (lease_id, host, sandbox_id).
• Release. Push the entry back on top. LIFO matters: recently used sandboxes stay busy, the bottom of the stack idles long enough to be evicted, and the pool shrinks on its own after a spike.

At peak that's 1.04M acquires plus ~1.04M releases a second, **33K ops/s per partition**, each a hash lookup and a pointer move. At ~1 ms per round trip it fits the 10 ms p99 warm budget easily.

Interviewer: "Two front-ends ask for checkout-api v42 at the same instant, and there's one idle sandbox. Who gets it?"

You: "Exactly one, because there's one writer. Only the partition owner mutates v42's idle stack, so the pop is serialized in memory with no distributed lock, and the loser takes the cold path. The second defence is the worker. The front-end sends the lease_id with the payload, and the host agent accepts it only if that sandbox is Idle on the host, whatever placement believes (D2). Placement can be wrong about who is idle. The worker can't be wrong about who is busy."

Quick check before moving on: