Section 10: Now Try to Break It - Interviewer Deep-Dive Challenges

Executive Summary: The design is complete, and the interviewer now tries to break it. Eight challenges arrive in quick succession: a function hot enough to melt its placement partition, a function that invokes itself forever, a placement owner dying during Tidewater's 09:00 burst, a tenant mining crypto, patching hosts that run 15-minute functions, a customer database that takes only 100 connections, a card charged by a function that then timed out, and a 10 GB ML image. Each answer has the same shape: the mechanism, the number that sizes it, the trade-off it accepts, and a senior line that states the principle. Almost every answer reuses something Sections 6 to 9 already built. That reuse is the signal.

The interviewer leans back. "I believe the happy path. Let's see what happens when things go wrong."

You: "Good. For each one I'll give the mechanism, the number, and what it costs us."

Challenge 1: The Hot Function

"Your hottest function does 50K invokes a second. All of them hash to one placement partition. It melts."

You: "It would. A partition carries about 16K acquires a second at peak (~33K ops/s with releases) (N26). One function at 50K/s (N23) would put three times an average partition's entire load on one owner. The answer is to split that function's pool, which is the flip trigger already written into D3. Past a threshold of its partition's capacity, the function version is re-keyed as hash(function_version, i) for i in 0..k-1, and its sandboxes are spread across those sub-partitions. Choose k so each share stays well under 16K/s. That's at least 4 for 50K/s, and I'd use 8 for headroom. Front-ends pick a sub-partition at random. If it has no idle sandbox, they try a second before paying for a cold start."

The concurrency limit still holds: each sub-partition draws an allowance of the function's and the account's concurrency from the coordinator, the same budget-leasing scheme Section 9 uses across partitions, approximate within a few percent and never unbounded.

The senior line: "Partition by key, and keep a way to split any single key. One hot key is the one load pattern that a hash scheme alone can't spread."

Quick check before moving on: