Section 8: Spending an Impossible Allowance - Deep Dive: Living on 150 Requests per Second

Executive Summary: Section 7 ended on the uncomfortable truth: the decision costs 2 µs and the data costs everything, because the only tap filling the mirror runs at 150 requests a second. This section is the machinery of spending that allowance well. One scheduler owns the entire vendor budget (D7): a priority queue over next_check_due ranked by traffic rank × staleness, drained through a token bucket at 120 rps, deduped to one in-flight check per IP, with verdicts batched into the mirror. The budget splits 60/40 between first-look and re-validation — we run the 10M-checks-a-day arithmetic in both failure directions to show why. Vendor hygiene (circuit breaker, backoff, Retry-After) keeps the supply alive, and D1 means the vendor's 99.5% availability costs freshness, never availability. Then the Staff move: naming the constraint as commercial, not physical, and negotiating a bulk feed (D9) — while proving the design stands without one.

The interviewer picks up Section 7's closing line. "You said the decision is free and the data is expensive. You get 150 requests a second against 100 million unique IPs a day. Show me who decides which question is worth asking."

One Scheduler Owns the Budget

You: "One component, by decree (D7): the crawl scheduler is the only process in the company allowed to speak to SentinelDB. The budget is a shared, exhaustible resource; the moment two teams spend it independently, nobody can account for it and the vendor's rate limiter starts answering for us."

The scheduler's state is Section 5's verdicts table — specifically an index on next_check_due. Two streams feed it. Edge observation batches report never-seen IPs with traffic counts; those enter with no verdict and a due date of now. Existing verdicts re-enter as their TTL class expires. Both land in one priority queue, and the priority function is deliberately crude: traffic rank × staleness. An IP sending 400 requests a second with no verdict outranks everything — that is exactly how 203.0.113.42 jumped the queue at 09:16 and was denied fleet-wide by 09:17. An IP seen once, three days ago, may never reach the front at all, and that is a feature, not starvation.

Three mechanics keep the spend honest. A token bucket at 120 rps meters the drain — 80% of the 150 midpoint, keeping headroom for retries and bursts, and never exceeding the vendor's stated ceiling without agreement. The 120 assumes the vendor sustains at least 150; if they enforce the 100 floor of the band, the bucket drops to 80 and every cadence in this section stretches proportionally. Dedupe guarantees one in-flight check per IP, because thirty PoPs observing the same new scraper within a minute is the common case, and thirty checks for one answer is budget arson. And verdicts are batch-written to the mirror — the vendor path is precious; the database path shouldn't add per-row overhead to it.

Quick check before moving on: