Section 4: Start, Write, Finish - API Design (with Pushback)
Executive Summary: Section 3 showed writes arriving as bursty bulk runs of up to 50,000 annotations a second, and reads at 6,500 QPS that must not notice. This section writes the two contracts that split those paths. Producers write through an operation: start it, send batches of up to 500, finish it with a declared count, and the run becomes visible all at once. Consumers get versioned CRUD, entity reads with a cursor, and a JSON search DSL with temporal operators. We walk char-detect v4's run on Night Heist and Priya's first query across the wire, then defend each shape under pushback.
The interviewer says: "Show me the API." Before you write a path, name the rule that shapes it. Section 1 settled that a re-run replaces the old results for a title atomically, never a mix. So a producer can't just write annotations. It has to write a run, and the API has to know where a run starts and ends.
The Producer Surface
| Purpose | Endpoint | Returns |
|---|---|---|
| Register a schema version | POST /v1/schemas |
201, or 400 if a minor version isn't additive |
| Start a run | POST /v1/operations |
201 {op_id, lease_expires_at}, 409 if one is in progress |
| Write a batch | POST /v1/operations/{op_id}/annotations:batch |
202, ≤ 500 items, ≤ 1 MB |
| Keep the lease alive | POST /v1/operations/{op_id}:heartbeat |
200 {lease_expires_at} |
| Finish | POST /v1/operations/{op_id}:finish {declared_count} |
202 {state: "SEALED"} |
| Cancel | POST /v1/operations/{op_id}:cancel |
200 {state: "CANCELLED"} |
| Check status | GET /v1/operations/{op_id} |
200 {state, declared_count, indexed_count} |
| Roll back a scope | POST /v1/scopes/{entity_id}/{schema}/{producer}:rollback |
200 {current_op} |
The scope is the triple (entity, schema, producer). One operation per scope can be open at a time. Here is char-detect v4 starting its run on Night Heist:
POST /v1/operations
{ "entity_id": "title:70451122", "schema": "character-appearance@2.3",
"producer": "char-detect", "producer_version": "v4" }
HTTP/1.1 201 Created
{ "op_id": "op_7Hq2", "lease_expires_at": "2026-10-31T14:30:00Z" }
The lease is 30 minutes. A run that stops heartbeating for that long is marked ABANDONED and never becomes visible. One item in its first batch is the Lena Cruz segment:
{ "producer_key": "shot-0412/face-1",
"temporal": { "start_ms": 2470000, "end_ms": 2512000 },
"spatial": { "frame_ms": 2470000, "x": 0.62, "y": 0.18, "w": 0.14, "h": 0.22 },
"data": { "character": "Lena Cruz", "confidence": 0.97 } }
There's no annotation_id in the item. The service derives it as hash(op_id, producer_key), so a retried batch overwrites the same 500 rows instead of adding 500 duplicates, and the declared count stays exact. A batch of 500 at 1.3 KB each is 650 KB, inside the 1 MB cap. At the 50,000/s peak that's only 100 batch requests a second.
The whole lifecycle, as the producer sees it:
The run becomes visible within 2 minutes p99 of the finish call. Section 8 builds what happens between SEALED and COMMITTED.
| Code | Meaning | What the producer does |
|---|---|---|
400 |
An item violates the schema; the body lists item index and field path | Fix the item; the rest of the batch was rejected too |
409 |
Another operation is open on this scope | Wait, or cancel the stuck one |
410 |
Operation not writable (SEALED, ABANDONED, CANCELLED) | Start a new operation |
412 |
Stale If-Match version (human edits) |
Re-read, re-apply |
429 |
Producer over its quota, with Retry-After |
Back off |
429 is a feature. Each producer has a token bucket, so one team's catalog-wide run can't starve another team's or slow the read path. Producers get a clear "slow down", not timeouts.
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson