Section 4: Offsets, Not Acks - API Design (with Pushback)
Executive Summary: The interface is small. A producer calls send(topic, key, value, headers) and gets back a partition and an offset. A consumer subscribes as part of a group, or assigns itself partitions, then calls poll() to get record batches from its current position, commit() to save that position, and seek() to move it. Ari's order-58213 comes back as partition 37, offset 1,284,553,901. The interesting part is what the API leaves out: no push, no per-message ack, no free choice of partition, no deleting one message. Each gap follows from one idea: a consumer's progress is one number per partition, and the consumer owns it.
Interviewer: "Section 3 says storage binds and reads are the biggest number. Show me the interface. What does Ari's checkout service actually call?"
You: "Three surfaces: producer, consumer, admin. I'll go through each, then defend the features I left out, because those are where you'll push."
The Producer: Send Returns an Address
send(topic="orders", key="order-58213", value=<order placed>, headers={trace-id})
-> (partition=37, offset=1,284,553,901)
The client hashes the key, murmur2(key) mod 64, which gives partition 37. It looks up the leader for that partition in its cached metadata (broker 12) and adds the record to a per-partition batch. The batch goes out after 5 ms of linger or once it fills to tens of KB, compressed with lz4 or zstd (Decision 9). Three knobs carry most of the weight:
| Knob | Our setting | What it buys | Flip trigger |
|---|---|---|---|
acks |
all (Decision 3) |
The reply comes only after every in-sync replica has the batch, with at least min.insync.replicas 2 of them | Lossy telemetry topics opt into acks=1 |
| idempotence | on for every acks=all producer (Decision 8) |
Producer id, epoch and per-partition sequence numbers let the broker drop a retried duplicate | None for acks=all topics. It requires acks=all, so the acks=1 telemetry topics (Decision 3's flip) give it up and may duplicate on retry |
| linger / batch size | 5 ms, tens of KB | One request carries hundreds of records instead of one | Sub-millisecond needs → linger 0, smaller batches |
That budget gives 8–12 ms typical and p99 ≤ 20 ms for an acks=all produce, which matches the requirement from Section 2. The call is asynchronous: send returns a future, and Ari's service keeps handling checkouts while the batch is in flight.
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson