Section 9: At Least Once, Exactly Once, and What Lies Between - Deep Dive: Consumer Groups & Exactly-Once

Executive Summary: Section 8 made an acknowledged write durable. This section covers the other half of delivery: who reads each partition, and how many times each record takes effect. A group coordinator hands each partition to exactly one member of a group and moves partitions when members come and go. Cooperative incremental rebalancing moves only the partitions that have to change hands. Committing offsets after processing gives at-least-once: a crash or a rebalance replays a little, never skips. The idempotent producer (producer id, epoch, per-partition sequence numbers) removes duplicates caused by retries. Transactions then make a read-process-write step atomic: frd-18's output to fraud-scores and its offset commit on orders succeed or fail together, and read_committed consumers stop at the last stable offset. That is exactly-once inside Kafka. Anything outside it needs an idempotent sink (Decision 8).

Interviewer: "Your consumers 'track their own offsets'. Who decides that frd-18 reads partition 37 and nobody else in the fraud group does?"

You: "The group coordinator. Membership first, then what a commit means, then the two mechanisms that turn 'at least once' into 'exactly once' for one specific pipeline."

Groups, Coordinators and Rebalances

Every group hashes to one partition of __consumer_offsets, and the broker leading that partition is the group's coordinator. Members send it a join, then a sync, then heartbeats. In the classic protocol one member, the group leader, runs the assignment strategy (range, round-robin, sticky, cooperative-sticky) and the coordinator hands each member its share. The fraud group has 32 consumers on 64 partitions, two each. frd-18 owns 36 and 37.

Now frd-18 stalls in a long GC pause. The whole JVM freezes, heartbeat thread included. If the pause ends inside the consumer session timeout (45 s by default in recent releases; it has changed over time), nothing happens. If it doesn't, the coordinator evicts frd-18 and rebalances. A separate limit, the max poll interval, catches the opposite case: heartbeats still flowing but the poll loop stuck.

The other 31 members keep consuming. Add static membership for rolling restarts. With a stable instance id, a frd-18 that restarts and rejoins inside the session timeout gets 36 and 37 back with no rebalance. The cost is that a member that really died holds its partitions for the full 45 s before they move. That's Decision 7. Rejected: eager rebalancing, which stops the world, and static manual assignment, which never heals. Flip trigger: very large groups with constant churn, where the newer broker-side consumer protocol (Section 12) wins.

Quick check before moving on: