Section 6: Thirty Brokers, Six Thousand Logs - High-Level Architecture
Executive Summary: The architecture has three planes. The data plane is 30 brokers, 10 per zone in 3 zones, hosting 6,000 partition logs three times over. Each broker leads about 200 and follows about 400. The control plane is 3 KRaft controllers, one per zone, which decide who leads what. The coordination plane for consumers is spread across the brokers: each group's coordinator is the broker leading the __consumer_offsets partition its group id hashes to. Clients cache which broker leads each partition and talk straight to it. Ari's batch goes to partition 37's leader, both followers pull it, the high watermark moves, and only then does the producer get its offset. Consumers fetch from their own position up to that high watermark, zero-copy from the page cache.
Interviewer: "You've got a data model and an API. Put the boxes on the board."
You: "Three planes, kept apart because they fail differently: brokers carry bytes, controllers carry metadata, coordinators carry group membership and offsets. Then one order through the write path and back out."
The Components
| Component | Count (our assumption) | Owns | Never does |
|---|---|---|---|
| Producer client (Ari's checkout service) | every emitting service | Partitioning by key, batching, retries, a metadata cache | Ask anyone but the leader to write |
| Broker | 30 (10 per zone) | ~600 replicas: ~200 as leader, ~400 as follower; a page cache of ≈ 100 GB | Track a consumer's position per message |
| KRaft controller | 3 (1 per zone) | Cluster metadata: topics, partition leaders, ISR, broker liveness | Touch a single record |
| Group coordinator | one broker per group | Membership, assignment generation, committed offsets | Push records to consumers |
Consumer client (ful-7, frd-18, …) |
per group | Its own fetch position, processing, offset commits | Read past the high watermark |
Notice what is missing: a load balancer. Only partition 37's leader can serve partition 37, so brokers are not interchangeable and a balancer would just add a hop. Instead every broker holds the full cluster metadata. A client asks any broker once, caches the leader map, and talks straight to leaders. If a leader moves, the old broker answers "not leader" and the client refreshes and retries. The flip trigger for a proxy is clients that can't speak the binary protocol, and that proxy sits at the edge, off this path.
Key Insight: The broker is deliberately simple. The 2011 paper's broker kept no per-consumer state; a consumer's position is one offset per partition that the consumer owns. So a broker serves its ~200 led partitions to every group without its work growing per reader (Decision 1). Fan-out costs network bytes, not broker bookkeeping.
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson