Section 7: Why Spinning Disks Win Here - Deep Dive: The Log — Sequential I/O, Page Cache & Zero-Copy

Executive Summary: Section 6 put 30 brokers on hard disks carrying 1.5 GB/s of produce and 7.5 GB/s of consumer reads at peak. Three techniques make that work. Sequential appends turn 150 MB/s of writes per broker into about 12.5 MB/s of streaming writes per disk. The OS page cache, not an in-process cache, holds about 11 minutes of tail in roughly 100 GB of RAM, so tail readers like fulfillment never touch a platter. Zero-copy sendfile sends cached bytes to the socket without passing them through the JVM. Then the case that breaks it: a three-day analytics replay that evicts the hot tail. Last, the flush policy that skips per-message fsync, safe only because of Decision 3's replication.

Interviewer: "You've put a read-heavy system on spinning disks. Convince me."

You: "Disks are slow at seeking and fast at streaming. This design never asks a disk to seek on the write path, and it keeps most reads off the disk entirely."

Sequential Appends: Never Ask the Disk to Seek

Random I/O pays a head movement per operation; streaming runs many times faster. Per-message broker state (ack flags, redelivery counts, delete-on-consume) means scattered updates. A log means one append at the end of one file.

Section 3's numbers: the cluster writes 4.5 GB/s at peak, so each of the 30 brokers writes 150 MB/s, spread over 12 disks in JBOD, or about 12.5 MB/s per disk. That's why Section 3 found that storage capacity binds, not disk throughput.

Interviewer: "600 replicas per broker is about fifty active segments per disk. Isn't that random I/O again?"

You: "Only if each message were its own write. The OS absorbs writes in the page cache and flushes dirty pages in large runs, each contiguous within one file, and Decision 9's batching makes the unit tens of kilobytes, not 1 KB. At 12.5 MB/s, fifty streams of large runs is close enough to streaming."

With linger at 5 ms, Ari's producer gathers every pending record for partition 37, order-58213 included, into one batch, compresses it once with lz4 or zstd, and sends one request. The broker assigns offsets (order-58213 lands at 1,284,553,901) and appends the batch as it arrived. In recent message formats it doesn't recompress a batch whose codec matches the topic's (a format conversion can still force it), and the consumer decompresses. Compression is end to end, so the 1 KB as stored is 1 KB on the wire, on disk and in cache.