Gossip Protocol — Part 2: Internals & Failure Detection

maang.io System Design Series

What is this?

Back to our office rumor mill. In Part 1 we watched news spread — each person, every 30 seconds, swaps what they've heard with a random colleague. Now zoom in on what exactly gets swapped and how each person keeps their story straight when two colleagues tell them contradicting versions. A good gossip isn't just repeating noise: they keep a little mental ledger — "I heard X from Priya at 2:05, and a fresher version from Sam at 2:07, so Sam's wins." And crucially, they notice when someone has gone quiet: "I haven't heard a peep out of Dana in twenty minutes… is she off sick?" That last instinct — deciding when silence means dead — turns out to be the subtlest and most important part of the whole machine.

This chapter is the machinery behind those instincts: the exact data structure each node gossips, the handshake that reconciles two nodes' views in one round trip, and the failure detector that converts "haven't heard from you lately" into an actionable "you're dead." We'll use Cassandra's implementation as the worked reference because it's the most widely-studied, but the ideas are the same in DynamoDB, Consul/Serf, and Redis Cluster.

The one-line idea: each node keeps a versioned ledger of everything it believes about every other node, and a gossip round is just two nodes merging ledgers — higher version wins, per item. Layered on top, a failure detector watches the heartbeat versions climb and raises a suspicion level when they stop, so "dead" is a tunable judgment, not a fragile timeout.