Section 7: The Update That Can't Go Wrong - Deep Dive: The Write Path, Max Semantics & Rebuilds

Executive Summary: The interviewer hands you the wheel, and you drive straight to the write path, because that's where a leaderboard quietly corrupts itself. Four walls: an atomic Lua script that raises a best and reads the rank in one step; the max property, which, behind one board applier's verdict and ban filter, makes duplicates, retries and reordering harmless and lets Kafka be written first; a composite score that breaks ties by time and stays exact below 2^53; and a rebuild that restores a 150-million-member board in five minutes. The lesson: when your update is a max, most of the hard problems in distributed writes turn into no-ops.

Forty minutes in, the interviewer caps the marker. "You've got the boxes. Where do you want to go deeper?" The tempting answer is sharding, because it sounds senior, but this design doesn't have that problem yet. The write path is the problem it has today: 17,400 runs a second at peak, each touching two boards, sent by phones on flaky networks that retry, duplicate and deliver out of order.

You: "I'd like to walk the write path end to end: how one run becomes a rank, why retries can't corrupt it, how ties resolve, and how we get the board back if Redis disappears. One property carries all four."

Wall One: Update and Rank in One Atomic Step

On each board, the Score Service must raise Kai's best if run r-7f3a91's 1,284,500 beats it, then report where Kai now stands. As two round trips, another player's write can land in between, and the rank you report isn't your write's. So it's one script, loaded once with SCRIPT LOAD and called with EVALSHA:

-- KEYS[1] = board key, e.g. lb:weekly:2026-W44
-- ARGV[1] = composite score (an integer below 2^53, sent as a string)
-- ARGV[2] = member (playerId, e.g. 48213377)
local changed = redis.call('ZADD', KEYS[1], 'GT', 'CH', ARGV[1], ARGV[2])
local r = redis.call('ZREVRANK', KEYS[1], ARGV[2], 'WITHSCORE')  -- {rank0, storedScore}
local mine = (tonumber(r[2]) == tonumber(ARGV[1])) and 1 or 0     -- is this run the best?
return { changed, r[1] + 1, r[2], mine }                           -- 1-based rank

Each line earns its place:

• GT updates an existing member only when the new score is greater. New members are still added, but a best is never lowered: Kai's all-time 1,350,200 is safe from today's 1,284,500.
• CH makes ZADD count members added or updated, not just added. Combined with GT, it returns 1 for a new best or a first appearance and 0 otherwise. CH has existed since Redis 3.0.2; GT, and so the pair, needs 6.2.
• ZREVRANK ... WITHSCORE (Redis 7.2 or later) returns the 0-based rank and the stored score in one command. On older versions, add a ZSCORE, which is an O(1) hash lookup.
• and 1 or 0 returns an integer 0 or 1. A Lua false would come back as a null element, and a nil would truncate the array.

Redis runs a script atomically on its single main thread, so no other command interleaves, and the rank is the rank of this write. Each board is a standalone primary with two replicas, not Redis Cluster, and its :pending and :shadow side keys live beside it, so every script touches keys on one node. (Moved into Cluster, they'd need hash tags: lb:{weekly:2026-W44}:pending.) The two boards live on two primaries, so the Score Service fires both scripts in parallel and the submit pays one Redis round trip, not two. At 2 ops per board per run, that's 34,800 ops/s on each primary at peak and 70,000 at the design surge, against a 100,000 budget.

Kai's weekly board returns changed 1 and rank 88,416 (up from 412,907). The all-time board returns changed 0 and rank 1,042,311. Both ranks ride back in the submit response: read-your-writes for free, with no replica lag to explain. Every other read goes to replicas and can be up to 2 s stale.

Interviewer: "CH already tells you whether it was an improvement. Why compute mine?"

You: "CH says whether this call changed the set. It doesn't say whether this run is the best. A retry or a failover replay can apply Kai's run before this call does, and then CH returns 0 even though Kai just set a new best. Comparing the stored composite with the run's own answers the question the player is asking."

Quick check before moving on: