Section 8: The Long Tail Has an Address - Deep Dive: Every Product's Rank in Every Category

Executive Summary: The Top-10 needed ten winners. The rank badge needs everyone's position: "#1,056 overall" only means something if you know the 1,055 products above it, in every one of 20,000 categories. This section builds the rank path in five dives: a decayed score maintained with one multiply per hour, the expansion of 40M products into 240M (product, category) rows, an hourly Spark job that assigns unique ranks even when one category holds 40M rows, a versioned snapshot flipped atomically, and the live sorted-set design we turn down on semantics, not memory.

Section 7 closed the speed path. The interviewer turns the page.

Interviewer: "Fine, the Top-10 is exact. Now the helmet. You told me it's #1,056 overall, #13 in Sports Equipment, #24 in Safety. Where does each of those numbers come from?"

You: "From the only part of this system that needs the whole order rather than its head. 40M ranked products, 20,000 categories, 240M rank rows, recomputed every hour. Every rank must be unique, consistent within a page, stable and fair, because sellers fight over these badges. I'll follow the data: score, memberships, sort, serve."

Dive 1: The Score - Decay, Not Windows

A rank is only as defensible as the number being sorted. We use exponentially decayed units with a 24-hour half-life, and the whole update is one line:

S(h) = S(h-1) × 0.97153 + netUnits(h)        # 0.97153 = 2^(-1/24)

Each hour every score loses 2.85% of its weight: after 24 hours a sale counts half, after 7 days 1/128 ≈ 0.0078, and after 30 days about 9.3 × 10⁻¹⁰, which is why the ranked set is simply "at least one unit in the trailing 30 days". Anything older has decayed to nothing anyway.

Take the helmet. At 20:00 its score is 5,710. Between 20:00 and 21:00 it sells 240 units, Priya's among them. The 21:00 run computes 5,710 × 0.97153 = 5,547.4, plus 240, giving 5,787.4. Decay took 162.6 and the hour added 240, a net gain of about 77 points while most neighbors merely replaced what decay took, so the helmet goes from #1,056 → #1,041 overall and #13 → #12 in Sports Equipment.

Each obvious alternative fails a merchandiser in a specific way:

Decay is also cheap: a trailing window must subtract the hour that's leaving, so it needs 30 days of hourly history, while the recurrence needs only last hour's score table (40M × 28 B ≈ 1.1 GB) and the newest hour of hourly_units. Those are counted units: the Lake Sink writes them from the Window Aggregator's counted-units side output, already deduplicated and capped per buyer, so the rank job inherits both rules instead of implementing them a second time.

One trap: a cancellation can arrive hours after its order was already folded in. It lands in the hour it arrives, tagged with its originalHour, so no closed partition is ever rewritten. The job subtracts it at its decayed weight, cancelledUnits × 0.97153^k, where k is the number of hourly folds since the run that first added those units (a 20:xx sale is folded at 21:00, so a cancellation applied at 23:00 has k = 2). The score then comes out exactly as if the order had never happened, which matches Section 7's rule of retracting from the original minute.

Interviewer: "Multiplying 40M scores every hour is fine in batch. Could you keep the same score in a stream?"

You: "Yes, with forward decay (Cormode, Shkapenyuk, Srivastava and Xu, 2009). Fix a landmark time L. A unit sold at time t adds e^(λ(t − L)) to the stored value, with λ = ln 2 / 24 ≈ 0.0289 per hour. Stored values only grow. The true decayed score at time T is the stored value × e^(−λ(T − L)), and that factor is the same for every product, so ordering by the stored value is ordering by the decayed score. Nothing is ever rescored. The catch is that stored values double every 24 hours, so a double would overflow at 2^1024, roughly 1,024 days out. Long before then you re-landmark: move L forward by, say, 7 days and multiply every stored value by 1/128. That's one uniform rescale, so it preserves order. That's the trick Dive 5's flip depends on."

Quick check before moving on: