Section 11: A Megaphone Pointed at Millions - Trust & Safety: Moderation, Spam & Abuse
Executive Summary: Section 10 handled failures from machines; this section handles failures from people. We map the threats, then layer the defense: rate limits, a 30 ms synchronous Safety Filter, Async Safety retractions through the existing pipeline, broadcaster and viewer controls, and raid detection. The organizing idea is reach: how strict a check must be depends on how many screens a comment can hit before anyone can pull it back.
The interviewer turns the page: "You've built a pipe that takes one comment to millions of screens in about two seconds. Now hand it to the internet. What goes wrong?"
You: "It becomes the best megaphone a spammer ever had. So moderation isn't one box; it's a series of filters, each priced by its latency, and the path with the most reach gets the strictest one."
The Threat Model
| Threat | Primary defense | Added latency | Failure mode if it slips through |
|---|---|---|---|
| Spam bots | Rate limits per user, IP and device; account-age gating | Microseconds, inline | Bots rotate accounts; IP and device limits carry the load |
| Copypasta floods | Normalized-text hash per video over 60 s | Inline, inside the 30 ms | Character tweaks dodge the hash; Async Safety clusters near-duplicates |
| Raids on small streams | Raid detection, then auto slow mode or followers-only | Seconds to trigger | The first burst lands before the trigger fires |
| Harassment of the broadcaster | Blocked words, blocked users, moderators (Settings Service) | Inline, once pushed | A fresh account repeats it; account-age gating slows that |
| Slurs and illegal content | Sync classifier, then heavier async models | ≤30 ms, then ~2 s | Visible on a small stream until retraction (p99 < 5 s) |
| Impersonation ("Aria Vance" lookalike) | Role badges taken from the token, never from the display name | None | Lookalike names pass; Async Safety flags names resembling the broadcaster's |
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson