Apache Flink — Part 2: Internals & Exactly-Once

maang.io System Design Series

What is this?

Back to our sorting office. In Part 1 we watched clerks handle letters off the belt, reason about postmarks, and seal date-labelled bins. Now we go behind the scenes: who assigns clerks to sections of the belt (the runtime), where a clerk keeps the running ledger when the pigeonholes overflow (state backends), and — the crown jewel — how a supervisor photographs the entire floor at one consistent instant while the belt keeps moving, so a blackout never loses or double-counts a letter (checkpoints via barrier snapshotting).

That last trick is what earns Flink its reputation. Getting a globally consistent snapshot of a distributed system without stopping it sounds impossible. Flink does it with an elegant algorithm — asynchronous barrier snapshotting — and if you understand nothing else from this chapter, understand that. It's the difference between "exactly-once" being a marketing slogan and an actual guarantee.

The one-line idea: Flink's runtime turns your dataflow into parallel subtasks, keeps per-key state locally (heap or RocksDB), and periodically floats checkpoint barriers through the stream. When every operator has snapshotted its state at the same logical barrier, Flink has a consistent global snapshot it can restart from — so on failure it rewinds the sources and restores state, reprocessing nothing more and nothing less than it must. That's exactly-once.