Apache Flink — Part 1: Foundations & Core Concepts

maang.io System Design Series

---

What is this?

Picture a postal sorting office that never closes. A conveyor belt runs down the middle of the room and letters pour onto it around the clock — there is no "end of day" when the pile is finally complete. A naïve office would wait until midnight, dump the whole day's mail into one giant heap, and only then start sorting. Every letter would sit for hours. That's batch processing.

A good sorting office doesn't do that. Clerks stand along the belt and handle each letter the instant it arrives — read the postmark, drop it in the right pigeonhole, update the running tally for that recipient. Nothing waits for a heap to form. The office also copes with reality: letters arrive out of order (a letter postmarked Monday can turn up Wednesday), so the office reasons about the date on the stamp, not the moment the letter hit the belt. And every so often a supervisor walks the floor taking a coordinated photograph of every pigeonhole, so that if the building loses power, they can restore the exact state and carry on without losing or double-counting a single letter.

That sorting office is Apache Flink. Flink is a distributed engine for processing unbounded streams of data — events that arrive continuously and forever — record by record, with low latency, correct handling of out-of-order data, durable state, and exactly-once guarantees. We'll return to this sorting office in every part of this topic; it maps onto every hard idea in Flink.

The one-line idea: Flink is a true stream processor — it acts on each event the moment it arrives (not in micro-batches), reasons about when events actually happened (event time + watermarks), remembers things across events (managed keyed state), and can snapshot that state consistently so it recovers from failure without losing or double-counting anything (exactly-once).

---

1. Why Flink exists — what was painful before

Ten years ago you had a bad choice to make between two worlds:

• Batch engines (Hadoop MapReduce, early Spark): great at chewing through a bounded pile of data, but they process it after the fact. You'd land a day of logs in HDFS and run a job at 2 a.m. Correct and high-throughput, but the answer is always hours stale. Our sorting office waiting for midnight.
• Early stream processors (Apache Storm): fast and record-at-a-time, so latency was low — but they were at-least-once by default (an event could be processed twice on failure, corrupting counts), had no built-in managed state (you bolted on an external database and prayed), and reasoned only in processing time (the moment a record was handled), so out-of-order data quietly produced wrong answers. (There's a sibling Apache Storm topic in this tier — it's the direct predecessor and a favourite interview contrast.)

Then Spark added Spark Streaming, but its trick was micro-batching: chop the stream into tiny batches (say every 0.5–1 second) and run a mini batch job on each. Clever and high-throughput, but it bakes in a latency floor (you wait for the batch boundary) and treats time as "which batch did this land in," which is again processing time, not event time.

Flink's bet was that you shouldn't have to choose. It delivers all of these at once, which none of the predecessors did:

1. Record-at-a-time, not micro-batch — genuinely low latency (milliseconds), not "wait for the next batch."
2. Event-time processing with watermarks — correct answers even when events arrive late or out of order.
3. First-class managed state — Flink stores and checkpoints your state for you, so a "count per user" survives crashes and rescaling.
4. Exactly-once state semantics — via consistent distributed snapshots (Part 2), so a failure never double-counts.
5. Batch as a special case of streaming — a bounded dataset is just a stream that ends, so one engine and one API does both.

💡 The clean way to hold it in your head: Storm gave you low latency without correctness; Spark Streaming gave you correctness without true low latency; Flink gives you both — record-at-a-time and exactly-once and event time — which is exactly why it became the default for serious stateful stream processing.

---

2. The core vocabulary (your mental model)

A handful of concepts unlock everything. Learn these names now; Part 2 opens each one up.

2.1 Streams and the dataflow graph

Everything in Flink is a DataStream — a possibly-infinite sequence of records. Your program is a dataflow graph: a source (where events come in — usually Apache Kafka; see that sibling topic), a chain of operators (transformations: map, filter, keyBy, window, aggregate), and a sink (where results go — a database, another Kafka topic, a data lake).

Flink runs each operator in parallel — many copies (subtasks), each handling a slice of the data. That's how one logical keyBy(userId).count() becomes hundreds of parallel counters across a cluster. The belt in our sorting office isn't one clerk; it's a whole floor of them, each owning a range of pigeonholes.

2.2 Event time vs processing time — the single most important distinction

Two clocks exist, and confusing them is the #1 source of wrong stream-processing answers:

• Event time — the timestamp embedded in the record: when the thing actually happened (the postmark on the letter). A click at 12:00:00 carries 12:00:00 even if it reaches Flink at 12:00:07 because the user's phone was in a tunnel.
• Processing time — the wall-clock time on the Flink machine when the record is handled (when the letter hit the belt). Simple and fast, but non-deterministic: replay the same data tomorrow and you get different window boundaries.

Real events arrive late and out of order — network delays, mobile buffering, retries. If you bucket "clicks per minute" by processing time, a click that happened at 11:59:58 but arrives at 12:00:03 lands in the wrong minute. Event time fixes this by bucketing on the postmark. But that raises a question: if you're bucketing "the 12:00 minute" by event time, how do you ever know you've seen all the 12:00 events so you can close the bucket? You can't wait forever.

2.3 Watermarks — how Flink knows time has moved on

A watermark is Flink's answer to that question. It's a special marker that flows inside the stream alongside the records, carrying a timestamp T and asserting: "I believe I've now seen all events with event time ≤ T; anything older is late."

In the sorting office, the supervisor periodically announces: "We have now received every letter postmarked before 3 p.m." The moment that announcement passes, clerks know it's safe to seal every bin dated before 3 p.m. and post the totals. That announcement is a watermark.

You configure how much lateness to tolerate — a bounded out-of-orderness of, say, 5 seconds means "emit a watermark for T once I've seen an event timestamped T + 5s." Bigger bound = more tolerance for stragglers but higher latency; smaller bound = faster results but more events declared "late." That knob is one of the central trade-offs of the whole system, and we tune it in Part 2.

2.4 Windows — turning an infinite stream into finite answers

You can't "count all clicks" on an infinite stream — the answer never finishes. So you slice the stream into windows:

• Tumbling — fixed size, non-overlapping. "Count per 1-minute window." Each event belongs to exactly one window. (Bins labelled by the hour.)
• Sliding — fixed size, overlapping, advancing by a smaller step. "Count over the last 5 minutes, updated every 1 minute." Each event belongs to several windows. (A 5-hour rolling view refreshed hourly.)
• Session — dynamic, gap-based. A window stays open while events keep coming and closes after a gap of inactivity (say 30 minutes of silence). Perfect for "a user's browsing session," whose length you can't know in advance.

A window fires (computes and emits its result) when the watermark passes the window's end — i.e. when Flink believes all the events for that window have arrived.

2.5 State — remembering across events

A pure filter is stateless — each event decides its own fate. But most useful stream logic remembers: running counts, the previous reading per sensor, the set of items in a cart, a fraud-detection sliding history. Flink gives you managed keyed state — after a keyBy(userId), each parallel subtask keeps a private slot of state per key (a pigeonhole per recipient), and Flink handles storing it, checkpointing it, and redistributing it when you rescale. You just call state.value() / state.update(). Part 2 is largely about how Flink stores that state (heap vs RocksDB) and snapshots it consistently.

---

3. How you actually use it — a short concrete example

Here's the classic "count events per key in event-time windows" in the DataStream API (Java-flavoured, trimmed to the essentials):

// A stream of ad clicks, keyed by campaign, counted per 1-minute event-time window.
DataStream<Click> clicks = env
    .fromSource(kafkaSource, WatermarkStrategy
        .<Click>forBoundedOutOfOrderness(Duration.ofSeconds(5))  // tolerate 5s of lateness
        .withTimestampAssigner((click, ts) -> click.eventTimeMillis), // event time = the postmark
        "clicks");

clicks
    .keyBy(click -> click.campaignId)                 // one pigeonhole per campaign
    .window(TumblingEventTimeWindows.of(Time.minutes(1)))
    .aggregate(new CountAggregate())                  // running count, managed by Flink
    .sinkTo(kafkaSink);                               // emit "campaign X had N clicks in minute M"

Read it top-down and it is the sorting office: pull letters off the belt (source), read the postmark and set a 5-second patience for stragglers (WatermarkStrategy), file by recipient (keyBy), seal one bin per minute (window), tally each bin (aggregate), post the totals (sink). Notice how little you had to say about storage, parallelism, or failure recovery — Flink owns all of that.

⚠️ The most common beginner bug is forgetting the WatermarkStrategy (or setting event time but never emitting watermarks). Your windows then never fire — the bins fill forever and no result ever comes out, because Flink is still waiting to be told "time has moved past this window." If a Flink job "produces no output," suspect watermarks first.

---

4. When to reach for Flink — and when not

Reach for Flink when the workload is genuinely streaming and stateful and correctness matters:

• Real-time analytics and dashboards on a firehose (clicks, telemetry, metrics) with event-time correctness.
• Stateful event processing — fraud detection, real-time recommendations, session analytics, anomaly detection — where you must remember history per key.
• Exactly-once pipelines where double-counting is unacceptable (billing, payments, inventory).
• Continuous ETL / streaming joins feeding a lake or search index in near-real-time.

Reach for something else when:

Instead you want…	Better tool	Why
Simple stream transforms, no cluster to run	Kafka Streams	A library embedded in your app, not a cluster. Lower ops burden; tied to Kafka; scales with your app, not independently.
Streaming bolted onto a big batch/analytics stack	Spark Structured Streaming	Micro-batch, but unifies with Spark SQL/ML on the same engine. Higher latency floor; fine if seconds are OK.
Pure large-scale batch analytics / warehouse queries	Spark / a warehouse (e.g. Azure Synapse Analytics)	If nothing is real-time, a batch engine or MPP warehouse is simpler and cheaper.
A legacy low-latency pipeline you already run	Apache Storm	Only if it exists already; for anything new, Flink supersedes it on correctness and state.

💡 Rule of thumb: if you find yourself saying "per-key state + event-time windows + can't afford to double-count," that's a Flink-shaped problem. If it's a stateless transform or you're happy with second-level micro-batch latency, a lighter tool will save you a cluster to operate.

---

5. Key takeaways

1. Flink is a true stream processor — it acts on each record as it arrives (not micro-batches like Spark Streaming), giving millisecond latency while still being correct.
2. Event time + watermarks are the heart of it: reason about when events happened (the postmark), and use watermarks to decide when you've seen enough to emit a result — correct answers despite out-of-order, late data.
3. Windows (tumbling, sliding, session) turn an infinite stream into finite results, firing when the watermark passes their end.
4. Managed keyed state lets operators remember across events, and Flink stores, checkpoints, and rescales that state for you — the foundation for the exactly-once guarantees we unpack in Part 2.
5. Reach for Flink when the problem is streaming and stateful and correctness-critical; reach for Kafka Streams, Spark Structured Streaming, or a batch engine when it isn't.

Next, Part 2 — Internals & Exactly-Once opens the machine: the JobManager/TaskManager architecture, how keyed state lives in RocksDB, and the star of the show — asynchronous barrier snapshotting, the checkpoint algorithm that makes exactly-once possible.