🛸 Hitchhiker's Guide — Phase 04: Apache Flink

Read this if: you want Flink's model — execution, time, state, checkpoints, exactly-once — in your head fast, and to be ready for the "diagnose this Flink job" interview round. Skim, read the WARMUP, build the engine.

0. The 30-second mental model

Flink processes records one at a time over a graph of operators, keeps keyed state, orders by event time using watermarks, snapshots the whole running job consistently with barrier checkpoints (so a crash resumes with exact state), and gets exactly-once output via idempotent or two-phase-commit sinks. One sentence: Flink = stateful event-time streaming with consistent snapshots; correctness is checkpoints + the right sink.

1. Execution model in one diagram

JobManager (coordinates, triggers checkpoints)
  └─ TaskManagers → task slots → operator subtasks
Source → keyBy(k) → window/process (KEYED STATE) → sink
         hash-partition      checkpoint barriers flow with data
parallelism = N subtasks/operator; chaining fuses adjacent ops

2. Watermarks (the thing everything hangs on)

W = max_event_time_seen − bounded_delay
window fires when  W ≥ window.end
multi-input op watermark = MIN across inputs  ← one idle partition stalls everything
                                                (fix: withIdleness)
late-but-allowed → re-emit correction; too-late → side output (never drop)

Trade: bigger delay = more complete first results, more latency, more state.

3. State & backends

State grows from: too many open windows (sliding / huge allowed lateness), unbounded key cardinality, no state TTL. State size is the cost center.

4. Checkpoints = consistent distributed snapshot

JobManager injects BARRIER N into sources
→ each operator, on receiving N on ALL inputs (ALIGNMENT), snapshots state async → S3
→ all confirm → checkpoint N COMPLETE
crash → restart from last complete checkpoint: restore state + rewind source offsets

Alignment buys exactly-once and is where it stalls under backpressure → unaligned checkpoints help.

5. Checkpoint vs savepoint

checkpoint = automatic, engine-owned, fast recovery, may be GC'd
savepoint  = manual, user-owned, portable → upgrades, rescale, migrate
upgrade ritual: savepoint → stop → start new version from savepoint (need stable operator UIDs)

6. Rescaling via key groups

key → key_group = hash(key) % maxParallelism   (fixed at job creation!)
key_group → task = kg * parallelism // maxParallelism   (contiguous ranges)
rescale = move whole key-group ranges between subtasks
GOTCHA: maxParallelism caps you forever — set it generously

7. Exactly-once OUTPUT (state alone isn't enough)

idempotent sink     → upsert/set-union/overwrite-by-id (Cassandra, KV)   ← simplest
two-phase commit    → pre_commit on barrier (pending, invisible)
                      commit on checkpoint-complete (visible, idempotent)
                      recover → re-commit pending  (files rename / Iceberg snapshot commit)
buffered-not-precommitted data is DROPPED on crash → correct (offset not advanced either)

8. The Round-2 failure chain (memorize this)

slow sink / hot-key skew / expensive process()
   → backpressure → buffers fill
       → checkpoint barriers slow + alignment ↑ → CHECKPOINT DURATION ↑
       → source throttled → KAFKA LAG ↑
huge allowed-lateness / sliding / no TTL / unbounded keys → many open windows → STATE ↑

Fix the CAUSE: async/scale the sink, re-key/salt the hot key, incremental aggregation, state TTL, rescale; unaligned checkpoints relieve alignment while you fix root cause.

9. Beginner mistakes that mark you

1. Ordering/aggregating by processing time → wrong on replay (P01 again).
2. No withIdleness → one quiet partition freezes all watermarks.
3. process() that buffers all elements instead of incremental aggregate.
4. No state TTL + unbounded keys → RocksDB grows forever.
5. Forgetting operator UIDs → savepoint won't restore after a code change.
6. Setting maxParallelism too low (or defaulting it) → can't scale up later.
7. Assuming checkpoints give exactly-once output (they give exactly-once state).

10. How this phase pays off later

• 2PC sink → P09 Iceberg/Delta exactly-once writes.
• Checkpoints in S3 / RocksDB LSM → P08 storage, P11 LSM internals.
• Broadcast state + Cassandra enrichment → P11 + capstone fraud detector (P16).
• The failure chain → P13 observability & incident response.

Read the WARMUP, build the engine, then P05: the other streaming engines (Kafka Streams, Spark SS, Akka/FS2) as variations — plus CDC and streaming joins.