Phase 04 — Apache Flink
Difficulty: ⭐⭐⭐⭐⭐ Estimated Time: 2 weeks (35–45 hours) Prerequisites: Phase 01 (time, exactly-once), Phase 02 (the source), Phase 03 (validated events)
Why This Phase Exists
Flink is the JD's reference engine for "low-latency stream processing … stateful operators … checkpointing … savepoints … exactly-once … RocksDB state backend … rescaling." It is also the subject of an entire interview round (Round 2: "a Flink job with rising checkpoint duration, increasing RocksDB state size, delayed watermarks, and growing Kafka lag — diagnose it"). If you can reason about Flink's execution and state model, you can reason about every stream processor; the others (P05) are variations.
This is the hardest phase in the streaming half, because correct stateful streaming is genuinely hard: you must get event time, out-of-order arrival, state, fault tolerance, and exactly-once output all correct simultaneously. The lab makes you implement the core, so the words "checkpoint" and "exactly-once sink" stop being slogans and become code you've debugged.
Concepts
- Execution model: job graph → operators → task slots & parallelism; the streaming dataflow; keyBy → keyed streams; chaining; the JobManager/TaskManager split.
- Time: event time vs processing time vs ingestion time; watermarks (how generated, per-partition watermarks and the min-across-inputs rule, idle sources); allowed lateness; side outputs.
- Windows: tumbling, sliding, session (merging), global; window assigners, triggers, evictors; the state cost of windows.
- State: keyed state vs operator state;
ValueState/ListState/MapState; state backends (heap vs RocksDB, incremental checkpoints); state TTL; why state size drives everything. - Fault tolerance: the Chandy-Lamport distributed snapshot; barriers & alignment (and unaligned checkpoints); checkpoints (engine-owned) vs savepoints (user-owned, portable); exactly-once vs at-least-once checkpointing.
- Exactly-once sinks: idempotent sinks vs two-phase commit (
pre_commiton barrier,commiton completion); transactional Kafka/file sinks; the Iceberg/Delta sink (P09). - Rescaling: key groups as the redistribution unit; max parallelism; restoring a savepoint at a new parallelism.
- Backpressure (P01): credit-based flow control; the symptoms — rising checkpoint duration, alignment time, and lag — and their causes.
- Flink SQL / Table API: declarative streaming; dynamic tables; the streaming-batch unification.
Labs
Lab 01 — Flink-Grade Stream Processor (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Build event-time windows + watermarks + allowed lateness + side output, keyed state with snapshot/restore (checkpoint recovery), key-group rescaling, and a two-phase-commit exactly-once sink |
| Concepts | Watermark-driven emission, checkpoint/savepoint, key groups, two-phase commit |
| How to Test | pytest test_lab.py -v — 15 tests incl. restore-continuity and crash-recovery exactly-once |
| Talking Points | What makes a checkpoint consistent? Why is output invisible until commit? Why key groups for rescaling? |
| Resume bullet | Implemented event-time windowing, checkpoint/restore state recovery, key-group rescaling, and a two-phase-commit exactly-once sink |
→ Lab folder: lab-01-flink-engine/
Lab 02 — Event-Time Stream Processing in Scala (implemented, sbt test)
| Field | Value |
|---|---|
| Goal | Reimplement the event-time core (tumbling windows, watermark, allowed lateness, side output, dedup) in idiomatic Scala, verified with ScalaTest — the JVM/Scala form of stream processing interviewers expect |
| Concepts | The same Flink semantics, expressed in Scala (case classes, immutable Acc, keyed processor) |
| How to Test | sbt test — 6 specs incl. order-independence under shuffle (real Scala, runs here) |
| Resume bullet | Implemented event-time windowing with watermarks, allowed lateness, and exactly-once dedup in production-style Scala (ScalaTest-verified) |
→ Lab folder: lab-02-scala-windows/
Extension Project A — Run real Flink (spec)
docker Flink cluster; a DataStream job consuming the P02 Kafka topic; keyed windows; a
RocksDB state backend; take a savepoint, change the job, restore from the savepoint; observe
the Flink UI's backpressure, watermark, and checkpoint metrics.
Extension Project B — Stateful fraud detector (spec; → capstone)
The JD's Problem 2: event-time sliding + session windows, broadcast rules state, Cassandra enrichment (P11), RocksDB, S3 checkpoints, savepoint-based rule upgrades, replay for validation. Sketch the topology and the state/keying plan.
Integrated-Scenario Hooks
- This engine consumes P02's validated topic and writes P09's Iceberg tables via the 2PC sink.
- Its checkpoints live in S3 (P08); its state backend is RocksDB (an LSM, P08/P11).
- Its enrichment reads Cassandra (P11); its rules arrive as broadcast state.
- It is the processing core of capstone Problems 1 and 2 (P16).
Guides in This Phase
Key Takeaways
- Watermarks turn "when is a window done?" into an explicit, tunable answer.
- A checkpoint is a consistent global snapshot; restore resumes with exact state — that's exactly-once state.
- Exactly-once output needs idempotent or two-phase-commit sinks; the engine alone can't.
- State size is the cost center; RocksDB, TTL, and key design exist to manage it.
- Rescaling works because key groups (not keys) are the redistribution unit.
Deliverables Checklist
- Lab 01 implemented; all 15 tests pass
- You can explain checkpoint barriers, alignment, and what stalls them
- You can explain the two-phase-commit sink protocol end to end
- You can diagnose the Round-2 scenario (rising checkpoint duration + state + lag)
- Extension A savepoint round-trip or Extension B topology sketch