Lab 02 — Event-Time Stream Processing in Scala

Phase: 04 — Apache Flink | Difficulty: ⭐⭐⭐⭐☆ | Time: 5–6 hours

Stream processing on the JVM is the in-demand Scala data-engineering skill — Flink, Kafka Streams, and Spark are Scala/JVM systems, and their operators are written in this style. This lab implements Flink's event-time core in idiomatic Scala (case classes, immutable accumulators, a keyed processor), verified with ScalaTest via sbt test. It mirrors Lab 01's Python flink-engine so you can compare the language to the semantics.

What you build

  • Windowing.assignWindow — tumbling [start, end) arithmetic (boundary → next window)
  • StreamProcessor — per-key event-time aggregation with watermark-driven emission, allowed-lateness corrected re-emission, late-event side output, and dedup for exactly-once effect
  • Windowing.finalResults — fold emissions+updates to the consumer's latest-per-window view

Run

sbt test

First run downloads Scala 2.13 + ScalaTest, then compiles and runs. Expected: 6 tests, all green, including the property that certifies the design — order independence under bounded shuffle (shuffled delivery == in-order delivery).

Key concepts (same as Lab 01, in Scala)

ConceptScala expression
WatermarkmaxTs - watermarkDelay; emission is watermark-driven
Allowed latenessemitted windows linger and re-fire isUpdate = true corrections
Side outputtoo-late events appended to lateEvents (never dropped)
Exactly-once effectseen set deduplicates by eventId before accumulation
ImmutabilityAcc is a case class; updates are .copy(...)

Success criteria

  • sbt test → all 6 specs pass.
  • You can read the Scala and map every line to the Flink concept (WARMUP).
  • You can explain why the shuffle test is the one that proves correctness.

Extensions

  • Make the processor purely functional: thread state through an immutable fold (no var/ mutable), returning (newState, emitted) — the FS2/Cats way (P12).
  • Add sliding and session windows (session needs merge-on-late-bridge).
  • Bound the seen dedup set by a TTL behind the watermark (P01's bounded-dedup tradeoff).
  • Wire it to real Flink: drop this logic into a KeyedProcessFunction[String, Event, ...] with ValueState/timers (the production form — see WARMUP Ch. 4).