Lab 01 — Mini Stream Processor

Phase: 02 — Data Engineering & Streaming | Difficulty: ⭐⭐⭐⭐☆ | Time: 5–7 hours

Watermarks, allowed lateness, and exactly-once-effect are folklore until you've implemented them. This lab builds the core of Flink's event-time machinery in ~150 lines — and proves it with the property that justifies the whole design: shuffled delivery produces identical results to ordered delivery.

What you build

  • assign_window — tumbling-window arithmetic ([start, end) — boundary events belong to the next window; a test enforces it)
  • StreamProcessor — per-key window accumulators with the full lifecycle: open → closeable (watermark passes the end: emit once) → retained for allowed lateness (late events re-emit corrected is_update=True results) → expired (further events go to the side output, never silently dropped)
  • Dedup by event_id — at-least-once delivery in, exactly-once effect out
  • flush — bounded-input end-of-stream semantics

Key concepts

ConceptWhat to understand
Event time vs processing timeResults depend only on event timestamps — which is exactly what the shuffle test verifies
Watermarkmax_ts_seen − delay: a moving claim that "nothing older is coming"; emission is watermark-driven, not arrival-driven
Allowed latenessEmitted ≠ done: windows linger and re-fire corrections; consumers must tolerate updates
Side outputToo-late data is information (client clocks, pipeline health) — route it, never drop it
Exactly-once effectDedup before accumulation makes duplicate delivery a no-op — the only "exactly-once" that exists

Files

FilePurpose
lab.pySkeleton with # TODO markers
solution.pyComplete reference; python solution.py runs an out-of-order + duplicate demo
test_lab.py13 tests incl. order-independence under shuffle and idempotence under duplication
requirements.txtpytest only — pure stdlib

Run

pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success criteria

  • All 13 tests pass — test_order_independence_under_bounded_shuffle and test_duplicates_under_shuffle_still_idempotent are the ones that certify understanding
  • You can explain why watermark_delay and allowed_lateness are different knobs (tolerated disorder before first emission vs tolerated corrections after)
  • You can state what the consumer of this stream must handle (updates! late topic!)

Extensions

  • Sliding windows (one event → multiple windows) and session windows (gap-based, with merge-on-late-bridge — significantly harder; do it to feel why)
  • Bound seen_ids with a TTL keyed to the watermark (real systems can't keep every id forever — what does expiring dedup state cost in guarantees?)
  • Replace the dict state with an interface and add a "spill to disk" implementation — the state-backend abstraction of real engines

References