Lab 01 — Streaming Joins & CDC Apply

Phase: 05 — Streaming Alternatives & CDC | Difficulty: ⭐⭐⭐⭐☆ | Time: 5–7 hours

Flink, Kafka Streams, and Spark Structured Streaming differ in API and runtime, but they all have to get the same four patterns correct. This lab implements them engine-agnostically: CDC materialization (change stream → table, idempotent by LSN), the as-of join (point-in-time correct stream↔dimension enrichment — the leakage killer), the interval join (stream↔stream within a time window), and sessionization.

What you build

  • apply_cdc — fold a Debezium-style change stream into a materialized table; apply only newer versions so duplicates and out-of-order delivery are no-ops (exactly-once effect on the table); deletes leave tombstone version memory so a stale insert can't resurrect a deleted key
  • temporal_join — enrich each event with the dimension value valid at the event's time (not the latest) — using a future value is temporal leakage
  • interval_join — pair two streams within [ts+lower, ts+upper]; bounded intervals bound the state
  • sessionize — gap-based session windows per key (the merge logic of Flink session windows, distilled)

Key concepts

ConceptWhat to understand
CDC → tablea change log materializes a table; idempotent by LSN/version
Tombstone memorykeep the version after delete so stale inserts can't resurrect
As-of joinpoint-in-time correctness; using a later dim value is leakage
Interval joinbounded window = bounded state for stream-stream joins
Sessionizationdynamic, gap-defined windows; the user-behavior shape

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success criteria

  • All 14 tests pass — test_cdc_idempotent_under_duplicate_and_shuffle and test_cdc_stale_insert_after_delete_rejected certify the CDC correctness model.
  • You can explain why the as-of join must use the event-time-valid dimension value, and what bug you get if you use the latest.
  • You can explain why a stream-stream join needs a bounded interval to be feasible.

Extensions

  • Add late-data handling to the as-of join (a dimension update that arrives after the event — do you re-emit? this is P04's allowed-lateness on the dimension side).
  • Implement the CDC snapshot + streaming bootstrap (initial full-table snapshot then switch to the change stream — Debezium's actual lifecycle).
  • Add a windowed stream-stream join with watermark-driven state cleanup (combine this lab with P04's watermark machinery).