Phase 05 — Streaming Alternatives & CDC
Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 1.5 weeks (24–32 hours) Prerequisites: Phase 04 (Flink — the reference engine these are compared against)
Why This Phase Exists
Flink is not the only stream processor, and a principal must be able to say why Flink and not Kafka Streams here, or Spark Structured Streaming there, or Akka Streams for that service. The JD lists them all — "Kafka Streams, Flink DataStream API, Flink SQL, Spark Structured Streaming, Akka Streams" — and names CDC platforms, streaming joins, enrichment, and sessionization as required patterns. This phase rounds out the streaming half: the engine landscape, the FP/JVM streaming libraries (Akka Streams, FS2 — foreshadowing P12), and the join/CDC patterns that every engine implements and that you build here once, engine-agnostically.
Concepts
- Kafka Streams: a library, not a cluster — embeds in your app; KStream vs KTable (the stream/table duality, P02 compaction made API); local state in RocksDB + changelog topics for fault tolerance; interactive queries; when "no cluster to run" wins.
- Spark Structured Streaming: micro-batch (and continuous) on the Spark SQL engine; checkpointing to a durable store; the same DataFrame API as batch (unification); higher latency than Flink but trivial if you're already on Spark.
- Akka Streams: Reactive-Streams back-pressured graphs on the actor runtime; typed, composable, in-process — the right tool for streaming inside a service (P12).
- FS2 (Functional Streams for Scala): pure, back-pressured streaming on Cats Effect — resource-safe, referentially transparent (P12).
- The engine decision matrix: latency, state size, exactly-once needs, ops model, language, ecosystem — Flink vs Kafka Streams vs Spark SS vs Akka/FS2.
- CDC: turning a DB's binlog/WAL into events (Debezium); snapshot + streaming bootstrap; applying CDC to a table idempotently (the lab); the OLTP→lakehouse bridge.
- Join patterns: stream↔table (temporal / as-of), stream↔stream (interval/windowed), enrichment from an external store (P11); the leakage and state-size traps.
- Sessionization and other stateful patterns (dedup, top-N, running aggregates).
Labs
Lab 01 — Streaming Joins & CDC Apply (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Implement CDC materialization (idempotent by LSN, tombstone memory), the as-of stream↔table join (point-in-time correct), the interval stream↔stream join, and gap-based sessionization |
| Concepts | Change-stream → table, temporal leakage, bounded-interval joins, session windows |
| How to Test | pytest test_lab.py -v — 14 tests incl. CDC idempotence under shuffle |
| Talking Points | Why must the dimension lookup be event-time-valid? Why keep version memory after a delete? Why bound a stream-stream join? |
| Resume bullet | Built engine-agnostic streaming join + CDC primitives: idempotent change-stream materialization, point-in-time stream-table enrichment, interval joins, sessionization |
→ Lab folder: lab-01-joins-cdc/
Extension Project A — Engine bake-off memo (spec)
One page: pick a real use case (clickstream sessionization; CDC replication; in-service enrichment) and choose Flink vs Kafka Streams vs Spark SS vs Akka/FS2 — with the deciding dial (P00) and an ADR.
Extension Project B — Debezium CDC pipeline (spec)
docker Postgres + Debezium + Kafka; capture row changes; materialize a table with your
apply_cdc; demonstrate snapshot→streaming handover and idempotent re-apply.
Integrated-Scenario Hooks
- This phase's CDC apply feeds the lakehouse upsert tables (P09) and Cassandra serving (P11).
- Its as-of join is the enrichment step of the capstone fraud detector (P16, JD Problem 2).
- Its Akka/FS2 preview is built for real in Scala in P12.
Guides in This Phase
Key Takeaways
- The engines differ in latency, ops model, and language — not in the patterns they must implement.
- Kafka Streams = a library (no cluster); Flink = lowest latency + richest state; Spark SS = free if you're on Spark; Akka/FS2 = in-service streaming.
- CDC materialization is idempotent-by-LSN; the stream/table duality is everywhere.
- Joins are where leakage (as-of) and unbounded state (intervals) bite — bound both.
Deliverables Checklist
- Lab 01 implemented; all 14 tests pass
- You can choose an engine for a stated use case with the deciding dial
- You can explain KStream vs KTable and the changelog-topic fault-tolerance model
- Extension A memo or Extension B CDC pipeline