🛸 Hitchhiker's Guide — Phase 05: Streaming Alternatives & CDC

Read this if: you know Flink (P04) and need the rest of the streaming world fast — the other engines, when to pick each, CDC, and the join patterns. Skim, read WARMUP, build the lab.

0. The 30-second mental model

Flink isn't the only stream processor — it's the heaviest. Kafka Streams is a library (no cluster), Spark Structured Streaming is micro-batch on the Spark engine, Akka Streams / FS2 are in-service back-pressured pipelines. They differ in ops and latency, not in the patterns: CDC (change stream → table), as-of joins, interval joins, sessionization. One sentence: pick the engine by operational fit; the patterns are the same everywhere.

1. Engine picker (one line each)

Flink           → low-latency, large state, strongest exactly-once; a cluster to run
Kafka Streams   → a LIBRARY in your app, state in RocksDB + changelog topics; no cluster
Spark Str. Str. → micro-batch (seconds), same API as batch Spark; free if you're on Spark
Akka Streams    → in-service, typed, back-pressured graphs (Reactive Streams)
FS2             → in-service, pure/resource-safe streams on Cats Effect  (P12)

2. KStream vs KTable (the duality, again)

KStream = events (append)        KTable = latest-value-per-key (compacted log → table)
toTable() / toStream() interconvert     state durability = changelog topic in Kafka

This is P02 compaction as a typed API; your apply_cdc is a KStream→KTable fold.

3. CDC in 5 bullets

  • DB commit log (binlog/WAL) → change events (Debezium): op=c/u/d, before/after, LSN.
  • No dual-write: you capture what the DB already committed.
  • Snapshot + streaming: full table snapshot, then stream from that position (no gap).
  • Apply idempotently by version/LSN: dup/out-of-order = no-op (P01).
  • Tombstone version memory: keep the version after delete so a stale insert can't resurrect the row. Key the topic by PK for per-row ordering.

4. Join patterns (and their traps)

stream↔table (AS-OF)   : value valid AT EVENT TIME, not latest → else TEMPORAL LEAKAGE
stream↔stream (INTERVAL): r.ts ∈ [l.ts+lower, l.ts+upper] → bound = bounded state
enrichment (external)   : async I/O + cache; the store may lag the stream

5. Sessionization

new session when (ts - last) > gap   (gap exclusive)
boundaries are DYNAMIC; a late event can BRIDGE two sessions into one (merge)

The shape for web sessions, trips, viewing sessions.

6. Beginner mistakes that mark you

  1. "Always Flink" — can't justify it; ignoring Kafka Streams / Spark SS fit.
  2. Enriching with the current dimension value → temporal leakage.
  3. Unbounded stream-stream joins → unbounded state.
  4. Blocking external lookups in an operator (no async I/O) → backpressure.
  5. CDC apply without LSN gating → duplicates/resurrected rows.
  6. Forgetting CDC needs PK-keyed topics for per-row ordering.

7. War stories

  • "Old events all show users as 'pro'." → enriched with current tier; needed as-of join.
  • "The CDC table has rows we deleted." → no tombstone version memory; a stale insert resurrected them.
  • "Our enrichment service fell over under load." → synchronous external lookups; switch to async I/O + cache.
  • "Kafka Streams app lost its state on redeploy." → no/short changelog retention; the changelog is the durability.

8. How this phase pays off later

  • CDC apply → P09 lakehouse upserts, P11 Cassandra/DynamoDB sinks.
  • As-of join → capstone fraud detector enrichment (P16).
  • Akka/FS2 → built for real in Scala (P12).
  • Engine picker → ADRs in P15.

Read WARMUP, build the lab, then P06: we cross from streaming into batch — Spark internals, the other half of the platform.