Warmup Guide — Streaming Alternatives & CDC

Zero-to-principal primer for Phase 05: the stream-processing engine landscape (Kafka Streams, Spark Structured Streaming, Akka Streams, FS2) and how to choose between them, Change Data Capture, and the join/sessionization patterns every engine must implement.

Table of Contents

Chapter 1: The Engine Landscape

P04 taught Flink as the reference. The others trade Flink's power for different operational shapes:

  • Flink: dedicated cluster, record-at-a-time, lowest latency, richest state & exactly-once. Most powerful, most to operate.
  • Kafka Streams: a library you embed in your service — no cluster — using Kafka itself for state durability. Lowest ops if you're already on Kafka.
  • Spark Structured Streaming: micro-batch on the Spark engine; same DataFrame API as batch; higher latency, but "free" if your org already runs Spark.
  • Akka Streams / FS2: in-process, back-pressured streaming inside a single service — for app-level pipelines, not cluster-scale analytics.

The principal point: these are not "better/worse," they're different operational commitments. The skill is matching the engine to the use case (Ch. 5).

Chapter 2: Kafka Streams — Streaming as a Library

Kafka Streams flips the model: instead of submitting a job to a cluster, you write a normal JVM application that links the Streams library and scales by running more instances (in the same consumer group). The model:

  • KStream = an unbounded stream of records (events). KTable = a changelog folded into the latest-value-per-key table (P02 compaction, as a first-class type). The two interconvert (toTable, toStream) — the stream/table duality made API. The lab's apply_cdc is a KStream→KTable fold.
  • Local state + changelog topics: stateful operations keep state in a local RocksDB store; for fault tolerance, every state change is also written to a changelog topic in Kafka, so a failed instance's state is rebuilt by replaying the changelog. Exactly-once is available via Kafka transactions (processing.guarantee=exactly_once_v2).
  • Interactive queries: because state is local, you can query it directly (a lightweight serving path).
  • When it wins: you're already on Kafka, you want no separate cluster to operate, and your scale fits "run more app instances." When it doesn't: cross-cluster sources, the very largest state, or sub-millisecond latency needs (Flink territory).

Chapter 3: Spark Structured Streaming — Micro-Batch on the SQL Engine

Spark Structured Streaming models a stream as an unbounded table that grows; each micro-batch processes the new rows with the same DataFrame/SQL API as batch Spark (P06). Properties:

  • Micro-batch means latency is bounded by the batch interval (typically hundreds of ms to seconds) — higher than Flink's record-at-a-time, but plenty for most analytics. A continuous mode exists for lower latency with weaker guarantees.
  • Checkpointing to a durable store (HDFS/S3) plus idempotent/transactional sinks gives exactly-once (the offsets processed and the output are tracked together).
  • Watermarks & windows exist (similar semantics to P04) but with micro-batch granularity.
  • When it wins: you already run Spark, your latency budget is seconds not milliseconds, and you want one engine and one API for batch and streaming (the unification). When it doesn't: hard low-latency or very large keyed-state, fine-grained timers — Flink.

Chapter 4: Akka Streams and FS2 — Functional, In-Service Streaming

These are not cluster engines; they're in-process streaming libraries for building back-pressured pipelines inside a service (an ingestion gateway, an enrichment microservice, a connector):

  • Akka Streams: a Reactive Streams implementation on the Akka actor runtime. You build a typed graph of Source → Flow → Sink with built-in backpressure (demand flows upstream — P01 Ch. 8 made concrete). Great for bounded-resource, composable in-service streaming with strong typing.
  • FS2 (Functional Streams for Scala): pure, referentially-transparent streams on Cats Effect (P12), with resource safety (a stream that opens a file/connection releases it even on failure or cancellation) and backpressure by construction. The functional-purist's choice.
  • These matter for this role because the JD explicitly lists Akka/Cats/ZIO/FS2 and asks you to build internal SDKs (P12). You'll often build the platform's in-service components (a validating ingestion sidecar, a CDC connector) with exactly these libraries, while the cluster-scale processing runs on Flink/Spark.

Chapter 5: Choosing an Engine

The decision matrix (and the dial each turns — P00):

NeedFlinkKafka StreamsSpark SSAkka/FS2
Latencymsms–ss (micro-batch)ms (in-proc)
Ops modelclusterlibrary (no cluster)cluster (yours already?)in-service
State sizehuge (RocksDB)large (RocksDB+changelog)mediumsmall (in-proc)
Exactly-oncestrongestyes (Kafka txns)yes (ckpt+sink)DIY
Batch+stream one APIvia Table APInoyesno
Best whenlow-latency stateful at scalealready-on-Kafka, no clusteralready-on-Sparkin-service pipelines

The principal answer is always contextual: "low-latency stateful analytics at scale → Flink; we're all-Kafka and want no cluster → Kafka Streams; we already run Spark and need seconds-latency with one API → Spark SS; it's a single service's internal pipeline → Akka/ FS2." Then the ADR.

Chapter 6: Change Data Capture

CDC turns a database's commit log (MySQL binlog, Postgres WAL) into a stream of row-level change events — the bridge from OLTP systems to the lakehouse and to event-driven consumers, without the dual-write problem (you don't write to both the DB and Kafka; you capture what the DB already committed).

  • Debezium is the dominant connector: it reads the binlog/WAL and emits change events (op = c/u/d, before/after images, source LSN) to Kafka.
  • Snapshot + streaming: on first run, CDC takes a consistent snapshot of the table (the current rows), then switches to streaming the log from the snapshot's position — so consumers get the full state and all subsequent changes, with no gap. (Extension B.)
  • Applying CDC (the lab): fold the change stream into a materialized table. The correctness keys: apply changes in LSN order, make it idempotent by version (a re-delivered or out-of-order change is a no-op — P01 again), and keep tombstone version memory so a stale insert can't resurrect a deleted row. This is exactly how a CDC sink keeps a lakehouse upsert table (P09) or a Cassandra table (P11) correct.
  • Ordering & keys: CDC events for a row must stay ordered → key the Kafka topic by the primary key (P02 per-key ordering). Schema changes in the source flow through as schema evolution (P03).

Chapter 7: Join Patterns

Joins are where streaming gets subtle, because at least one side is unbounded:

  • Stream ↔ Table (temporal / "as-of" join): enrich each event with a dimension value — but the value valid at the event's time, not the latest. Using the latest is temporal leakage: you'd attribute an event to a dimension state that didn't exist yet (a user's current tier applied to last month's event). The lab's temporal_join does the point-in-time-correct lookup; this is the streaming twin of the feature-store point-in-time-join leakage bug.
  • Stream ↔ Stream (interval / windowed join): join two streams where the records fall within a bounded time interval of each other (click→purchase within 30 min). The bound is what makes it feasible — each side's state need only be retained for the interval width (Flink's interval join). Unbounded "join everything to everything" is unimplementable at scale; the bound is the design.
  • Enrichment from an external store: look up a key in Cassandra/DynamoDB (P11) per event — use async I/O (don't block the operator) and a cache; mind the consistency (the store may lag the stream).

Chapter 8: Sessionization and Stateful Patterns

  • Sessionization (the lab): group a key's events into sessions separated by an inactivity gap (a 30-min silence ends the session). Boundaries are dynamic (unlike fixed windows), and a late event can bridge two sessions into one (the merge logic Flink session windows implement). The natural shape for "web session," "trip," "viewing session" features.
  • Other stateful patterns you'll implement on these engines: dedup (P01's seen-ids, with bounded TTL state), top-N / leaderboards (keyed sorted state), running aggregates (incremental, not buffer-all), pattern detection (CEP / MATCH_RECOGNIZE — the fraud-rule sequences). All share the lesson: bound the state (TTL, window, gap), or it grows forever (P04 Ch. 4).

Lab Walkthrough Guidance

Lab 01 — Streaming Joins & CDC Apply, suggested order:

  1. apply_cdc — version-gated apply; delete with tombstone memory; then the idempotent-under-shuffle and stale-insert-after-delete tests (the correctness core).
  2. temporal_join — sort dim versions per key, bisect for the latest valid_from ≤ ts; None before the first version (no leakage).
  3. interval_join — per-key, the [ts+lower, ts+upper] window; validate lower ≤ upper.
  4. sessionize — per-key, split when ts − last > gap (gap exclusive — the boundary test).

Success Criteria

You are ready for Phase 06 when you can, from memory:

  1. Contrast Flink, Kafka Streams, Spark SS, and Akka/FS2 and pick one for a stated use case.
  2. Explain KStream vs KTable and Kafka Streams' changelog-topic fault tolerance.
  3. Explain CDC, the snapshot+streaming bootstrap, and the idempotent-by-LSN apply.
  4. Explain the as-of join and the temporal-leakage bug it prevents.
  5. Explain why stream-stream joins must be bounded and how sessionization differs from fixed windows.

Interview Q&A

Q: When would you NOT use Flink? When its operational weight isn't justified. If we're already all-in on Kafka and the use case fits "run more app instances," Kafka Streams removes a whole cluster to operate. If we already run Spark and our latency budget is seconds with batch+stream sharing one API, Spark Structured Streaming is the lower-friction choice. If it's a single service's internal back-pressured pipeline, Akka Streams or FS2 keeps it in-process. Flink earns its operational cost when you need low-latency, large keyed-state, fine-grained timers, and the strongest exactly-once — otherwise a lighter engine is the principal choice.

Q: How does CDC stay correct under retries and reordering? Each change carries a monotonic source LSN/version, and the apply is version-gated: a change is applied only if its version exceeds the last applied version for that key, so duplicates and out-of-order delivery are no-ops — exactly-once effect on the materialized table. Deletes keep tombstone version memory so a stale insert that arrives after a delete (with a lower version) can't resurrect the row. Order is preserved per row by keying the CDC topic on the primary key. The snapshot+streaming handover ensures no gap between the initial state and the change stream.

Q: A teammate enriched events with users.current_tier. What's wrong? Temporal leakage. They joined each event to the user's current tier, but the correct value is the tier that was valid at the event's timestamp — an as-of join. Using the current value means a user who upgraded last week has all their old events mislabeled as "pro," which corrupts any time-based analysis or training data (it's the same point-in-time bug as a feature store). The fix is a temporal join keyed by event time against the dimension's validity intervals.

References