Warmup Guide — Data Engineering & Streaming

Zero-to-expert primer for Phase 02: logs and streams as the substrate of ML systems — Kafka's model, event time and watermarks, delivery semantics, and the data-quality machinery that keeps features trustworthy.

Table of Contents

Chapter 1: The Log — One Abstraction Under Everything

From zero: an append-only, ordered sequence of records, each addressed by an offset. That's the whole abstraction — and it unifies the field: Kafka topics are logs; database replication streams are logs; a data warehouse table is a log compacted by key; "batch" is reading a bounded slice of a log, "streaming" is reading its unbounded tail. The senior framing (Kleppmann's): batch and streaming are not two systems but two consumption modes of one log — which is why architectures that treat them as separate pipelines (the old Lambda architecture's dual codebases) rot, and why "Kappa-style" (one streaming pipeline, replay the log for backfills) won.

For ML specifically, the log is the source of truth for training data: features and labels are derived views, and every "we can't reproduce last month's training set" incident traces to deriving views without retaining or versioning the log positions they came from. Hold that; Phase 03 builds on it directly.

Chapter 2: The Kafka Model in Working Detail

The vocabulary you must own (transferable to Pulsar/Kinesis/PubSub — names differ, model rhymes):

  • Topic = a named log, split into partitions for parallelism. Ordering is guaranteed within a partition only — total order across a topic does not exist at scale, by design.
  • Keys: a record's key hashes to a partition (the PMC track's partitioner lab is literally this code) — so per-key ordering is the guarantee you engineer with: all events for user_42 land in one partition, in order. Choosing the key is choosing your ordering and your skew (hot keys → hot partitions).
  • Offsets + consumer groups: each group tracks its position per partition; partitions are divided among the group's consumers (rebalancing on join/leave). Replay = rewind offsets — the property that makes reprocessing and backfills possible at all.
  • Retention/compaction: time/size-based deletion, or log compaction (keep latest per key — a changelog that doubles as a table).
  • Producer acks (acks=all + min ISR — durability), idempotent producers, and transactions exist and matter — Chapter 6 places them.

Chapter 3: Event Time vs Processing Time

The distinction the entire phase hangs on:

  • Event time: when the thing happened (timestamp in the record, set at origin).
  • Processing time: when your system sees it.

They diverge constantly: mobile clients buffer offline for hours; a partition lags; a backfill replays last week at full speed (processing time = now, event time = last week). Any computation defined in processing time changes its answer depending on infrastructure weather — "orders per hour" computed by arrival time double-counts during a replay and undercounts during an outage.

ML's stake: features like "purchases in the last 24h" are event-time statements; computing them in processing time creates training data that doesn't match reality and — worse — training/serving skew when the serving path's timing differs from the batch path's (Phase 03's central topic). The rule: aggregate in event time; use processing time only for operational metrics about the pipeline itself.

Chapter 4: Windows

Unbounded streams need bounded questions. The window taxonomy:

  • Tumbling: fixed-size, non-overlapping ([12:00, 12:05), [12:05, 12:10)). Assignment is arithmetic: start = ts - (ts % size). The default for counts/sums.
  • Sliding/hopping: fixed size, overlapping starts (size 1h, hop 5min) — one event belongs to multiple windows; smoother but size/hop × the state.
  • Session: gap-based (a window closes after N min of silence per key) — dynamic boundaries, merge logic when a late event bridges two sessions; the natural shape for user behavior features.

Window state is the cost center: every open window per key holds an accumulator; state size = keys × windows-in-flight — the reason streaming engines obsess over state backends and why your lab keeps explicit window lifecycle (open → closeable → expired) rather than "keep everything forever."

Chapter 5: Watermarks and Lateness

The hard question: with out-of-order arrival, when is a window done? Wait forever = no results; close at the boundary = drop every straggler.

The watermark is the answer: a monotonic estimate flowing with the stream that asserts "no events with timestamp ≤ W are still coming." Common construction: W = max_event_time_seen − bounded_delay (the heuristic: we believe disorder is bounded by, say, 2 minutes). When W passes a window's end, the window emits.

Lateness policy completes the design:

  • Events older than W but within allowed lateness of their window: the window re-fires with an updated (corrected) result — consumers must tolerate updates.
  • Events beyond allowed lateness: routed to a side output (late-data topic) — never silently dropped; late data is information (about client clocks, pipeline health) and sometimes must be reconciled in batch.

The trade is explicit and product-owned: delay (larger watermark lag = more complete first results) vs completeness vs state size (longer lateness = more retained windows). Saying that sentence with numbers is the difference between using Flink and understanding it. The lab implements exactly this machinery, minus the distribution.

Chapter 6: Delivery Semantics — What Exactly-Once Actually Means

The honest ladder:

  • At-most-once: fire and forget — losses on failure. Rarely acceptable.
  • At-least-once: retry until acked — duplicates on failure. The practical default; the system must then handle duplicates.
  • "Exactly-once" is precisely: at-least-once delivery + idempotent or transactional processing = exactly-once effect. Nothing transmits each message exactly once over a faulty network (the Two Generals say hi); what's engineerable is that retries don't change the result.

The two implementation patterns:

  1. Idempotency: processing is a no-op the second time — dedup by a stable event id (the lab's approach: a seen-ids set with TTL), or naturally idempotent sinks (upsert by key, set-union accumulators).
  2. Transactions: atomically commit (output + consumer offset) together — Kafka's transactional producer/read_committed consumers, or the classic outbox pattern at service boundaries.

The interview-grade summary: "exactly-once is a property of the pipeline's effect on state, purchased with idempotency or atomic offset+output commits — not a network guarantee."

Chapter 7: Schemas, Contracts, and Evolution on Streams

A topic without a schema is an outage on a delay timer. The machinery:

  • Schema registry pattern: producers register schemas (Avro/Protobuf/JSON-schema); consumers resolve by id; the registry enforces compatibility rules on evolution — backward (new readers read old data), forward (old readers read new data), full. This is the PMC track's compatibility discipline (its Phase 11 skew matrix applies verbatim to topic evolution: writers and readers upgrade at different times, always).
  • Safe evolution: add optional fields with defaults; never repurpose/rename in place; deprecate then remove over announced versions. Breaking change = new topic (the major-version analog).
  • Where Phase 01's library plugs in: contract validation at the consumer edge — schema-valid but semantically wrong records (negative prices, impossible ages) go to quarantine (Ch. 8), with the path+rule aggregation becoming your data-quality metrics.

Chapter 8: Data Quality, Quarantine, and Backfills

  • The quarantine pattern: validation failures route to a dead-letter/quarantine topic with the reason attached — the pipeline keeps flowing, the bad data is preserved for diagnosis, and quarantine rate becomes the alert (a step change in rejects = upstream broke something; you found out in minutes, not at model-metric time — the entire point).
  • Quality dimensions to monitor per stream: volume (events/min vs historical), schema-validity rate, semantic-validity rate (per rule!), freshness (event-time lag), key cardinality drift. These are Phase 12's drift detectors aimed at inputs — same math, earlier in the pipe, cheaper to act on.
  • Backfills/reprocessing (Extension B's subject): the log makes recomputation possible; correctness requires versioned outputs (write agg_v2 alongside agg_v1, cut consumers over, then retire), deterministic logic (event-time windows — Ch. 3's payoff: a replay produces identical results because nothing depends on arrival time), and idempotent sinks (Ch. 6). The test of a well-built pipeline is literally "can you replay it" — your lab's order-independence test is the miniature of this property.

Lab Walkthrough Guidance

Lab 01 — Mini Stream Processor, suggested order:

  1. Value objects: Event(key, ts, value, event_id), WindowKey(key, start, end), WindowResult — frozen (Phase 01 habits).
  2. Window assignment (tumbling arithmetic) — unit-test boundaries (ts exactly at a window edge belongs to the next window: [start, end)).
  3. The processor loop without lateness: ingest → update accumulator; advance watermark (max_ts − delay); emit windows whose end ≤ watermark.
  4. Allowed lateness: emitted windows stay in state until end + allowed_lateness ≤ watermark; a late-but-allowed event re-emits a corrected result (mark it as an update). Then: too-late events → side output.
  5. Dedup: event_id seen-set consulted before accumulation — then run the duplicate-delivery test.
  6. The big test (write it before the code if you dare): shuffle a fixed event set (bounded disorder), process, and assert identical final results to in-order processing — order-independence is the property everything above exists to buy.

Success Criteria

You are ready for Phase 03 when you can, from memory:

  1. Explain the log abstraction and why batch/streaming are consumption modes of it.
  2. State Kafka's ordering guarantee precisely and what key choice controls.
  3. Give an ML-relevant example where event-time vs processing-time changes the answer.
  4. Define watermark, allowed lateness, and side output, and articulate the delay/completeness/state triangle.
  5. Define exactly-once-effect and both purchase methods (idempotency, transactional offset+output).
  6. Describe the quarantine pattern and name four stream-quality metrics.

Interview Q&A

Q: Your "orders per hour" feature doubled during a pipeline replay. What went wrong and how do you fix it permanently? The aggregation was keyed by processing time (arrival), so replayed events counted in today's windows — and possibly counted twice without dedup. Fix: event-time windows (replay lands in the original windows), idempotent accumulation by event id, and versioned output for the backfill cutover. The follow-up trap is "just don't replay" — wrong: replayability is the system's most valuable property; the pipeline was at fault, not the replay.

Q: Design the lateness policy for a fraud-features stream. What questions do you ask first? Product questions, not tech ones: what's the cost of an incomplete-but-fast feature vs a complete-but-slow one (fraud scoring wants fast — small watermark delay, accept corrections); can downstream consume updates (re-emission) or only finals; what's the actual disorder distribution (measure event-time lag percentiles before choosing the delay); and what happens to too-late events (for fraud: they're a signal in themselves — route to investigation, never drop). Policy = numbers chosen against those answers, monitored thereafter.

Q: The team says "Kafka gives us exactly-once, so we're fine." Correct them precisely. Kafka's idempotent producer gives exactly-once append per partition under retries; transactions give atomic offset+output commits within Kafka. The moment your consumer writes to an external system (a feature store, a DB), you're back to designing idempotency yourself — upserts, dedup keys, or an outbox. "Exactly-once" is an end-to-end property of effects, re-established at every boundary — and the boundary your ML pipeline cares about (the feature store) is exactly where the slogan stops applying on its own.

References

  • Kleppmann, Designing Data-Intensive Applications — ch. 11 (streams) is this phase's textbook; ch. 4 (encoding/evolution) for Ch. 7
  • Kreps, The Log: What every software engineer should know (2013) — the Chapter 1 essay
  • Akidau et al., The Dataflow Model (VLDB 2015) — windows/watermarks/lateness, from the source; or Streaming Systems (Akidau, Chernyak, Lax) for the book form
  • Kafka documentation: design section — delivery semantics, transactions
  • Flink docs: event time and watermarks — the production implementation of your lab
  • Confluent Schema Registry docs — compatibility rules
  • PMC track Phase 07 WARMUP (partitioning) and Phase 11 (the skew matrix — applies to schema evolution verbatim)