Phase 02 — Data Engineering & Streaming
Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (35–45 hours) Roles Supported: the JD's "data pipelines … event-driven services, streaming data" — the layer every ML system stands on
Why This Phase Exists
Models are downstream of data, and senior MLEs are accountable for the data system: where events come from, what ordering and delivery guarantees they carry, how batch and streaming views reconcile, and how quality is enforced before features are computed. The hardest production bugs in ML (silent feature corruption, double-counted events, training on data that arrived late) are streaming-semantics bugs — so this phase builds a stream processor by hand, because watermarks and exactly-once-effect only become real when you've implemented them.
Concepts
- Batch vs streaming as views of the same log; the Kafka mental model (topics, partitions, offsets, consumer groups, keys → ordering)
- Event time vs processing time — the distinction everything else hangs on
- Windows: tumbling, sliding, session; watermarks and allowed lateness
- Delivery semantics: at-most/at-least/exactly-once-effect (idempotency + dedup)
- Schemas as contracts on topics (the Phase 01 library, applied); schema evolution
- Data quality gates and quarantine paths; backfills and reprocessing
Labs
Lab 01 — Mini Stream Processor (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Build a windowed event-time aggregator: tumbling windows, watermark-driven emission, allowed-lateness updates, late-event side-output, and idempotent processing under duplicate delivery |
| Concepts | Event time vs processing time, watermarks, window lifecycle (open→closeable→expired), dedup for exactly-once effect |
| Steps | 1. Event/Window value objects; 2. window assignment; 3. watermark advance + emission; 4. allowed lateness (re-emission) vs too-late (side output); 5. dedup by event id; 6. out-of-order and duplicate stress tests |
| How to Test | pytest test_lab.py — 13 tests incl. out-of-order delivery producing identical results to in-order |
| Talking Points | Why watermarks instead of timers? What does "exactly-once" actually promise? Where do late events go and who decides? |
| Resume Bullet | Implemented an event-time stream aggregator with watermarks, allowed lateness, and idempotent delivery semantics; verified order-independence under shuffled and duplicated delivery |
→ Lab folder: lab-01-stream-processor/
Extension Project A — Kafka Hands-On (spec)
Run single-node Kafka (docker); produce keyed events; demonstrate per-key ordering, consumer-group rebalancing, and replay-from-offset; wire your Phase 01 contracts as a validating consumer with a quarantine topic.
Extension Project B — Backfill Design Memo (spec)
Design (1 page + diagram) the reprocessing story for your lab's aggregates: a bug in the aggregation logic shipped for a week — how do you recompute without double-counting, what's versioned, and how do consumers cut over?
Guides in This Phase
Deliverables Checklist
- Lab 01 implemented; all 13 tests pass
- Extension A demo script + notes (or equivalent experience)
- Extension B memo