Capstone 02 — Streaming Feature Platform with Gated Retraining

Composes: Phase 01 (contracts) · Phase 02 (stream processor) · Phase 03 (feature store) · Phase 08 (orchestrator) · Phase 12 (drift + retraining FSM) Time: 2–4 weeks

The platform-engineering capstone: events in, point-in-time-correct features out, and a retraining loop that cannot launder broken data into a model.

What you ship

event stream (P02 generator: late events, duplicates, planted drift)
   │
   ├─► contract validation (P01) ──quarantine──► dead-letter ledger
   │
   ├─► windowed aggregation w/ watermarks (P02) ──► online store (P03)
   │                                                  │ parity check
   └─► append-only event log ──► offline store ◄──────┘
                                      │
                         point-in-time training frames (P03)
                                      │
                    orchestrated train/validate DAG (P08, cached, lineaged)
                                      │
                  retraining policy FSM (P12): drift- & schedule-triggered

Stage list

1. Stream generator: sessions of events with controlled out-of-orderness, duplicate rate, a schema-break injection point, and a planted distribution shift at a known timestamp.
2. Walking skeleton: events → one windowed feature → online store → a trivial model retrained on schedule. End to end, day one.
3. Hardening: contracts at ingestion (violations → quarantine with reasons — never silent drops); dedup + watermark policy from Phase 02; the online/offline parity test as a continuous check.
4. Training frames: point-in-time joins over the offline store; prove leak-freedom with the Phase 03 leakage test adapted to this data.
5. Orchestration: the Phase 08 DAG runs ingest-validate → build-frames → train → validate → register; show content-hash caching skipping unchanged stages and lineage answering "which model saw the broken events?"
6. The loop: Phase 12 FSM wired to the drift monitor over the planted shift; demonstrate both guard rails — (a) schema break + drift → INCIDENT, not retraining; (b) challenger that fails current-data validation → INCIDENT.

Money table (fill with your runs)

Injected-failure demo (required)

The schema-break injection: contracts quarantine it; the drift monitor flags the survivors; the FSM lands in INCIDENT with reasons; the lineage taint query names every downstream artifact. One incident, four mechanisms, fully narrated.

Honest-limitations paragraph (required)

In-memory stores vs real Kafka/Redis/warehouse; single-process watermarks vs distributed ones; exactly-once effect via dedup, not transactional semantics.