Phase 03 — Feature Stores & Training-Serving Skew
Difficulty: ⭐⭐⭐⭐⭐ Estimated Time: 2 weeks (35–45 hours) Roles Supported: the JD's "feature stores" named capability — and the home of ML's most expensive bug class (leakage and skew)
Why This Phase Exists
Two failure modes silently destroy more production ML value than any modeling choice:
- Temporal leakage: training features computed with information that wasn't available at prediction time — offline metrics soar, production disappoints, and nobody knows why for months.
- Training-serving skew: the offline feature pipeline and the online one compute almost the same thing — different defaults, different windows, different staleness — and the model meets features at serving time that it never saw in training.
The feature store is the architectural answer to both: one feature definition, point-in-time-correct offline materialization for training, and a low-latency online store for serving — with parity between them measured, not assumed. The flagship lab builds all three pieces.
Concepts
- Feature definitions as code; entities, feature views, timestamps (event vs created)
- Point-in-time (as-of) joins — the leakage killer, in exact semantics
- Offline store (training materialization) vs online store (serving lookup); freshness/staleness budgets
- Training-serving skew taxonomy: logic skew, data skew, temporal skew
- Online/offline parity testing and skew detection (the lab implements both)
- Feature freshness SLOs, TTLs, defaults-on-miss (and their skew implications)
- Build-vs-adopt landscape: Feast/Tecton/SageMaker FS — what they actually provide
Labs
Lab 01 — Mini Feature Store (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Build a feature store core: register feature views over timestamped event data, produce point-in-time-correct training frames (as-of joins with TTL), serve online lookups, and verify online/offline parity + detect skew |
| Concepts | As-of join semantics (strictly ≤ label time), TTL expiry, online materialization, parity checking, distribution-skew detection (PSI) |
| Steps | 1. Feature view registration; 2. as-of join (the heart — no future rows, TTL honored); 3. leakage tests (a future-leaking join must be caught); 4. online store materialize + lookup; 5. parity checker (online value == offline value at same instant); 6. PSI-based skew detector between training frame and serving log |
| How to Test | pytest test_lab.py — 12 tests incl. an adversarial leakage case and a deliberate skew injection |
| Talking Points | Why is as-of ≤ not <? (it's a choice — defend yours.) What breaks when online TTL ≠ offline TTL? Why is parity a test, not an assumption? |
| Resume Bullet | Built a point-in-time-correct feature store core with online/offline parity verification and PSI-based skew detection; adversarial tests prove temporal-leakage immunity |
→ Lab folder: lab-01-feature-store/
Extension Project A — Feast Hands-On (spec)
Define the lab's feature views in Feast (local provider); materialize, fetch historical features and online features; compare its as-of semantics and TTL handling to yours; write the one-page "what Feast gives you / what you still own" memo.
Extension Project B — Skew Post-Mortem (spec)
Write the post-mortem (PMC-track template) of a realistic skew incident: offline
imputed missing days_since_last_order with the median, online defaulted to 0.
Reconstruct the metric impact mechanism, the detection that should have existed,
and the parity test that prevents recurrence.
Guides in This Phase
Deliverables Checklist
- Lab 01 implemented; all 12 tests pass
- Extension A memo (or equivalent Feast/Tecton experience)
- Extension B post-mortem