Lab 01 — Mini Feature Store

Phase: 03 — Feature Stores & Training-Serving Skew | Difficulty: ⭐⭐⭐⭐⭐ | Time: 6–8 hours

Temporal leakage and training-serving skew are ML's most expensive bug class. This lab builds the machinery that prevents both: point-in-time-correct joins, an online store with declared defaults, parity verification, and PSI skew detection — with an adversarial leakage test where the naive join provably leaks and yours provably doesn't.

What you build

  • FeatureView / FeatureStore.register — feature definitions as code, with TTL and default as part of the definition (not the serving code — that's how the median-vs-zero skew incident happens)
  • get_historical — the as-of join: largest τ ≤ label_ts within TTL, validated against pandas.merge_asof as a differential oracle
  • materialize / get_online — latest-within-TTL per key; defaults on miss
  • parity_check — online value == offline recomputation at the same instant
  • psi — distribution-skew detection between training frame and serving log

Key concepts

ConceptWhat to understand
As-of semanticsτ ≤ t inclusive, TTL as explicit staleness→missingness; every choice here is contract
The adversarial testNaive latest-join returns the future (3.0, 6.0); the correct join returns the past (1.0, 5.0) — same data, same code path, opposite epistemics
Parity as a testOnline and offline are two implementations until proven one — the check is the proof, run in CI and on production samples
PSIQuantile-binned distribution comparison; thresholds are calibrated folklore, not statistics-sacred

Files

FilePurpose
lab.pySkeleton with # TODO markers (naive leaking join provided as the foil)
solution.pyComplete reference; python solution.py runs a demo
test_lab.py12 tests: boundary/TTL edges, merge_asof oracle, adversarial leakage, parity, PSI
requirements.txtnumpy, pandas, pytest

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success criteria

  • All 12 tests pass — test_leakage_adversarial_naive_join_differs_from_asof is the certificate
  • You can defend the ≤ boundary convention and say when it self-leaks (availability vs occurrence timestamps)
  • You can explain what breaks if online TTL ≠ the TTL used in training joins

Extensions

  • Lag-aware historical joins: join as-of label_ts − materialization_lag to train against what serving actually sees (WARMUP Ch. 5's subtlety, implemented)
  • Multi-view get_historical returning a full training frame
  • A streaming materializer fed by your Phase 02 stream processor's window results

References