Lab 01 — Feature Store with Point-in-Time-Correct Joins

Phase 27 · Lab 01 · Phase README · Warmup

The problem

A feature store looks trivial until you build a training set. Then you meet the single hardest correctness bug in applied ML: label leakage through time.

Each training row has an event timestamp — the instant the label happened (a transaction was approved, a user churned). The features for that row must be the values that were known at that instant. If your join reaches for the current value of a feature — or any value stamped after the event — you have trained the model on information it will never have at inference time. It scores beautifully offline and falls apart in production. Nobody sees it in code review, because the join looks right; it's only wrong along the time axis.

The fix is the point-in-time-correct join: for each (entity, event_time), pick the latest feature value whose feature timestamp is <= event_time, and never anything newer. Layer on a TTL (a feature older than some age is stale and shouldn't be served), a materialize step that pushes the latest values to a fast online store, and the store's whole reason to exist — the offline value the model trains on and the online value it serves on come from one definition, so there is no training-serving skew — and you have the core of Feast / Vertex AI Feature Store / Databricks Feature Store, built from stdlib.

What you build

File map

Run it

pytest test_lab.py -v                       # your lab.py (red until you implement)
LAB_MODULE=solution pytest test_lab.py -v   # the reference (green)
python solution.py                          # worked example

Success criteria

• get_historical_features uses the latest value at-or-before the event time and never a value stamped after it — proven by a test where a later "correction" is not used.
• The boundary is inclusive: a feature stamped exactly at event_time is used; one tick later is not.
• Tie-break is deterministic: two feature rows at the same timestamp resolve to the later-ingested one (last write wins).
• TTL rejects a stale feature (age > ttl → None), and the boundary (age == ttl) is still fresh.
• A cold-start entity (no value before the event) yields None, and features never leak across entities.
• Multiple feature views — including views with different entity keys — join into one training row correctly.
• materialize populates the online store with the latest value per entity; get_online_features serves it, returns None for unknown entities, and applies a serve-time freshness TTL when now is given.
• Training-serving consistency: an offline point-in-time query at/after the newest feature time returns exactly what the online store serves.
• All 30 tests pass under both lab and solution.

How this maps to the real stack

• get_historical_features is Feast's headline API, and the point-in-time join is exactly what Feast (and Tecton, and Databricks/Vertex feature stores) do under the hood: an as-of join (a.k.a. ASOF JOIN in DuckDB/Snowflake, merge_asof in pandas) between your entity/label dataframe and each feature table, keyed on entity and bounded by feature_timestamp <= event_timestamp. Our nested scan is the O(rows·features) honest version of what a warehouse does with a sort-merge as-of join.
• TTL is a real Feast field on a FeatureView: it bounds how far back the as-of join may reach, so a feature that stopped updating doesn't silently serve a months-old value.
• materialize is a real Feast command (feast materialize) that reads the offline store and writes the latest values to an online store (Redis, DynamoDB, Bigtable) for millisecond serving. Our in-memory dict is that online store's mechanism, minus the network.
• Training-serving skew is the failure this whole design prevents, and it is the #1 reason feature stores exist. The online store serving the same value the offline join produced is the literal definition of "no skew" — we assert it directly.
• Entity / feature reference model ("view:feature") mirrors Feast's feature references and Vertex/Databricks feature-lookup syntax; resolving different references to different views (and different entity keys) in one query is how real feature retrieval assembles a wide training row from many sources.

Limits of the miniature. Real stores use actual timestamps (with timezones and late-arriving data), a columnar offline store (Parquet/Delta/BigQuery) with a real as-of join, and a networked online store with its own consistency and TTL-eviction semantics. Ours uses integer ticks, an in-memory list, and a synchronous materialize so the time-correctness mechanism — not the I/O — is the thing on display. We also model one feature value per (entity, timestamp); production handles multiple entities per view (composite keys) and streaming ingestion.

Extensions (your own machine)

• Add an on-demand feature transform: a feature computed at request time from other features (e.g. amount / avg_amount), and prove it produces the same value offline and online.
• Replace the nested scan with a sort-merge as-of join: pre-sort each view's rows by (entity, timestamp) and binary-search the event time, then benchmark against the scan.
• Add composite entity keys (join on (user_id, merchant_id)), matching real fraud-feature views.
• Wire the offline store to DuckDB and re-express the point-in-time join as a real ASOF JOIN, keeping lab.py's pure-Python version as the reference oracle for a differential test.

Interview / resume signal

"Built a feature store with a point-in-time-correct historical join: for each (entity, event-time) it takes the latest feature value at-or-before the event and never a future value, so training sets carry zero label leakage — plus TTL-based staleness, materialization to an online store, and a test that proves offline and online serve the identical value (no training-serving skew). This is the correctness core of Feast / Vertex AI Feature Store / Databricks Feature Store."