Warmup Guide — Feature Stores & Training-Serving Skew

Zero-to-expert primer for Phase 03: why feature platforms exist, the exact semantics of point-in-time correctness, the taxonomy of training-serving skew, and the parity machinery that keeps offline and online worlds honest.

Table of Contents

Chapter 1: The Problem Feature Stores Solve

From zero: a feature is a model input computed from raw data — "user's order count, last 30 days." Without shared infrastructure, every team computes features twice: once in a batch job for training, once in service code for serving. The twice is the disease:

  • The two implementations drift (different languages, different engineers, different bug fixes) → skew (Ch. 6).
  • Training-set construction re-solves the hardest problem — what was this feature's value at each historical label's moment? — per project, usually wrongly (Ch. 3).
  • Features aren't discoverable or reusable; the same "30-day order count" exists five times with five definitions.

A feature store is the deduplication of this work: define once (feature view as code), materialize twice (offline history for training, online latest for serving), guarantee the seams (point-in-time correctness offline, freshness online, parity between them). The JD names it because owning this layer is what "end-to-end ML systems" means in practice.

Chapter 2: The Anatomy — Definitions, Offline, Online

The standard architecture (Feast/Tecton/SageMaker all rhyme):

  • Entity: the join key — user_id, (user_id, merchant_id).
  • Feature view: a named set of features over an entity, sourced from a timestamped table/stream, with metadata: event timestamp (when the fact became true — Phase 02's event time), optional TTL (how long a value stays valid), owner, version.
  • Offline store: the full timestamped history (warehouse/lake) — the substrate for point-in-time training joins.
  • Online store: key→latest-value (Redis/DynamoDB-class) — single-digit-ms reads at serving time, populated by a materialization job (batch or streaming from Phase 02's pipelines).
  • The two consumption APIs that define the contract: get_historical_features(entity_df_with_timestamps) → training frame (PIT-correct), and get_online_features(entity_keys) → serving vector (fresh-as-materialized).

Everything else (registries, UIs, monitoring) decorates these five pieces — and the lab builds exactly the five.

Chapter 3: Temporal Leakage — the Expensive Bug

The setup: training data is constructed retrospectively. Each row is (entity, label_timestamp, label) and needs feature values. The naive join — "latest feature value per entity" — uses values computed after the label event: information from the future.

Why it's so damaging: the leaked feature is often causally downstream of the label ("orders_30d" computed after the churn date encodes the churn), so offline metrics inflate spectacularly; serving can't access the future, so production delivers the un-inflated reality; and the gap reads as "model degradation," sending teams chasing drift for months. It is the CV track's leakage chapter (Phase 02) in its most virulent form — every feature is a potential leak vector because every feature has a timestamp.

The discipline: training joins are as-of joins (Ch. 4), always; and the test suite contains an adversarial case — a feature table constructed so that a naive join produces detectably different output than a correct one (the lab's test_leakage_adversarial). A feature pipeline without that test is unverified.

Chapter 4: Point-in-Time Joins, Exactly

The semantics, precisely (the lab implements these to the letter):

For each entity row with label timestamp $t$, the joined feature value is the one with the largest event timestamp $\tau$ such that $\tau \le t$ and $t - \tau \le \text{TTL}$ — else null/default.

The decisions hiding in that sentence (each is a real design choice — defend yours):

  • ≤ vs <: can a feature whose event timestamp equals the label instant be used? If the "feature event" is the same business event that generates the label (an order that updates order_count and also triggers the label), ≤ self-leaks. Common resolution: ≤ but with feature timestamps set to availability time (when the value was queryable), not occurrence time — which also absorbs pipeline delay honestly. Know both conventions; state which you use.
  • TTL: without it, a 4-year-old value joins happily ("most recent" ≠ "relevant"). TTL turns staleness into explicit missingness — which the model then handles by a declared default (Ch. 5's skew tie-in).
  • Created vs event time: if values can be backfilled/corrected, the join must use what was known at the time (event time + created time both ≤ t) to reproduce the serving view truthfully — the bitemporal wrinkle; at minimum know it exists.
  • Implementation shapes: sort-merge per entity (pandas merge_asof — the lab's reference oracle), or point-in-time window functions in SQL (ROW_NUMBER() OVER (PARTITION BY entity ORDER BY τ DESC) filtered to τ ≤ t).

Chapter 5: The Online Store — Freshness, TTL, Defaults

The serving side's physics:

  • Freshness = now − event_timestamp of the served value. Its budget is a product decision per feature ("30-day order count" tolerates hours; "items in cart now" tolerates seconds → streaming materialization, Phase 02).
  • The materialization job is a lag source: batch-hourly materialization means online values are 0–60+ min stale — and training data joined as-of exactly will be systematically fresher than serving reality. The honest fix: train against the feature values as serving would have seen them (lag-adjusted as-of joins) or accept and measure the gap. This subtlety is senior-interview gold.
  • Miss policy: key absent (new user) or TTL-expired → serve what? The default must be the same value the training pipeline used for missingness — the median-vs-zero mismatch is the classic skew incident (Extension B). Defaults are part of the feature definition, not the serving code.
  • Read-path engineering: batch lookups per request (one round trip for all entities/views), p99 budgets, hot-key caching — ordinary low-latency engineering (the serving phase deepens it).

Chapter 6: Training-Serving Skew — a Taxonomy

"Skew" is three distinct diseases; name them separately because the fixes differ:

  1. Logic skew: two implementations of "the same" feature differ — language ports, library versions, different null handling, different window arithmetic (calendar-month vs rolling-30d). Fix: one definition executed by both paths (the feature store's reason to exist), plus parity tests (Ch. 7).
  2. Temporal skew: training joins at label time; serving reads materialized-lagged values (Ch. 5) — same logic, different effective timestamps. Fix: lag-aware training joins; freshness monitoring.
  3. Distribution skew: the live entity/feature distribution drifts from the training frame's (new market launches, marketing campaign changes the user mix) — nobody's bug, still your problem. Fix: detection (Ch. 7's PSI) wired to Phase 12's retraining policy.

The Google-coined umbrella ("training-serving skew") hides this trichotomy; the taxonomy is what makes incidents debuggable.

Chapter 7: Parity Testing and Skew Detection

The two instruments the lab builds:

  • Parity testing (logic+temporal skew): for a sample of (entity, timestamp) pairs, compute the feature offline (as-of join into history) and fetch it online (as materialized at that moment — in tests, materialize-then-lookup) and assert equality within tolerance. Run it in CI on every feature-definition change and continuously on a trickle of production traffic (log online values; compare against the offline recomputation nightly). A feature without parity coverage is an unverified rumor — the same epistemics as every "export-verify" ritual in this curriculum.
  • Distribution-skew detection (PSI — Population Stability Index): bin the training distribution; compare serving-log proportions:

$$\text{PSI} = \sum_b (p_b^{serve} - p_b^{train}) \cdot \ln\frac{p_b^{serve}}{p_b^{train}}$$

Folklore thresholds: <0.1 stable, 0.1–0.25 watch, >0.25 act. It's KL symmetrized by use, not statistics-sacred — calibrate per feature against incident history. Phase 12 generalizes this into the full drift/monitoring system; here it guards one seam.

Chapter 8: Build vs Adopt, and Operating Realities

  • What the platforms actually give you: registry + definition language, the PIT join implemented correctly, materialization orchestration, online-store adapters (Feast: open-source, bring-your-own infra; Tecton: managed + streaming transforms; cloud-native ones: integration over features). What you still own: feature quality (Phase 02's gates), defaults/TTL decisions, parity-in-CI, skew monitoring, and cost (online stores at high QPS are real money).
  • Adopt when: multiple teams share features, PIT correctness is currently hand-rolled per project, or online serving exists at all. Don't adopt when: one batch model, no online serving — a warehouse table + merge_asof discipline is the right size (resisting platform-for-platform's-sake is senior judgment).
  • Operating gotchas with scars attached: backfilling a corrected feature (version the view; never mutate history in place — Phase 02 Ch. 8), entity-key cardinality explosions (cross-product entities), and on-call for materialization lag (it's a freshness SLO, treat it like one).

Lab Walkthrough Guidance

Lab 01 — Mini Feature Store, suggested order:

  1. FeatureView registration over a timestamped DataFrame; validate inputs with Phase 01 instincts (required columns, timestamp dtype).
  2. The as-of join — the heart. Implement per-entity sort + searchsorted (or verify against pd.merge_asof as your oracle — differential testing). Get the ≤ boundary and the TTL exactly right; the tests probe both edges.
  3. The adversarial leakage test: construct data where naive-latest ≠ as-of and watch your implementation pass while the naive join (provided as a foil) fails.
  4. Online store: materialize(as_of_time) (latest value per key within TTL) + get_online(keys) with declared defaults on miss.
  5. Parity checker: offline value at time T == online value materialized at T, over a random sample.
  6. PSI skew detector + the deliberate-skew test (inject a shifted serving log; detector must fire; un-shifted must not).

Success Criteria

You are ready for Phase 04 when you can, from memory:

  1. State the as-of join semantics in one precise sentence including TTL, and defend the ≤-with-availability-timestamps convention.
  2. Explain why leakage inflates offline metrics specifically (causal downstream-ness of leaked features) and what the adversarial test looks like.
  3. Recite the three-way skew taxonomy with one fix each.
  4. Explain the materialization-lag subtlety and the lag-aware training answer.
  5. Write the PSI formula and its use-not-sacred thresholds.
  6. Argue build-vs-adopt for two scenarios (multi-team online recsys; single batch churn model).

Interview Q&A

Q: Offline AUC is 0.92; the deployed model performs like 0.71. Walk through your investigation. Leakage first (the magnitude says so): audit every feature's timestamp semantics — recompute the training frame with a strict as-of join and re-measure offline; a big drop confirms it (the classic: a feature computed post-label). If offline survives: skew taxonomy in order — parity-test logic (sample entities, compare offline recompute vs logged online values), check temporal skew (materialization lag vs training-join freshness), then distribution skew (PSI on serving logs vs training frame). The structure — leakage, then logic, then temporal, then distribution — is the answer.

Q: Why isn't get_latest(entity) an acceptable training join even with "fresh" data? Because training rows have different timestamps: latest-as-of-now joins tomorrow's feature values onto last month's labels — future information by construction. Each label gets the value as of its own moment. The only case where latest is fine is when feature values never change — at which point the timestamp discipline costs nothing anyway. (Then sketch the merge_asof.)

Q: Your online store misses 8% of lookups (new users). What's the full story of handling that correctly? Declare the default in the feature definition (not service code); ensure the training pipeline produced the identical value for missing history (the median-vs-zero incident is exactly this failure); consider a "missingness indicator" feature so the model can learn the new-user pattern explicitly (CV-track Phase 02's imputation lesson); monitor miss rate as a first-class metric (a step change means a materialization outage or an entity-key bug, not suddenly more new users).

References

  • Feast documentation — read "point-in-time joins" and TTL semantics against your implementation
  • pandas merge_asof docs — your oracle's exact semantics
  • Sculley et al., Hidden Technical Debt in ML Systems (2015) — the pipeline-jungle and skew sections
  • Uber Michelangelo and [Palette feature store] writeups — the original production accounts
  • Tecton blog: What is a Feature Store — vendor-flavored but precise on anatomy
  • Breck et al., The ML Test Score — the parity-testing rubric items
  • CV track Phase 02 WARMUP — the leakage taxonomy this chapter specializes