Design 01 — Feature Store & ML Data Platform

"Design the data layer for ML at a company where 30 teams ship models." Phases 01–03 and 08 built every component; capstone 02 assembled them.

1. Requirements

Functional: teams define features once and consume them in training and serving; training sets are point-in-time correct; online reads at single-digit ms; batch + streaming sources; feature discovery/reuse across teams.

Non-functional: online p99 ≤ 10 ms at 50k QPS aggregate; offline frames over 2+ years of history; freshness from daily (batch) to seconds (streaming); correctness > everything (a silently skewed feature is worse than an outage).

The two problems this platform exists to solve (state them up front): training-serving skew and point-in-time correctness. Every architecture decision serves one of them.

2. Architecture

                      feature definitions (code, in a registry)
                                   │ one definition → both paths
            ┌──────────────────────┼──────────────────────┐
   batch sources (warehouse)  streaming sources (Kafka)   │
            │                      │                      │
   batch materialization     stream processor             │
   (orchestrated DAG, P08)   (windows/watermarks, P02)    │
            │                      │                      │
            ▼                      ▼                      ▼
   OFFLINE STORE  ◄── parity checker ──►  ONLINE STORE   metadata/
   (columnar, full history)              (KV, latest)    lineage
            │                                  │
   point-in-time training frames        serving reads
   (as-of joins, P03)                   (feature vectors at p99 ≤ 10ms)

3. Deep dives (where interviews go)

Point-in-time joins (the make-or-break): every feature row carries an effective timestamp; a training frame for (entity, t_label) joins each feature's latest value with effective_ts ≤ t_label − serving_lag. The serving-lag term is the subtle part — training must see what serving would have had, including pipeline delay. Implementation: validity intervals (effective_ts, expiry) + as-of join (the Phase 03 lab); at warehouse scale, ASOF JOIN / window functions over partitioned history.

Skew defenses, ranked: (1) single definition — the registry's feature code generates both materializations; two codepaths is the original sin; (2) continuous parity sampling — online value vs offline recompute for sampled (entity, ts), alert on divergence rate (the Phase 03 detector); (3) the strongest form: log-and-wait — log the feature vector actually served, train on the logs; right answer for request-scoped features (model inputs at decision time), at the cost of waiting for history to accumulate. Offer all three; recommend registry + parity for shared features, log-and-wait for per-request context.

Online store: KV (Redis/Dynamo-class), key = (entity_type, entity_id), value = latest vector per feature group, TTLs matched to freshness contracts. Hot-key mitigation (popular items): client-side caching with short TTL. Offline store: lakehouse tables partitioned by date, append-only, time travel — the data side of reproducibility (Phase 08 Ch. 9).

Streaming features (velocity counters, session aggregates): the Phase 02 machinery — event-time windows, watermarks for lateness, idempotent updates. State the freshness-vs-completeness tradeoff explicitly per feature: a fraud counter takes the watermark early (freshness wins); a billing aggregate waits.

4. The platform-product layer (what makes it senior)

• Registry as the contract: definitions versioned, owned, documented, searchable; consumption tracked (who uses feature X — the deprecation question, Phase 11 PMC-style lifecycle).
• Data contracts at every ingestion (Phase 01): schema/range/volume gates; violations quarantine, never silently drop.
• Lineage (Phase 08): feature → sources, model → features; the taint query ("this upstream table was wrong Tuesday") is an incident-response requirement, not a luxury.
• Cost governance: materialization compute is the platform's bill; usage tracking retires dead features (the most common feature-store pathology: write-only features nobody reads).

5. Failure modes (volunteer these)

6. Evolution

Streaming-first (batch as backfill of the stream); embedding features (vector columns + ANN serving — converges with Design 02's infrastructure); cross-team feature marketplace with quality tiers; push-down of PIT joins into the warehouse engine as data volumes grow.