« Track Overview · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 27 — Data & Feature Engineering at Scale: Feature Stores, Dataflow & Warehouse ML

Answers these JD lines: "Experience with BigQuery ML and Dataflow for data processing and machine learning workflows," "Experience with Databricks for data engineering, model development, and analytics workflows," "Strong understanding of data preprocessing, feature engineering, and large-scale data processing," "Build and optimize end-to-end ML pipelines for data ingestion." This is the data and feature layer the rest of the ML stack stands on: Phase 25's cloud ML platforms train and serve models, Phase 26's MLOps tracks and promotes them — but every one of those models eats features, and features come from pipelines. Get this layer wrong and no amount of model quality or deployment discipline saves you.

Why this phase exists

Ask a production ML team where their outages come from and the honest answer is rarely "the model." It's the data: a feature computed one way for training and another way for serving, a training set silently joined against future values, a pipeline whose batch and streaming halves drifted apart, a scaler re-fit on serving data. Three ideas — each a lab — do most of the work of preventing that:

1. The feature store and the point-in-time-correct join. One feature definition feeding both training (offline, historical) and serving (online, latest), with the historical join constrained to values known at or before each training row's event time. That constraint is what keeps label leakage out of your training set; the shared definition is what keeps training-serving skew out of your production traffic (Lab 01 — the star lab).
2. The Dataflow model: batch and streaming are one thing. Apache Beam's transform algebra (Map/Filter/FlatMap/GroupByKey/CombinePerKey) plus event-time windowing (fixed, sliding) and a watermark that decides when a window fires. Write the pipeline once; run it over a bounded file or an unbounded stream and get the same answer (Lab 02).
3. Warehouse-native ML with model-owned preprocessing. BigQuery ML's CREATE MODEL / ML.PREDICT / ML.EVALUATE, with a TRANSFORM clause fit on training data and stored inside the model so serving cannot apply different preprocessing — skew prevented structurally, not by code review (Lab 03).

Around the labs, the Warmup covers what interviewers actually probe: the data→feature→model→serving lifecycle, the lakehouse (Databricks, Delta Lake, medallion bronze/silver/gold), feature engineering techniques, data quality gates, shuffle/skew/partitioning at scale, and orchestration (Airflow, dbt).

Dataflow/Beam vs Spark/Databricks vs warehouse SQL ML

Three ways to process data into features and models. Interviewers listen for whether you pick by constraint, not by fashion.

Concept map

• Lifecycle — raw events → ingestion → transformation → features → training sets (offline) and serving lookups (online) → models → predictions, with validation gates between stages.
• Feature store — feature views, entities, offline vs online store, point-in-time joins, TTL/freshness, materialization, on-demand vs precomputed features (Lab 01; Feast, Vertex AI Feature Store, Databricks Feature Store).
• Dataflow model — PCollections/PTransforms, ParDo, GroupByKey as the shuffle, event time vs processing time, fixed/sliding/session windows, watermarks, triggers, exactly-once (Lab 02; Apache Beam, Google Dataflow, Flink).
• Lakehouse — Delta Lake ACID + time travel, medallion architecture (bronze/silver/gold), Photon, Unity Catalog; Spark's shuffle, partitioning, and skew (Warmup §8–§9).
• Warehouse ML — CREATE MODEL, TRANSFORM, ML.PREDICT/ML.EVALUATE, hash-based splits (Lab 03; BigQuery ML, Redshift ML, Snowflake ML).
• Also: feature engineering techniques (encoding, scaling, binning, crosses, categorical embeddings, windowed aggregations), data quality and expectations, orchestration (Airflow/dbt) — covered in the Warmup, not built as labs.

The labs

Integrated scenario (how this shows up at work)

Your team ships a fraud model. It must score transactions in 50 ms, retrain weekly, and — after last quarter's incident — never again train on a feature that "knew" the outcome. Clickstream and transaction events flow through a Dataflow pipeline (Lab 02) that computes sliding-window aggregates ("transactions in the last 10 minutes, updated every minute") keyed by user, with the watermark deciding when each window's aggregate is final. Those aggregates land in the feature store (Lab 01): the offline store keeps every timestamped value, and the weekly training job builds its dataset with get_historical_features — a point-in-time join, so each labeled transaction sees only feature values known at that moment. materialize pushes the latest values to the online store the 50 ms scoring path reads, and the TTL guarantees a stalled pipeline serves nothing rather than three-day-old counts. The analytics team, meanwhile, trains a churn model in the warehouse (Lab 03) — the data's already there, TRANSFORM keeps their scaling consistent, and nobody exports a CSV. The retrained fraud model itself is tracked and promoted through Phase 26's MLOps pipeline and served on Phase 25's platform — this phase is what feeds them.

Deliverables checklist

• Lab 01 green under LAB_MODULE=solution pytest and your own lab.py (30 tests); you can explain point-in-time correctness with a timeline drawing.
• Lab 02 green (30 tests); you can state what a watermark is in one sentence and why sliding windows put one event in several windows.
• Lab 03 green (32 tests); you can explain why TRANSFORM inside the model prevents skew.
• You can define training-serving skew and label leakage without confusing them.
• You can sketch the medallion architecture and say what each layer is for.
• You can argue Dataflow vs Databricks vs BigQuery ML for a given workload from the table above.

Key takeaways

• The point-in-time join is the feature store's reason to exist. "Latest value at-or-before the event time, never after" is one sentence that separates people who've built training sets from people who've read about them.
• Batch is streaming with a bounded source. Windowing + watermark + trigger is the shared vocabulary; Beam/Dataflow, Flink, and Spark are dialects of it.
• Skew dies structurally, not by discipline. One feature definition (store), one transform owned by the model (warehouse ML) — both beat "remember to apply the same scaler."
• The warehouse is a legitimate ML platform for tabular, SQL-adjacent problems — and the wrong one for deep learning or sub-100ms serving. Knowing which is the senior signal.
• The senior framing: "models are downstream of features, and features are downstream of pipelines — so I engineer correctness where the leverage is: the join, the window, and the transform."