« Phase 27 README · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 27 Warmup — Data & Feature Engineering at Scale

Who this is for: someone who has built agent infrastructure (Phases 00–17), framework internals (18–24), cloud ML platforms (Phase 25) and MLOps (Phase 26) — and now needs the layer all of it eats from: the data and feature pipelines. By the end you will be able to explain, from first principles, how a feature store prevents label leakage and training-serving skew, how the Beam/Dataflow model unifies batch and streaming under windows and watermarks, how the Databricks lakehouse and Delta Lake actually work, when warehouse-native ML (BigQuery ML) is the right call, and how features are engineered, validated, and moved at scale — with the mechanism, not the marketing. No cloud account, no Spark cluster, no network — everything in the labs is mechanism.

Table of Contents

  1. The data-to-feature-to-model-to-serving lifecycle
  2. Batch vs streaming processing
  3. The Beam/Dataflow model: pipelines, PCollections, PTransforms
  4. Windowing: fixed, sliding, session
  5. Watermarks, triggers, and exactly-once
  6. The feature store: why it exists
  7. Point-in-time-correct joins: the leakage problem
  8. Freshness, TTL, and on-demand vs precomputed features
  9. The lakehouse: Spark, Delta Lake, and the medallion architecture
  10. BigQuery ML and warehouse-native ML
  11. Feature engineering techniques
  12. Data quality and validation
  13. Large-scale processing patterns: partitioning, shuffle, skew
  14. Orchestration: Airflow and dbt
  15. Common misconceptions
  16. Lab walkthrough
  17. Success criteria
  18. Interview Q&A
  19. References

1. The data-to-feature-to-model-to-serving lifecycle

Every production ML system, whatever it predicts, is the same pipeline wearing different clothes:

raw events ──► ingestion ──► transformation ──► features ──┬──► offline: training sets ──► model
 (clicks,      (batch loads,  (clean, join,      (defined     │
  txns, logs)   CDC, streams)  aggregate)         once)       └──► online: serving lookups ──► predictions

Raw events land from applications, databases (via change-data-capture), and logs. Ingestion gets them into a processable substrate — object storage, a warehouse, a message bus. Transformation cleans, deduplicates, joins, and aggregates them. The output that matters to ML is features: named, typed, versioned values describing an entity at a moment in time ("user 42's transaction count over the trailing 7 days, as of Tuesday 14:03").

The subtlety that shapes this whole phase: features have two consumers with opposite needs. Training needs historical feature values — thousands of entities, each as of a different past moment — read in bulk from an offline store where latency is irrelevant but time-correctness is everything. Serving needs the current value for one entity in single-digit milliseconds from an online store where history is irrelevant but freshness is everything. Nearly every data-side ML failure is a violation of one of two invariants across those consumers:

  • Correctness through time (offline): a training row must only see feature values that existed at its event time — §7.
  • Consistency across paths (offline vs online): the value the model trained on and the value it serves on must come from the same definition — §6.

Downstream of features sit the layers previous phases built: training and serving platforms (Phase 25), and the tracking/registry/CI/CD/drift loop (Phase 26). Monitoring closes the circle: drift detected in serving traffic (Phase 26) usually sends you here, to the pipeline, not to the model.

2. Batch vs streaming processing

Batch processes a bounded dataset: the input is complete before you start, so you can sort it, scan it twice, and know when you are done. Streaming processes an unbounded dataset: the input never ends, so "done" doesn't exist — you can only be done with a region of time. That single difference (bounded vs unbounded) generates every other contrast people recite: batch optimizes throughput and cost (process a day of data in one efficient pass at 2 a.m.); streaming optimizes latency (a fraud feature that updates within seconds of the transaction).

Two clocks run through every discussion of streaming, and confusing them is the classic junior mistake:

  • Event time — when the thing happened (the transaction's timestamp).
  • Processing time — when your pipeline saw it (arrival at the worker).

Events arrive out of order and late — a phone that was offline uploads yesterday's events today. Any aggregation that means anything to the business ("transactions between 14:00 and 14:10") is over event time, which immediately raises the question §5 answers: how long do you wait for stragglers before declaring a time region complete?

Historically, teams that needed both modes ran the Lambda architecture: a batch pipeline for correct-but-slow results and a parallel streaming pipeline for fast-but-approximate ones, merged at query time — two codebases computing "the same" feature, which is training-serving skew's favorite breeding ground. The Kappa architecture counter-proposal: keep only the stream, and reprocess history by replaying the log. The Dataflow model (§3) is the synthesis the industry converged on: one set of transforms with windowing and watermark semantics, where a bounded input is just a stream whose watermark jumps to infinity when the file ends. Batch is a special case of streaming — Lab 02 proves this with an assert.

3. The Beam/Dataflow model: pipelines, PCollections, PTransforms

Apache Beam is the open-source SDK of the Dataflow model (from Google's FlumeJava and MillWheel lineage, published as the 2015 VLDB "Dataflow Model" paper); Google Cloud Dataflow is its fully managed runner. The same Beam pipeline also runs on Flink or Spark runners — the model is the portable part.

  • A Pipeline is a DAG of transforms.
  • A PCollection is an immutable, potentially unbounded collection of elements. The crucial design decision: every element carries an event timestamp, and, after windowing, a window assignment. Data and when it happened travel together — that is what makes event-time processing composable.
  • A PTransform turns PCollections into PCollections. The workhorse is ParDo — "parallel do" — an arbitrary per-element function that can emit zero, one, or many outputs. Map (1→1), FlatMap (1→N), and Filter (1→0/1) are ParDo with convenience wrappers, which is exactly how Lab 02 implements them.
  • GroupByKey is the shuffle: it collects every value sharing a key onto the same worker. It is the expensive primitive — an all-to-all network move — and the one that gives streaming its statefulness. Critically, in a windowed pipeline GroupByKey groups by (window, key), not by key alone: the same user in two different 10-minute windows is two separate groups.
  • CombinePerKey is GroupByKey plus an associative, commutative combiner (sum, max, count). Associativity is not a nicety: it lets the runner pre-combine partial results on each worker before the shuffle ("combiner lifting"), shrinking network traffic from one-element-per-input to one-accumulator-per-worker. When an interviewer asks "why does Beam want a CombineFn instead of a lambda over the whole list," this is the answer.

What the managed runner (Dataflow) adds is ops, not semantics: it fuses adjacent ParDos into single stages, autoscales workers on backlog, rebalances stragglers dynamically, checkpoints state, and upgrades pipelines in place. The semantics — windows, watermarks, triggers — are the model's, and they are what Lab 02 builds.

4. Windowing: fixed, sliding, session

You cannot aggregate an infinite stream; you can only aggregate windows of it. Windowing assigns each element, by its event timestamp, to one or more finite time regions; every keyed aggregation then happens per (window, key).

  • Fixed (tumbling) windows partition the timeline into equal, non-overlapping intervals: with size 10, event time 7 falls in exactly [0,10), event time 12 in [10,20). One event, one window. Right for "per-minute request counts," billing periods, hourly rollups.
  • Sliding windows have a size and a period; a new window starts every period. When period < size the windows overlap, and one event lands in \(\lceil size/period \rceil\) windows: with size 10 / period 5, event time 7 belongs to [0,10) and [5,15). This is how you compute "the trailing 10 minutes, refreshed every 5" — the moving averages and rolling counts that dominate fraud and recommendation features. The overlap is a real cost multiplier: each element is processed once per window it belongs to.
  • Session windows are per-key and data-driven: a session extends while events for that key keep arriving within a gap of each other, and closes after gap of silence. Unlike fixed and sliding windows, sessions merge — two provisional windows that touch become one — which makes them the mechanically hardest windowing type (and a deliberate extension in Lab 02, not part of the core).

Windowing is assignment only — it labels elements with time regions. It says nothing about when results are emitted. That is the trigger's job, and triggers need a clock that measures event-time progress: the watermark.

5. Watermarks, triggers, and exactly-once

The watermark is the runner's moving estimate of event-time completeness: "I believe I have now seen (nearly) all events with timestamps earlier than T." When the watermark passes the end of a window, that window is probably complete and can fire. Two things to be precise about:

  • A watermark is a heuristic for most real sources. A perfect watermark exists only when the source can promise ordering (e.g. a log with monotonic timestamps). For a Pub/Sub topic fed by mobile devices, the watermark is a statistical guess — and events behind the watermark ("late data") will happen.
  • The watermark advances with the slowest input. One stalled partition holds back the watermark for the whole pipeline, which is why "my windows stopped firing" usually means "one source stopped moving," not "the pipeline is broken."

Triggers decide what to do as the watermark and the data evolve. The default — AfterWatermark — fires a window once, when the watermark passes its end; that is exactly what Lab 02's StreamingRunner.advance_watermark implements, including firing each (window, key) exactly once and refusing a backwards watermark. Real Beam lets you trade latency against completeness around that default: early firings (emit speculative partial results every N seconds while the window is still open), late firings (re-emit when late data arrives within an allowed_lateness horizon), and an accumulation mode — does a re-fire emit the corrected total (accumulating) or just the delta (discarding)? Downstream consumers must know which they are getting; billing pipelines and dashboards want different answers.

Exactly-once is the most abused term in streaming. It does not mean each message is delivered exactly once over the network (impossible in general); it means each element's effect on the result is applied exactly once, even across retries, worker crashes, and redeliveries. Dataflow achieves this internally by checkpointing shuffle state and deduplicating retried work by unique IDs. But the guarantee is only as good as the pipeline's edges: a sink that appends blindly turns internal exactly-once into external at-least-once. End-to-end exactly-once requires an idempotent or transactional sink — write keyed by a deterministic ID so replays overwrite instead of duplicate, or commit results and offsets atomically. Same lesson as Phase 08's durable workflows: replays are inevitable; make effects idempotent.

6. The feature store: why it exists

A feature store is the answer to three failures that every ML team meets independently, usually in this order:

  1. Training-serving skew. The training pipeline computes txn_count_7d in SQL over the warehouse; the serving path computes it in application code over a cache. The two disagree — different time zones, different null handling, different dedup rules — and the model sees a feature distribution in production that it never saw in training. Accuracy quietly degrades, and nothing errors. Google's Rules of ML dedicates multiple rules to exactly this failure. The fix is structural: one feature definition, materialized to both paths.
  2. Reuse. The fifth team to need "account age in days" writes the fifth subtly different implementation. A registry of named, owned, documented features turns that into a lookup.
  3. Governance. Which models consume this feature? Can we delete this column (GDPR)? What is its freshness SLA? Without a registry, these are archaeology; with one, they are queries.

The anatomy, common to Feast, Tecton, Vertex AI Feature Store, Databricks Feature Store, and SageMaker Feature Store:

  • A registry of feature views (Lab 01's FeatureView): each view binds an entity (the join key — user_id, merchant_id), a set of feature columns, a timestamp column, and metadata like TTL and ownership.
  • The offline store — the full timestamped history of feature values, in the warehouse/lake (BigQuery, Snowflake, Delta/Parquet). Serves get_historical_features for training-set construction (§7).
  • The online store — the latest value per entity, in a low-latency KV store (Redis, DynamoDB, Bigtable). Serves get_online_features on the request path.
  • Materialization — the scheduled process (Feast's feast materialize) that copies the newest offline values into the online store. Lab 01 builds all four pieces.

The store does not compute features — pipelines (Beam/Dataflow, Spark, dbt) do that and write to the store. The store's job is definitions, storage duality, and the join semantics that make both consumers correct.

7. Point-in-time-correct joins: the leakage problem

This is the section to over-learn: it is the #1 correctness problem a feature store solves and a near-guaranteed senior interview question.

A training set starts as an entity dataframe: one row per labeled event — an entity key, an event timestamp, and the label. The task is to attach feature values. The wrong way is a plain join against the current feature table: that stamps today's values onto last month's events. The subtle-wrong way is joining the nearest timestamp. The right way is the point-in-time join (an as-of join): for each entity row, take the feature value with the largest feature timestamp that is ≤ the event timestamp — the newest value that was actually known at the moment the event happened — and nothing newer, ever.

A concrete timeline, the one Lab 01's tests enforce:

feature: user_tier   t=0: "silver"        t=100: "gold"
event:   user churns at t=50, label=1

correct  (as-of t=50):  tier="silver"     ← what the world knew at t=50
leakage  (latest):      tier="gold"       ← a value from the FUTURE of the event

Why "leakage"? Because the future value often encodes the label. Suppose users get upgraded to "gold" as part of a churn win-back campaign — then tier="gold" at training time is a proxy for "this user churned," the model learns to read it, scores spectacularly in offline evaluation... and collapses in production, where at prediction time the upgrade hasn't happened yet. That is label leakage through time: information from after the event contaminating the features. It is vicious precisely because it improves offline metrics — your validation set is contaminated the same way — so nothing warns you until production.

Mechanically, warehouses and libraries implement the as-of join as a sort-merge: sort both sides by (entity, timestamp), and for each entity row binary-search the feature side. DuckDB and Snowflake expose it as ASOF JOIN; pandas as merge_asof; Feast compiles get_historical_features into exactly this against BigQuery/Snowflake/Spark. Lab 01 implements the same semantics as an explicit scan with a deterministic (timestamp, ingestion_seq) tie-break — small enough to read, strict enough to test the boundaries (a value stamped exactly at the event time is usable; one tick later is not).

Two adjacent leaks to name in an interview: re-fit leakage — fitting scalers/encoders on the full dataset (including test rows) before splitting, so test-set statistics bleed into training (§10's TRANSFORM and Lab 03 address this); and post-event mutation — a "raw" table that gets updated in place (a chargeback flag written onto the original transaction row), destroying the historical values a point-in-time join needs. The cure for the latter is append-only, timestamped feature logs — which is why feature stores ingest timestamped rows, never updates.

8. Freshness, TTL, and on-demand vs precomputed features

A feature value is an assertion about the world at a time. The further you get from that time, the weaker the assertion — and for fast-moving features ("transactions in the last 10 minutes"), an old value is not just weak but wrong. Feature stores encode this as TTL (time-to-live) on the feature view, and it bites in both stores:

  • Offline: the point-in-time join finds the newest value at-or-before the event time — but if that value is older than the TTL, it returns null instead. Better for the model to learn from an honest "unknown" than from a stale count masquerading as current.
  • Online: at serve time, a materialized value whose age exceeds the TTL is withheld. The operational consequence is deliberate: a stalled materialization pipeline makes features disappear (visibly, alarms fire) rather than silently serving three-day-old values (invisibly, the model degrades). Lab 01 implements both checks and tests both boundaries.

Freshness also drives the precomputed vs on-demand split:

  • Precomputed (materialized) features are calculated by pipelines ahead of time and looked up at serving. Cheap and fast to serve, arbitrarily expensive to compute — but always as stale as the pipeline interval. This is the default for aggregations over history.
  • On-demand features are computed at request time from data only available then — the current transaction's amount, its ratio to the user's (precomputed) 7-day average, a distance between the request's coordinates and the account's home. Zero staleness, but the computation must run inside the serving latency budget, and — the skew trap again — the same function must be applied when building training sets. Feast's on-demand feature views and Tecton's realtime features exist precisely to keep that function single-sourced.
  • Streaming features (a Beam/Flink job updating the online store within seconds) sit between the two: near-fresh aggregations at the cost of running a streaming pipeline — the Lab 02 → Lab 01 composition in the phase README's integrated scenario.

9. The lakehouse: Spark, Delta Lake, and the medallion architecture

The 2010s split data infrastructure in two: data lakes (files on object storage — cheap, schemaless, ML-friendly, and consistency-free) and warehouses (SQL engines — transactional, governed, fast, and historically closed/expensive for ML-scale raw data). The lakehouse is Databricks' synthesis: keep the data in open formats on object storage, add a transactional table layer on top, and run both SQL analytics and ML from one copy.

Delta Lake is that table layer: Parquet data files plus a transaction log (the _delta_log directory — ordered JSON commits, periodically compacted into checkpoints). Every write is a new set of data files plus an atomic log entry; readers reconstruct the table state from the log. From that one mechanism falls out everything Delta is known for:

  • ACID on object storage — concurrent writers use optimistic concurrency on the log; readers never see partial writes.
  • Time travel — old log entries and files persist (until vacuumed), so SELECT ... VERSION AS OF 42 reads the table as it was. For ML this is quietly enormous: a training set defined as (query + table version) is reproducible — Phase 26's lineage story, provided by the storage layer.
  • Schema enforcement and evolution — writes that don't match the schema fail (or evolve it explicitly), killing the silent-corruption failure mode of raw file dumps.
  • MERGE / upserts, OPTIMIZE and compaction, Z-ORDER clustering — the operational toolkit for CDC ingestion and read performance.

Spark is the engine that processes it: a driver plans a DAG of transformations over partitioned data; stages split at shuffle boundaries (every wide operation — groupBy, join — is a shuffle, same as Beam's GroupByKey); executors run tasks per partition. Photon is Databricks' C++ vectorized re-implementation of the Spark SQL execution layer — same API, several-fold faster scans and joins, no code changes. Structured Streaming expresses the Dataflow ideas (§3–§5) in Spark's dialect: an unbounded table, micro-batch execution, watermarks for state eviction.

The medallion architecture is the lakehouse's data-quality layering convention:

  • Bronze — raw, append-only, as-ingested (schema-on-read, full fidelity; your replay/audit layer and the append-only substrate §7's history needs).
  • Silver — cleaned, deduplicated, conformed, joined; typed tables with enforced schemas — the layer feature pipelines read.
  • Gold — business-level aggregates and feature/serving tables — what dashboards, analysts, and the feature store's offline tables consume.

The point is not the names; it is the contract per layer (bronze: nothing dropped; silver: quality gates passed — §12; gold: business definitions applied) and cheap reprocessing — bug in silver logic? Rebuild silver from bronze; the raw history is still there.

Databricks Feature Store rounds out the JD line: feature tables are Delta tables registered in Unity Catalog (governance and lineage for free), synced to online stores for serving — and its signature anti-skew move is packaging feature lookups with the MLflow model: the logged model records which features to fetch, so the serving endpoint fetches them itself and callers cannot pass hand-computed (skewed) values.

10. BigQuery ML and warehouse-native ML

BigQuery ML's premise: for tabular ML on data that already lives in the warehouse, the highest cost is not training — it is the export pipeline: moving data out to a training environment, keeping the copy fresh, re-implementing preprocessing at serving, and securing every hop. So: train where the data is, in SQL.

CREATE OR REPLACE MODEL ds.churn_model
TRANSFORM(
  ML.STANDARD_SCALER(tenure_months) OVER () AS tenure_scaled,
  ML.STANDARD_SCALER(monthly_spend) OVER () AS spend_scaled,
  platform,                      -- categoricals are one-hot encoded automatically
  churned
)
OPTIONS(model_type='logistic_reg', input_label_cols=['churned'])
AS SELECT tenure_months, monthly_spend, platform, churned FROM ds.users
WHERE MOD(ABS(FARM_FINGERPRINT(CAST(user_id AS STRING))), 10) < 8;  -- deterministic 80% split

SELECT * FROM ML.EVALUATE(MODEL ds.churn_model,
  (SELECT * FROM ds.users
   WHERE MOD(ABS(FARM_FINGERPRINT(CAST(user_id AS STRING))), 10) >= 8));

SELECT * FROM ML.PREDICT(MODEL ds.churn_model, TABLE ds.new_users);

Three load-bearing details, each built in Lab 03:

  • The TRANSFORM clause is stored inside the model. The scaler's mean/std and the encoder's vocabulary are fit on the training query and embedded; ML.PREDICT re-applies them automatically. A caller physically cannot serve with different preprocessing than training — training-serving skew for the preprocessing step is prevented structurally.
  • The FARM_FINGERPRINT split is the warehouse idiom for reproducibility: a row's train/test side is a pure function of its own key, so the split survives reruns, engine upgrades, and dataset growth — no shuffle, no seed to forget.
  • Deterministic, table-in/table-out verbs: ML.PREDICT returns the input rows plus predicted_* columns; ML.EVALUATE returns a metrics row (precision, recall, accuracy, f1_score, log_loss, roc_auc for classifiers); ML.WEIGHTS and ML.EXPLAIN_PREDICT expose the model's internals to SQL.

Model breadth is wider than people assume: linear/logistic regression, k-means, boosted trees (XGBoost under the hood), DNNs (TensorFlow under the hood), ARIMA_PLUS for time series, matrix factorization — plus imported TensorFlow/ONNX models and remote models that call a Vertex AI endpoint (including LLMs) from SQL. Redshift ML and Snowflake (Snowpark ML) are the same idea on other warehouses.

When it fits: tabular data already in the warehouse; SQL-fluent team; batch or moderate-latency scoring; propensity/churn/LTV/segmentation-class problems. When it doesn't: sub-100ms online serving straight from the warehouse (export the model, or pair with an online feature store — Lab 01), unstructured data at scale, custom architectures, or heavy hyperparameter search — that is Vertex AI / Databricks territory (Phase 25). The senior answer is a routing answer, not a loyalty answer.

11. Feature engineering techniques

The catalog every JD means by "strong understanding of feature engineering" — with the correctness constraint attached to each, because the constraint is the senior part:

Numeric transforms.

  • Standardization (z-score): \( x' = (x - \mu)/\sigma \); min-max scaling to [0,1]. The constraint: \(\mu, \sigma\) are fit on training data only and reused verbatim at serving (Lab 03's Transform) — re-fitting on the serving batch is skew by construction.
  • Log / power transforms for heavy-tailed quantities (spend, counts); clipping to percentile caps so one whale doesn't own the gradient.
  • Binning/bucketing — fixed-width or quantile — converts a numeric into a categorical, letting linear models capture non-linear structure and making the feature robust to outliers; bucket boundaries are training-time artifacts (same constraint).

Categorical encodings.

  • One-hot: one indicator per vocabulary entry; the vocabulary is fixed at training time, and an unseen category at serving must map to all-zeros, not an error (Lab 03 tests this). Fine to ~thousands of categories, then the width hurts.
  • Hashing trick: hash the category into a fixed number of buckets — no vocabulary to store, graceful on unseen values, at the price of collisions. (The same trick as this track's deterministic hashing embedder in Phase 05.)
  • Learned embeddings: for high-cardinality categoricals (user IDs, SKUs, merchants), train a dense vector per category as part of the model — the recommender-systems workhorse.
  • Target encoding (replace the category with the mean label for that category): powerful and dangerous — computed naively it leaks the label into the feature. It must be computed out-of-fold or with smoothing, and is the canonical "leakage hides in clever features" example.

Interactions and time.

  • Feature crosses (country × device_type): explicit interaction terms for models that can't learn them (linear); trees and DNNs largely learn their own.
  • Windowed aggregations — counts, sums, means, distinct-counts over trailing windows — are the highest-value features in most transactional domains, and they are exactly Lab 02's sliding windows feeding Lab 01's store. Their event-time correctness is the whole game: a "7-day count" computed with future rows included is §7's leak wearing an aggregate's clothes.
  • Ratios and deltas ("this transaction ÷ trailing average") — often on-demand features (§8), since they need the current request.

12. Data quality and validation

Feature pipelines fail quieter than services: a schema change upstream doesn't throw a 500, it fills a column with nulls and your model's accuracy sags a week later. The countermeasures are gates between pipeline stages, in escalating sophistication:

  1. Schema validation — types, required columns, nullability, value ranges, referential integrity. Delta's schema enforcement (§9) gives you this at the storage layer; Beam/Spark jobs assert it explicitly at ingestion. Lab 01 and Lab 03 both raise structured SchemaErrors for exactly this class.
  2. Expectations — declarative assertions on data, not just shape, in the style of Great Expectations: expect_column_values_to_not_be_null("user_id"), expect_column_values_to_be_between("amount", 0, 100000), row-count deltas vs yesterday, distinct-count bounds, freshness ("max(event_ts) within 2h of now"). dbt ships the same idea as tests: on models (§14); TFDV (TensorFlow Data Validation) infers a schema and flags anomalies and train/serve distribution divergence.
  3. Distributional checks — compare today's feature distribution against a reference window (population stability index, KL divergence, simple quantile bands). This is input drift detection, the upstream sibling of Phase 26's prediction-drift monitoring — catching it here, before training or materialization, is strictly cheaper.

Where the gates go: bronze→silver (reject/quarantine malformed rows into a dead-letter table — never silently drop), silver→gold (business-rule expectations), before training (the training set itself validated — label balance, leakage tripwires like features perfectly correlated with the label), and before materialization (don't push a broken batch into the online store; a TTL'd store, §8, at least fails visibly if you do). The organizational upgrade is the data contract: the producing team commits to schema and semantics at the source, so the gate moves from your pipeline to their CI.

13. Large-scale processing patterns: partitioning, shuffle, skew

"Large-scale data processing" in a JD decodes to a handful of mechanical realities:

  • Partitioning is the unit of parallelism and of pruning. Storage is partitioned (by date, region: dt=2026-07-04/) so queries touching one day read one directory — partition pruning is why the same query costs 100× less with a WHERE dt = .... Compute is partitioned so N workers each process 1/N of the data — until a shuffle forces them to talk.
  • The shuffle (Beam's GroupByKey, Spark's wide dependencies, SQL's hash join/group-by) is the all-to-all exchange where every worker sends each record to the worker owning its key. It costs network, serialization, and disk spill, and it is where big jobs die. The levers: pre-aggregate before shuffling (combiner lifting, §3), broadcast joins (ship a small table to every worker instead of shuffling the big one), and shuffle less often (partition data by the join key at write time).
  • Skew is the shuffle's pathology: keys are Zipf-distributed, so one worker gets the celebrity user / null key / default merchant and runs for hours while the rest idle. The symptom is unmistakable — "the job is 99% done for 45 minutes." Fixes: salting (split the hot key into key#0..key#k sub-keys, aggregate in two stages), isolating known hot keys onto a broadcast path, or engines' automatic mitigation (Spark AQE's skew-join splitting).
  • Incremental processing — the difference between recomputing the world nightly and processing only what changed: new partitions only, MERGE upserts from CDC feeds, dbt incremental models, streaming as the limit case. The correctness caveat ties back to §5 and §7: late data means yesterday's partition isn't necessarily final, so incremental pipelines reprocess a trailing window (e.g. "last 3 days") — and append-only feature logs make that reprocessing safe.
  • File formats and layout: columnar formats (Parquet) give column pruning, compression, and predicate pushdown (skip row groups whose min/max exclude the filter). The small-files problem — thousands of tiny files from streaming writes — murders scan performance until compaction (OPTIMIZE, §9) fixes it.

14. Orchestration: Airflow and dbt

Someone has to run all of this in the right order, at the right time, with retries — and rerun last Tuesday when a bug is found.

Apache Airflow is the general-purpose orchestrator: pipelines as Python-defined DAGs of tasks with dependencies, a scheduler that runs them per schedule, retries with backoff, sensors that wait on external conditions ("the upstream partition exists"), and backfills — re-running historical intervals. Two disciplines matter for ML correctness: tasks should be idempotent (a retried task must not double-write — same lesson as §5's sinks), and scheduling should be data-interval-aware (a run for July 4 processes July 4's partition, regardless of when it executes — which is what makes backfills meaningful). Cloud Composer is managed Airflow on GCP; Dagster and Prefect are the modern alternatives with first-class data assets.

dbt owns the in-warehouse transformation layer: models are SQL SELECTs materialized as tables/views, ref() builds the dependency DAG, dbt test runs data expectations (§12), incremental models process only new rows, and docs/lineage come from the same code. dbt is how the §9 silver/gold layers are typically built and tested in warehouse-centric stacks.

The division of labor: dbt transforms inside the warehouse; Airflow orchestrates across systems — trigger the Dataflow job, wait for it, run dbt, materialize the feature store (Lab 01's materialize as a nightly task), kick off training (Phase 26's pipeline). For feature correctness, the orchestrator's backfill semantics are load-bearing: backfilling a feature table must write rows timestamped as of their data interval, so §7's point-in-time joins remain truthful about what was knowable when.

15. Common misconceptions

  • "Training-serving skew is data drift." No. Skew is your two code paths computing the same feature differently (a bug you shipped); drift is the world changing under a correct pipeline (a fact you must adapt to). Skew is fixed by unifying definitions (feature store, model-owned TRANSFORM); drift is managed by monitoring and retraining (Phase 26).
  • "A point-in-time join is a join on the nearest timestamp." It is at-or-before, never after. "Nearest" can reach forward — that's leakage with extra steps.
  • "Label leakage would show up in my metrics." Backwards: leakage inflates offline metrics, because the validation set is contaminated identically. Suspiciously good is a symptom.
  • "A feature store is a database." It's a definition registry plus dual materialization plus join semantics. The databases (warehouse offline, KV online) are pluggable parts.
  • "Streaming replaces batch" / "batch and streaming need separate codebases." The Dataflow model's point is that they are one semantics; batch is a bounded stream (Lab 02's equivalence test). Teams still run both cadences — with one definition of the logic.
  • "The watermark guarantees completeness." It's a heuristic estimate. Late data exists; allowed-lateness and accumulation mode are how you handle it, not deny it.
  • "Exactly-once means the network delivers each message once." It means each element's effect is applied once — and it's only end-to-end if the sink is idempotent/transactional.
  • "One-hot everything; embeddings are for text." High-cardinality categoricals want hashing or learned embeddings; one-hot at 10⁶ categories is a memory bomb.
  • "Warehouse ML is toy ML." BigQuery ML's boosted trees are XGBoost; its DNNs are TensorFlow. The real limits are serving latency and unstructured data, not model quality.
  • "Delta Lake is a database engine." It's a table format — Parquet plus a transaction log on object storage. Engines (Spark, Trino, DuckDB) read it; that openness is the point.

16. Lab walkthrough

Build the three miniatures in order; each isolates one layer, everything is stdlib, offline, and deterministic (integer tick timestamps, hashed splits, zero-init training — no wall clock, no random seed to forget).

  1. Lab 01 — Feature Store (the star). FeatureView with entity/features/TTL, offline ingestion of timestamped rows, get_historical_features doing the point-in-time join (at-or-before, TTL-checked, deterministic tie-breaks), materialize, and get_online_features with serve-time freshness — plus the training-serving-consistency test. 30 tests.
  2. Lab 02 — Dataflow Pipeline. PCollection with Map/Filter/FlatMap/GroupByKey/CombinePerKey (grouping per (window, key)), FixedWindows and overlapping SlidingWindows, and a StreamingRunner whose watermark fires each window exactly once — ending in the batch==streaming equivalence proof. 30 tests.
  3. Lab 03 — Warehouse ML. CREATE MODEL as deterministic logistic regression (zero-init full-batch gradient descent), the model-owned TRANSFORM (z-score + one-hot, fit on train only, unseen categories → all-zeros), ML.PREDICT / ML.EVALUATE with hand-checkable metrics, and FARM_FINGERPRINT-style hash splits. 32 tests.

Run each with LAB_MODULE=solution pytest test_lab.py -v first (green reference), then fill your lab.py to match, then read solution.py's main() output.

17. Success criteria

  • You can define training-serving skew and label leakage — and the difference between skew and drift — without hesitating.
  • You can draw the point-in-time join on a timeline and state the at-or-before rule in one sentence.
  • You can explain offline vs online stores, TTL in both, and what materialization does.
  • You can define PCollection, ParDo, GroupByKey, and why CombinePerKey wants associativity.
  • You can explain fixed/sliding/session windows and compute how many windows an event lands in.
  • You can say what a watermark is, what fires a window, and what happens to late data.
  • You can sketch Delta Lake's transaction log and the medallion layers, and say what each buys.
  • You can write a CREATE MODEL with TRANSFORM from memory and explain why TRANSFORM prevents skew.
  • You can diagnose a skewed shuffle from its symptom and name two fixes.
  • All three labs pass under both lab and solution (92 tests total).

18. Interview Q&A

Q: What is training-serving skew, and how do you prevent it? A: The model trains on features computed by one code path (batch SQL over the warehouse) and serves on features computed by another (application code at request time); any disagreement — null handling, time zones, windows — means the serving distribution differs from training and quality quietly degrades. Prevention is structural, not disciplinary: one feature definition materialized to both offline and online stores (feature store), preprocessing fit at training and stored inside the model (BigQuery ML's TRANSFORM, sklearn Pipelines), or the model fetching its own features so callers can't pass hand-computed ones (Databricks Feature Store's model packaging). And distinguish it from drift: skew is my bug; drift is the world changing.

Q: Walk me through a point-in-time-correct join. Why not just join the latest feature values? A: Each training row has an entity and an event timestamp. For each row, take the feature value with the largest feature timestamp that is at-or-before the event timestamp — never anything later — optionally rejecting values older than a TTL. Joining "latest" stamps today's values onto historical events: if the feature moved because of the label (a tier upgrade triggered by churn), the model learns to read its own answer key, offline metrics inflate, and production collapses. That's label leakage, and it's invisible in evaluation because the validation set is contaminated identically.

Q: What is a watermark, and what happens to data that arrives behind it? A: The runner's moving estimate of event-time completeness — "I've probably seen everything earlier than T." The default trigger fires a window when the watermark passes its end. It's a heuristic, so late data happens: within the configured allowed-lateness you can re-fire a corrected pane (accumulating or discarding mode — downstream must know which); beyond it, the element is dropped and counted. A watermark that stops advancing usually means one stalled source partition, and it holds every downstream window open.

Q: Fixed vs sliding vs session windows — when would you use each? A: Fixed for non-overlapping periodic aggregates (per-minute counts, hourly rollups) — one event, one window. Sliding for trailing-window features refreshed more often than their span ("last 10 minutes, every minute") — one event lands in size/period windows, which multiplies processing cost. Session for activity-burst analytics (user sessions with a 30-minute inactivity gap) — per-key, data-driven windows that merge, which makes them the mechanically hardest.

Q: What does exactly-once actually guarantee in a streaming pipeline? A: That each element's effect on results is applied once despite retries and redelivery — not that the network delivers once, which is impossible in general. Internally the runner gets there with checkpointed state and dedup of retried work; end-to-end it additionally requires an idempotent or transactional sink — deterministic keys so a replayed write overwrites rather than duplicates, or results and offsets committed atomically. Internal exactly-once with an append-blindly sink is at-least-once where it counts.

Q: Why do feature stores have two stores, and how do they stay consistent? A: Opposite access patterns: training reads bulk history as of many past timestamps (offline — warehouse/lake); serving reads one entity's latest values in milliseconds (online — KV). Consistency comes from both being materialized from the same definitions and ingestion stream: materialization copies the newest offline values online, and the invariant worth testing is that an offline point-in-time query "as of now" equals the online read — that equality is the no-skew guarantee (Lab 01 asserts it directly).

Q: Explain the medallion architecture and what each layer is for. A: Bronze: raw, append-only, full-fidelity ingestion — the audit/replay substrate; nothing dropped, schema optional. Silver: cleaned, deduplicated, conformed, schema-enforced — where quality gates run and feature pipelines read. Gold: business-level aggregates and feature/serving tables. The value is the per-layer contract plus cheap reprocessing: a silver bug is fixed by rebuilding from bronze. On Databricks each layer is Delta tables, so you get ACID, schema enforcement, and time travel per layer.

Q: When is BigQuery ML the right choice over Vertex AI or Databricks? A: When the data is already in BigQuery, the problem is tabular (churn, propensity, LTV, segmentation), the team is SQL-fluent, and scoring is batch or moderate-latency — then training where the data lives eliminates the export pipeline, and TRANSFORM kills preprocessing skew. Wrong when you need sub-100ms online serving from the model (export it or pair with an online feature store), heavy unstructured data, custom architectures, or serious hyperparameter search — that's Vertex/Databricks (Phase 25). It's a routing decision by workload, not a platform loyalty test.

Q: Your distributed job is "99% done" for an hour — one task still running. What is it and what do you do? A: Shuffle skew: a hot key (celebrity user, null default, one giant tenant) put a disproportionate share of records on one worker. Confirm from the task-level input-size distribution. Fixes: salt the hot key into k sub-keys and aggregate in two stages; pre-combine before the shuffle; broadcast-join if one side is small; isolate known hot keys to a separate path; or lean on the engine's automatic skew handling (Spark AQE). Also ask whether the null/ default key should be in the aggregation at all.

Q: Precomputed vs on-demand features — how do you decide? A: Precompute anything derived from history (windowed aggregations): cheap to serve from the online store, at the cost of pipeline-interval staleness — bounded by TTL so a stalled pipeline fails visibly. Compute on-demand anything needing the current request (the transaction's amount, its ratio to a precomputed average): zero staleness, but it must fit the serving latency budget and — the skew trap — the identical function must run when building training sets, which is why stores support on-demand feature views as first-class, single-sourced definitions.

Q: How do you make a train/test split reproducible over a growing warehouse table? A: Hash a stable row key and bucket it — MOD(ABS(FARM_FINGERPRINT(CAST(id AS STRING))), 10) < 8 — so a row's side is a pure function of its key: no shuffle order, no seed, and a row never migrates sides as data grows (Lab 03 tests exactly that property). Random splits without a persisted assignment silently reshuffle on every run, which makes metric deltas between runs meaningless.

Q: What does Delta Lake time travel buy an ML team specifically? A: Reproducible training data: a training set pinned as (query + table version) can be re-materialized exactly, which turns "we can't reproduce last month's model" into a lookup — the data half of Phase 26's lineage. It also gives instant rollback from a bad pipeline write and lets you diff feature distributions across versions when debugging drift. The caveat: VACUUM deletes old files, so retention must outlive your reproducibility window.

19. References

  • Akidau et al., The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing (VLDB 2015).
  • Tyler Akidau, Streaming 101 and Streaming 102 (O'Reilly Radar) — the windows/watermarks/triggers canon. https://www.oreilly.com/radar/the-world-beyond-batch-streaming-101/
  • Apache Beam — Programming Guide (PCollections, ParDo, windowing, triggers). https://beam.apache.org/documentation/programming-guide/
  • Google Cloud Dataflow — documentation (managed runner: autoscaling, exactly-once, streaming engine). https://cloud.google.com/dataflow/docs
  • BigQuery ML — Introduction and end-to-end user journey. https://cloud.google.com/bigquery/docs/bqml-introduction
  • BigQuery ML — CREATE MODEL statement and the TRANSFORM clause. https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-create
  • Feast — documentation (feature views, point-in-time joins, materialization, on-demand features). https://docs.feast.dev/
  • Vertex AI Feature Store — documentation. https://cloud.google.com/vertex-ai/docs/featurestore
  • Databricks — Feature Store / Feature Engineering in Unity Catalog documentation. https://docs.databricks.com/en/machine-learning/feature-store/index.html
  • Armbrust et al., Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores (VLDB 2020); Delta Lake documentation. https://delta.io/
  • Databricks — Medallion architecture. https://www.databricks.com/glossary/medallion-architecture
  • Apache Spark — documentation (RDDs/DataFrames, shuffles, Structured Streaming, AQE). https://spark.apache.org/docs/latest/
  • Sculley et al., Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015) — pipeline jungles, entanglement, and why the data layer is the debt.
  • Google — Rules of Machine Learning (training-serving skew rules). https://developers.google.com/machine-learning/guides/rules-of-ml
  • Great Expectations — documentation (expectations, checkpoints, data docs). https://docs.greatexpectations.io/
  • dbt — documentation (models, ref(), tests, incremental models). https://docs.getdbt.com/
  • Apache Airflow — documentation (DAGs, scheduling, backfills, idempotency). https://airflow.apache.org/docs/