« Phase 27 README · Warmup · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 27 — Hitchhiker's Guide

The compressed practitioner tour. Read the WARMUP for the mechanism; this is the stuff you say in the meeting.

30-second mental model

Models are downstream of features; features are downstream of pipelines. Three tools own that territory: Beam/Dataflow (one transform algebra over batch AND streaming — windows, watermarks, triggers; batch is just a bounded stream), the feature store (one feature definition, materialized twice: offline history for training, online latest for serving — joined point-in-time-correctly so training never sees the future), and warehouse ML (BigQuery ML: train where the data lives, with preprocessing stored inside the model so serving can't skew). The two demons this layer exists to kill: label leakage (training on information from after the event) and training-serving skew (two code paths computing "the same" feature differently). The senior move: "I engineer correctness where the leverage is — the join, the window, and the transform."

The concepts to tattoo on your arm

The distinctions that signal seniority

• Skew vs drift → skew is my two code paths disagreeing (fix: one definition); drift is the world changing (fix: monitor + retrain, Phase 26). Confusing them wastes a retrain on a bug, or a bug-hunt on the economy.
• Point-in-time vs nearest-timestamp → "nearest" can reach forward. At-or-before, full stop.
• Event time vs processing time → aggregate by when it happened, not when you saw it. Everything in streaming falls out of this.
• Throttle vs hold → a stalled watermark doesn't drop data, it holds windows open. "Windows stopped firing" = "one source stopped moving."
• Feature store vs database → it's definitions + dual materialization + join semantics; the stores underneath are pluggable.
• Warehouse ML vs platform ML → BigQuery ML for tabular-SQL-batch; Vertex/Databricks (Phase 25) for deep learning, custom architectures, low-latency serving. Routing decision, not loyalty.

The one-liners

# Feast: the two calls that ARE the feature store (Lab 01 builds both)
training_df = store.get_historical_features(          # point-in-time join, offline
    entity_df=labels_df,                               # entity keys + event_timestamps
    features=["user_activity:txn_count_7d", "user_profile:country"],
).to_df()
online = store.get_online_features(                    # latest values, online
    features=["user_activity:txn_count_7d"], entity_rows=[{"user_id": 42}])

# Beam: windowed count — identical code for batch and streaming (Lab 02 builds the engine)
(p | beam.io.ReadFromPubSub(topic)                     # or ReadFromText → same transforms
   | beam.Map(lambda e: (e["user_id"], 1))
   | beam.WindowInto(beam.window.SlidingWindows(600, 60))
   | beam.CombinePerKey(sum))

-- BigQuery ML: train + evaluate + predict without the data leaving (Lab 03 builds the verbs)
CREATE MODEL ds.churn TRANSFORM(ML.STANDARD_SCALER(tenure) OVER () AS t, platform, churned)
OPTIONS(model_type='logistic_reg', input_label_cols=['churned']) AS
SELECT tenure, platform, churned FROM ds.users
WHERE MOD(ABS(FARM_FINGERPRINT(CAST(id AS STRING))), 10) < 8;
SELECT * FROM ML.EVALUATE(MODEL ds.churn);
SELECT * FROM ML.PREDICT(MODEL ds.churn, TABLE ds.new_users);

War stories

• The fraud model that aced backtesting and face-planted on launch. Training joined "current account standing" onto historical transactions — and standing had been updated by the fraud team after each chargeback. The feature was the label wearing a trench coat. Offline AUC 0.97; production coin-flip. The fix was boring: append-only feature log + point-in-time join. The lesson wasn't the join — it was that nothing in evaluation catches leakage, because the validation set is contaminated identically.
• The "same" feature, twice. Warehouse SQL computed txn_count_7d with BETWEEN event_ts - 7d AND event_ts; the serving path counted rows in a Redis list trimmed by a cron job that ran... mostly. Two definitions drifted 4% apart, model quality sagged for a month before anyone looked. Feature store materialization made both paths one definition — and the online/offline consistency check (Lab 01's test) became a CI gate.
• The dashboard that "lost" a day of revenue. A mobile SDK buffered events offline and uploaded a day late — behind the watermark, past allowed-lateness, silently dropped. Streaming counts disagreed with the nightly batch by exactly the late data. Fixes: longer allowed-lateness on money-adjacent pipelines, dead-letter for dropped-late elements, and the batch job kept as the reconciliation source of truth.

Vocabulary

feature view / entity / event timestamp · offline store / online store / materialization · point-in-time (as-of) join · TTL / freshness · on-demand feature · label leakage · training-serving skew · PCollection / PTransform / ParDo / GroupByKey / CombinePerKey · event time vs processing time · fixed / sliding / session windows · watermark / trigger / allowed lateness / accumulating vs discarding · exactly-once / idempotent sink · lakehouse / Delta Lake / transaction log / time travel · medallion (bronze/silver/gold) · Photon · shuffle / skew / salting / broadcast join / partition pruning · CREATE MODEL / TRANSFORM / ML.PREDICT / ML.EVALUATE · FARM_FINGERPRINT split · expectations / data contract · Airflow backfill / dbt model.

Beginner mistakes

1. Joining "latest" feature values onto historical training rows — leakage, the classic.
2. Fitting the scaler/encoder on the full dataset before splitting — test statistics bleed into training.
3. Target-encoding a categorical with in-fold label means — leakage in feature's clothing.
4. Treating suspiciously high offline metrics as good news instead of a leakage tripwire.
5. Aggregating by processing time and wondering why replays change the numbers.
6. Two codebases for batch and streaming "because they're different" — that's how skew is born.
7. No TTL on fast features — a stalled pipeline silently serves last Tuesday forever.
8. Random train/test splits re-rolled every run — metric deltas between runs become noise.
9. Ignoring the one straggler task — that's a hot key, and it's eating your cluster.
10. Calling it "the Databricks database" or "Delta the engine" — it's a table format; precision here is a shibboleth.