« Phase 27 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor

Phase 27 — Staff Engineer Notes: Owning the Data & Feature Layer

Anyone can write a SELECT that joins features onto labels. The gap between using a feature store and being trusted to own the data layer is judgment about four things nobody hands you a default for: whether to build or buy a feature platform, which store backs which access pattern, where to compute each feature, and — the one that gets escalated to you — whether a suspiciously good model is a breakthrough or a leak. This doc is about that judgment and the signal that proves you have it.

The decisions a staff engineer actually owns here

Point-in-time correctness is a platform guarantee, not a code-review habit. The junior stance is "we're careful with our joins." The staff move is to make leakage structurally impossible: an append-only, timestamped feature log (never in-place updates that destroy history), a compiled as-of join every training set is built through, and a leakage tripwire in the training-set validation (a feature perfectly correlated with the label is a red flag, not a jackpot). You own the invariant "latest value at-or-before the event time, never after," and you own making sure no engineer can route around it.

Skew dies structurally or it does not die. "Remember to apply the same scaler" is not a control; it is a hope. The three real controls, in order of strength: one feature definition materialized to both stores (feature store), preprocessing fit at training and stored inside the model (BigQuery ML's TRANSFORM, sklearn Pipelines), and the model fetching its own features so callers cannot pass hand-computed values (Databricks Feature Store's model packaging). You choose which control the platform enforces, and you make it the path of least resistance so the skew-prone path takes more effort than the safe one.

Store selection outlives the model. The model is a config string you can swap; the offline warehouse and online KV store are infrastructure you provision, scale, secure, and pay for. Reuse a store you already operate, keep data residency where compliance needs it, and treat the online store's TTL as a product decision — how stale is too stale for this feature — not a default.

Precompute versus on-demand is a latency-budget decision with a skew trap. Precompute anything derived from history (windowed aggregations): cheap to serve, stale by the pipeline interval, bounded by TTL so a stall fails visibly. Compute on-demand anything needing the current request (a transaction's ratio to a precomputed average): zero staleness, but it must fit the serving latency budget and the identical function must run when building training sets — which is why on-demand feature views exist as first-class single-sourced definitions and not as two copies of a formula.

Warehouse ML versus a training platform is a routing answer. BigQuery ML fits when the data is already in the warehouse, the problem is tabular (churn, propensity, LTV, segmentation), the team is SQL-fluent, and scoring is batch or moderate-latency. It is the wrong tool for sub-100ms serving from the warehouse, unstructured data at scale, custom architectures, or serious hyperparameter search — that is Vertex/Databricks. Naming the axis is the signal; "BigQuery ML is best" is the anti-signal.

The decision framework, out loud

• Event-time streaming correctness matters — sessions, watermarks, late data, sub-minute freshness → Beam/Dataflow (or Flink) feeding the online store.
• Big transformations plus ML on one governed platform, team lives in notebooks and Delta → Databricks lakehouse with the medallion layering and Feature Store.
• Tabular data already in the warehouse, SQL-speaking team, moderate model needs → warehouse-native ML (BigQuery ML / Redshift ML / Snowflake ML).
• You need reproducible training data with lineage → a table format with time travel (Delta) so a training set is (query + version), and retention set to outlive your reproducibility window.
• Two teams keep reimplementing the same feature differently → a feature store, because the problem is definition drift, and a registry is the fix.

Code-review red flags

• A plain equijoin or "nearest timestamp" join building a training set. Both reach into the future — the former always, the latter exactly when the feature moved late. Demand <= event_ts.
• A scaler or encoder .fit() called on the full dataset before the train/test split. Re-fit leakage: test-set statistics bleed into training. Fit on train only, freeze, reuse.
• Target encoding computed in-fold. It leaks the label into the feature. Out-of-fold or smoothed, or not at all.
• A "raw" table updated in place (a chargeback flag written onto the original transaction row). It destroys the historical values a point-in-time join needs. Feature logs are append-only.
• A serving feature computed in application code that also exists as a training SQL expression. Two code paths, guaranteed skew. One definition, two materializations.
• A random train/test split with no persisted assignment on a growing table. It reshuffles on every run and makes metric deltas between runs meaningless. Hash the key.
• A backfill that stamps feature rows with today's timestamp instead of the data interval's. It silently makes every future point-in-time join lie about what was knowable when.
• README.md chapter links, an un-TTL'd fast-moving feature, a streaming sink that appends blindly — small tells the author has not internalized the boundaries.

War stories worth carrying

• The model that was too good. A fraud model scored spectacularly offline and collapsed in production. The feature tier="gold" was set by a churn win-back campaign that ran after the churn — the model had learned to read its own answer key through a point-in-time-incorrect join. Offline metrics never warned because the validation set was contaminated identically. The fix was one predicate; the lesson is that suspiciously good is a symptom.
• The Lambda-architecture drift. A batch pipeline and a streaming pipeline computed "the same" 7-day count with subtly different null handling. They agreed for months, then a schema change moved them apart, and the model degraded with no error and no alarm. Collapsing to one Dataflow definition removed the thing that could drift.
• The stalled materialization nobody noticed. A materialization job silently failed for three days; because someone had set the online store to serve last-known values instead of withholding on TTL, the model served three-day-old counts and quietly degraded. The fix was to make stale features disappear — a visible cold-start failure beats an invisible wrong answer.

The interview / architecture-review signal

What a reviewer listens for: do you separate correctness-through-time from consistency-across- paths, and do you enforce each structurally? Can you draw the point-in-time join on a timeline and state the at-or-before rule in one sentence? Can you distinguish skew (your bug) from drift (the world changing)? Can you explain why batch is a bounded stream and why that equivalence is testable? Can you name why TRANSFORM inside the model kills preprocessing skew, and when the warehouse is the wrong ML platform? Candidates who can join two tables are common; the person who knows where the clock and the filter live in each of the three labs, and who reaches for a structural cure over a discipline, is who gets handed "make our feature layer trustworthy."

Closing takeaways

1. Models are downstream of features, and features are downstream of pipelines — engineer correctness where the leverage is: the join, the window, and the transform.
2. Leakage and skew die structurally or not at all. The <= event_ts filter, one definition, and the model-owned transform beat every "remember to."
3. The store outlives the model. Store selection, TTL, and data residency are the durable decisions; the model is a config string.
4. Choose visible failure over quiet wrong answers. A withheld stale feature that fails to a cold-start path is safer than a stale number masquerading as current.
5. Every "which platform" answer is a routing answer keyed on a constraint — event-time streaming, governed lakehouse, or warehouse-native SQL — not a loyalty test.