« Phase 27 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Staff Notes

Phase 27 — Core Contributor Notes: How the Real Systems Are Built

This is the maintainer's-eye view of the actual libraries our miniatures imitate — Feast, Apache Beam / Google Dataflow, BigQuery ML, and Delta Lake. The labs faithfully reproduce the shapes; the real systems earn their complexity in the seams. Where I describe a pattern rather than a documented internal, I say so — do not quote as fact an implementation detail you cannot verify.

Feast: the registry is the product, the stores are pluggable

The mental model that trips people up is "Feast is a database." It is not. Feast is a registry plus a compiler plus an orchestrator over storage engines it does not own. The registry — a serialized proto blob in object storage or a database — holds your FeatureView, Entity, and FeatureService definitions. Everything else is pluggable: the offline store is a provider (BigQuery, Snowflake, Redshift, Spark, DuckDB, files), and the online store is another (Redis, DynamoDB, Bigtable, Postgres, sqlite). Our lab collapses all of this into two Python dicts, which loses the single most important production property — that the definition is decoupled from the storage, so the same feature view can be backed by BigQuery in one environment and DuckDB in a test.

The seam a committer cares about is get_historical_features. Our miniature runs an explicit scan with a (timestamp, _seq) tie-break. Feast instead compiles the point-in-time join to native SQL and pushes it down to the offline store — it does not pull billions of rows into Python. The entity dataframe (your labels + event timestamps) is uploaded as a temp table, and Feast generates a templated query per provider: a window function or ASOF-style join that, for each entity row, selects the feature row with the max event_timestamp <= entity event_timestamp within the TTL, with a deterministic tie-break on the created-timestamp column. The _seq we invented is Feast's created_timestamp doing the same job — breaking ties among rows sharing an event timestamp so the join is a total order. feast materialize [START] [END] is a separate path that reads the offline store for a time range and writes latest-per-entity into the online store; the online read is a pure KV lookup with a serving-time TTL check. The non-obvious gotcha real users hit: materialization windows and TTL interact, and a materialization that does not cover a gap leaves the online store serving nothing (correctly — the TTL withholds it) while the offline store still has the value. Our lab's serve-time TTL check is faithful; its full-history in-memory materialization is not.

Apache Beam / Dataflow: the model is portable, the runner does the hard part

Lab 02 builds the model — PCollections, windowing as assignment, GroupByKey as the shuffle, watermark-triggered emission. What it cannot build, and what the real system is mostly about, is the runner. The Dataflow model (Akidau et al., VLDB 2015, from Google's FlumeJava and MillWheel lineage) is deliberately split so that Beam is the portable SDK and Dataflow/Flink/Spark are interchangeable runners. The seams:

  • Combiner lifting. Our combine_per_key is group_by_key then a fold — one place, whole list in memory. The real runner exploits the CombineFn's associativity and commutativity to pre-combine partial accumulators on each worker before the shuffle, so the network moves one accumulator per worker instead of one element per input. This is why Beam wants a CombineFn (with create_accumulator/add_input/merge_accumulators/extract_output) and not a lambda over the full list — the four-method shape exists so the runner can split the fold across the network boundary. Missing this is the classic "why is my streaming job's shuffle so expensive" bug.
  • Watermark propagation. Our advance_watermark is a single monotone number the test drives. In a real pipeline the watermark is computed and propagated through the DAG: each stage derives its output watermark from the minimum of its input watermarks and any buffered state, and the source's watermark is a heuristic (perfect only for ordered sources like a log with monotone offsets; a statistical estimate for Pub/Sub fed by mobile devices). The for a real runner must handle that our lab elides: late data behind the watermark — allowed-lateness horizons, late-firing panes, and the accumulation-mode choice (does a re-fire emit the corrected total or just the delta?). Downstream consumers must be told which; billing and dashboards want different answers.
  • Exactly-once is a runner property, not a wire property. Dataflow achieves it by checkpointing shuffle state and deduplicating retried bundles by unique IDs. But it is only end-to-end if the sink is idempotent or transactional. Our lab's _fired set is a faithful miniature of "apply each pane's effect exactly once"; the real guarantee spans worker crashes, bundle retries, and redelivery.

Dataflow the managed runner adds ops, not semantics: it fuses adjacent ParDos into stages, autoscales on backlog, rebalances stragglers by dynamic work rebalancing, and upgrades pipelines in place. The windows/watermarks/triggers are the model's; that is the part the lab teaches, and it is the transferable part across Flink and Spark Structured Streaming.

BigQuery ML: TRANSFORM is a stored, versioned preprocessing graph

Lab 03's anti-skew move — fit statistics on training rows, store them in the model object, reuse them at predict by object identity — is exactly what BigQuery ML's TRANSFORM clause does, but the real implementation makes it structural in a way a Python object cannot. When you CREATE MODEL with a TRANSFORM, BigQuery fits the preprocessing (the ML.STANDARD_SCALER's mean/std, the implicit one-hot vocabulary, bucket boundaries) on the training query and embeds those fitted statistics inside the model resource. ML.PREDICT then re-applies the identical transform automatically — a caller physically cannot serve with different preprocessing, because the preprocessing travels with the model, not with the calling code. Our lab reproduces this by making the same Transform object run inside both create_model and predict_proba; the sharp edge the real system handles that we simplify is that BigQuery ML supports a wide model catalog under the same verbs — logistic/linear regression, k-means, boosted trees (XGBoost under the hood), DNNs (TensorFlow), ARIMA_PLUS, matrix factorization, plus imported ONNX/TF models and remote models that call a Vertex endpoint (including LLMs) from SQL. The FARM_FINGERPRINT split idiom — a row's train/test side is MOD(ABS(FARM_FINGERPRINT(key)), n), a pure function of its own key — is the warehouse's answer to reproducible splits: no shuffle, no seed, a row never migrates sides as the table grows. Our lab uses md5(str(key)) % 100; same property, different hash.

Delta Lake: a table format, not an engine

The most common misconception, worth stating for a committer: Delta Lake is not a database engine — it is a table format. Parquet data files plus an ordered transaction log (the _delta_log directory of JSON commits, periodically compacted into checkpoints). Every write is a new set of data files plus one atomic log append; readers reconstruct table state by replaying the log. Everything Delta is known for falls out of that one mechanism: ACID via optimistic concurrency on the log, time travel because old files and log entries persist until VACUUM, schema enforcement because the write path validates against the logged schema, and MERGE/OPTIMIZE/ Z-ORDER as operations on the file set. The openness is the point — Spark, Trino, and DuckDB all read the same format. Databricks Feature Store's signature move builds on this: feature tables are Delta tables in Unity Catalog, and logging a model packages the feature lookups with the MLflow model so the serving endpoint fetches features itself and callers cannot pass hand-computed (skewed) values — a stronger structural anti-skew guarantee than a shared definition alone.

What our miniatures deliberately simplify

  • Feast: two dicts instead of pluggable offline/online providers; a Python scan instead of a compiled, pushed-down SQL as-of join; full-history in-memory materialization instead of windowed materialization jobs.
  • Beam: a single-process runner with a hand-driven watermark instead of a distributed runner with propagated heuristic watermarks, combiner lifting, dynamic rebalancing, and late-data handling.
  • BigQuery ML: deterministic zero-init logistic regression instead of the full model catalog; an in-Python Transform object instead of statistics embedded in a managed model resource.
  • Delta: not built at all in the labs — but the append-only, timestamped-rows discipline the feature store depends on is exactly what Delta's transaction log provides in production.

Know the shape and the seams, and each system's real documentation reads as confirmation rather than surprise.