« Phase 26 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Staff Notes

Phase 26 — Core Contributor Notes: How the Real MLOps Tools Are Built

This is the maintainer's-eye view of the actual tools — MLflow, Weights & Biases, SageMaker's tracking/registry/monitor stack, Vertex AI Model Monitoring, Evidently, Kubeflow — not our miniature. Where the labs use an injected window index and in-memory dicts, the real systems are databases, object stores, and scheduled distributed jobs, and the interesting engineering is in the seams. Where I describe a pattern rather than a documented internal, I say so; do not quote implementation details you cannot verify.

MLflow tracking: two stores, one filter language

The design decision a committer should internalize first is that MLflow splits a run into a backend store (a SQL database or file tree holding params, metrics, tags, run metadata) and an artifact store (object storage — S3/GCS/Azure Blob — holding the bytes). Params, metric points, and tags are small and queryable; artifacts are large and content-addressed. Our lab collapses both into one Run object, which is faithful to the data model and drops the storage split.

Two details Lab 01 mirrors deliberately. Metrics are time series: MLflow's log_metric(key, value, step) appends (step, value), so loss is a learning curve and "the run's accuracy" is a projection over the log — a store that kept only scalars could never render the overfitting hook. Params are strings because they exist for display and diffing, not arithmetic. And search_runs takes a filter string ("metrics.accuracy > 0.9 and params.optimizer = 'adam'") that MLflow parses into per-field conditions — our Condition(kind, key, op, value) is that parsed form, and the AND-fold with an absent-key-returns-False rule is the correct set semantics a real SQL WHERE also produces.

The stages-to-aliases migration is the API evolution to know

Lab 02 implements MLflow's classic stages (None/Staging/Production/Archived) with a transition state machine and the single-Production invariant. The real evolution matters and interviewers probe it: MLflow deprecated stages in favor of model aliases and tags. The reason is a genuine design lesson — stages hardcode a specific lifecycle (exactly four, one Production) that not every organization shares, while an alias is just a named movable pointer (champion, challenger, default) you can define as many of as you need and repoint atomically. SageMaker took a different road entirely with ModelApprovalStatus (PendingManualApproval -> Approved/Rejected) plus an EventBridge trigger on the flip; Vertex uses aliases from the start. Three products, one mechanism — a mutable pointer to an immutable version — and the "correct" spelling is the one the org's deployment automation listens for. Our stage machine is the clearest teaching form; know that the modern idiom is aliases.

Weights & Biases: content-addressed artifacts with a lineage graph

W&B is worth contrasting because it is a managed service you buy where MLflow is infrastructure you run. Its distinguishing engineering is Artifacts: content-addressed, deduplicated, and wired into a lineage DAG so you can walk from a model back through the dataset artifact and the run that produced it. That lineage graph is the production form of Lab 01's hash_artifact plus lineage tags — same idea (a hash proves bit-identity; pointers record provenance), industrialized into a browsable graph. W&B's other real strength is the sweep/curve UI, which is why researchers reach for it and platform teams weigh the hosted-data tradeoff.

SageMaker and Vertex model monitoring: baseline, schedule, violate

The cloud monitors productize exactly Lab 03's shape — reference distribution, live window, scalar distance, threshold — with real operational seams:

• SageMaker Model Monitor computes a baseline from training data (emitting statistics.json and constraints.json), runs scheduled monitoring jobs that compare captured serving traffic (from Data Capture) against the baseline, and writes CloudWatch violations that can fan out through EventBridge to a retraining pipeline. The baseline-then-schedule-then-violate pattern is our DriftMonitor constructed once with reference histograms, then evaluate_window per window, then a RetrainingTrigger.
• Vertex AI Model Monitoring draws this phase's exact distinction that the lab teaches: skew detection (serving vs training data) versus drift detection (serving vs earlier serving data), and it defaults to Jensen–Shannon divergence for numeric features and chi-square/L-infinity-style distances for categorical ones. The JS choice is a real committer's decision: it is the symmetrized, bounded cousin of KL, which is why production tools prefer it over raw KL divergence.

The sharp edge both systems hit and Lab 03's psi/kl_divergence teach: KS and chi-square become too powerful at scale — with a large enough window, a statistically significant p-value fires on a practically meaningless shift — which is why large-scale monitors switch from significance tests to effect-size measures (PSI, JS, Wasserstein). Our lab builds PSI and KL because their mechanism is pure arithmetic over bins; the README's extensions walk KS.

Evidently: pick the test per column

Evidently is the open-source drift/quality-report specialist, and its non-obvious engineering is per-column test selection: point it at a reference and a current dataset and it chooses the statistical test by column type and sample size — PSI/JS/Wasserstein for numeric, chi-square for categorical, switching to effect-size distances once samples are large — then emits HTML dashboards or CI test suites. Lab 03's DriftReport is that skeleton: the same reference-vs-current comparison, with the test-selection logic simplified to the two detectors the lab builds by hand.

Kubeflow and the caching seam that crosses into this phase

Kubeflow Pipelines owns orchestration (containerized ML DAGs on Kubernetes), not tracking or registry — teams pair it with MLflow. The seam relevant here is the same caching-correctness gotcha Phase 25 met: KFP keys execution caching on the component spec, so a code change that doesn't change the spec can replay stale outputs. It is the reason the retraining pipeline node in this phase's loop needs the same "version the image into the key or disable caching on external-state steps" discipline.

Data validation: the same detectors pointed upstream

TFDV (schema inference + validation) and Great Expectations (expectation suites) are the productized form of §13's data-axis CI. The elegant maintainer's observation the lab makes concrete: the drift detectors of Lab 03 are the training-data validators of §13 pointed the other direction — the same PSI that watches serving traffic can gate a training batch, and a failed validation halts the pipeline before the training step spends GPU-hours. Same histogram/threshold machinery, aimed at a training batch instead of a live window.

What our miniature deliberately simplifies

• One in-memory Run object instead of MLflow's backend-store / artifact-store split.
• The MLflow stage machine instead of the modern alias/tag idiom (and instead of SageMaker's approval status + EventBridge, or Vertex's aliases).
• Fixed bin edges for inspectability where production tools use reference quantiles; a Laplace epsilon where real smoothing has more tuning.
• An injected _eval_count window index instead of a scheduled job on a wall clock reading Data Capture from object storage.
• PSI and KL built by hand instead of a per-column test selector that also knows KS, chi-square, and Wasserstein.
• A synchronous RetrainingTrigger object instead of a CloudWatch violation → EventBridge → pipeline (or Jira ticket) fan-out.

Know the shape and the seams — metrics as series, the movable-pointer registry, baseline-schedule- violate monitoring, effect-size-over-significance at scale — and every vendor's docs read as confirmation of one lifecycle loop wearing different logos rather than six unrelated products.