« Phase 26 README · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 26 Warmup — MLOps: Experiment Tracking, Model Registry, CI/CD & Drift Monitoring

Who this is for: someone who has built agent infrastructure (Phases 00–17), framework internals (18–24), and evaluation harnesses (Phase 11) — and now needs the discipline that wraps any model, classical or LLM, in a governed lifecycle: how a training run becomes a queryable experiment, how an experiment becomes a registered version, how a version earns Production through a gate, how a deployed model is watched for drift, and how a threshold breach becomes a retraining ticket instead of a quiet quality collapse. By the end you can derive PSI on a whiteboard, explain why CI for ML tests three things instead of one, and say precisely where this phase ends and Phase 14's runtime observability begins. No MLflow install, no cloud account — everything in the labs is mechanism.

Table of Contents

  1. What MLOps is and why ML is not just software
  2. The three axes of change: code, data, model
  3. The MLOps maturity model
  4. Experiment tracking from first principles
  5. The model registry: versions, stages, approval, rollback
  6. CI/CD/CT: what ML adds to continuous delivery
  7. Evaluation gates: blocking promotion on regression
  8. Model monitoring: operational vs statistical
  9. Drift in depth: data, prediction, concept, label
  10. Drift detectors: PSI, KL divergence, KS test, chi-square
  11. Training-serving skew
  12. Retraining strategies: closing the loop
  13. Feature and data validation
  14. Reproducibility and determinism
  15. The tool landscape
  16. Common misconceptions
  17. Lab walkthrough
  18. Success criteria
  19. Interview Q&A
  20. References

1. What MLOps is and why ML is not just software

MLOps is the engineering discipline that takes a machine-learning model from a notebook to a governed, monitored, continuously-improving production system — the ML-specific extension of DevOps. DevOps gave software a repeatable path from commit to deploy (CI/CD, infrastructure as code, observability). MLOps has to solve everything DevOps solves plus a set of problems software does not have, because an ML system's behavior is not fully specified by its code.

The canonical statement of the problem is Sculley et al.'s Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015): in a real ML system, the model is a small box in the middle of a much larger diagram — data collection, verification, feature extraction, serving infrastructure, monitoring — and most failures happen in the boxes around the model. Their sharpest observation is CACE: Changing Anything Changes Everything. A model's behavior is a function of its training data, so an innocuous upstream change (a field's units, a filter in an ETL job, a shifted user population) silently changes the model, with no diff to review and no compiler to complain.

That is why "we have CI" is not "we have MLOps." Your app's CI proves the code still behaves; nothing in it proves the model still behaves, because the model's behavior was never written down in code to begin with — it was learned from data that has since moved. MLOps closes that gap with four mechanisms, which are exactly this phase's four topics: track every training run so models are reproducible records, not files (§4); register and stage models so exactly one governed version serves (§5); gate promotion on evidence, in CI (§6–7); and monitor the deployed model statistically, feeding retraining when the world moves (§8–12).

2. The three axes of change: code, data, model

Software has one axis of change: code. Version it (git), test it (CI), deploy it (CD), and the system is governed. An ML system changes along three axes, and a change on any one can break the other two:

  • Code — the training pipeline, the feature transformations, the serving wrapper. Versioned in git like any code, but with a twist: feature code runs twice (training and serving), and if the two paths diverge you get training-serving skew (§11).
  • Data — the training set, the feature distributions, the label definitions. Data is the substrate the model's behavior is compiled from, and it changes without anyone committing anything: upstream schema changes, seasonal cycles, product launches, user-mix shifts. Data needs its own versioning (a dataset hash or snapshot id in every run's lineage) and its own tests (§13).
  • Model — the trained artifact itself. Retraining the same code on new data produces a different model; the same code and data with a different seed can too (§14). The model must be versioned independently of the code that produced it, because one training pipeline at one commit produces many models over its lifetime.

The consequences follow mechanically. Versioning must cover all three axes and record the links between them — a model version points at a code version and a data version, which is what "lineage" means (§4). Testing must cover all three — CI for ML validates data, code, and model (§6). Monitoring must cover all three — the code can be healthy (Phase 14's dashboards all green) while the data has drifted and the model is confidently wrong (§8). Every serious MLOps tool is, at bottom, bookkeeping for these three axes and the arrows between them.

3. The MLOps maturity model

The framing the industry converged on comes from Google Cloud's MLOps: Continuous delivery and automation pipelines in machine learning — three levels, defined by what is automated:

Level 0 — the manual process. Training happens in notebooks, driven by a person. The "deployment" is handing a model file to another team. Deploys are rare, releases are scary, and there is no monitoring beyond "did anyone complain." The defining property: the connection between training and serving is a human. Most teams honestly start here; the failure mode is staying here after the model matters.

Level 1 — ML pipeline automation. The unit of deployment stops being a model and becomes a training pipeline: an automated, parameterized sequence (validate data → prepare features → train → evaluate → validate model → register) that can run without a human, on a schedule or a trigger. This is what makes continuous training (CT) possible — the system retrains itself on fresh data. Level 1 requires the machinery this phase builds: experiment tracking (or you can't compare pipeline runs), a model registry (or the pipeline has nowhere governed to put its output), data validation (or the pipeline happily trains on garbage), and monitoring (or nothing triggers the pipeline).

Level 2 — CI/CD pipeline automation. Level 1 automated running the pipeline; Level 2 automates changing it. The pipeline itself goes through CI (unit tests for feature logic, tests for data schemas, tests that the training loop converges on a fixture) and CD (the new pipeline is deployed to run in production), so a data scientist's improvement to the feature code travels to production through the same tested, gated path as any software change. At Level 2 you have all three C's: CI (test data + code + model), CD (deploy pipelines and models through gates), CT (retrain continuously) — and the loop in §12 runs without heroics.

The interview-ready compression: Level 0 deploys a model by hand; Level 1 deploys an automated pipeline that can retrain the model; Level 2 puts the pipeline itself under CI/CD. The labs build Level-2 machinery in miniature: tracking (Lab 01), registry + gate (Lab 02), drift-triggered retraining (Lab 03).

4. Experiment tracking from first principles

Training a model is an experiment: you fix a configuration, run a stochastic process over data, and observe outcomes. Experiment tracking is the systematic record of those experiments, and its unit is the run — one training execution. A run carries five things, and knowing why each exists is the difference between using MLflow and understanding it:

  • Parameters — the inputs you chose: learning rate, architecture, regularization, data version. Set once per run, they answer "what did we try?" MLflow stores them as strings, which looks lazy until you realize params exist for display and diffing, not arithmetic.
  • Metrics — the outcomes: loss, accuracy, AUC. The crucial design decision — which Lab 01 makes you implement — is that a metric is a time series, not a scalar: every logged value carries a step, so loss is a learning curve you can plot, and "the run's accuracy" is just the latest point of the curve. A tracker that kept only final values could not show you the overfitting hook in a validation curve — half the diagnostic value.
  • Artifacts — the outputs: model weights, plots, evaluation reports, the exact config file. Real trackers store the bytes in object storage and content-address them (a hash of the bytes), which buys deduplication and, more importantly, proof: if the hash matches, the artifact is bit-identical. Lab 01's hash_artifact is that mechanism, undisguised.
  • Lineage — the pointers out of the run: the git commit of the training code, the dataset version/hash, the environment. Lineage is what makes a run reproducible (§14) and what makes a registry meaningful (§5): a model version with no run behind it is provenance-free.
  • Grouping — runs belong to experiments (a named question: "churn-model architecture sweep"), so a hundred runs are a table, not a pile.

On top of the record sit the queries: search ("all runs with metrics.accuracy > 0.9 and params.optimizer = 'adam'" — MLflow compiles exactly this filter string into per-field conditions, Lab 01's Condition), compare (a side-by-side param/metric diff across selected runs — every tracker's most-used screen), and best-run selection (best_run("val_loss", mode="min") — model selection as a deterministic query, including the tie-break). The production significance is blunt: when the promotion gate (§7) asks "does the candidate beat the incumbent?", both numbers come from the tracking store. No tracking, no gate, no MLOps.

5. The model registry: versions, stages, approval, rollback

The tracking store answers "what did we run?" The model registry answers the operational question: "which model is live, which is next, and how did it get there?" Its unit is the model version: a registered name (churn-classifier) with monotonically increasing versions (v1, v2, ...), each an immutable artifact pointing back (lineage) at the tracking run that produced it.

The registry's second concept is the stage — a lifecycle state per version. The classic MLflow vocabulary, which Lab 02 implements: None (registered, unvetted) → Staging (under evaluation/shadow traffic) → Production (serving) → Archived (superseded, kept for audit and rollback). Two rules give the lifecycle its integrity:

  1. Transitions are a state machine, not a free-for-all. You cannot move Production → None; you archive it. An explicit transition table (Lab 02's ALLOWED_TRANSITIONS) makes illegal moves errors instead of incidents.
  2. At most one Production version per model name. Promoting v9 to Production auto-archives v7 — the single-Production invariant. Without it, "which model is serving?" has two answers, which is zero answers.

Around the lifecycle sit approval and rollback. Approval is the human/CI checkpoint — SageMaker's registry makes it a first-class field (ModelApprovalStatus: PendingManualApproval → Approved/Rejected), and deployment automation listens for the flip. The modern MLflow idiom generalizes stages into aliases (champion, challenger) you move between versions — different vocabulary, same mechanism: a mutable pointer to an immutable version. Rollback is why versions are never deleted on supersession: when v9 misbehaves in production, restoring v7 is moving a pointer back — one call, seconds — because the artifact, its lineage, and its metrics are all still in the registry. Lab 02's rollback records each promotion as (promoted, displaced) precisely so undo is deterministic.

6. CI/CD/CT: what ML adds to continuous delivery

App CI/CD answers: does the code pass its tests, and can we ship it safely? ML keeps both questions and adds a third C — and the differences run deeper than the acronym.

CI for ML tests three things, not one. (The reference here is Breck et al.'s ML Test Score rubric.) Data tests: does the training data match its schema, are feature distributions within expected ranges, are there enough rows, is the label leak-free (§13)? Code tests: the ordinary unit tests, plus ML-specific ones — feature transforms are deterministic, the training loop can overfit a tiny fixture (a ten-example sanity check that the gradient actually flows). Model tests: the trained candidate meets minimum quality on a held-out set, beats a naive baseline, satisfies fairness slices, stays within latency/size budgets. A pipeline that only runs code tests is app CI wearing an ML costume.

CD for ML deploys two different things. At Level 1 you deploy a model (through the registry gate, §7). At Level 2 you also deploy the training pipeline itself — the pipeline is code, it goes through CI, and "release" means the new pipeline definition starts producing tomorrow's models. This is the inversion that makes senior candidates stand out: in mature MLOps you ship pipelines, not models; models are what the shipped pipeline emits.

CT — continuous training — has no software analogue. Software does not rot when the world changes; models do (§9). CT is the standing capability to retrain on fresh data without a human driving: on a schedule, or on a trigger from the drift monitor (§12). CT is also why ML CD needs gates more than app CD does — an automated pipeline retraining weekly will occasionally produce a worse model (bad data week, upstream breakage), and the gate is what stops CT from becoming continuous degradation.

One more honest difference: an ML "build" is not reproducible by default. Compile the same commit twice, get the same binary; train the same commit twice, get two different models unless you did the determinism work of §14. CI for ML has to decide what "same" means — usually "metrics within tolerance," not "bytes identical" — and that choice must be explicit.

7. Evaluation gates: blocking promotion on regression

The promotion gate is where MLOps stops being bookkeeping and starts being enforcement. The JD language — "approve model sign-off packages", "block promotion on metric regression" — is this section, and Lab 02 builds it as a pure function.

A gate worth the name has three families of check, ALL of which must pass:

  1. Primary metric, with a margin. The candidate must beat the current Production model on the metric you actually optimize (AUC, accuracy, revenue-weighted whatever) by at least min_improvement. The margin is not pedantry: eval metrics are noisy, and a gate with a zero margin will happily promote noise, churning your production model weekly for no real gain. The first-model-ever path (no incumbent) passes by construction — there is nothing to regress.
  2. Guardrail metrics, with tolerances. The metrics you must not lose while chasing the primary: recall on a protected slice, latency, calibration, cost per prediction. The gate's defining property — and Lab 02's sharpest test — is that a guardrail regression blocks even when the primary metric improves. A model that's 2 points better on AUC and 30ms slower than the budget does not ship. Each guardrail carries its own direction (higher-is-better vs lower-is-better) and tolerance.
  3. Required checks. Pass/fail attestations that must be present and true: the fairness audit ran and passed, the latency budget test ran and passed, security review signed off. "Missing" fails exactly like "failed" — an absent fairness report is not a passing fairness report.

The gate's output is not a boolean; it is the sign-off package: a structured result naming every check, its outcome, and the evidence (candidate vs incumbent values, margins, tolerances) — Lab 02's GateResult. That artifact is what an approver approves, what an auditor reads, and what the 2 a.m. investigation pulls up. And the gate must be pure: evaluate never mutates the registry, promote applies the transition only on pass, and a blocked candidate stays in Staging with Production untouched.

This is the same discipline as Phase 11's eval-harness regression gates, one layer down the stack: Phase 11 builds the machinery that produces trustworthy evaluation numbers (judges, rubrics, regression suites); this gate consumes those numbers to govern the model artifact's lifecycle. Same idea for LLM systems and classical ML alike — the gate doesn't care whether the metric came from a judge or a confusion matrix.

8. Model monitoring: operational vs statistical

Once a model serves, two different questions need watching, and conflating them is the classic junior mistake:

Operational monitoring asks: is the service healthy? Request rate, error rate, latency percentiles, saturation — the golden signals, plus ML-flavored ones like feature-fetch latency and prediction-cache hit rate. This is Phase 14's territory — meters, traces, dashboards, the cost per request — and everything there applies unchanged to a model service.

Statistical monitoring asks: is the model still right? — and here is the trap that makes ML monitoring genuinely hard: a wrong model looks healthy. Every request returns 200, latency is great (the model computes the same arithmetic whether or not its assumptions hold), throughput is fine — and the predictions are garbage, because the input distribution moved (§9). No operational signal fires. You need statistical signals: distributions of inputs compared against a reference, distributions of outputs, and — when labels arrive — realized quality.

The two also differ in tempo and response. Operational alerts are seconds-to-minutes and page an SRE to fix infrastructure. Statistical alerts are hours-to-weeks (you need a window of traffic to compare distributions, and labels lag), and their response is not "restart the pod" — it is "open a retraining ticket, investigate the upstream data" (§12). A mature team runs both dashboards, staffed by the right people: Phase 14's dashboard tells you the service is up; this phase's dashboard tells you the model is still telling the truth.

9. Drift in depth: data, prediction, concept, label

"Drift" is four distinct phenomena. Set up notation: the model learned \(P(y \mid x)\) from training data drawn from \(P_{train}(x, y)\); live traffic is drawn from \(P_{live}(x, y)\). Factor the joint as \(P(x, y) = P(x),P(y \mid x)\) and each drift type is a statement about which factor moved:

  • Data drift (covariate shift) — \(P(x)\) changed; \(P(y \mid x)\) did not. The inputs look different: your users got younger, a marketing campaign changed the traffic mix, an upstream service started sending cents instead of dollars. The model may still be correct on each input (the relationship holds) but is now extrapolating into regions it saw rarely in training, where its error bars are silently wide. Detectable without labels — compare live feature distributions to the training reference (§10). Earliest, cheapest warning.
  • Prediction drift — the distribution of the model's outputs \(P(\hat{y})\) changed. Your churn model used to flag 8% of users; this month it flags 24%. Either the inputs moved (data drift showing up downstream) or the model is misbehaving — either way, the business impact of the model changed, which is why teams alert on it: it is label-free, one distribution instead of hundreds of features, and closest to what stakeholders feel.
  • Concept drift — \(P(y \mid x)\) itself changed: the same input now means something else. Fraudsters adapted their behavior to look like your legitimate users; a pandemic changed what "normal purchasing" means; a competitor's launch changed which customers churn. This is the killer, because inputs can look perfectly stable while the model quietly becomes wrong — no data drift, no prediction drift, pure relationship change. Detecting it honestly requires labels (realized outcomes), which usually arrive late; the standard proxy is a rolling accuracy/AUC on labeled feedback compared to a baseline — Lab 03's accuracy-drop signal. Concept drift also has shapes worth naming (from Gama et al.'s survey): sudden (a policy change), gradual (slow adoption), incremental (continuous shift), and recurring (seasonality — which you should model, not "fix").
  • Label drift (prior shift) — \(P(y)\) changed: the base rate moved. Churn goes from 5% to 15% in a recession even if who-churns-given-features is stable. Label drift breaks calibrated probabilities and any threshold you tuned to the old base rate.

The triage ladder in production: data drift monitors run label-free on everything (broad, cheap, noisy); prediction drift watches the output distribution (label-free, focused); concept drift via delayed-label quality tracking is the ground truth that confirms or clears the earlier warnings. All three run in Lab 03's monitor, because each catches what the others miss: benign covariate shift can fire data-drift alarms while accuracy holds (drift ≠ damage), and pure concept drift can collapse accuracy with zero input drift.

10. Drift detectors: PSI, KL divergence, KS test, chi-square

Every detector reduces to one shape: a reference distribution (training data or a trusted window), a live window, and a scalar score of how far apart they are, compared to a threshold. Know four detectors, with formulas.

PSI — Population Stability Index

The industry workhorse, born in credit-risk model validation. Bin the feature (fixed edges or reference quantiles — production tools use quantiles, §Lab 03 uses fixed edges for inspectability), compute each bin's share in the reference (\(q_i\)) and the live window (\(p_i\)), then:

\[ \mathrm{PSI} ;=; \sum_{i=1}^{B} (p_i - q_i),\ln!\frac{p_i}{q_i} \]

Each term is non-negative (when \(p_i > q_i\) both factors are positive; when \(p_i < q_i\) both are negative), so \(\mathrm{PSI} \ge 0\), with equality iff the distributions match on every bin — and it grows monotonically with the size of the shift, which Lab 03's tests assert. Zero bins would blow up the logarithm, so every implementation smooths (Laplace: add a small \(\epsilon\) per bin, renormalize). The canonical thresholds, worth stating in exactly these words: PSI below 0.1 — no significant shift; 0.1 to 0.2 — moderate shift, investigate; above 0.2 — significant shift, act. PSI is symmetric-ish in structure (it is the sum of the two directed KL divergences, a.k.a. Jeffreys divergence) and works unchanged on categorical features (bins = categories).

KL divergence

The information-theoretic parent:

\[ D_{KL}(P ,|, Q) ;=; \sum_{i} p_i ,\ln!\frac{p_i}{q_i} \]

— the expected extra nats to encode samples from \(P\) using a code optimized for \(Q\). Always \(\ge 0\) (Gibbs' inequality), zero iff identical, asymmetric (\(D_{KL}(P|Q) \ne D_{KL}(Q|P)\) in general), and unbounded, which is why practitioners often prefer its symmetrized, bounded cousin Jensen–Shannon divergence (Vertex AI's numeric-drift default). The connection to PSI is exact: \(\mathrm{PSI}(P, Q) = D_{KL}(P|Q) + D_{KL}(Q|P)\).

KS test — Kolmogorov–Smirnov

Binning-free, for numeric features: compare the two empirical CDFs and take the largest vertical gap,

\[ D ;=; \sup_x \left| F_{live}(x) - F_{ref}(x) \right| \]

Reject "same distribution" at significance \(\alpha\) when \(D > c(\alpha) \sqrt{\tfrac{n+m}{n,m}}\) with \(c(0.05) \approx 1.36\) (for samples of size \(n\) and \(m\)). Strengths: no bin edges to choose, sensitive anywhere in the distribution. The production gotcha: with very large windows the KS test becomes too powerful — statistically significant \(p\)-values for practically meaningless shifts — which is why tools like Evidently switch from significance tests to effect-size distances (PSI, Wasserstein) once samples are big.

Chi-square

The categorical significance test. With observed live counts \(O_i\) and expected counts \(E_i\) (the reference proportions scaled to the live sample size):

\[ \chi^2 ;=; \sum_{i=1}^{B} \frac{(O_i - E_i)^2}{E_i} \]

compared against the \(\chi^2\) distribution with \(B-1\) degrees of freedom. Same large-sample caveat as KS. Rule of thumb for choosing: PSI or JS divergence for effect-size monitoring at scale; KS (numeric) and chi-square (categorical) when you want a significance test on modest samples. Lab 03 builds PSI and KL — the two whose mechanism is pure arithmetic over bins — and its README's extensions walk you through KS.

11. Training-serving skew

Drift is the world changing under a correct pipeline. Training-serving skew is your own pipeline disagreeing with itself: the features the model sees in production are computed differently from the features it was trained on. Same user, same moment, different numbers — the model is answering a question phrased differently than the one it studied.

The three classic sources: duplicated feature logic — training features built in Python/Spark, serving features rebuilt in Java/Go for latency, and the two implementations disagree on avg_purchase_30d (timezone handling, null semantics, rounding); temporal leakage — training features computed from a data warehouse that has future information relative to prediction time (the training row "knows" things the serving path cannot yet know), so offline metrics are inflated and production quality mysteriously misses them; environment gaps — different library versions, different preprocessing defaults, a tokenizer updated in one place.

Why it deserves its own section: skew looks exactly like drift on a monitoring dashboard — distributions of "the same feature" differ between training and serving — but the fix is entirely different (fix the pipeline, don't retrain; retraining on skewed features just bakes the bug in). The detection is symmetric though, which is elegant: log the features actually served, and run the same distribution comparison (§10) between served features and training features. That is precisely what SageMaker Model Monitor's data-quality baseline and Vertex AI's skew detection do — Vertex explicitly distinguishes skew detection (serving vs training data) from drift detection (serving vs earlier serving data). The prevention is architectural: compute each feature once, in one place, for both training and serving — the core argument for a feature store (Feast, Vertex Feature Store, SageMaker Feature Store) and for "log the features at prediction time and train on the logs."

12. Retraining strategies: closing the loop

Detection without response is a dashboard nobody looks at. The last mile of MLOps is the policy for when to retrain, and there are three, in ascending order of automation:

  • Scheduled retraining — cron: weekly, nightly. Simple, predictable, capacity-plannable, and it wastes compute when nothing changed while reacting up to a full period late when something did. The right default when drift is roughly continuous (recommendation freshness) or labels arrive on a natural cycle.
  • Triggered retraining — retrain when monitoring says so: a drift score crosses its threshold, realized accuracy drops past tolerance, or a business rule fires (new product category launched). Reactive, efficient, and only as good as the monitor — which is why Lab 03's monitor emits a first-class RetrainingTrigger ticket carrying which signal fired, the score, the threshold, the severity, and the window: everything the on-call (or the pipeline) needs to decide and to audit the decision later. Real systems wire this as: monitor → alert → EventBridge/PubSub → pipeline execution, or → Jira ticket when a human should look first (the JD's "initiate retraining tickets when thresholds are breached" is literally this).
  • Continuous / online learning — the model updates on every batch or every example. Maximum freshness, minimum governance: a poisoned or buggy data stream walks straight into the model, evaluation becomes a moving target, and rollback means "to when?". Rare in practice outside ads and recsys; most teams get 95% of the benefit from triggered retraining with a good gate.

Two disciplines make triggered retraining humane. Cooldown/dedupe: drift is a state, not an event — a distribution that shifted stays shifted, and a naive monitor re-alerts every window. One sustained drift must produce one ticket per cooldown period (Lab 03's _should_fire), or your team learns to ignore the channel — the same alert-fatigue math as any paging system. The gate still applies: a retrained model enters the registry as a candidate and passes the same promotion gate (§7) as any other — retraining triggered by bad data would otherwise ship a model trained on the same bad data. And after promotion, re-baseline the monitor on the new training distribution, or the monitor will forever compare live traffic to a reference the new model no longer represents.

That is the closed loop, end to end: monitor (Lab 03) → trigger → retrain → track (Lab 01) → register (Lab 02) → gate (Lab 02) → promote → serve → re-baseline → monitor. Draw this circle on a whiteboard and you have summarized the phase.

13. Feature and data validation

The cheapest place to stop a bad model is before training. Data validation is CI's data-axis test suite (§6), and it comes in three strengths:

  • Schema validation — the contract: expected columns, types, nullability, categorical domains. Catches the upstream rename, the type change, the suddenly-null field. TensorFlow Data Validation (TFDV) infers a schema from training data and validates every future batch against it; Great Expectations calls the same idea an "expectation suite."
  • Range and distribution checks — semantics, not just shape: age in [0, 130], purchase_amount non-negative, category frequencies within tolerance of the reference, missing-value rate under a ceiling. Notice these are the drift detectors of §10 pointed at training data — the same PSI that watches serving traffic can gate a training batch.
  • Cross-column and leakage checks — the expensive class: label not derivable from a feature (target leakage), train/test split honest with respect to time and entities (no user in both), feature timestamps strictly before label timestamps (§11's temporal leakage).

The operational rule: a failed validation halts the pipeline before the training step spends GPU-hours, and its report names the violated expectation — the same fail-closed, structured-error discipline as everything else in this track. In Lab 03 the mechanism appears in mirror image: the same histogram/threshold machinery, aimed at live windows instead of training batches.

14. Reproducibility and determinism

A result you cannot reproduce is an anecdote. Reproducibility in ML has levels, and honesty about which one you've achieved is itself a seniority signal:

  1. Bookkeeping reproducibility — you recorded everything: code commit, data version/hash, params, environment, seed. This is what tracking (§4) buys, and it is the mandatory floor — without the record, no stronger level is even attemptable.
  2. Statistical reproducibility — rerunning the pipeline produces a model with equivalent metrics (within tolerance), though not identical bytes. This is the practical production target, and the tolerance must be explicit (it feeds the gate's margin, §7).
  3. Bitwise reproducibility — same bytes out. Requires seeding every RNG (Python, NumPy, the framework), deterministic kernels (GPU parallel reductions reorder float additions, and float addition is not associative — cudnn.deterministic-style flags trade speed for order), single-threaded or order-fixed data loading, and pinned environments. Expensive; reserved for regulated domains and debugging.

The labs live at level 3 by construction, because the track's rules (see the Lab Standard) forbid the sources of nondeterminism: run ids are counters (not UUIDs), time is an injected tick (not wall clock), artifacts are content hashes, and there is no unseeded randomness anywhere. That is not a toy simplification — it is the same seam production teams use (record/replay, fixed seeds in CI, content-addressed artifacts) to make ML testable, and Lab 01's determinism test (same script → byte-identical run records) is the assertion form of the whole idea.

15. The tool landscape

Six names cover most JD checklists. What matters in an interview is knowing which problem each owns and where they overlap:

  • MLflow (Databricks, open source) — the default open-source answer to §4 + §5: Tracking (runs/params/metrics/artifacts, search_runs), Model Registry (versions, stages/aliases, transition API), plus packaging (MLmodel format) and serving adapters. Self-hostable anywhere; the natural first tool, and the one Labs 01–02 mirror most directly.
  • Weights & Biases — experiment tracking as a managed product, with the best-in-class UI for sweeps, learning curves, and collaborative reports; content-addressed Artifacts with lineage graphs; a model registry via aliases. Where MLflow is infrastructure you run, W&B is a service you buy; researchers tend to love it, platform teams weigh the hosted-data tradeoff.
  • Kubeflow — ML pipelines on Kubernetes (§6's Level-1/2 automation): Kubeflow Pipelines expresses the training DAG as versioned, containerized components with tracked runs. It owns orchestration, not tracking/registry per se (teams pair it with MLflow). Heavyweight; the choice when you already live on Kubernetes.
  • SageMaker Pipelines + Model Registry + Model Monitor (AWS) — the integrated AWS story: Pipelines for the training DAG, the Registry with first-class ModelApprovalStatus for §5's approval flow, Model Monitor for §8–11 (data-quality baselines from training data, scheduled comparisons, CloudWatch violations → EventBridge → retraining). The all-in choice for AWS shops — same platform logic as Phase 24's Bedrock reasoning, one layer over.
  • Vertex AI (Google) — the GCP equivalent: Vertex Pipelines (Kubeflow-lineage), Model Registry, and Vertex AI Model Monitoring, whose docs draw this phase's exact distinction: skew detection (serving vs training) vs drift detection (serving vs earlier serving), with JS divergence for numeric and L-infinity/chi-square-style distances for categorical features.
  • Evidently (open source) — the drift/quality-report specialist: point it at a reference and a current dataset and it computes per-column drift (PSI, KL, KS, chi-square, Wasserstein — chosen per column type and sample size), data-quality checks, and performance reports, as HTML dashboards or CI test suites. Lab 03's DriftReport is its skeleton.

The composition that actually ships: one tracker (MLflow or W&B), one registry (usually the tracker's), one orchestrator (Kubeflow/SageMaker/Vertex pipelines — whichever cloud you are), one monitor (Evidently self-hosted, or the cloud's). The senior take: these are four roles, and any tool résumé-matching beats naming — say which role each fills and the interview goes fine.

16. Common misconceptions

  • "MLOps is DevOps for the training code." It is DevOps for three axes — code, data, model — and the data axis is the one with no diff, no compiler, and most of the incidents.
  • "The model registry is artifact storage." Storage keeps bytes; the registry keeps lifecycle: versions, stages, the single-Production invariant, approval, lineage, rollback. S3 is storage; "v9 is Production, promoted by the gate on evidence X, v7 one call away" is a registry.
  • "A better primary metric means promote." Not past a real gate: guardrail regressions (latency, fairness-slice recall, calibration) block regardless of the primary win, and required checks fail when missing, not just when failing.
  • "Drift means the model is broken." Drift is a distribution statement, not a quality statement. Benign covariate shift fires drift alarms while accuracy holds; that is why drift signals trigger investigation/retraining tickets, not automatic panic — and why concept-drift (label-based) signals are the confirmatory tier.
  • "No drift alarms means the model is fine." Pure concept drift moves \(P(y \mid x)\) without moving \(P(x)\) — inputs look identical, accuracy collapses. Input monitoring alone is necessary, never sufficient; you need delayed-label quality tracking too.
  • "PSI and the KS test are interchangeable." PSI is an effect-size measure (how big is the shift), KS is a significance test (is the shift nonzero) — and at production sample sizes KS finds statistically-significant-but-meaningless shifts, which is why large-scale monitors default to PSI/JS-style distances.
  • "Retraining fixes drift." Retraining fixes drift if the new training data reflects the new world and the pipeline is correct. It does not fix training-serving skew (that's a pipeline bug you'd be baking in), and an ungated retrain on a corrupted feed ships the corruption.
  • "We retrain on a schedule, so we don't need monitoring." The schedule tells you when you last retrained, not whether it worked or whether the world moved mid-cycle; and without monitoring you cannot even answer "did the retrain help."
  • "This is Phase 14 again." Phase 14 watches the service (tokens, dollars, latency, traces); this phase watches the model (versions, gates, distributions, realized quality). A healthy service serving a wrong model is precisely the failure mode only this phase catches.

17. Lab walkthrough

Build the three miniatures in order — they compose into §12's closed loop.

  1. Lab 01 — Experiment Tracking. Implement the tracking store: start_run (sequential ids), log_param/log_metric (with step history) / log_artifact (content hash), experiment grouping, Condition.matches + search_runs, compare_runs, and best_run(metric, mode). 28 tests: recording, curves, hashing, search, selection, and a same-script-same-records reproducibility assertion.
  2. Lab 02 — Registry & Promotion Gate. Implement transition_stage over the ALLOWED_TRANSITIONS state machine with the single-Production invariant and rollback, then the PromotionGate: primary-metric margin, guardrail non-regression (direction-aware), required checks, pure evaluate + effectful promote. 26 tests: every gate outcome including block-on-guardrail-despite-primary-win, first-model-ever, and rollback.
  3. Lab 03 — Drift & Retraining. Implement normalize (smoothed distributions), psi and kl_divergence, then DriftMonitor.evaluate_window: data drift per feature, prediction drift over output categories, the concept-drift accuracy-drop proxy, and trigger emission with per-signal cooldown/dedupe. 27 tests: PSI properties (zero/monotone/nonnegative), all three drift types, dedupe, threshold configurability, determinism.

Run each with LAB_MODULE=solution pytest test_lab.py -v first (green reference), then fill your lab.py, then read solution.py's main() output — Lab 03's worked example walks a no-drift window, a drifted window (three tickets), and a persisted window (zero new tickets, cooldown).

18. Success criteria

  • You can name the three axes of ML change and give a failure a pure-code CI cannot catch.
  • You can describe MLOps maturity Levels 0/1/2 and what each level automates.
  • You can state what a run records (params, metric step history, artifacts, lineage) and why artifacts are content-hashed.
  • You can draw the registry stage machine, state the single-Production invariant, and explain rollback as pointer-restore.
  • You can list the three check families of a promotion gate and say why a guardrail regression blocks despite a primary-metric win.
  • You can write the PSI formula from memory, state the 0.1/0.2 bands, and name when you'd use KL/JS vs KS vs chi-square.
  • You can define all four drift types via \(P(x)\), \(P(\hat{y})\), \(P(y \mid x)\), \(P(y)\), and say which need labels.
  • You can distinguish drift from training-serving skew and give the different fixes.
  • You can compare scheduled vs triggered vs continuous retraining and explain cooldown/dedupe.
  • All three labs pass under both lab and solution (81 tests total).

19. Interview Q&A

Q: What makes MLOps different from DevOps? A: Software changes along one axis — code — and DevOps governs it with versioning, CI, CD. An ML system changes along three: code, data, and model. The model's behavior is learned from data, so it changes when the data changes, with no commit and no diff — Sculley's CACE, "changing anything changes everything." MLOps extends versioning, testing, and monitoring to all three axes, and adds the one loop software doesn't have: continuous training, because models degrade as the world moves even when nobody touches the code.

Q: Walk me through the MLOps maturity levels. A: Level 0 is manual — notebooks, hand-off deployment, no monitoring; the training-serving connection is a human. Level 1 automates the training pipeline itself — parameterized, runnable on schedule or trigger, which is what makes continuous training possible; it needs tracking, a registry, data validation, and monitoring as prerequisites. Level 2 puts the pipeline under CI/CD: changes to feature code or training logic travel through tested, gated automation. The compression: Level 0 deploys models by hand, Level 1 deploys an automated pipeline, Level 2 automates changing the pipeline.

Q: What does an experiment tracking system actually store, and why do metrics have steps? A: Per run: parameters (config), metrics as time series — each value with a step, so loss is a learning curve, not a scalar — artifacts stored content-addressed (a hash proves an artifact is bit-identical), and lineage tags (code commit, data version). Steps matter because half the diagnostic value is the curve's shape: an overfitting hook in validation loss is invisible if you only kept final values. And the tracker feeds the promotion gate — both "candidate" and "incumbent" numbers come from it.

Q: What's the difference between a model registry and artifact storage? A: Storage keeps bytes; the registry keeps lifecycle. A registry gives each model name monotonically-versioned, immutable versions with lineage to their training runs, a stage per version (None/Staging/Production/Archived) governed by a transition state machine, an at-most-one-Production invariant with auto-archive on promote, approval flow, and rollback as a pointer-restore. "The model is in S3" answers none of: which version serves, who approved it, on what evidence, and how do we undo it.

Q: Design a promotion gate for me. A: Three check families, all mandatory. One: primary metric — the candidate beats the current Production model by a configured margin, not just beats it, because eval noise would otherwise churn promotions; with no incumbent, first-model-ever passes by construction. Two: guardrails — metrics that must not regress beyond tolerance, each direction-aware (latency lower-is-better, recall higher-is-better); a guardrail regression blocks even if the primary improves. Three: required checks — fairness audit, latency budget — where missing fails like failing. Output is a structured sign-off package naming every check and its evidence; evaluation is pure, promotion applies the stage transition and auto-archives the incumbent, and rollback is one call.

Q: How does CI differ for ML? A: It tests three things instead of one. Data: schema, ranges, distributions, leakage. Code: normal unit tests plus ML-specific ones — deterministic feature transforms, can-overfit-a-tiny-fixture sanity checks. Model: the trained candidate clears quality thresholds, beats a baseline, holds on fairness slices, fits latency/size budgets. Plus a reproducibility caveat: an ML "build" isn't bitwise-reproducible by default, so CI must define "same" — usually metrics-within-tolerance.

Q: Explain the four kinds of drift. A: Factor the joint as \(P(x)P(y \mid x)\). Data drift: \(P(x)\) moved — inputs look different; detectable without labels. Prediction drift: the model's output distribution moved — also label-free, closest to business impact. Concept drift: \(P(y \mid x)\) itself moved — same inputs now mean different outcomes; the dangerous one, because inputs can look stable while accuracy collapses, and honestly detecting it needs labels, which lag. Label drift: the base rate \(P(y)\) moved, which breaks calibration and tuned thresholds. Production monitoring layers them: broad label-free input monitoring, output- distribution monitoring, and delayed-label quality tracking as the confirmatory tier.

Q: Write down PSI and interpret it. A: Bin the feature; with reference shares \(q_i\) and live shares \(p_i\): \(\mathrm{PSI} = \sum_i (p_i - q_i)\ln(p_i/q_i)\). Every term is non-negative, so PSI is zero iff the binned distributions match and grows with the shift; smooth the bins so no share is zero. Bands: under 0.1 stable, 0.1–0.2 moderate — investigate, over 0.2 significant — act. It's the symmetrized KL divergence (sum of both directions), and it works on categorical features with categories as bins.

Q: When would you use the KS test vs PSI? A: KS is a significance test on numeric samples — the max gap between empirical CDFs against a critical value scaling like \(\sqrt{(n+m)/nm}\) — no binning needed, good on modest samples. At production scale it becomes too powerful: tiny, meaningless shifts hit significance. PSI (or Jensen–Shannon) is an effect-size measure with actionable bands, so large-scale monitors default to it. Chi-square plays KS's role for categorical data, same large-sample caveat.

Q: What's training-serving skew and how is it different from drift? A: Drift is the world changing under a correct pipeline; skew is the pipeline disagreeing with itself — serving computes features differently than training did (duplicated logic in two languages, temporal leakage, environment gaps). It looks like drift on a dashboard, but the fix is opposite: retraining on skewed features bakes the bug in — you fix the pipeline. Detection: log served features, compare against training features with the same distribution tests. Prevention: compute each feature once for both paths — the feature-store argument. Vertex AI literally ships both as separate modes: skew (serving vs training) and drift (serving vs past serving).

Q: Scheduled vs triggered vs continuous retraining? A: Scheduled is cron — simple, plannable, wastes compute when nothing changed and reacts late when something did; right when drift is continuous or labels have a natural cycle. Triggered retrains when monitoring breaches a threshold — efficient and reactive, but only as good as the monitor, and it needs cooldown/dedupe so one sustained drift raises one ticket, not one per window. Continuous/online updates on every batch — maximum freshness, minimum governance, rare outside ads/recsys. In all cases the retrained model re-enters through the promotion gate — CT without a gate is continuous degradation risk — and after promotion you re-baseline the monitor.

Q: Your drift dashboard is all green but customers say the model is wrong. What do you check? A: Green input monitoring rules out (detected) data drift, so: concept drift first — pull labeled feedback / delayed outcomes and check realized accuracy against baseline, since \(P(y \mid x)\) can move with stable inputs. Then training-serving skew — compare served feature values against training values; the monitor may be watching the offline pipeline while the online path computes different numbers. Then coverage gaps: features not monitored, segments averaged away (aggregate PSI can hide a drifted subpopulation), thresholds set too loose, or a reference that was re-baselined wrongly. And confirm the complaint isn't operational — Phase 14's territory — before blaming the model.

Q: How do tracking, registry, gate, and monitor compose into one system? A: As a loop. The monitor scores live windows against a reference and, on a threshold breach, emits a deduped retraining ticket. The retraining pipeline runs and logs its run — params, curves, artifacts, lineage — to the tracker. The run's model is registered as a new version in Staging. The gate compares it to the serving incumbent using tracked metrics: primary-with-margin, guardrails, required checks; on pass it promotes and auto-archives, on fail the incumbent keeps serving. After promotion the monitor re-baselines on the new training distribution. Rollback stands by as a pointer-restore. That circle — monitor, trigger, retrain, track, register, gate, promote, re-baseline — is Level-2 MLOps in one sentence.

20. References

  • Sculley, D., et al. — Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015). The CACE principle and the "the model is the small box" diagram.
  • Google Cloud Architecture Center — MLOps: Continuous delivery and automation pipelines in machine learning (the maturity Levels 0/1/2 framing). https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning
  • Breck, E., et al. — The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction (IEEE Big Data 2017). The data/model/infrastructure/monitoring test taxonomy behind §6.
  • Gama, J., et al. — A Survey on Concept Drift Adaptation (ACM Computing Surveys, 2014). The sudden/gradual/incremental/recurring drift taxonomy.
  • MLflow documentation — Tracking (runs, params, metrics, artifacts, search_runs) and Model Registry (versions, stages, aliases, transitions). https://mlflow.org/docs/latest/
  • Weights & Biases documentation — Experiments, Artifacts (content-addressed lineage), Model Registry. https://docs.wandb.ai/
  • Kubeflow documentation — Kubeflow Pipelines (containerized ML DAGs on Kubernetes). https://www.kubeflow.org/docs/
  • Amazon SageMaker Developer Guide — SageMaker Pipelines; Model Registry (ModelApprovalStatus); Model Monitor (baselines, scheduled monitoring, CloudWatch violations). https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines.html and https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor.html
  • Google Cloud Vertex AI documentation — Model Monitoring (training-serving skew detection vs prediction drift detection; distance measures per feature type). https://cloud.google.com/vertex-ai/docs/model-monitoring/overview
  • Evidently documentation — data drift reports and test suites (per-column statistical tests: PSI, KL, KS, chi-square, Wasserstein; defaults by column type and sample size). https://docs.evidentlyai.com/
  • TensorFlow Data Validation (TFDV) documentation — schema inference and data validation for ML pipelines (the §13 mechanism, productionized).
  • Karpathy-adjacent classic on PSI's credit-scoring origins: Siddiqi, N. — Credit Risk Scorecards (Wiley) — the source of the 0.1/0.2 PSI bands.
  • This track: Phase 11 — Agent Evaluation, Judge & Regression Gates (the evaluation machinery the gate consumes) and Phase 14 — Cost, Latency & Observability (the operational-monitoring complement to §8).