Warmup Guide — MLOps: Tracking, Registry, Drift & CI/CD

Zero-to-senior primer for Phase 17. We start from "why does an ML system rot while a software system stays put" and end with the platform that makes a model operable: experiment tracking (with MLflow in depth), the model registry and versioning, packaging and serving, pipelines and orchestration, the feature store and training-serving skew, CI/CD for ML, production monitoring and drift, the LLMOps additions, and the maturity model that tells you how much of this you actually need. Every term: what it is → why it exists → how it works under the hood → production significance → the misconception that bites people. Then the lab walkthrough, success criteria, interview Q&A, and references.

Table of Contents

• Chapter 1: What MLOps Is, and Why ML Systems Rot Differently
• Chapter 2: Experiment Tracking — Runs, Params, Metrics, Artifacts, Lineage
• Chapter 3: MLflow in Depth — The Four Components
• Chapter 4: The Tracking Landscape — W&B, Neptune, Comet, DVC
• Chapter 5: The Model Registry & Versioning
• Chapter 6: Model Packaging & Serving
• Chapter 7: Pipelines & Orchestration
• Chapter 8: The Feature Store & Training-Serving Skew
• Chapter 9: CI/CD for ML — Versioning, Eval Gates, "Tests Are Evals"
• Chapter 10: Monitoring & Observability — Drift, Decay, the Feedback Loop
• Chapter 11: LLMOps — Prompts, Eval-Driven Dev, Token/Cost Tracking
• Chapter 12: The Maturity Model & the Org/Culture Reality
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: What MLOps Is, and Why ML Systems Rot Differently

From zero. MLOps is DevOps for machine-learning systems: the practices, tooling, and culture that take a model from a notebook to a reliable, observable, continuously-improvable production service — and keep it there. But it is not just "DevOps with a model in the build." The reason it's a separate discipline is that an ML system has a property no ordinary software has: its behavior is defined by data it learned from, and that data keeps changing after you ship.

Why ML systems rot differently. Hold three properties in your head; everything in this phase exists to manage one of them.

1. Data dependencies, not just code dependencies. A traditional program is f(input) where f is fixed by code. A model is f_θ(input) where θ (the weights) were learned from a dataset. So the artifact you deploy is code + weights + the data and config that produced the weights. If you can't version all three, you can't reproduce a result, can't bisect a regression, and can't pass an audit. Sculley et al. call the resulting mess "the high-interest credit card of technical debt" — entanglement (changing one feature changes everything, "CACE: Changing Anything Changes Everything"), hidden feedback loops, undeclared consumers of your model's outputs, and data pipelines held together with glue code.

2. Silent failure. A null-pointer bug throws; you see a stack trace, a 500, an alert. A stale model throws nothing. It keeps returning confident, well-formed, wrong answers as the world drifts away from its training distribution, and your uptime dashboard stays green the whole time. The failure mode is economic, not exceptional — you lose money/quality slowly and invisibly. This is why you must monitor the predictions and the input distribution, not just the latency and error rate.

3. Training-serving skew. You compute features one way in the training pipeline (batch, in pandas, over a warehouse) and another way at serving time (online, in a service, per request). The two implementations drift apart — a different default-fill, a timezone, a units bug, a leak of future information — so the model sees different inputs in production than it trained on and silently underperforms. Your offline metric was honest; the skew made it a lie.

The senior's mental model. A model is a perishable asset on a closed loop, not a static binary. The loop is data → train → track → register → deploy → monitor → (drift) → retrain. The job of MLOps is to make that loop automated (you can turn it without heroics) and observable (you know which part is on fire). The phase's three labs own the three hard arcs: track+register (lab-01), monitor (lab-02), deploy safely (lab-03).

Common misconception. "MLOps is just putting the model behind a REST API." Serving is one box in the diagram. The reason the model needs a platform is the loop above — provenance, drift, safe release, and retraining — none of which a FastAPI wrapper gives you.

Chapter 2: Experiment Tracking — Runs, Params, Metrics, Artifacts, Lineage

What it is. Experiment tracking is the system of record for every training trial. The unit is a run: one execution of a training/eval script, which records the parameters it used, the metrics it produced, the artifacts it emitted, and the tags/metadata (who, when, which git commit, which dataset version) that situate it. A tracking server is, in essence, a structured, queryable log of "we tried this and got that."

Why it exists. Without it, model development is a graveyard of un-reproducible notebooks and filenames like model_final_v2_REALLY_final.pkl. The questions tracking answers — which hyperparameters gave the best validation AUC? what produced the model now in production? can I reproduce last quarter's result? — are unanswerable from memory and Slack. Tracking turns ML development from folklore into a database query.

Under the hood — the four record types, and why their mutability differs.

• Params are the inputs: hyperparameters and config (learning rate, depth, dataset version). They are immutable and single-valued per run. Re-logging a different value for the same key is treated as an error, because a run's parameters must be a faithful record of what produced its metrics. If params could change, lineage would lie. (Lab-01 enforces exactly this.)
• Metrics are the outputs, and they are a time-series: each metric is a list of (step, value) observations, so you keep the whole learning curve (loss per epoch, val-AUC per checkpoint), not just the last number. This is what lets you plot training, detect overfitting, and let a best_run query pick the right checkpoint.
• Artifacts are blobs: the serialized model, a confusion-matrix PNG, a sample of predictions, the feature importances. They're stored by name.
• Tags are mutable free-form metadata: owner, git sha, dataset version, "sweep-v1." Mutable because you annotate runs after the fact; they're not part of the immutable provenance contract.

Lineage and reproducibility. The payoff of tracking is lineage: given a model, walk back to the run, and from the run read the exact params, metrics, code commit, and data version that produced it. Reproducibility is a triple — code (git) + data (a dataset/feature version) + config (params). Tracking owns the config and metric history; you bolt on data versioning (Chapter 4) and code versioning (git) to complete it. A run with a captured git sha and dataset hash is reproducible; a run with neither is an anecdote.

Production significance. Tracking is the cheapest, highest-leverage thing in MLOps. It is the first thing a serious team installs and the thing that pays off at 2 a.m. when someone asks "what changed?" The model registry (Chapter 5) is literally built on top of it — a registered model version points back to a run.

Common misconception. "We'll just log to a CSV / TensorBoard." TensorBoard visualizes one run's curves beautifully but is not a queryable, multi-run, team-shared system of record with a registry attached. The moment two people or two weeks are involved, you need a real tracking store.

Chapter 3: MLflow in Depth — The Four Components

What it is. MLflow is the de-facto open-source MLOps backbone: an Apache-licensed platform with four loosely-coupled components you can adopt independently. Knowing the four — and the storage split underneath them — is table stakes for any MLOps interview.

1) MLflow Tracking. The API and server for runs. The core calls:

import mlflow
mlflow.set_experiment("fraud-classifier")
with mlflow.start_run() as run:                 # opens a run, auto-ends on exit
    mlflow.log_param("lr", 0.05)                # immutable input
    mlflow.log_metric("val_auc", 0.91, step=3)  # time-series output
    mlflow.log_artifact("confusion_matrix.png") # a blob
    mlflow.set_tag("git_sha", "abc123")

Programmatically you query through MlflowClient (search_runs, get_run, …). Lab-01 is a direct reimplementation of this API surface.

The storage split (the thing juniors miss). A tracking server has two stores:

• Backend store — the structured data: experiments, runs, params, metrics, tags. Lives in a database or filesystem (--backend-store-uri, e.g. sqlite:///mlflow.db or a Postgres URL).
• Artifact store — the blobs: models, images, files. Lives in object storage (--default-artifact-root, e.g. s3://bucket/mlflow or a local dir).

They're separate because metrics are small and transactional (a DB job) while artifacts are large and immutable (an object-store job). Lab-01 keeps artifacts inline for simplicity but the README calls out the split — name it in the interview.

2) MLflow Models. A standard packaging format so a model can be saved once and served many ways. A saved model is a directory with an MLmodel YAML file that declares one or more flavors — different ways to load the same model:

flavors:
  python_function:               # the universal flavor: a predict(df) -> df contract
    loader_module: mlflow.sklearn
  sklearn:                       # the native flavor: load the real sklearn object
    sklearn_version: 1.4.0
signature:                       # input/output schema (columns + dtypes)
  inputs: '[{"name": "amount", "type": "double"}, ...]'

The python_function (pyfunc) flavor is the universal contract — any MLflow model exposes a predict() you can call without knowing the framework — which is what lets one serving runtime serve sklearn, XGBoost, PyTorch, and a custom model identically. The signature pins the input/output schema so a malformed request fails fast instead of silently mis-predicting.

3) MLflow Model Registry. The lifecycle layer on top of Models: named registered models, each an append-only list of versions, each version bound to the run that produced it (lineage), with a stage (None/Staging/Production/Archived) you transition through. This is Chapter 5 and lab-01's ModelRegistry. The registry is what turns "a file" into "the thing in production, version 7, promoted by Alice on the 3rd, traceable to run-0042."

Note on stages vs aliases. Classic MLflow used the four fixed stages. MLflow ≥ 2.9 deprecates fixed stages in favor of model aliases (@champion, @challenger) plus tags, for more flexible blue/green and multi-model patterns. Learn the stage state machine first (it's the cleaner mental model and what lab-01 builds); know aliases are where the tool is going.

4) MLflow Projects. A convention (an MLproject file) for packaging code so a run is reproducible — it declares the entry points, parameters, and environment (conda/pip/Docker) so mlflow run . reproduces a training run on any machine. Less universally adopted than the other three, but it's the component that completes the reproducibility story (the code side of the triple).

Production significance. MLflow is the lingua franca; even teams on W&B or a cloud platform often speak "MLmodel flavors" and "registry stages." If you can explain the four components and the backend/artifact split, you sound like you've run a platform.

Common misconception. "MLflow is the model server." MLflow can serve a model (mlflow models serve), but its value is the format and registry; in production you usually load the registry's pyfunc model into a real serving runtime (Chapter 6), not run MLflow's dev server.

Chapter 4: The Tracking Landscape — W&B, Neptune, Comet, DVC

What they are. MLflow has company. The differences are real and interview-relevant.

DVC deserves its own paragraph because it solves a different axis. Tracking versions runs; DVC versions the data and the pipeline. It stores large files as content-addressed blobs in a remote (S3/GCS) and keeps small .dvc pointer files in git, so git checkout of an old commit restores both the code and the exact data that went with it. DVC + git is how you make the data leg of the reproducibility triple real. (Delta Lake / lakeFS / Iceberg "time travel" do the same at warehouse scale.)

Under the hood — what's the same. All trackers share the run/param/metric/artifact model; the differences are hosting (SaaS vs self-host), scale (Neptune's many-runs focus), polish (W&B's UI), and scope (Comet adds monitoring, DVC adds data versioning). The mechanism lab-01 builds is the common denominator.

Production significance. The choice is usually MLflow (self-host, open, registry) vs W&B (hosted, best UI, deep-learning teams) — plus DVC for data versioning regardless. Don't adopt three overlapping trackers; that's resume-driven tool sprawl (Chapter 12).

Common misconception. "DVC and MLflow compete." They're complementary: DVC = data/pipeline versioning, MLflow = run tracking + registry. Mature teams run both.

Chapter 5: The Model Registry & Versioning

What it is. A model registry is the central catalog of deployable models. A registered model is a named entity (fraud-model); under it live immutable, auto-incrementing versions (v1, v2, …), each bound to the run that produced it; each version sits in a stage that you transition through a defined lifecycle. It's the source of truth for "what is deployable, what is live, and what produced it."

Why it exists. Without a registry, "the production model" is a .pkl in someone's S3 bucket with no provenance, no version history, no rollback, and no record of who approved it. The registry gives you versioning (an append-only history), stages (a controlled path to production), lineage (version → run → params/data), approvals (a human gate before Production), and rollback (re-promote the previous Production version). It is the operational backbone of the whole phase.

Under the hood — the stage state machine. The four stages and the legal transitions between them (lab-01 enforces exactly this):

        ┌──────────────────────────────────────────────────┐
        ▼                                                    │
     ┌──────┐  promote  ┌─────────┐  promote  ┌────────────┐ │ re-stage
     │ None │──────────▶│ Staging │──────────▶│ Production  │ │
     └──────┘           └─────────┘           └────────────┘ │
        │  ▲                 │  ▲                    │        │
        │  │demote           │  │demote     archive  │        │
        ▼  │                 ▼  │                    ▼        │
     ┌──────────────────────────────────────────────────┐   │
     │                    Archived  ────────────────────────┘
     └──────────────────────────────────────────────────┘
   Legal: None→{Staging,Production,Archived}; Staging→{Production,Archived,None};
          Production→{Archived,Staging}; Archived→{Staging,None}.
   ILLEGAL: Archived→Production directly (must be re-staged & re-validated first).

The state machine is the point. You cannot resurrect an Archived model straight to Production — it must go back through Staging so it's re-validated, because the world has moved since it was archived. And archive_existing=True when promoting a new version to Production atomically demotes the incumbent to Archived, which is how the registry guarantees exactly one Production version is live. That guarantee is the registry's reason to exist; in lab-01 it's the "soul test."

Rollback falls out of this for free: to roll back, you transition_stage the previous version back to Production (with archive_existing=True to demote the bad one). No retraining, no redeploy of code — just flip the registry pointer.

Lineage is the other payoff: get_model_lineage(name, version) walks version → run → the params, metrics, and data that produced it. This is the answer to the most common MLOps fire: "what model is in prod, and what trained it?"

Production significance. The registry is where MLOps stops being "training tooling" and becomes "operations." Promotion, rollback, audit, and on-call all live here. If you build one thing in this phase, build this; lab-01 does.

Common misconception. "Versioning models is just saving files with version numbers." A registry adds the state machine, the lineage, the approvals, and the single-Production-version invariant — the operational semantics, not just a filename suffix.

Chapter 6: Model Packaging & Serving

What it is. Packaging turns trained weights into a portable, loadable artifact with a defined input/output contract; serving runs it to answer prediction requests. The landscape splits along two axes: what format and online vs batch.

Formats and runtimes (the landscape to be able to name):

Online vs batch (the deployment-pattern axis):

• Online (real-time) serving — a request comes in, you return a prediction in milliseconds (fraud check at checkout). Latency-sensitive; needs a live service, autoscaling, and the feature-store online path (Chapter 8).
• Batch (offline) inference — score a whole table on a schedule (nightly churn scores written to a warehouse). Throughput-sensitive, latency-irrelevant; just a scheduled job (Chapter 7).
• Streaming — score events off a queue (Kafka) as they arrive; the middle ground.

The format/runtime is downstream of this choice: batch is often "load the pyfunc model in a Spark job"; online is "Triton/KServe/BentoML behind a load balancer."

Under the hood — why a format matters. The MLmodel/ONNX abstraction decouples training (many frameworks) from serving (one runtime). You train in PyTorch, export ONNX or pyfunc, and the server doesn't need PyTorch. This is the same decoupling the registry provides at the lifecycle level — both exist to keep "what produced it" separate from "how we run it."

Production significance. Choosing batch vs online is a cost and architecture decision (online needs a feature store and a live fleet; batch needs neither). Choosing the runtime is a latency and ops decision. Seniors pick the simplest one that meets the latency SLO — batch if you can, online only if you must.

Common misconception. "Everything must be real-time." A huge fraction of ML value is delivered by a nightly batch job writing scores to a table. Online serving is expensive operational surface; don't pay for it unless the use case needs sub-second freshness.

Chapter 7: Pipelines & Orchestration

What it is. An ML workflow is a DAG (directed acyclic graph) of steps: ingest → validate → featurize → train → evaluate → register. An orchestrator schedules that DAG, runs steps in dependency order, retries failures, caches unchanged steps, and gives you observability into each run. It's the automation substrate of the whole loop.

The tools (name them and their flavor):

Under the hood — the three things an orchestrator gives you.

1. Dependency execution. It runs steps in topological order and parallelizes independent branches. You declare what depends on what; it figures out when to run each.
2. Scheduling & triggering. Cron-style ("retrain nightly") or event-based ("retrain when the drift monitor fires" — the closed loop of Chapter 10).
3. Caching & idempotency. If the featurization step's inputs didn't change, skip it and reuse the cached output. This is what makes iterating on a 6-step pipeline tolerable — you don't re-run the 2-hour ingest to fix the eval step.

Production significance. The orchestrator is where "I ran a script" becomes "the pipeline runs reliably, on a schedule, with retries and lineage." It's also where the retrain trigger lives — the arc that closes the MLOps loop. Lab-03's pipeline DAG is a miniature of this.

Common misconception. "Airflow is for ML pipelines." Airflow is a general workflow scheduler; it's batch/cron-centric and not data-aware. Dagster/Kubeflow/Metaflow were built for ML/data and add asset awareness, containerization, and versioning. Pick by whether you need general scheduling or data-asset semantics.

Chapter 8: The Feature Store & Training-Serving Skew

The problem it solves. Recall training-serving skew (Chapter 1): features computed one way for training and another way for serving drift apart, and the model silently underperforms. A feature store is the system that makes the two paths compute features identically and serve them at the right freshness.

What it is. A feature store has two synchronized paths over the same feature definitions:

• the offline store — historical feature values (in a warehouse), used to build training sets;
• the online store — the latest feature values (in a low-latency KV store like Redis), used to serve predictions in milliseconds.

You define a feature once; the store materializes it to both. Feast is the open-source reference; cloud platforms (Vertex, SageMaker, Databricks, Tecton) have managed ones.

Under the hood — point-in-time correctness (the hard part). When you build a training set, each training example must see the feature values as they were at that example's timestamp — not the latest values. If you join in a feature computed after the label event, you've leaked the future into training ("label leakage"), and your offline metric is fantasy. A feature store does a point-in-time (as-of) join: for each event at time t, fetch the feature value valid at t. This is the single most error-prone thing in feature engineering, and it's why a feature store is worth the operational cost.

Production significance. Feature stores fight the two most expensive ML bugs: training-serving skew (same definition both paths → no skew) and leakage (point-in-time joins → honest offline metrics). They also give feature reuse across teams. But they're heavy infrastructure — Chapter 12 says don't adopt one until skew is actually hurting you.

Common misconception. "A feature store is just a feature cache." The cache (online store) is the easy half. The hard, valuable half is the point-in-time join for training-set construction and the single definition shared across offline and online — that's what kills skew.

Chapter 9: CI/CD for ML — Versioning, Eval Gates, "Tests Are Evals"

What it is. CI/CD for ML extends continuous integration/delivery to the ML triple. Google's "CD4ML" (Continuous Delivery for Machine Learning) frames it as a discipline for reproducibly producing and safely releasing models. The twist: in software, CI runs unit tests; in ML, CI must run evaluations, because "the code compiles" says nothing about whether the model is good.

Under the hood — what gets versioned and tested.

1. Three things versioned together: code (git), data (DVC/Delta), model (registry). A CI run pins all three so its result is reproducible.
2. Data validation tests — schema, ranges, null rates, distribution checks on the incoming data before you train (garbage-in guard).
3. The eval gate ("tests are evals"). Before a candidate model is allowed to promote, it must clear a quality bar vs the incumbent on a held-out eval set — and not regress on safety or on important slices (tie this to Phase 16's eval harness). If the candidate scores higher overall but worse on a protected slice, the gate blocks the promotion. This is the ML equivalent of "tests must pass before merge" — except the test is an eval, and a green eval is the merge criterion. Lab-03 builds exactly this gate.
4. Continuous training (CT). The fully automated loop: new data → retrain → eval-gate → canary → promote, triggered by a schedule or a drift alert.

The GitHub Actions shape. In practice this is a workflow that, on a PR or a trigger, pulls the pinned data, trains, runs the eval suite, and fails the job if the gate isn't cleared — then, on the protected branch, registers the new version and kicks off the canary release. The eval suite is a pytest-shaped thing where each "test" asserts a metric threshold.

Production significance. Eval gates are what stop a well-meaning teammate from shipping a regression. They convert "we think it's better" into "it provably clears the bar, gated in CI." This is the single most senior idea in ML release engineering.

Common misconception. "CI for ML means the training script runs without error." That's necessary and useless — the script running tells you nothing about model quality. The gate must be an eval against a threshold, not an exception check.

Chapter 10: Monitoring & Observability — Drift, Decay, the Feedback Loop

What it is. Production ML monitoring watches the model's behavior and inputs, not just the service's health. It answers "is the model still good?" — a question uptime can't. Lab-02 builds the core of it.

Under the hood — the kinds of drift.

• Data (covariate) drift — the input distribution P(X) changes (new user demographics, a new product, seasonality). The model may still be valid but is now extrapolating.
• Concept drift — the relationship P(y|X) changes (fraud patterns evolve to evade your model; "spam" means something new). The mapping the model learned is now wrong even on familiar inputs.
• Prediction drift — the output distribution shifts (your fraud rate suddenly triples). A cheap early-warning proxy when you don't have labels yet.

The detectors (the math lab-02 implements).

• PSI (Population Stability Index) — bin a feature; compare the live distribution to a reference (training) distribution: ( \text{PSI} = \sum_i (p_i^{live} - p_i^{ref}) \ln \frac{p_i^{live}}{p_i^{ref}} ). Rules of thumb: < 0.1 stable, 0.1–0.25 moderate shift, > 0.25 significant drift — investigate.
• KS (Kolmogorov-Smirnov) two-sample statistic — the max gap between the two empirical CDFs; a distribution-free test for "are these two samples from the same distribution?"
• Performance decay — when labels eventually arrive (often delayed — you learn the true fraud label weeks later), track the real metric (AUC/accuracy) over time and alarm when it falls below a floor. This is the ground truth; drift metrics are the leading indicator you watch while you wait for it.

The feedback loop. Monitoring isn't passive. When PSI or decay crosses a threshold, it should fire a retrain trigger (Chapter 7's orchestrator) — closing the loop monitor → retrain → register → deploy. That automated closure is what separates a mature platform from a dashboard someone occasionally looks at. The tools: Evidently (open-source drift/quality reports), Arize, WhyLabs, Fiddler (hosted ML observability).

Production significance. This is the chapter that prevents the silent-failure money-leak of Chapter 1. Your real SLOs are drift and performance metrics, not just p99 latency. A green latency dashboard over a drifting model is the most expensive kind of "fine."

Common misconception. "If accuracy is fine, there's no drift problem." Labels are often delayed by weeks; by the time accuracy drops you've been wrong for a month. Watch the input drift (PSI/KS) as the leading indicator and the delayed accuracy as confirmation.

Chapter 11: LLMOps — Prompts, Eval-Driven Dev, Token/Cost Tracking

What it is. LLMOps is MLOps specialized for LLM applications, where you usually don't train the model — you compose prompts, retrieval, and tools around a frontier model. The lifecycle shifts: less "train and register weights," more "version prompts, eval the system, and track tokens/cost." This ties directly to Phase 16 (evaluation & observability).

What changes vs classic MLOps.

• The "model" is a prompt + config. Your versioned, registry-managed artifact is often a prompt template + few-shot examples + model id + temperature + tool definitions, not weights. Prompt versioning (and a prompt registry) is the LLMOps analogue of the model registry.
• Eval-driven development. You can't unit-test "is this answer good," so you build an eval suite (LLM-as-judge, reference matching, rubric scoring — Phase 16) and treat passing the eval set as the release gate. "Tests are evals" (Chapter 9) is the LLMOps workflow.
• Token & cost tracking. Every call has a token count and a dollar cost; tracking cost per request and latency per request is a first-class production metric (it's your variable cost).
• Tracing. A single user turn fans out into prompt → retrieval → tool calls → completion; you need distributed tracing of that chain to debug and to compute cost. Langfuse (open) and LangSmith (LangChain) are the LLM-native observability/tracing/eval platforms; they are to LLMOps what MLflow + Evidently are to classic MLOps.

Under the hood — why tracing is the unit. In classic ML the unit of observability is a run or a prediction. In LLMOps it's a trace: a tree of spans (each prompt, retrieval, tool call) with inputs, outputs, tokens, latency, and cost. You version prompts, attach eval scores to traces, and watch cost/quality over time. Langfuse/LangSmith store traces the way MLflow stores runs.

Production significance. When a senior says "we run LLMs in prod," they mean: prompts are versioned, there's an eval suite gating changes, every request's tokens/cost/latency are tracked, and traces are searchable. Without those, an LLM app is a demo.

Common misconception. "LLMs don't need MLOps because we don't train them." You don't version weights, but you do version prompts, gate on evals, track cost, and trace requests — arguably more operational discipline, because the failure modes (hallucination, prompt regressions, cost blowouts) are subtle and the system is non-deterministic.

Chapter 12: The Maturity Model & the Org/Culture Reality

What it is. Google's MLOps maturity model gives names to how automated your loop is:

• Level 0 — Manual. Notebooks, hand-offs, scripts run by humans. A model ships rarely; no CI/CD, no monitoring. Most teams start here, and it's fine for a first model.
• Level 1 — ML pipeline automation. The training pipeline is automated and parameterized; you have continuous training triggered by data/schedule, plus basic monitoring. Retraining doesn't need a human to run a notebook.
• Level 2 — CI/CD pipeline automation. The pipeline itself is built, tested, and deployed through CI/CD; eval gates, canary, automated rollback. The loop runs with minimal human touch.

The org/culture reality (the part nobody puts on a slide). Most "MLOps problems" are organizational, not technical:

• Resume-driven tool sprawl. Teams adopt a feature store, three trackers, and Kubeflow before they've shipped a second model, because the tools are exciting. The result is a platform nobody can operate. Match the maturity level to the actual need. Tracking + a registry (lab-01) is ~80% of the value; add monitoring (lab-02) when models hit production; add CI/CD/canary (lab-03) when releases get scary; add a feature store only when skew is measurably hurting you.
• Ownership gaps. Data scientists hand a model "over the wall" to engineers; nobody owns the loop. The senior AI engineer's role is often precisely to own the loop end to end.
• "We have no idea which model is in prod." The single most common real failure — solved by the boring core (registry + lineage), not by a platform purchase.

Production significance. The senior judgment in this phase is not "deploy Kubeflow." It's "diagnose the maturity level we need and build the smallest platform that gets us there." Knowing all the tools is table stakes; knowing which to skip is what gets you hired.

Common misconception. "More MLOps tooling = more mature." Maturity is about automation of the loop and observability of the model, not the size of your tool inventory. A team with tracking, a registry, and a drift alert is more mature than one with ten tools and no idea what's in prod.

Lab Walkthrough Guidance

The phase has three labs; you build lab-01 here, and the WARMUP equips you for all three.

Lab-01 — MLflow-style Tracking & Registry (this lab). Suggested order (matches the file):

1. Run.latest_metric + TrackingStore ids — Chapter 2. The id counter is the determinism trick: run-0001, run-0002, no uuid.
2. create_experiment / create_run — namespace + run creation; validate parents exist.
3. log_param (immutable) / log_metric (step history) / log_artifact / set_tag — Chapter 2's mutability contract. The param-immutability check is a correctness test.
4. search_runs / best_run / compare_runs — model selection; best_run is the soul of an HPO loop, and it uses the latest metric value.
5. ModelRegistry — Chapter 5. create_model_version auto-increments; transition_stage enforces _ALLOWED_TRANSITIONS; archive_existing=True is the registry soul test; lineage resolves version → run → params/metrics.

Run red → green: pytest test_lab.py -v, then LAB_MODULE=solution pytest test_lab.py -v, then python solution.py for the worked sweep-and-promote example.

Lab-02 — Drift Detection & Monitoring. You'll implement PSI and the two-sample KS statistic (Chapter 10), a binned reference-vs-live comparator, a delayed-label performance-decay tracker, and an alert rule that returns a retrain trigger when a threshold is crossed. Watch for the ( \ln(p^{live}/p^{ref}) ) zero-bin guard (smooth empty bins) — the numerical-stability lesson.

Lab-03 — CI/CD Deployment Pipeline. You'll build a pipeline DAG (validate → eval-gate → canary → promote/rollback), an eval gate that blocks promotion unless the candidate clears the incumbent on the eval set and doesn't regress a guard metric (Chapter 9), a canary that steps traffic and watches a guard, and automatic rollback on breach (Chapter 5's "re-promote previous"). The soul test: a candidate that wins overall but regresses a slice is blocked.

Success Criteria

• Lab-01's tests pass against your lab.py and against LAB_MODULE=solution.
• You can explain why ML systems rot in place (data dependencies, silent failure, training-serving skew) and name the loop data→train→track→register→deploy→monitor→retrain.
• You can state the mutability contract: params immutable, metrics time-series, tags mutable — and why params re-logged with a different value must raise.
• You can name MLflow's four components and the backend-vs-artifact store split, and explain the MLmodel/pyfunc flavor abstraction.
• You can draw the registry stage state machine, name the illegal edge (Archived→Production), and explain what archive_existing=True guarantees.
• You can compute a PSI and a KS statistic and say what each detects; you can describe an eval gate + canary + rollback release and the role of a feature store in killing skew.
• You can place a team on the maturity model and argue what tooling to skip.

Interview Q&A

• "Why is MLOps different from DevOps?" — Behavior is set by data, not just code (version the triple); failures are silent (monitor predictions, not uptime); training and serving paths skew. Software ships once; ML is a loop that rots in place.
• "Walk me through experiment tracking. Why are params immutable but metrics not?" — Params are the run's inputs and must be a faithful record of what produced the metrics, so re-logging a different value is an error; metrics are outputs and a time-series (full step history) so you can plot curves and pick the best checkpoint.
• "Name MLflow's four components." — Tracking (runs/params/metrics/artifacts), Models (the MLmodel format + flavors, esp. pyfunc), Model Registry (versions + stages + lineage), Projects (reproducible code packaging). Bonus: backend store (DB) vs artifact store (object storage).
• "Explain the model registry stage state machine and rollback." — None→Staging→Production→ Archived; Archived can't jump straight to Production (re-stage to re-validate); archive_existing demotes the incumbent so exactly one Production version is live; rollback = re-promote the previous version with archive_existing=True.
• "What's training-serving skew and how do you prevent it?" — Features computed differently in the train vs serve paths diverge, so the model sees different inputs in prod than in training. A feature store with a single feature definition materialized to both offline and online stores (and point-in-time joins for training sets) prevents skew and leakage.
• "How do you do CI/CD for a model — what's an eval gate?" — Version code+data+model; on a trigger, train and run an eval suite; block promotion unless the candidate clears the incumbent's bar and doesn't regress safety/slices ("tests are evals"); then canary + auto-rollback.
• "Data drift vs concept drift — how do you detect each?" — Data drift: P(X) shifts, detect with PSI/KS on inputs (no labels needed). Concept drift: P(y|X) shifts, detect with performance decay once labels arrive. Use input drift as the leading indicator while you wait for delayed labels.
• "What does PSI tell you and what's a concerning value?" — Population Stability Index measures distribution shift per binned feature; <0.1 stable, 0.1–0.25 moderate, >0.25 significant — investigate/retrain.
• "Batch vs online serving — how do you choose?" — Latency SLO. Batch (scheduled scoring to a table) is cheaper and simpler; choose online only when you need sub-second freshness, accepting a live fleet + feature-store online path.
• "How is LLMOps different?" — You version prompts (not weights), gate on an eval suite, track tokens/cost/latency per request, and trace the prompt→retrieval→tool chain (Langfuse/ LangSmith). Arguably more operational discipline, not less.
• "It's 2 a.m., the model is making bad calls — what do you do?" — Check the registry for the live version and its lineage (what data/params), check the drift monitor (is P(X) shifted), roll back to the previous Production version via the registry (no retrain), then investigate.
• "How much MLOps tooling should a team have?" — Match the maturity level. Tracking + registry first (~80% of value), monitoring when models hit prod, CI/CD/canary when releases get scary, a feature store only when skew bites. Tool sprawl is a smell, not maturity.
• "How do you make a result reproducible?" — Version the triple: code (git), data (DVC/Delta), config/model (tracking + registry). A run with a git sha + dataset hash + params is reproducible.
• "What's the most common real MLOps failure?" — "We don't know which model is in prod / what trained it." Solved by the boring core — registry + lineage — not a platform purchase.

References

• Sculley et al., Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015) — the foundational "ML code is a small box; the plumbing is the system" paper (CACE, glue code, feedback loops).
• Google Cloud, MLOps: Continuous delivery and automation pipelines in ML — the maturity model (levels 0/1/2) and continuous training.
• ThoughtWorks, Continuous Delivery for Machine Learning (CD4ML) — the CI/CD-for-ML discipline.
• MLflow documentation — Tracking, Models (the MLmodel format & flavors), Model Registry, Projects; backend vs artifact store; model aliases (≥ 2.9).
• Weights & Biases, Neptune, Comet docs — the hosted tracking landscape.
• DVC documentation — data & pipeline versioning (Git for data).
• Feast documentation — the open-source feature store; point-in-time joins, online/offline.
• Evidently documentation — drift/data-quality reports (PSI, KS, and the test suites).
• BentoML / KServe / Seldon / Triton / TorchServe docs — the serving landscape.
• Airflow / Dagster / Prefect / Kubeflow Pipelines / Metaflow docs — orchestration.
• Langfuse / LangSmith docs — LLM tracing, prompt management, eval (ties to Phase 16).
• "Rules of Machine Learning" (Martin Zinkevich, Google) — the field's most-quoted practical wisdom.