Warmup Guide — Training Orchestration & MLOps

Zero-to-expert primer for Phase 08: the five mechanisms inside every ML platform — DAGs, content-addressed caching, retries, lineage, and promotion gates — plus the surrounding practice: experiment tracking, registries, and CI/CD for models.

Table of Contents

Chapter 1: What MLOps Actually Is

Strip the vendor marketing and MLOps is one sentence: applying software delivery discipline to artifacts that depend on data as well as code. The "as well as data" clause is what makes it a distinct field:

  • Code changes are diffable and reviewable; data changes are silent and continuous — so validation must be runtime, not review-time (Phase 01's contracts, Phase 12's drift detection).
  • A binary either passes tests or doesn't; a model is statistically good — so gates are metric thresholds with baselines, not assertions (Phase 09 of the model-accuracy track built exactly this for accuracy; this phase generalizes it).
  • Builds are seconds; training runs are hours — so caching and incremental recomputation are economics, not conveniences (Chapter 3).

The platform zoo (Airflow, Dagster, Kubeflow, MLflow, SageMaker, Vertex) all implement the same five mechanisms this phase teaches. Learn the mechanisms; the YAML dialects follow in an afternoon.

Chapter 2: Pipelines as DAGs

From zero: a pipeline is tasks + dependencies = a directed graph; requiring it be acyclic (a DAG) is what makes execution decidable — a cycle means "A needs B needs A," which is a specification bug, and your orchestrator must detect and reject it, not hang (the lab tests this).

The mechanics:

  • Topological sort produces a valid execution order: repeatedly emit a node whose dependencies are all emitted (Kahn's algorithm: maintain in-degrees, pop zeros). Multiple valid orders exist; independent tasks are the parallelism opportunity.
  • Fan-out/fan-in: one upstream feeding N independent tasks (per-country training) then merging — the shape that makes DAGs strictly more useful than scripts.
  • Tasks are functions of their inputs — the discipline that makes everything else (caching, retries, lineage) possible. Hidden state (a task reading a global path another task mutates "by convention") is the original sin of pipeline design; every input must be an explicit, declared edge.
  • Real-system mapping: Airflow tasks declare >> dependencies (scheduling-only edges — data passing is on you); Dagster's assets and Kubeflow's typed inputs/outputs make the data edges first-class, which is the modern direction and the lab's design.

Chapter 3: Content-Addressed Caching

The economic core of every serious pipeline system (also: Bazel, Nix, DVC, dbt):

The key insight: a task's output is fully determined by (its code, its parameters, its inputs). So cache by content:

$$\text{cache_key} = H(\text{task code version} ,|, \text{params} ,|, H(\text{input}_1) ,|, \cdots)$$

Key consequences, each of which the lab turns into a test:

  • Hash, not timestamps: timestamps (Make's method) lie under re-runs, clock skew, and re-materialized-but-identical data. Content hashes are ground truth: same bytes, same key, skip.
  • Invalidation propagates automatically: change one param in task B → B's key changes → B re-runs → B's output hash changes → downstream keys change → exactly the affected subgraph re-executes. Nothing upstream re-runs. This is the whole feature, derived from the key structure — no invalidation logic exists anywhere.
  • The code-version term: if the key omits the task's code identity, editing the logic silently serves stale artifacts — the most common homemade-cache bug (production sightings: weekly). Real systems hash the function source, the container digest, or demand an explicit version= bump (the lab's approach).
  • Nondeterminism poisons caching: a task with an unseeded RNG produces different outputs for the same key — downstream caching becomes incoherent. Determinism (Ch. 9) is a caching requirement, not just a science one.

Chapter 4: Idempotency and Retries

Infrastructure fails transiently (OOM, spot preemption, network); orchestrators retry. Retrying is only safe if tasks are idempotent — running twice has the same effect as once:

  • Write outputs to a staging location, then atomically rename/commit — a task killed mid-write must not leave a half-artifact that a retry then treats as done (or that downstream reads). This is the same atomic-commit pattern as the PMC track's release staging repos and Kafka's log: stage, verify, publish.
  • Classify failures: transient (timeout, 5xx, OOM-on-noisy-neighbor) → retry with backoff; permanent (validation error, missing input, code bug) → fail fast, no retry. Retrying a permanent failure wastes hours and hides the bug in retry noise; the lab's retry policy takes an is_transient classifier for exactly this reason.
  • Bounded retries with backoff + jitter — unbounded retries turn one incident into a queue flood (the thundering-herd lesson every distributed-systems curriculum repeats because every on-call relearns it).

Chapter 5: Artifact Lineage

The question lineage answers is the one every incident review asks: "this model is misbehaving — what data, code, and params produced it?" And its inverse: "this upstream table was corrupted Tuesday — which models are tainted?"

The mechanism (the lab's ledger): every task execution appends an immutable record — (artifact_hash, task, code_version, params, input_artifact_hashes, timestamp). Lineage queries are then graph walks: upstream closure of a model = everything it was made from; downstream closure of a dataset = everything made from it (the taint query). Append-only matters: ledgers you can edit are ledgers you can't trust during an incident — the same audit-trail discipline as Phase 10's vote parser in the PMC track.

Production mapping: MLflow's run/artifact metadata, OpenLineage events, SageMaker/Vertex lineage tracking — all this record, at enterprise scale.

Chapter 6: Experiment Tracking

Training is experimentation; untracked experiments are unrepeatable anecdotes. The minimum viable record per run:

  • Params (all of them — including the ones you "didn't change": defaults drift), metrics (logged over steps, not just finals — curves diagnose what finals hide), artifacts (the model, the eval report, the confusion matrix PNG), environment (git SHA, package lock hash, data version), seed.
  • The discipline that makes runs comparable: same eval data + same metric code = comparable; anything else is two different experiments. Pin the eval set by content hash in the run record (the Phase 09 model-accuracy lesson, recurring).
  • What separates tracking from a CSV: queryability ("all runs on dataset X with lr<0.01, by AUC") and artifact linkage (click from metric to model). That's the whole product surface of MLflow/W&B — Extension A has you build it in 100 lines to demystify it.

Chapter 7: Model Registries and Promotion

A registry is a state machine over model versions (the PMC track's issue-FSM pattern, applied to artifacts):

candidate → staging → production → archived
  • Transitions carry gates: candidate→staging requires the offline quality gate (metric ≥ baseline on the pinned eval set, fairness slices pass); staging→ production requires online verification (shadow or canary — Phase 09 of this track); any→archived records why (superseded / regressed / deprecated).
  • Versions are immutable; promotion moves pointers, never rewrites artifacts — rollback is then a pointer move too (the deployment-safety property that matters at 2am).
  • The registry is the deployment interface: serving infrastructure pulls "production" by alias, not by version number — which is what makes promotion and rollback deployment-free operations.

Chapter 8: CI/CD for ML

What "CI" means when the artifact is a model — the four test layers (Extension B builds them):

  1. Code tests — ordinary pytest; fast; on every commit.
  2. Data validation — contracts on training inputs (schema, ranges, volumes, class balance); on every pipeline run. A training run on garbage data that succeeds is the worst outcome — it produces a plausible-looking bad model.
  3. Training smoke test — train on a tiny fixture dataset in CI (<2 min): does the loss decrease, does the pipeline run end-to-end, does the artifact serialize/deserialize? Catches the 80% of training bugs that need no GPU.
  4. Quality gates — full training (scheduled or on-merge): metric vs pinned baseline with significance (Phase 11's statistics), golden-prediction diffs on a fixed probe set (catches "metric same, behavior different"), latency/size budget checks.

CD then = registry promotion + progressive rollout (canary/shadow, Phase 09) + automated rollback on guardrail breach (Phase 12). The pattern sentence for interviews: models ship like code, but with statistical gates where code has assertions, and with data validation where code has type checks.

Chapter 9: Reproducibility as a System Property

Reproducibility is engineered, layer by layer, in descending order of pain caused when missing:

  1. Data: snapshot or version every training input (DVC, lakehouse time-travel, or plain immutable dated paths). "The table changed since training" is the #1 irreproducibility cause and the #1 lineage query.
  2. Code: git SHA in every run record; no uncommitted-state training (the tracker should refuse or loudly tag dirty runs).
  3. Environment: lock files + container digests (Phase 01 Ch. 9).
  4. Seeds: one seed in the config, fanned to every RNG; logged.
  5. Hardware nondeterminism (GPU atomics, cuDNN autotuning): bound it — deterministic flags where affordable, or tolerance-based golden tests where not. Bit-exactness is sometimes the wrong goal; statistically indistinguishable with a stated tolerance is the honest one.

Lab Walkthrough Guidance

Lab 01 — Mini Orchestrator, suggested order:

  1. Task + Pipeline.add_task + topological sort with cycle detection — get ordering tests green first; Kahn's algorithm, and a cycle raises with the cycle named (error messages as documentation).
  2. Execution without caching: run tasks in order, pass declared inputs, record results. The execution-counter test fixture shows you exactly what runs.
  3. Cache keys per Chapter 3's formula (the provided stable_hash handles hashing; your job is what goes in the key). Make the hit/miss tests pass, then the propagation test — if propagation fails, your key forgot upstream output hashes.
  4. Retries: the flaky-task fixture fails N times then succeeds; implement bounded retries with the transient/permanent classifier; the give-up test wants the original exception surfaced, not a retry wrapper.
  5. Ledger + lineage: append execution records; implement lineage(artifact) as the upstream graph walk; the taint test walks downstream.

Throughout: tasks are pure functions in the tests — notice how much machinery that purity buys you; that's Chapter 2's discipline made visceral.

Success Criteria

You are ready for Phase 09 when you can, from memory:

  1. Run Kahn's algorithm on paper and state why cycles must be rejected loudly.
  2. Write the cache-key formula and derive invalidation-propagation from it; name the code-version trap.
  3. Define idempotency, the stage-then-commit pattern, and the transient/permanent retry split.
  4. Sketch the lineage ledger schema and both closure queries (provenance, taint).
  5. Recite the four CI layers for ML with what each catches.
  6. Map each lab mechanism to its Airflow/MLflow/DVC equivalent in one line each.

Interview Q&A

Q: Design CI/CD for a churn model retrained weekly. Layers: code tests on PR; data contracts on the weekly snapshot (schema, volume, label rate — block training on violation); training smoke test on a fixture in PR CI; the weekly run gated by metric-vs-baseline with significance on a pinned eval set plus golden-prediction diff; artifact to registry as candidate; shadow against production for N days (Phase 09); auto-promote on parity, page on divergence; rollback = registry pointer move. Lineage recorded at every step so a bad weekly snapshot is traceable to the models it touched. Each clause of that answer is a chapter of this guide.

Q: Your nightly pipeline re-trains everything every night even when nothing changed. Fix it properly. Content-addressed caching: key each task on code version + params + input content hashes; nightly runs then skip every task whose key is unchanged — the pipeline becomes incremental automatically, and a one-table change re-runs exactly its downstream closure. The prerequisites to call out: tasks must be pure (declared inputs only) and deterministic (seeded), else the cache is incoherent — which is usually the actual engineering work hiding inside "add caching."

Q: Two runs with identical configs produced different models. Walk through it. Order of suspects: data changed underneath (no snapshot/version pinning — check lineage first), unseeded randomness (init, shuffling, dropout, augmentation — audit the seed fan-out), environment drift (library upgrade between runs — compare lock hashes), then GPU nondeterminism (atomics/cudnn autotune — bounded, small, and only credible once the first three are excluded; people blame it first because it's blameless, but it's rarely the real cause).

References