Phase 08 — Training Orchestration & MLOps
Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (40–50 hours) Roles Supported: the JD's "deep expertise in MLOps and platform engineering, including model registries, feature stores, CI/CD, infrastructure as code, experiment tracking, and automated validation"
Why This Phase Exists
Every ML platform — Airflow+MLflow stacks, Kubeflow, Vertex, SageMaker Pipelines — is the same five ideas wearing different YAML: a DAG of tasks, content-addressed caching so unchanged work is never redone, retries because infrastructure flakes, an artifact ledger answering "where did this model come from," and gates that block bad artifacts from promotion. Engineers who have only used these platforms configure them by superstition; engineers who have built one configure them by mechanism. The flagship lab builds one in ~300 lines.
The WARMUP covers the surrounding practice: experiment tracking, model registries and promotion lifecycles, CI/CD for ML (what "test" even means when the artifact is a model), and reproducibility discipline.
Concepts
- DAGs: topological execution, cycle detection, fan-in/fan-out
- Content-addressed caching: hash(code, params, upstream artifact hashes) → skip; cache invalidation as dependency propagation
- Idempotency and retries; transient vs permanent failure
- Artifact lineage: model → data → code provenance, queryable
- Experiment tracking: params/metrics/artifacts per run; what makes runs comparable
- Model registry: stages (candidate → staging → production), promotion gates
- CI/CD for ML: data validation, training smoke tests, quality gates, golden predictions; the train-on-merge pattern
- Reproducibility: seeds, pinned environments, data snapshots/versions
Labs
Lab 01 — Mini Pipeline Orchestrator (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Build miniflow: declare tasks with dependencies, execute in topological order with content-hash caching, retries with backoff, and an artifact ledger supporting lineage queries — the engine inside every MLOps platform |
| Concepts | Topological sort, cycle detection, content-addressed cache keys, invalidation propagation, idempotent task contracts, lineage |
| How to Test | pytest test_lab.py — 13 tests: ordering, cycles, cache hit/miss semantics, upstream-change propagation, retry/give-up behavior, lineage trace |
| Talking Points | Why hash inputs rather than timestamps? What makes a task safely retryable? Why must cache keys include the code version? |
| Resume Bullet | Built a DAG pipeline orchestrator with content-addressed caching, retry semantics, and artifact lineage — then mapped each mechanism onto Airflow/MLflow equivalents in production use |
→ Lab folder: lab-01-mini-orchestrator/
Extension Project A — Experiment Tracker (spec)
A track(run) context manager logging params/metrics/artifacts to a SQLite ledger; a
compare(run_a, run_b) diff report; a leaderboard query. Then map every feature to
its MLflow equivalent and write the one-pager "what MLflow actually stores."
Extension Project B — Model CI Pipeline (spec)
A GitHub Actions (or local make ci) pipeline for one of the earlier phases' models:
data-contract validation (Phase 01's library!) → training smoke test (tiny data, must
converge) → quality gate (metric ≥ baseline from a stored golden) → package artifact
with lineage metadata. Deliberately break each stage; verify each gate fires.
Guides in This Phase
- WARMUP.md — the full zero-to-expert guide
Deliverables Checklist
- Lab 01 implemented; all 13 tests pass
- Extension A tracker or documented MLflow equivalent
- Extension B pipeline with all gates demonstrated firing