« Phase 26 README · Warmup · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes
Phase 26 — Hitchhiker's Guide
The compressed practitioner tour. Read the WARMUP for the mechanism; this is the stuff you say in the meeting.
30-second mental model
Software changes on one axis (code); ML changes on three (code, data, model), and the data
axis changes with no commit and no diff — that's why MLOps exists. The machinery is a loop:
track every training run (params, metric curves, content-hashed artifacts, lineage) →
register the model as a version with a stage (None/Staging/Production/Archived) →
gate promotion in CI (beat the incumbent by a margin, regress no guardrail, pass required
checks) → serve → monitor statistically (PSI/KL/KS/chi-square against a reference) → on a
threshold breach, emit ONE deduped retraining ticket → retrain → back to track. The senior
move: "Phase 14's dashboard says the service is healthy; this one says the model is still right.
Different failure modes, both mandatory."
The numbers to tattoo on your arm
| Number | Meaning |
|---|---|
| PSI < 0.1 | stable — no significant shift |
| PSI 0.1–0.2 | moderate shift — investigate |
| PSI > 0.2 | significant shift — act (ticket / retrain) |
| 3 | axes of change: code, data, model |
| 3 | things ML CI tests: data + code + model |
| 3 | maturity levels: manual → automated pipeline (CT) → CI/CD of the pipeline itself |
| 1 | Production version per model name, ever (auto-archive on promote) |
| 1 | ticket per sustained drift (cooldown/dedupe — not one per window) |
The PSI formula, because someone will ask: \(\mathrm{PSI} = \sum_i (p_i - q_i)\ln(p_i/q_i)\) over bins — ≥ 0, zero iff identical, monotone in shift size, smooth the zero bins.
The concepts to keep straight
| Concept | One line | Maps to |
|---|---|---|
| Run | one training execution: params + metric step history + hashed artifacts + lineage | Lab 01 |
| Experiment | named group of runs; search/compare/best-run are queries over it | Lab 01 |
| Model version | immutable artifact, auto-incremented per name, points at its run | Lab 02 |
| Stage | lifecycle state machine; illegal transitions are errors | Lab 02 |
| Promotion gate | primary+margin, guardrail non-regression, required checks — ALL must pass | Lab 02 |
| Rollback | pointer-restore to the displaced version; seconds, not a retrain | Lab 02 |
| Data drift | \(P(x)\) moved; label-free; earliest warning | Lab 03 |
| Prediction drift | \(P(\hat{y})\) moved; label-free; closest to business impact | Lab 03 |
| Concept drift | \(P(y \mid x)\) moved; needs labels; the killer — inputs can look fine | Lab 03 |
| Label drift | \(P(y)\) base rate moved; breaks calibration and thresholds | Warmup §9 |
| Training-serving skew | your pipeline disagreeing with itself; fix the pipeline, do NOT retrain | Warmup §11 |
| CT | continuous training — the loop only ML has | Warmup §12 |
The distinctions that signal seniority
- Tracking vs registry → tracking answers "what did we run"; the registry answers "what's live and how it got there." Different objects (run vs version), different queries.
- Storage vs registry → S3 keeps bytes; a registry keeps lifecycle (stages, approval, invariants, rollback).
- Gate vs "metrics look better" → a gate has a margin (noise would promote itself), and a guardrail regression blocks even when the primary metric wins. Missing check = failed check.
- Drift vs damage → drift is a distribution statement, not a quality statement. Covariate shift can be benign; concept drift can hit with zero input drift. That's why you monitor all three tiers.
- Drift vs skew → world moved vs pipeline bug. Retraining fixes the first and bakes in the second.
- PSI vs KS → effect-size measure vs significance test. At production scale KS flags meaningless shifts; monitors default to PSI/JS.
- Phase 14 vs Phase 26 → runtime observability (tokens, latency, traces) vs model lifecycle (versions, gates, distributions). A healthy service can serve a wrong model.
The tool one-liners
MLflow: open-source tracking + registry; search_runs("metrics.accuracy > 0.9"), stages →
aliases. W&B: managed tracking with the best sweep/curve UI; content-addressed Artifacts.
Kubeflow: training DAGs on Kubernetes; orchestration, not tracking. SageMaker: Pipelines +
Registry (ModelApprovalStatus) + Model Monitor (baseline → CloudWatch violations → EventBridge →
retrain). Vertex AI: same stack GCP-flavored; its Model Monitoring splits skew (vs training)
from drift (vs past serving). Evidently: open-source drift reports/test suites; picks the
statistical test per column type. Four roles — tracker, registry, orchestrator, monitor — name the
role, then the tool.
War stories
- The model that was "fine" for five months. All-green ops dashboards, rising customer complaints. No input drift either — pure concept drift: fraudsters had adapted to the model. Only the delayed-label accuracy tracker caught it. Lesson: input monitoring is necessary, never sufficient.
- The retrain that made everything worse. Distributions shifted, auto-retrain fired, quality dropped further. Root cause: an upstream job had started writing cents instead of dollars — the "drift" was a data bug, and retraining baked it in. The fix was a schema/range check that should have halted the pipeline (Warmup §13), not a fresher model.
- The alert channel everyone muted. Drift monitor with no cooldown, re-alerting every 10-minute window on the same sustained shift: 400+ pings in three days; team muted it; real incident missed two weeks later. One sustained drift = one ticket. Dedupe is not a nicety.
- The 40-minute rollback that should have been 40 seconds. No registry — "rollback" meant finding last quarter's artifact on someone's laptop and hand-deploying it. A registry makes rollback a pointer move; that incident is why.
Vocabulary
run / experiment / params / metric step history / artifact (content hash) / lineage · model version / stage / transition / single-Production invariant / alias / approval / rollback · CI/CD/CT / training pipeline / sign-off package / primary metric / margin / guardrail / required check · reference vs live window / PSI / KL / JS / KS / chi-square / binning / smoothing · data / prediction / concept / label drift / training-serving skew · scheduled / triggered / continuous retraining / cooldown / dedupe / re-baseline · maturity Level 0/1/2 / CACE.
Beginner mistakes
- Logging only final metrics — the learning curve is half the diagnostic value.
- Calling S3 a model registry — no stages, no invariant, no rollback, no registry.
- A gate with no margin — congratulations, you now promote noise weekly.
- Skipping guardrails because "AUC went up" — and shipping a 2× latency regression.
- Monitoring only inputs — concept drift walks straight past you.
- Retraining on drifted-looking data without checking for skew/data bugs first.
- No cooldown on drift alerts — the channel gets muted, then the real one is missed.
- Forgetting to re-baseline the monitor after promoting the retrained model.