« Phase 26 README · Warmup · Hitchhiker's · Principal Deep Dive · Core Contributor · Staff Notes

Phase 26 — Deep Dive: MLOps — Tracking, Registry, CI/CD, and Drift

The load-bearing idea of this phase is that a model's whole lifecycle reduces to three data structures and one debounce: an append-only run record whose metrics are time series, a per-name stage machine with a single-Production invariant, and a scalar distance over binned distributions gated by a cooldown. Everything the labs build — TrackingStore, ModelRegistry, DriftMonitor — is a faithful miniature of those three shapes. This doc traces each at the level you have to reason about when you implement it: the actual fields, the ordering rules, the invariants, the complexity, and where the naive version breaks.

A metric is a time series, not a scalar

The Run in Lab 01 carries five fields, and the one that matters mechanically is the split between metrics (latest value per key) and metric_history (the full list[tuple[int, float]] per key). log_metric does the load-bearing work:

history = run.metric_history.setdefault(key, [])
if step is None:
    step = len(history)          # auto-increment per key
history.append((step, float(value)))
run.metrics[key] = float(value)  # "current" == last written

The invariant is that metrics[key] is always the value of the last log_metric call for that key, while metric_history[key] retains every (step, value) pair. This is not redundancy — it is the difference between a tracker that can render a learning curve and one that cannot. A store that kept only the scalar could never show you the overfitting hook where validation loss turns up while training loss keeps falling; half the diagnostic value of tracking lives in the shape of the curve, and the shape is only recoverable if you keep the series. The append-only history also makes the store a log, not a mutable cell: you never overwrite a step, you append the next observation, and "the run's accuracy" is a projection (last) over that log.

Two other fields are deliberately typed to teach a point. params is dict[str, str] — everything stringified — because params exist for display and diffing, never arithmetic; storing 0.01 as "0.01" is exactly MLflow's choice and it removes an entire class of float-equality bugs from search. artifacts maps a logical name to hash_artifact(content) — sha256(bytes)[:12] — which is content addressing in miniature: same bytes always produce the same digest, so an unchanged artifact is provably unchanged across runs, and the reproducibility test asserts exactly that (run-1's model.pkl hash equals a freshly-built store's hash for the same bytes).

Search compiles to a condition AND-fold; selection is an argmin with a tiebreak

search_runs is where a tracker stops being a dict and becomes a query engine. A Condition is the compiled form of MLflow's "metrics.accuracy > 0.9" filter string — a (kind, key, op, value) record where kind selects which namespace to read (metrics/params/tags/status) and op indexes a fixed operator table _OPS. Condition.matches(run) reads the field, applies the operator, and — critically — returns False (not an error) when the key is absent: a run that never logged accuracy does not match accuracy > 0.9, which is the correct set semantics. search_runs then AND-folds the conditions (all(c.matches(r) for c in conditions)), so the query is a conjunction over per-field predicates. Complexity is O(runs × conditions) — a linear scan, which is honest about what a real tracker's SQL WHERE clause does at small scale and what its indexes optimize at large scale.

Ordering carries two invariants worth internalizing. order_by is a (kind, key, direction) tuple, and a run missing the sort key sorts last regardless of direction (the sort key prefixes a missing_rank of 0/1 so present values always precede absent ones). Ties break by run_id ascending, so the result order is total and deterministic — no run's position depends on dict iteration or wall-clock insertion beyond the counter. best_run(metric, mode) is the same discipline distilled to an argmin: filter to runs that logged the metric, then min(candidates, key=lambda r: (-r.metrics[m], r.run_id)) for mode="max" (negate to turn max into min) or (r.metrics[m], r.run_id) for min. The run_id tiebreak is not cosmetic — it is what makes "the best run" a function, so model selection feeding the promotion gate is reproducible rather than order-dependent.

The registry is a state machine with one hard invariant

Lab 02's ModelRegistry stores dict[str, list[ModelVersion]] — versions auto-increment per name (len(versions) + 1), each immutable and carrying a run_id lineage pointer plus a snapshot of its run's metrics (the numbers the gate later reasons about). The lifecycle is the explicit table ALLOWED_TRANSITIONS, a directed graph over None -> Staging -> Production -> Archived with the back-edges that matter (Archived -> Staging for restore, Production -> Staging for demotion). Making the graph a data structure means an illegal move like Production -> None is an InvalidTransitionError at the call site, not an incident discovered later.

The single hard invariant is at most one Production version per name. transition_stage enforces it structurally: promoting to Production finds the incumbent via get_production, and if one exists you must pass archive_existing=True, which archives it and pushes (promoted, archived_prev) onto a per-name promotion stack. That stack is the entire reason rollback is O(1) and deterministic — it pops the last (promoted, archived_prev), archives the promoted version, and restores the displaced one to Production. Rollback deliberately bypasses ALLOWED_TRANSITIONS (it is an operational override), which is why versions are never deleted on supersession: the artifact, its lineage, and its metrics all stay resident so undo is a pointer move, not a re-registration.

The gate is a pure conjunction; promotion is the only mutation

The PromotionGate separates decision from effect. evaluate is pure — it reads the registry, runs every check, and returns a GateResult (passed plus the full list[CheckResult]), touching nothing. promote calls evaluate and applies the Production transition iff result.passed. This purity is the invariant that makes the gate testable and safe: a blocked candidate provably stays in Staging with the incumbent still serving, because the mutating path is guarded by the pure decision.

The decision itself is a conjunction of three check families, all of which must pass:

Primary metric + margin. cand >= base + min_improvement for mode="max" (or cand <= base - min_improvement for min). The margin is not pedantry — with a zero margin, eval noise promotes noise and churns Production weekly. The first-model-ever path (incumbent is None) passes by construction: there is nothing to regress against.
Guardrail non-regression, direction-aware. Each Guardrail computes a regression as base - cand (higher-is-better) or cand - base (lower-is-better) and passes iff regression <= tolerance. The sharp property: a guardrail regression blocks even when the primary metric improves — a candidate 5 points better on accuracy but 30ms over the latency budget does not ship, because passed = all(checks), not "primary wins."
Required checks. Each name must be checks.get(name) is True — so missing fails exactly like false. An absent fairness report is not a passing fairness report.

There is also a stage-0 gate: the candidate must already be in Staging. The whole GateResult is the sign-off package — every check named, its outcome, and the candidate-vs-incumbent evidence — which is what an approver approves and what the 2 a.m. investigation reads.

Drift is a scalar distance over bins, plus a debounce

Lab 03 reduces "the world moved" to arithmetic over histograms. bin_index uses bisect_right, so a value equal to an edge goes to the upper bin and binning is total over all reals; normalize applies Laplace smoothing, p_i = (count_i + eps) / (total + eps·n_bins), so no bin is exactly zero (a zero bin blows up the logarithm). PSI is then:

\[ \mathrm{PSI} ;=; \sum_i (p_{\text{live},i} - p_{\text{ref},i}),\ln\frac{p_{\text{live},i}}{p_{\text{ref},i}} \]

Every term is non-negative (both factors share sign), so PSI is 0 iff the binned distributions match and grows monotonically with the shift — which the tests assert by feeding [a + shift] and checking PSI increases with shift. KL divergence D(live ‖ ref) = Σ p_live·ln(p_live/p_ref) is the asymmetric parent; PSI is its symmetrized sum. Bands: below 0.1 stable, 0.1–0.2 investigate, above 0.2 act.

The DriftMonitor precomputes reference histograms once at construction, then evaluate_window scores three signals: data drift per feature (PSI, iterated in sorted name order for determinism), prediction drift (categorical PSI over model outputs — label-free), and a concept-drift proxy (accuracy drop on labeled feedback — the one signal needing labels). Severity is HIGH at score >= 2·threshold, else MODERATE.

The debounce is the part that separates monitoring from noise. Drift is a state, not an event — a shifted distribution stays shifted, and a naive monitor re-alerts every window. _should_fire gates emission: fire only if the signal never fired or (_eval_count - _last_fired[key]) > cooldown. _eval_count is an injected logical clock (the window index), not a wall clock, so one sustained drift produces one ticket per cooldown period. The worked example proves it: window 2 emits three triggers, window 3 (same drifted data) emits zero — same breach, deduped. Strip the domain and the pattern is exactly a paging system's alert suppression; the arithmetic that decides whether to alert is PSI, and the logic that decides whether to page again is the cooldown.