« Phase 26 README · Warmup · Hitchhiker's · Deep Dive · Core Contributor · Staff Notes

Phase 26 — Principal Deep Dive: MLOps as a Production Control System

The Deep Dive traced the three data structures and the debounce — the append-only run record, the single-Production stage machine, the PSI-over-bins scalar with a cooldown. This doc is about the system those structures compose into: the closed lifecycle loop as a production control system, how it scales, what fails and how far the blast radius reaches, and which "looks wrong" decisions are load-bearing. The through-line: software changes on one axis and DevOps governs it; ML changes on three — code, data, model — and the data axis moves with no commit, no diff, and no compiler, so MLOps is the control system that closes the loop the compiler can't.

The loop is the architecture

Draw the circle once and you have the whole phase: monitor → trigger → retrain → track → register → gate → promote → serve → re-baseline → monitor. Each node is one of the labs' miniatures, and the arrows are the contract. The architectural claim is that this is not a pipeline (a line with a start and an end) but a control loop (a cycle with a setpoint and feedback): the setpoint is "the model is still right," the sensor is the drift monitor, the actuator is the retraining pipeline, and the gate is the safety interlock that stops the actuator from making things worse. Deployment is the start of the job, not the end — the world drifts away from the training snapshot the moment you ship, and the loop exists to measure and close that gap continuously. This is why "we have CI" is not "we have MLOps": app CI governs the code axis and assumes the other two hold still. They don't.

The maturity model is the scaling path

The Google Level 0/1/2 framing is not trivia — it is the capacity ladder for the loop:

• Level 0 — the connection between training and serving is a human. The loop exists only in someone's head, and it fails silently when that person is on vacation.
• Level 1 — the training pipeline is automated and parameterized, which is what makes continuous training possible. This is the first level where the loop can run without heroics, and it requires the machinery this phase builds — tracking (or you can't compare runs), a registry (or the pipeline has nowhere governed to put output), monitoring (or nothing triggers it).
• Level 2 — the pipeline itself goes under CI/CD, so a data scientist's feature-code improvement travels to production through the same tested, gated path as any software change. The inversion senior candidates name: you ship pipelines, not models — models are what the shipped pipeline emits.

The scaling insight is that each level buys the next by adding a governed edge to the loop, and most teams' real problem is staying at Level 0 after the model starts to matter.

The scaling and tempo envelope

The three surfaces of the loop scale on different clocks, and conflating their tempos is the classic architectural mistake:

• Tracking is write-heavy and append-only — every run streams params, metric step histories, and content-hashed artifacts. It scales like a log with a query layer on top; search_runs is an O(runs × conditions) scan at small scale that a real backend answers with a SQL WHERE and indexes at large scale.
• The registry is low-volume and correctness-critical — a handful of transitions per model, each guarded by the single-Production invariant. It scales trivially; what it must never do is lose the invariant.
• Monitoring runs on the slowest and most heterogeneous clock. Operational signals (Phase 14's territory) are seconds-to-minutes; statistical drift needs a window of traffic to compare distributions (hours), and concept drift needs labels, which lag by however long ground truth takes to arrive (days to weeks). A principal sizes the drift window against noise (too small and PSI is jittery; too large and it smears a real shift) and accepts that the label-based tier is structurally late — which is exactly why you run the cheap label-free tiers in front of it.

Failure modes and blast radius

The dangerous failures here are quiet correctness regressions where every operational dashboard stays green.

• A wrong model looks healthy. Every request returns 200, latency is fine, throughput is fine — and the predictions are garbage because P(x) moved. No operational signal fires; only a statistical one does. Blast radius: every decision the model drives, for as long as nobody looks at the distribution dashboard.
• Retrain on a data bug. Distributions "shift," auto-retrain fires, quality drops further — because the shift was an upstream job writing cents instead of dollars, and retraining baked the corruption in. Blast radius: the new model plus the false confidence that you "responded to drift." The interlock is a schema/range check that halts the pipeline before training spends GPU-hours.
• The muted channel. A drift monitor with no cooldown re-alerts every window on the same sustained shift; 400 pings later the team mutes it, and the mute is permanent, so the next real incident sails through a silenced channel. Blast radius: everything, because you've disabled the sensor.
• The gate with no margin. Eval noise clears a zero-margin bar, so Production churns weekly on statistical noise dressed as improvement. Blast radius: stability itself, plus the audit trail that now records meaningless promotions.
• Skew misdiagnosed as drift. Served features are computed differently than training features; the dashboard shows a distribution gap, someone retrains, and the retrain bakes the pipeline bug in. Blast radius: compounding, because the fix (repair the pipeline) is the opposite of the reflex.

Cross-cutting concerns

Governance and audit. The registry plus IAM plus audit logs is the control system: every model in production got there through an authorized, logged, lineage-tracked transition, and you can prove it after the fact. The GateResult sign-off package — every check named, its evidence, candidate-vs-incumbent values — is what an approver approves and what the incident investigation reads. "Governance" is not a policy document; it is this artifact plus the invariant that produced it.

Cost. Two shapes: the compute to score drift windows and store the reference/live histograms (cheap, continuous), and the durable copy of captured serving payloads used to compute drift and realized accuracy (the same double-edged Data-Capture surface as Phase 25/24 — exactly what you need for monitoring, and possibly-sensitive data to govern).

Observability boundary. Phase 14 watches the service (tokens, dollars, latency, traces); this phase watches the model (versions, gates, distributions, realized quality). A healthy service serving a wrong model is precisely the failure only this phase catches. Two dashboards, two on-call rotations, both mandatory.

Multi-tenancy / fan-out. One monitor per model-and-segment, because aggregate PSI hides a drifted subpopulation — averaging a hot segment against a stable one can keep the top-line number serene while a tenant silently degrades.

The "looks wrong but is intentional" decisions

• Params stored as strings. 0.01 as "0.01" looks lazy; it is deliberate — params exist for display and diffing, never arithmetic, and stringifying removes an entire class of float-equality bugs from search. It is exactly MLflow's choice.
• The gate is pure; only promote mutates. evaluate reads the registry and returns a result touching nothing; promote applies the transition iff passed. That separation is what makes a blocked candidate provably stay in Staging with the incumbent serving.
• Rollback bypasses the transition table. Undo is an operational override, so versions are never deleted on supersession — the artifact, lineage, and metrics stay resident and rollback is a pointer move, not a re-registration.
• The clock is an injected logical counter. _eval_count is the window index, not wall time, which is what makes "one ticket per sustained drift per cooldown" deterministic and testable — the same seam as the drift being a state (it stays shifted) rather than an event.
• Content-addressed artifacts. A hash proves bit-identity; an unchanged artifact is provably unchanged across runs, which is the floor reproducibility that every stronger guarantee builds on.

Where the loop fits the platform decision

The lifecycle loop has been stable for a decade because it is the logic of governed ML, not a framework fashion — the tools rotate (MLflow/W&B for tracking and registry, Kubeflow/SageMaker/ Vertex for orchestration, Evidently and the clouds' monitors for drift) but they are four roles wearing logos, and every one of them is runs, versions, stages, gates, and reference-vs-live distributions underneath. The composition that ships: one tracker, one registry (usually the tracker's), one orchestrator (your cloud's), one monitor. And it composes upward — an LLM feature rides the same discipline: prompt/config versions in a registry-like store, shadow/canary rollouts for model or prompt changes, evaluation pipelines gating promotion with a Phase 11 judge metric. The vendors relabel it "LLMOps"; the loop is unchanged. Draw the circle on a whiteboard and you have summarized both the phase and the job.