« Phase 25 README · Warmup · Hitchhiker's · Principal Deep Dive · Core Contributor · Staff Notes

Phase 25 — Deep Dive: Cloud ML Platforms

The load-bearing mechanism in this phase is not "the cloud runs your model." It is a seam: in all three labs the one expensive, non-deterministic thing — a training epoch, an inference call, a pipeline step body — is injected as a pure function, and everything the platform actually owns is a deterministic state machine wrapped around that seam. train_step(hyperparameters, epoch, state) -> state, predict(record) -> output, StepFn(resolved_inputs) -> outputs. Once the learning is a pure function, the lifecycle becomes fully traceable, and the lifecycle is the product. This doc traces the four mechanisms — training state machine, autoscaling controller, LRU serving cache, and the pipeline DAG engine — at the level a person implementing them has to reason about: data shapes, ordering, invariants, and the exact place the naive version breaks.

The training job is a resumable state machine, and the invariant is byte-identity

TrainingJob carries status ∈ {Pending, InProgress, Completed, Failed, Stopped}, a current_epoch, and a Checkpoint(epoch, state, metric). ManagedTrainingRunner._run_epochs drives it. The subtle part is not the happy path; it is resume, and the correctness bar is that a spot-interrupted-and-resumed run produces a byte-identical artifact_hash to an uninterrupted run. compute_artifact_hash is sha256({hp, final_state, epochs}), so "identical artifact" is literally "identical final state dict."

Walk the trace with max_epochs=5, checkpoint_every=2, interrupt_before_epoch=3:

1. start: Pending → InProgress. checkpoint is None, so state = {}, start_epoch = 1.
2. epoch 1 runs → state updated; 1 % 2 != 0, no checkpoint.
3. epoch 2 runs → state updated; 2 % 2 == 0, so checkpoint = Checkpoint(epoch=2, state=copy). The dict(state) copy matters — a reference would let later epochs mutate the "durable" checkpoint.
4. epoch 3: interrupt_before_epoch == 3 fires before the step. status → Stopped, current_epoch reset to checkpoint.epoch = 2, interruptions += 1. Everything computed in epochs after the last checkpoint is gone — which is nothing here, because we checkpointed at 2.
5. resume: requires Stopped + a non-null checkpoint. state = dict(checkpoint.state), start_epoch = checkpoint.epoch + 1 = 3. Epochs 3, 4, 5 run.

The final state equals the straight run's because the step is a pure function of (hyperparameters, epoch, state) and we re-entered at exactly the right epoch with exactly the checkpointed state. The off-by-one is the whole bug class. If resume set start_epoch = checkpoint.epoch (replay epoch 2 again) the loss decays one extra step and the hash diverges; if it set start_epoch = checkpoint.epoch + 2 it skips epoch 3 and diverges the other way. Because default_train_step is a contraction (loss *= (1 - lr) each epoch), any double-count or skip is silently a different model with plausible metrics — undetectable without the hash assertion. That is why the platform contract is "resume must be idempotent w.r.t. the interruption," not "resume must not crash."

The finite compute pool is the second state machine. TrainingCluster holds in_use against instance_count; place refuses (CapacityError) when available() <= 0. The non-obvious invariant: a Stopped (spot-interrupted) job still holds its instance — it is mid-lifecycle, awaiting resume, not done. Only Completed/Failed decrement in_use. So 0 <= in_use <= instance_count holds across interruptions, and a one-instance cluster with one interrupted job correctly rejects a second job until the first resumes and completes.

HPO is deterministic search over the same seam

HyperparameterTuner reuses ManagedTrainingRunner and picks a best Trial by objective. The three strategies share one candidate source, _grid (cartesian product over sorted keys, so enumeration order is stable), and differ only in traversal:

• grid takes the first max_trials of the product — v^k blowup made explicit.
• random shuffles/samples the grid under a seeded random.Random(seed); deterministic given the seed, and it samples without replacement from the grid so it can't re-test a point.
• bayesian is the interesting one: warm up on the first min(2, budget) grid points, then repeatedly pick the untried candidate whose normalized coordinate (index within each param's candidate list) is closest by squared Euclidean distance to the best-so-far, min tie-broken by index. It keeps the shape of exploit-near-the-best while staying exactly reproducible; the docstring is honest that a real acquisition function adds an uncertainty/exploration term.

_select_best tie-breaks deterministically — Minimize breaks ties on lowest trial_id, Maximize on highest — because two trials can hit identical objectives and "best" must be a function, not a coin flip.

The registry is an adjacency-list state machine with a uniqueness invariant

ModelRegistry stores versions per model package group; each register appends version = len(versions) + 1 (monotone, immutable). The stage machine is a literal adjacency map, _ALLOWED_TRANSITIONS: None → {Staging, Archived}, Staging → {Production, Archived, None}, Production → {Archived, Staging}, Archived → {None}. None → Production is absent — you cannot skip staging — and Archived → Production is absent — a retired model must be restored to None first. Any illegal edge raises InvalidTransitionError.

The load-bearing invariant lives in transition: promoting version v to Production with archive_existing=True scans the group and archives any other Production version first. So at most one version per group is ever Production — which is exactly what makes get_latest_production a total, unambiguous answer to "what is deployed?" Delete that scan and the registry can report two production versions, and every downstream consumer (endpoint, pipeline) that reads "the production version" now has a nondeterministic input.

The autoscaler is a clamp plus hysteresis, and the tick is injected

Endpoint.desired_instances is pure arithmetic: clamp(ceil(concurrency / target_per_instance), min, max), with concurrency <= 0 pinned to min. The controller lives in observe(concurrency, now_tick), and the mechanism that separates it from a naive instance_count = desired is cooldown hysteresis against an injected clock:

• compute desired; if it equals instance_count, return (no action).
• pick the cooldown by direction: scale_out_cooldown if growing, scale_in_cooldown if shrinking.
• if _last_scale_tick is not None and now_tick - _last_scale_tick < cooldown, hold — return the unchanged count.
• else commit: set instance_count = desired, stamp _last_scale_tick = now_tick.

Trace the worked example (min=1, max=5, target=10, out_cd=3, in_cd=5): observe(45, 0) → ceil(45/10)=5, first action, commits 5 at tick 0. observe(5, 1) → desired=1, scaling in, 1 - 0 = 1 < 5, holds at 5 (scale-in cooldown). observe(5, 9) → 9 - 0 = 9 >= 5, commits 1. observe(9999, 20) → clamps to 5. The asymmetric defaults are deliberate: scale-in cooldowns run longer because shedding capacity early risks an outage on the rebound, while adding late only costs latency. Injecting now_tick instead of reading a wall clock is what makes any of this testable — the same trick as the training runner's injected step.

The serving cache is an LRU list with touch-on-use

MultiModelEndpoint keeps _registry (all known models) separate from _loaded (an ordered list, least-recent first). _ensure_loaded is the real MME memory-pressure lifecycle:

• already loaded → remove then append (touch: move to most-recent end).
• not loaded, at cap → pop(0) (evict least-recently-used), record evict_events.
• append, record load_events.

The invariant is len(_loaded) <= max_loaded_models. The list-based LRU is O(n) per touch (remove is a scan) — fine at MME cardinality, and the point is the policy, not the constant. The bug the touch guards against: without move-to-end on a hit, a frequently-served model drifts to the front of the list and gets evicted despite being hot — thrashing. invoke routes by name, _ensure_loaded, then calls the model's predict; the cold-load tax (the seconds a real fetch from object storage costs on a miss) is exactly the load_events a caller would pay for.

The pipeline engine: derived edges, Kahn order, value-keyed cache, skip cascade

Pipeline is the densest mechanism. Four moving parts:

1. Edges are derived, not declared. _deps_of reads each node's inputs values, splits on the first ., and treats the prefix as a producer step (unless it's params). Unknown producers raise at definition-adjacent time. The DAG is a consequence of the "step.output" references, never drawn by hand.
2. Kahn's algorithm, name-order tie-broken. topological_order computes each node's dep set (a branch-gated step gets its gating ConditionStep added as an extra dep), then repeatedly emits the sorted set of zero-dep nodes and removes them. If a round finds no ready node while nodes remain, that is a cycle → CycleError. The sorted tie-break makes execution order deterministic across runs.
3. The cache key is over resolved values, not references. compute_cache_key(name, resolved_inputs, params) hashes the step name plus the actual input values it received plus the params it reads. Two different producers that emit the same value therefore hit the same cache entry — content-addressed, not identity-addressed. The _cache dict survives across runs, which is what makes re-running a pipeline after editing one step re-execute only the changed step and its descendants. The honest gotcha, called out in the code: the step name stands in for the container image + command that real KFP/SageMaker fold into the key — so a code change invisible to the key replays stale outputs.
4. Skip cascades; it does not fail. A ConditionStep records branch_taken[name] = predicate(...). _is_skipped marks a step skipped if it is gated by a branch that wasn't taken, or if any of its producers was itself skipped. That second clause is the cascade: prune one branch and the entire subgraph below it is Skipped, and the run still Succeeded. This is why a pipeline can legitimately finish green with its register step never run — skip-vs-fail semantics are load- bearing, not cosmetic.

Trace the canonical run: preprocess → {train_a, train_b} → evaluate → accuracy_gate → register|notify. Order is deterministic; train_a/train_b fan out after preprocess; evaluate fans them in. On run 2 with identical params, every step's key is in _cache → executed() is empty, all cached. On run 3 with changed raw_rows, preprocess's resolved input differs → miss → new outputs → every descendant's resolved input differs → cascade of misses; and a higher threshold flips the gate so register is skipped and notify runs. Every behavior falls out of the same three rules: derived edges, value-keyed cache, skip cascade.

What to hold onto

Strip the vendor names and the mechanisms are provider-agnostic: make the lifecycle a deterministic state machine over an injected pure function; make resume idempotent to the interruption or you silently ship a different model; keep exactly one production pointer; scale with a clamp plus asymmetric hysteresis on an injected clock; evict LRU with touch-on-use; and derive the DAG, key the cache on values, and cascade skips. Learn the mechanism and every cloud's docs read as the same machine with different nouns.