« Phase 25 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Staff Notes

Phase 25 — Core Contributor Notes: How the Real Platforms Are Built

This is the maintainer's-eye view of the actual services — SageMaker, Vertex AI, Azure ML, and the Kubeflow/MLflow machinery underneath them — not our miniature. Where the labs inject a tick and process a batch in submission order in memory, the real systems are distributed control planes fronting heterogeneous compute fleets, and the interesting engineering is in the seams. Where I describe a pattern rather than a documented internal, I say so; do not quote implementation details you cannot verify.

The training job: an async control-plane resource with a container contract

A CreateTrainingJob call does not run your code — it records intent. The control plane validates, enqueues (InProgress after Pending), acquires the requested instances, pulls your image (or wraps your script in a framework container), stages the data channels from object storage onto a mounted path, and runs the container to the contract: read inputs from the mount, write the artifact to the output path, exit 0. Everything the platform "manages" is orchestration around that contract — scheduling, provisioning, log/metric shipping, retry-on-infra-failure, artifact upload. The learning is still your container's problem. Our lab collapses this into a synchronous ManagedTrainingRunner, which is faithful to the lifecycle state machine and deliberately drops the container/staging/async seams.

Managed Spot is where the contract gets sharp. SageMaker's interruption is a distinct signal, not a failure: the platform reclaims the instance, and on capacity return re-supplies the last checkpoint synced to your checkpoint_s3_uri. The committer's constraint is that the sync is yours to get right — the platform copies the path you designate on the cadence you write; if your loop checkpoints rarely, you redo more epochs, and if your resume logic replays or skips one, you have silently trained a different model. Our lab makes this the byte-identical-hash assertion; the real system gives you the machinery and trusts your loop to be idempotent to the interruption.

Hyperparameter tuning: a service around the same function

SageMaker Automatic Model Tuning, Vertex's Vizier-backed tuning, and Azure sweep jobs all exploit the same f(hyperparameters) -> metric shape our HyperparameterTuner does, but the production seams are richer: early stopping (Hyperband-style successive halving kills a dominated trial at epoch k instead of paying all epochs), parallel-trials-vs-total-budget (Bayesian strategies degrade as parallelism rises because each pick is made with fewer observed results — a genuine tension the lab's sequential simulation hides), and warm start from a prior job. Vizier is a real Gaussian-process/acquisition-function service; our "bayesian" strategy keeps the shape (warm-up, then exploit-near-the-best by proximity) while staying exactly deterministic, and the docstring is honest that a real acquisition function adds the uncertainty term the lab omits.

The registry speaks three dialects of the same idea

All three ship a registry, and a committer should read all three as one mechanism — a mutable pointer to an immutable version — wearing different spellings:

• SageMaker groups versions in a model package group and carries ModelApprovalStatus (PendingManualApproval -> Approved/Rejected). The load-bearing design choice: the approval event is what triggers CD — EventBridge listens for the status flip and kicks a deploy pipeline. Approval is a first-class field, not a stage.
• MLflow uses stages (None/Staging/Production/Archived) — exactly Lab 01's state machine — and is the de-facto open standard Azure ML speaks natively. Note the real evolution: newer MLflow deprecates stages in favor of aliases/tags (champion, default as movable pointers), because stages hardcode a lifecycle that not every org shares.
• Vertex uses version aliases from the start — same movable-pointer idea, third spelling.

Our miniature implements the MLflow stage machine with auto-archive-on-promote, which is the clearest teaching form; the real systems diverge on whether the live pointer is a stage, an approval status, or an alias, and an interviewer will hand you whichever vocabulary you didn't say.

Endpoints, MME, and deployment guardrails

The serving seams are where SageMaker's à-la-carte breadth shows:

• Autoscaling is Application Auto Scaling, target-tracking the SageMakerVariantInvocationsPerInstance metric with scale-out/scale-in cooldowns — our Endpoint.observe clamp-plus-hysteresis with an injected tick is this controller with the wall clock removed. The asymmetric cooldowns are real: scale-in runs longer because shedding early risks a brownout on the rebound.
• Multi-model endpoints route by the TargetModel header, lazy-load from an S3 prefix on a miss (the cold-load tax), and evict under memory pressure — the real eviction is byte accounting, not the model-count cap our _ensure_loaded uses as a stand-in. Vertex approximates co-hosting via a DeploymentResourcePool; Azure ML via multiple deployments behind one endpoint.
• Deployment guardrails generalize canary to all-at-once / canary / linear traffic shifting with CloudWatch-alarm auto-rollback between steps; shadow tests mirror live traffic to a candidate whose responses are logged and never returned. Our lab's injected healthy boolean is that CloudWatch alarm, and it tests both promote and rollback because a rollback path never exercised is a rollback path that does not work.

Pipelines: three DSLs, one derived-DAG engine

The orchestration layer is where "three dialects" is most literal. SageMaker Pipelines uses step properties — TrainingStep.properties.ModelArtifacts.S3ModelArtifacts is a deferred reference resolved at runtime, exactly what makes Lab 03's "train.model" strings derive edges instead of you drawing them. Vertex Pipelines is managed Kubeflow — you author with the KFP SDK, Vertex executes, and every artifact is tracked in ML Metadata with lineage (the production big brother of Lab 03's artifact dict). Azure ML v2 uses a component DSL with input/output bindings.

The sharp edge every committer eventually meets is caching correctness. Production keys include the container image and command; KFP keys on the component spec, so a code change that doesn't change the spec can replay stale outputs — which is why teams version images into the key or force-disable caching on steps that read external state. Our compute_cache_key hashes the step name plus resolved input values plus params, and the code is honest that the name stands in for the image+command the real systems fold in — so a code change invisible to the key replays stale outputs in the miniature exactly as it would in KFP. Cycle detection at definition time (CycleError before anything runs) is real: a cyclic pipeline is a definition bug, not a runtime surprise.

Feature stores and the skew they exist to kill

SageMaker Feature Store, Vertex Feature Store (BigQuery-backed), and Azure's managed store (built on the open-source Feast model) all answer one problem: features must be computed identically offline (training, point-in-time-correct over history) and online (serving, low-latency per request), or you get training-serving skew — a silent accuracy killer. The two phrases that show you understand the design are point-in-time correctness and one definition, two stores. The lab doesn't build this, but it is the seam every serious pipeline eventually needs.

What our miniature deliberately simplifies

• Injected pure-function training step / predict / step body instead of real containers, GPUs, and network fabric — determinism over fidelity, on purpose.
• A synchronous, submission-order BatchTransform; real batch is distributed and unordered — the README calls out the honesty gap explicitly.
• A model-count cap in MME instead of byte-level memory accounting.
• One in-memory registry with the MLflow stage machine instead of three vendor dialects with approval events and EventBridge wiring.
• A name-based cache key standing in for the container image + command a real key hashes.

Know the shape and the seams — the derived DAG, the checkpoint contract, the target-tracking controller, the movable-pointer registry — and every vendor's docs read as confirmation of one machine with different nouns rather than five unrelated products to memorize.