« Phase 25 README · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 25 Warmup — Managed Cloud AI Platforms: SageMaker, Vertex AI & Azure ML

Who this is for: someone who has built agent infrastructure from scratch (Phases 00–17), framework internals (Phases 18–23), and the AWS GenAI platform layer (Phase 24, Bedrock) — and now needs to speak fluently about the layer enterprise ML JDs still lead with: the general ML lifecycle platform. By the end you will be able to explain, from first principles, how a managed training job actually works (and why spot + checkpointing is the economic core of it), how hyperparameter tuning searches a space, what a model registry is for, the four serving shapes and three deployment-safety strategies, what a pipeline engine really executes under the DSL, and how SageMaker, Vertex AI, and Azure ML differ — with a real decision framework and a clean answer for where Bedrock and Azure OpenAI sit relative to all of it. No cloud account, no SDK, no network — everything in the labs is mechanism.

Table of Contents

  1. What a managed ML platform is: control plane vs data plane
  2. The managed training job: bring code, rent the lifecycle
  3. Spot capacity, checkpointing, and distributed training
  4. Hyperparameter tuning jobs
  5. The model registry
  6. Real-time endpoints and autoscaling
  7. Multi-model endpoints
  8. Batch, asynchronous, and serverless inference
  9. Deployment safety: shadow, canary, blue/green, A/B
  10. ML pipelines: DAG, artifacts, caching, conditions
  11. Feature stores, notebooks, and AutoML
  12. SageMaker vs Vertex AI vs Azure ML: deep comparison and decision framework
  13. Where Bedrock and Azure OpenAI fit: the GenAI seam
  14. The cost model
  15. The security model
  16. Common misconceptions
  17. Lab walkthrough
  18. Success criteria
  19. Interview Q&A
  20. References

1. What a managed ML platform is: control plane vs data plane

A managed ML platform is a cloud service that runs the ML lifecycle — train, tune, register, deploy, orchestrate — on infrastructure you rent per use, behind APIs you don't operate. AWS calls its version SageMaker, Google calls its version Vertex AI, Microsoft calls its version Azure Machine Learning. The JD you're prepping for says "Vertex AI or Azure ML or SageMaker" with an explicit or because the three are the same platform shape wearing three uniforms — learn the shape once and the vendor differences become vocabulary.

The first structural idea, carried straight over from Phase 24's Bedrock split: every one of these platforms divides into a control plane and a data plane, and the division predicts almost everything about how each API behaves.

  • Control plane — the slow-moving administrative surface: create a training job, purchase or configure compute, register a model version, create an endpoint, define a pipeline, set an autoscaling policy. Calls are infrequent, strongly IAM-gated, usually made by CI/CD or a human, and their latency doesn't matter. CreateTrainingJob, CreateEndpoint, CreateModelPackage, CreatePipeline live here.
  • Data plane — the hot path: InvokeEndpoint (SageMaker), predict/rawPredict (Vertex), scoring a managed online endpoint (Azure ML). High-frequency, latency-sensitive, autoscaled, and the path every inference dollar flows through.

Two consequences worth internalizing. First, the control plane is asynchronous by design: creating a training job or endpoint returns immediately with a resource in a Creating/Pending state, and you poll or subscribe for the lifecycle to advance — every lab in this phase models that lifecycle explicitly (Pending → InProgress → Completed/Failed in Lab 01) because thinking in state machines is the difference between "I called the API" and "I understand the platform." Second, the two planes fail differently: a control-plane outage means you can't deploy — a data-plane outage means you're down. Mature teams alarm on them separately.

One more identity-level point: these platforms are model-agnostic by construction. The same training job that fits a gradient-boosted churn model will fine-tune a transformer; the same endpoint that serves a scikit-learn pickle will serve an LLM if you give it a big enough GPU. That is exactly what distinguishes this phase from Phase 24: Bedrock is a model catalog you invoke; SageMaker/Vertex/Azure ML are a lifecycle you run arbitrary models through. §13 draws that boundary precisely.

2. The managed training job: bring code, rent the lifecycle

The atomic unit of every managed ML platform is the training job. Strip vendor naming (SageMaker CreateTrainingJob, Vertex AI CustomJob, Azure ML command job) and the contract is identical:

  1. You bring the code — either a framework script the platform wraps in a prebuilt container (a PyTorch/TensorFlow/sklearn "framework container"), or a fully custom container image (BYOC: bring your own container). The platform doesn't care what's inside; it cares about the contract: read inputs from a mounted path, write the model artifact to an output path, exit 0 on success.
  2. You declare the compute — instance type and count (ml.g5.2xlarge × 4 in SageMaker dialect; machine type + accelerator on Vertex; a compute cluster/instance type on Azure ML). You never provision, patch, or SSH the machines; the platform acquires them for the job's duration and releases them at exit — you pay per second of job runtime.
  3. You declare the data channels — S3/GCS/Blob URIs the platform mounts or streams into the container as input directories.
  4. You pass hyperparameters — a string map handed to your script (argv/env), which matters because it makes the job a function: same code + same data + same hyperparameters should give the same artifact. That function-like shape is what tuning jobs (§4) and pipeline caching (§10) both exploit.
  5. The platform owns the lifecycle — scheduling, container pull, data staging, log and metric shipping (CloudWatch / Cloud Logging / Azure Monitor), failure capture, artifact upload, and the status machine: Pending/Queued → InProgress/Running → Completed/Succeeded or Failed/Stopped.

Lab 01 builds exactly this: a TrainingJob with that status machine, a TrainingCluster with a finite instance pool (managed compute is finite — real accounts hit instance quotas and ResourceLimitExceeded long before "the cloud is infinite" survives contact), and the one determinism trick this whole track uses: the training step is injected as a pure function train_step(hyperparameters, epoch, state) -> state. Real training is the expensive, non-deterministic part; injecting it makes the lifecycle — the thing the platform actually owns — fully testable. That seam isn't a toy simplification: reproducible training (seeded, deterministic-ops) is a real production discipline, and the platforms assume your job is at least restartable (next section) even when it isn't bit-reproducible.

The senior distinction to carry into interviews: the platform manages the lifecycle, not the learning. If your script diverges, overfits, or OOMs, that's still your problem — what you bought is that scheduling, provisioning, logging, artifact plumbing, and retry-on-infra-failure are not your problem.

3. Spot capacity, checkpointing, and distributed training

Spot/preemptible capacity: the economics

GPU time dominates training cost, and all three clouds sell steeply discounted interruptible capacity: AWS Spot (SageMaker Managed Spot Training), GCP Spot/preemptible VMs, Azure low-priority/spot. Discounts are commonly in the 60–90% range versus on-demand (treat any exact number as market-priced and time-varying), with one string attached: the capacity can be taken back at any time with minimal warning.

That string is why checkpointing is not an optional nicety but the load-bearing mechanism. The contract, which Lab 01 implements and tests end-to-end:

  • Your training loop writes a checkpoint (model weights + optimizer state + epoch counter) to a designated path every N epochs/steps; the platform syncs that path to durable storage (S3 in SageMaker's case).
  • On interruption, everything in memory since the last checkpoint is gone. The job isn't failed — it's interrupted, a distinct state.
  • When capacity returns, the platform restarts your container, re-supplies the last checkpoint, and your script resumes from it — redoing the epochs since the checkpoint.
  • The correctness bar: a resumed run must produce the same final artifact an uninterrupted run would have. Lab 01 asserts byte-identical artifact hashes for the interrupted-and-resumed run versus the straight run — if your resume logic replays an epoch twice or skips one, the hashes diverge and the test catches it, which is precisely the bug class this contract exists to prevent.

The design lever is checkpoint frequency: checkpoint every epoch and you waste I/O and storage; checkpoint rarely and every interruption burns the redone epochs. The expected wasted work per interruption is on the order of half the checkpoint interval — so you size the interval against the interruption rate and the epoch cost, which is a genuinely good interview whiteboard exercise: cheap epochs + flaky capacity → checkpoint often; expensive epochs + stable capacity → checkpoint less, and consider on-demand for the final convergence run.

Distributed training: data parallel vs model parallel

When one device can't train the model fast enough (or at all), the job goes distributed, and the first fork in the road is what you split:

  • Data parallelism — every worker holds a full replica of the model; the batch is sharded across workers; each computes gradients on its shard; gradients are averaged (the all-reduce collective) and every replica applies the same update, staying in lockstep. This is the default because it's simple and scales well until the model itself no longer fits on one device. Communication cost per step scales with model size (you're exchanging gradients), which is why interconnect bandwidth starts mattering at scale.
  • Model parallelism — the model itself is split. Two sub-flavors: pipeline parallelism (layers are staged across devices; micro-batches flow through the stages like an assembly line, and the scheduling problem is keeping "bubbles" of idle time small) and tensor parallelism (single layers' weight matrices are sharded; each matmul becomes a distributed matmul). You reach for this when the model doesn't fit in one device's memory — the LLM regime.
  • The hybrid everyone actually runs at scale: sharded data parallelism (ZeRO/FSDP-style — weights, gradients, and optimizer state sharded across data-parallel workers and re-gathered just-in-time), which the platforms productize (SageMaker's distributed training libraries, Vertex's Reduction Server for faster all-reduce, Azure ML's DeepSpeed integration).

What the platform contributes to all of this is orchestration, not math: launching N containers with the right rank/world-size environment, wiring the network fabric, restarting failed workers, and (crucially) making spot + checkpointing work under distribution — an interruption of one worker interrupts the lockstep of all of them, so checkpoint/resume becomes even more valuable. Lab 01's single-worker miniature deliberately keeps distribution as an extension exercise: the lifecycle and checkpoint mechanics are the same; only the collectives are new.

4. Hyperparameter tuning jobs

Training's function-like shape — f(hyperparameters) → metric — makes hyperparameter search a platform primitive: a tuning job (SageMaker Automatic Model Tuning, Vertex AI's Vizier-backed hyperparameter tuning, Azure ML sweep jobs) fans out N trials, each a full training job with a different point from a declared search space, reads back an objective metric from each, and reports the best. The three canonical strategies, all in Lab 01:

  • Grid search — the exhaustive cartesian product of every candidate value. Complete but combinatorially explosive: \(k\) parameters with \(v\) values each is \(v^k\) trials. Fine for 2–3 discrete knobs; hopeless beyond that.
  • Random search — sample points from the space (uniformly or per-parameter distributions). The classic Bergstra–Bengio result explains why this beats grid at equal budget in practice: when only a few dimensions matter, random search covers each individual dimension with far more distinct values than a grid does, instead of wasting trials re-testing the same value of the important parameter against many values of unimportant ones.
  • Bayesian optimization — treat \(f\) as an expensive black box; fit a cheap surrogate model (typically a Gaussian process or tree ensemble) over the trials so far; pick the next point by an acquisition function that balances exploitation (near the best-so-far) against exploration (high surrogate uncertainty). Better sample efficiency when trials are expensive, at the cost of sequential dependence (less parallelism) and machinery. Lab 01's simulated Bayesian strategy keeps the shape — warm-up trials, then choose-next-by-proximity- to-best — while staying exactly deterministic, and its docstring is honest that a real implementation adds the uncertainty term.

Platform refinements worth naming in an interview: early stopping of unpromising trials (SageMaker's early stopping, Hyperband-style successive halving on all three clouds) — kill a trial whose learning curve is dominated at epoch k rather than paying for all epochs; parallel trials vs total budget (a tuning job runs max_parallel trials at once toward max_trials total — Bayesian strategies degrade as parallelism rises because each pick is made with fewer observed results); and warm start from a previous tuning job on related data.

The one thing the tuner cannot fix: a bad objective. Tune on training accuracy and it will happily select the most overfit trial; the objective must be a validation metric, and the final comparison belongs on a held-out test set — a two-sentence answer that separates people who have tuned real models from people who have read about it.

5. The model registry

A training job emits an artifact; a model registry turns artifacts into governed, versioned, deployable entities. All three platforms ship one (SageMaker Model Registry, Vertex AI Model Registry, Azure ML's registered models — with MLflow's Model Registry as the de-facto open standard Azure ML natively speaks), and they all answer the same four questions:

  1. What versions exist? Models group into a named lineage (SageMaker's model package group; Vertex's model with versions; MLflow's registered model), and each registration increments an immutable version pointing at the artifact + its metadata (metrics, container, signature).
  2. Which version is live? Two vocabularies coexist and you should be able to speak both. Stages (the MLflow classic that Lab 01 implements): None → Staging → Production → Archived, with transitions as a state machine — promoting a new version to Production auto-archives the incumbent so "what's in production" has exactly one answer. Approval status (SageMaker's dialect): a version is PendingManualApproval → Approved/Rejected, and the approval event is what triggers a CI/CD deployment pipe (EventBridge → deploy). Vertex uses version aliases (champion, default) as movable pointers — same idea, third spelling. Newer MLflow versions themselves are moving from stages toward alias/tag-based workflows, which is worth mentioning to show currency.
  3. Where did it come from? Lineage: which training job, which hyperparameters, which data, which code produced this version. Lab 01 records source_job, hyperparameters, and the artifact hash on every registration, because the 2 a.m. question — "prod is misbehaving; what exactly is running and how was it built?" — must be answerable from the registry alone.
  4. Who may promote? Registries hang approval workflows and access control off stage/status transitions — the governance seam where "a model" becomes "a change to production" and inherits all the ceremony code changes get.

The architectural point that makes everything downstream click: the registry — not the endpoint — is the source of truth. Endpoints, batch jobs, and pipelines all consume whatever version the registry designates; deployment tooling should read "latest approved/production version" (Lab 01's get_latest_production) rather than hardcoding artifact paths. Once you see it that way, §9's deployment strategies are just different ways of changing which version the traffic sees, and §10's condition-gated RegisterModel step is the pipeline writing into the source of truth.

6. Real-time endpoints and autoscaling

A real-time endpoint is a managed HTTPS service wrapping your model: SageMaker real-time inference (an EndpointConfig of one or more production variants behind InvokeEndpoint), Vertex AI online prediction (an Endpoint with one or more DeployedModels and a traffic split), Azure ML managed online endpoints (an endpoint with one or more deployments and traffic percentages). Persistent instances, always warm, single-digit-to-low-hundreds-of-milliseconds latency — and billed per instance-hour whether or not requests arrive, which is the economic fact the rest of the serving menu (§7–§8) exists to mitigate.

Autoscaling keeps the fleet sized to the load. The dominant pattern is target tracking (SageMaker via Application Auto Scaling on the SageMakerVariantInvocationsPerInstance metric; Vertex on utilization/QPS-per-replica targets; Azure ML scale rules): you pick a per-instance load target, and the controller solves the arithmetic

\[ \text{desired} = \mathrm{clamp}\left(\left\lceil \frac{\text{load}}{\text{target per instance}} \right\rceil,\; \text{min},\; \text{max}\right) \]

then applies hysteresis: a scale-out cooldown (don't add capacity again until the last add has had time to absorb load) and a scale-in cooldown, typically longer (don't shed capacity on a momentary dip — flapping burns cold starts and risks brownout on the rebound). Lab 02's Endpoint.observe is this exact controller with an injected tick clock, and its tests pin the behaviors that matter operationally: scale-out under load, clamp at max, hold during cooldown, scale-in only after the cooldown elapses, never below min.

Two production notes that signal experience. Scale-in is the dangerous direction — scaling out late costs latency, scaling in early costs an outage on the next burst; that's why the defaults are asymmetric. And min capacity is an availability decision, not a cost decision: min=1 means a single instance failure is a 100% capacity loss while the autoscaler reacts; for anything customer-facing, min≥2 across zones is the boring right answer.

7. Multi-model endpoints

Now invert the economics: not one big model, but hundreds or thousands of small ones — a model per tenant, per region, per SKU. A dedicated endpoint each (2× instances for availability) at any realistic instance price is a five-to-six-figure monthly bill for a fleet that is mostly idle.

The multi-model endpoint (SageMaker MME) is the platform answer: one endpoint, one shared fleet, many models. The mechanism — which Lab 02 implements literally — is a cache:

  • All model artifacts live in object storage (S3 prefix), not in memory.
  • A request names its model (TargetModel header). If it's loaded, serve it.
  • If not, lazy-load: fetch from S3, load into the container's memory, then serve — the cold-load tax, seconds of extra latency on first hit.
  • Memory full? Evict the least-recently-used model to make room. Lab 02's _ensure_loaded — touch-on-use, evict-LRU-at-cap, transparent reload after eviction — is the real MME lifecycle with a model-count cap standing in for byte-accounting.

The fit is therefore: many models, same framework/container, individually low and tolerably bursty traffic, cold-start tolerance. The anti-fit: one hot model with strict p99 (dedicated endpoint), or models that must be isolated from each other for compliance (separate endpoints — MME co-locates tenants in one process space, which is a real trust-boundary consideration to raise in any multi-tenant design conversation, echoing Phase 13). Vertex's closest analogue is co-hosting models on shared resources via a DeploymentResourcePool; Azure ML approximates the pattern with multiple deployments behind one endpoint. The idea also generalizes upward: serving many LoRA adapters over one base LLM is MME economics applied to fine-tunes.

8. Batch, asynchronous, and serverless inference

The rest of the serving menu is different answers to "how often, how big, how latency-tolerant":

  • Batch transform / batch prediction (SageMaker Batch Transform, Vertex batch prediction, Azure ML batch endpoints): no persistent endpoint at all. Spin up a fleet, map an input dataset from object storage through the model, write outputs back, tear down. Zero idle cost, hours of latency, perfect for nightly scoring, backfills, and offline evaluation. Real batch is distributed and unordered; Lab 02's BatchTransform processes in deterministic input order precisely so tests can assert on results — the README calls out that honesty gap. The fail-fast-vs-continue-on-error switch in the lab mirrors a real operational choice: a poison record shouldn't necessarily kill a nine-hour job at hour eight.
  • Asynchronous inference (SageMaker async endpoints): a queue in front of a model for large payloads or long-running single inferences (seconds to minutes — big documents, video). You get a ticket; results land in S3; the endpoint can scale to zero when the queue empties.
  • Serverless inference (SageMaker Serverless, and the scale-to-zero configurations of the other clouds): no instance management at all — capacity materializes per request, billed per invocation-duration, with the classic serverless trade: cold starts on the first request after idle. Right for spiky, low-average-volume models; wrong for steady high throughput, where per-invocation pricing crosses over dedicated-instance pricing (same crossover shape as Phase 24's On-Demand-vs-Provisioned-Throughput math, one layer down).

The four-way decision, compressed: steady interactive traffic → real-time; many small models → MME; big offline datasets → batch; spiky/rare or huge-payload → serverless/async. Being able to walk a workload through that menu — with the cost signature of each — is exactly the "deployment" line of the JD.

9. Deployment safety: shadow, canary, blue/green, A/B

A new model version is a production change, and mature platforms ship traffic-engineering primitives so you never do the naive thing (delete old, deploy new, pray). The three strategies — all built and tested in Lab 02 — form a progression of increasing exposure:

Shadow (SageMaker shadow tests / shadow variants; Azure ML traffic mirroring): the candidate receives a copy of live traffic; its responses are logged and compared but never returned. Zero user risk by construction — the two invariants Lab 02's tests pin are that the caller always receives the primary's output and that a crash in the shadow changes nothing for the user. Shadow answers "does the candidate behave on real traffic?" (latency, error rate, output drift vs incumbent) before any user ever depends on it. Its limit: you can't measure user reaction to outputs nobody sees.

Canary: give the candidate a small real slice — say 10% — watch health signals, then promote (100%) or roll back (0%). The blast radius of a bad model is capped at the canary weight times the exposure window. The production version wires the decision to alarms (CloudWatch alarms in SageMaker's deployment guardrails; error-rate/latency monitors elsewhere) — Lab 02's injected healthy boolean is that alarm, and the suite tests both paths, because a rollback path you've never exercised is a rollback path that doesn't work. SageMaker's guardrails generalize this to linear shifting (10% → 25% → 50% → 100% with health gates between steps); Vertex expresses canaries as endpoint trafficSplit percentages; Azure ML as traffic percentages across deployments on one endpoint.

Blue/green: run the full new environment (green) in parallel with the old (blue), flip all traffic atomically, keep blue warm; rollback is flipping back — instant and total, no re-provisioning, no partial states. The price is double capacity for the overlap window. Lab 02's controller captures the three semantics that matter: staging green moves no traffic, switch is atomic, rollback is instant.

A/B testing is the fourth idea, often conflated with canary but different in intent: a canary is a safety mechanism (asymmetric: promote or abort, as fast as possible), while an A/B test is a measurement mechanism — run variants at fixed splits long enough to reach statistical significance on a business metric, with sticky assignment so a user sees one variant consistently. SageMaker production variants' weights, Vertex traffic splits, and Azure ML traffic percentages all express both; what differs is what you're reading off the experiment and how long you let it run.

Composition, which is the actual senior answer: shadow first (free learning), canary second (bounded risk), promote; blue/green when you need instant total rollback and will pay for it; A/B when the question is "which is better for the business," not "is this safe."

10. ML pipelines: DAG, artifacts, caching, conditions

One model, retrained weekly, with evaluation gates and registration — that's not a script, that's an orchestrated workflow, and all three clouds converged on the same abstraction: a DAG of steps connected by artifacts. SageMaker Pipelines, Vertex AI Pipelines (which is managed Kubeflow Pipelines — you author with the KFP SDK and Vertex runs it), Azure ML pipelines (v2 component DSL). Lab 03 builds the engine underneath all three DSLs, and its four ideas are the whole story:

  1. The DAG is derived, not drawn. A step declares its inputs as references to other steps' outputs (Lab 03's "train.model" strings; SageMaker step properties like TrainingStep.properties.ModelArtifacts.S3ModelArtifacts; KFP's typed component I/O; Azure ML's input/output bindings). The engine extracts the dependency edges from those references, topologically sorts (Lab 03 uses Kahn's algorithm with a deterministic name-order tie-break), and rejects cycles at definition time — CycleError before anything runs, because a cyclic "pipeline" is a definition bug, not a runtime surprise.
  2. Artifacts, not shared memory. Steps run as isolated containers on remote compute; they communicate exclusively through named, stored artifacts (object-store URIs in production; Lab 03's run.artifacts["step.output"] in miniature). This is what makes steps independently retryable, parallelizable, cacheable, and auditable — Vertex tracks every artifact in ML Metadata with lineage, the production big brother of Lab 03's artifact dict.
  3. Step caching. Before running a step, compute a cache key over everything that determines its output — the step's identity and its resolved input values and parameters (production keys also include the container image/command). Hit → skip execution, replay the recorded outputs; miss → run and record. This is why re-running a six-hour pipeline after editing only the last step takes minutes, and Lab 03's tests pin both directions (identical rerun executes nothing; one changed input re-executes) plus the key properties themselves. The foot-gun to name out loud: a cache key that misses a real input serves you stale results — KFP keys on the component spec, so changing code without changing the spec can replay old outputs, which is exactly why production teams version container images in the key or force-disable caching on steps that read external state.
  4. Conditions. "Register only if the metric clears the bar" is a first-class branch node (SageMaker ConditionStep, KFP dsl.If). The untaken branch is skipped, not failed — and the skip cascades to everything downstream of it, which Lab 03 tests explicitly because skip-vs-fail semantics are what let a pipeline "succeed with the register branch not taken." Fan-out/fan-in falls out of the DAG for free: two trainers after one preprocess, an evaluator joining both.

Lab 03's main() assembles the canonical shape — preprocess → two parallel trainers → evaluate → condition(accuracy > threshold) → register-or-notify — which is, not coincidentally, the same train→evaluate→gate→register pattern SageMaker's own MLOps examples ship, with Lab 01's registry as the thing the final step writes into.

11. Feature stores, notebooks, and AutoML

Three more platform components you should recognize and place, covered at doc level:

Feature stores (SageMaker Feature Store, Vertex AI Feature Store, Azure ML managed feature store — the latter built on the open-source Feast model). The problem is real and specific: features must be computed identically at training time (offline, over history, in batch) and at serving time (online, per request, in milliseconds), and any drift between the two implementations is training/serving skew — a silent accuracy killer. A feature store is the dual-database answer: an offline store (warehouse-shaped: point-in-time-correct historical values for building training sets without label leakage) and an online store (key-value- shaped: latest value per entity at low latency), both fed by one definition of the feature. The two phrases that signal you actually understand it: point-in-time correctness and one definition, two stores.

Notebooks (SageMaker Studio; Vertex AI Workbench / Colab Enterprise; Azure ML compute instances + notebooks): managed JupyterLab-family environments with cloud identity and data access baked in. Platform-relevant for two reasons: they're the on-ramp (exploration happens here, and the platform wants that exploration one API call away from a real training job), and they're a governance surface (identity-bound access to data from a managed environment, instead of credentials scattered across laptops). The discipline point survives vendor choice: notebooks are for exploration; anything that must run twice belongs in a job or a pipeline.

AutoML (SageMaker Autopilot / Canvas, Vertex AutoML, Azure Automated ML): give it a labeled table (or images/text), and it runs the search you'd otherwise hand-build — feature preprocessing × algorithm × hyperparameters — and emits a leaderboard plus a deployable best model, often with generated notebooks showing what it did. Two correct senior takes at once: it's a genuinely strong baseline generator (an afternoon to "roughly the ceiling of standard approaches on this table," which either ships as a v1 or calibrates how much your custom work is worth), and it's not a substitute for problem framing, leakage detection, or the judgment about which errors are expensive. AutoML output should flow into the same registry → evaluate → gate → deploy machinery as everything else — it's a different way to produce a candidate, not a different lifecycle.

12. SageMaker vs Vertex AI vs Azure ML: deep comparison and decision framework

The phase README carries the feature-by-feature table; this section is the judgment layer an interviewer is actually probing for.

AWS SageMaker is the à-la-carte maximalist: the broadest menu of independently adoptable serving/training options (multi-model endpoints, serverless, async, deployment guardrails, shadow tests are all first-class SKUs) and the deepest integration with the AWS operational envelope (IAM, VPC, KMS, CloudWatch, EventBridge). Costs of that breadth: the service surface is large and occasionally overlapping (Studio vs classic notebooks; multiple deployment paths to the same outcome), and the GenAI story lives in a separate product — Bedrock (§13). Pick it when the organization already runs on AWS and wants maximum control over each lifecycle stage.

GCP Vertex AI is the integrated single-surface play: training, registry, endpoints, pipelines, feature store, and the GenAI layer (Gemini, Model Garden) under one product name and one metadata/lineage system. Its pipelines being managed KFP means the workflow layer is an open standard you can run elsewhere; its data gravity with BigQuery is the strongest of the three (BigQuery ML lets SQL analysts train/score models inside the warehouse — the same JD names it explicitly); Gemini's multimodal/long-context capability is native, not bolted on. Costs: smaller third-party ecosystem than AWS, and organizations not already on GCP rarely move for Vertex alone. Pick it when data lives in BigQuery, when you want ML + GenAI on one platform, or when Gemini is a product requirement.

Azure ML is the enterprise-workflow play: first-class MLflow nativity (tracking and registry speak the open standard out of the box), a strong component/pipeline reuse story, and — decisively for many buyers — residence inside the Microsoft enterprise estate: Entra ID (Azure AD) identity, Microsoft 365 governance, and Azure OpenAI next door for frontier-model access (§13). Costs: historically more ceremony (workspaces, compute targets) and a v1→v2 API transition that shows in older docs. Pick it when the organization's identity, compliance, and procurement are already Microsoft, or when OpenAI-day-one access via Azure OpenAI anchors the AI strategy.

The decision framework, stated as you'd say it in the room: the lifecycle is commoditized — all three train, tune, register, serve, and orchestrate competently — so the decision is gravity, not features. Where does the data live (BigQuery → Vertex; a lake on S3 → AWS; Synapse/Fabric → Azure)? Where does identity and compliance already sit (Entra ID and Microsoft procurement → Azure)? What does the GenAI dependency look like (OpenAI models → Azure; model breadth under one AWS envelope → Bedrock+SageMaker; Gemini/multimodal → Vertex)? And what does the team already know — platform fluency compounds, and a competent team on their home cloud beats a confused team on the "best" one. Answer those four and the platform picks itself; then the feature table settles the residual ties (fleets of small models → SageMaker MME; SQL-centric org → BigQuery ML; MLflow-standardized shop → Azure ML).

13. Where Bedrock and Azure OpenAI fit: the GenAI seam

The JD line reads: "Azure OpenAI or AWS Bedrock for GenAI" — alongside the ML platform line, not inside it. That's the seam to keep sharp:

  • AWS: SageMaker is the ML lifecycle platform; Bedrock (Phase 24, in depth) is the separate managed foundation-model platform — catalog, Converse, capacity modes, Guardrails, Knowledge Bases. They overlap at the edges (SageMaker JumpStart serves open-weight LLMs on endpoints you manage; a fine-tuned Llama can be trained on SageMaker and imported into Bedrock via Custom Model Import), but the division of labor is: your models → SageMaker; someone else's frontier models as a service → Bedrock.
  • Azure: Azure ML is the lifecycle platform; Azure OpenAI (now folded into the Azure AI Foundry umbrella, Phase 24 §13's comparison) is the managed frontier-model service — OpenAI's models with Azure identity, networking, and compliance wrapped around them. Same seam, same logic.
  • GCP is the exception that proves the rule: Vertex AI folds the GenAI layer into the ML platform — Gemini and Model Garden are Vertex APIs, sharing the registry/endpoint/pipeline machinery. One platform, both layers.

Why the seam exists at all: a hosted frontier model has no training job, no artifact you own, no registry version of yours — the lifecycle collapses to "choose model, prompt/ground/fine-tune at the margin, govern usage," which wants catalog-and-invocation machinery (Phase 24), not train-register-deploy machinery (this phase). And the composition answer that lands in interviews: you still run the LLM application through this phase's discipline — prompt/config versions in a registry-like store, shadow/canary rollouts for model or prompt changes (Lab 02's mechanics apply verbatim to swapping gpt-4o for its successor), and evaluation pipelines gating promotion (Lab 03's condition step with an LLM-judge metric from Phase 11). Both platform docs now push the same idea under "LLMOps"/"GenAIOps" — the vocabulary changed; the shape of the discipline didn't.

14. The cost model

The pricing shape (exact rates drift; the shape doesn't):

  • Training: instance-seconds for the job's duration. Levers: spot (§3's 60–90% class discount, paid for in checkpoint discipline), right-sizing (GPU utilization is the metric — a starved input pipeline burns GPU-hours on I/O waits), and early stopping in tuning jobs.
  • Real-time endpoints: instance-hours, 24/7, load or no load — the always-on tax. Levers: autoscaling min counts (availability floor, §6), MME consolidation (§7), serverless for spiky tails (§8).
  • Batch: instance-hours only while the job runs — zero idle by construction.
  • Serverless: per-invocation duration/memory — cheap at low volume, crossing over dedicated instances at sustained volume.

The worked example every platform conversation eventually reaches — "the model is cheap; the endpoint is expensive." One modest ml.m5.xlarge-class instance at an illustrative \$0.23/hour is \(0.23 \times 24 \times 30 \approx \$166\)/month; make it a two-instance minimum for availability and it's ~\$330/month per model — before a single request arrives. Ten such models: ~\$3,300/month. The same ten models on one two-instance MME: still ~\$330. That one division — always-on fleet cost versus per-model traffic — is the whole economic argument for MME, serverless, and batch, and it's the math to volunteer when someone asks "should every model get an endpoint?" (No.)

For training: a 4-GPU job at an illustrative \$12/hour for 20 hours is \$240 on-demand; at a 70% spot discount it's \$72 — if checkpointing makes interruptions cheap. If an interruption costs you a from-scratch restart, one bad day erases the discount, which is why §3's checkpoint-interval reasoning is a cost topic, not just a reliability topic.

15. The security model

The same trust-boundary discipline as Phase 24 §10, applied to the lifecycle platform — the four legs an enterprise security review will walk:

  • Identity/authorization: IAM roles (AWS), service accounts (GCP), Entra ID + RBAC (Azure) gate every control-plane verb (who may create jobs, register versions, deploy endpoints) and every data-plane call. The platform-specific sharp edge: a training job itself runs with a role (SageMaker's execution role) — it can read your data lake with whatever breadth you granted, so the job's role deserves the same least-privilege scrutiny as any service's. Registry stage/approval transitions (§5) are authorization points too: "who may promote to production" is an IAM policy, not a convention.
  • Network isolation: private connectivity so training and inference traffic never touches the public internet — VPC configuration for SageMaker jobs/endpoints and VPC endpoints (PrivateLink) for its APIs; Private Service Connect / VPC Service Controls perimeters on GCP (the latter guarding against exfiltration, not just ingress); private endpoints + managed VNets on Azure ML. Regulated-workload reviews start here.
  • Encryption: at-rest encryption of artifacts, checkpoints, registry entries, and endpoint storage with customer-managed keys (KMS / Cloud KMS / Key Vault) when the compliance regime requires holding your own keys; TLS in transit; inter-node traffic encryption available for distributed training on sensitive data.
  • Audit: CloudTrail / Cloud Audit Logs / Azure Activity Log record the control plane — who created which job, who approved which version, who flipped which traffic split. Data-plane content capture (SageMaker Data Capture logging endpoint requests/responses for drift monitoring) is the same double-edged sword as Phase 24's invocation logging: exactly what you need for monitoring and forensics, and a durable copy of possibly-sensitive payloads to govern accordingly.

One integrated sentence for interviews: "the registry plus IAM plus audit logs is the control system — every model in production got there through an authorized, logged, lineage-tracked transition, and I can prove it after the fact." That's what "governance" means when it's concrete.

16. Common misconceptions

  • "SageMaker/Vertex/Azure ML and Bedrock/Azure OpenAI are competing products." Different layers: lifecycle platform for models you train/own versus managed serving of frontier models. They compose; only Vertex merges the two surfaces (§13).
  • "Managed training means the platform tunes my model." It manages the lifecycle (scheduling, provisioning, artifacts, retries). Divergence, overfitting, and OOMs are still yours. Tuning is a separate primitive you configure (§4).
  • "Spot training is risky." Undisciplined spot is risky. With checkpointing sized to the interruption rate, it's the single biggest cost lever in training — and the resumed run should produce the identical artifact (Lab 01 proves it).
  • "Grid search is the rigorous choice." At equal budget random search usually dominates in high dimensions (Bergstra–Bengio), and Bayesian beats both when trials are expensive and sequential. Grid is fine for 2–3 discrete knobs only.
  • "The registry is a model S3 bucket with labels." It's the control system: versions, stage/approval state machines, lineage, and the authorization point deployments key off. Treat it as storage and you rebuild governance ad hoc per team.
  • "Autoscaling makes capacity a solved problem." Cooldowns, scale-in risk, cold starts, and min-capacity availability floors are all judgment calls the autoscaler cannot make for you (§6).
  • "Canary and A/B testing are the same thing." Canary is a safety mechanism (small slice, promote-or-rollback, fast); A/B is a measurement mechanism (fixed splits, significance, sticky assignment, business metric) (§9).
  • "Shadow testing measures user impact." By construction it measures system and output behavior only — users never see shadow responses, so anything requiring user reaction needs a canary/A-B (§9).
  • "Pipeline caching just works." Caching is only as correct as its key. Keys that omit a real input — external state, a code change invisible to the component spec — replay stale results silently (§10). Know what's in your platform's key.
  • "AutoML replaces ML engineers." It automates the search; it doesn't frame the problem, catch leakage, weigh error costs, or own production. Strong baseline generator, not a lifecycle (§11).

17. Lab walkthrough

Build the three miniatures in order; each isolates one third of the platform and injects the effectful dependency (training step, predict function, step function) as a pure callable so everything is deterministic and offline.

  1. Lab 01 — Training Jobs & the Model Registry. The training-job state machine on a finite TrainingCluster, spot interruption + checkpoint/resume (byte-identical artifact proven by hash), grid/random/Bayesian HyperparameterTuner with deterministic best-trial selection, and a ModelRegistry with the stage state machine, auto-archive-on-promote, and lineage. 30 tests.
  2. Lab 02 — Endpoints & Deployment Safety. Target-tracking autoscaling with cooldown hysteresis on an injected tick clock, a MultiModelEndpoint with lazy-load + LRU eviction, deterministic BatchTransform, and the deployment-safety trio — canary (both promote and rollback paths), blue/green (atomic switch, instant rollback), shadow (zero user impact, crash-isolated). 31 tests.
  3. Lab 03 — ML Pipelines. The DAG engine: dependencies derived from "step.output" references, Kahn's-algorithm ordering with cycle detection, artifact passing, content-keyed step caching (hit replays, miss re-runs), conditional branches with cascading skips, fan-out/fan-in, and the canonical train→evaluate→gate→register pipeline run down both branches. 24 tests.

Run each with LAB_MODULE=solution pytest test_lab.py -v first (green reference), then fill your lab.py to match, then read solution.py's main() output.

18. Success criteria

  • You can state the control-plane / data-plane split and give two operational consequences.
  • You can describe the training-job contract (code, compute, data, hyperparameters, lifecycle) in all three clouds' vocabulary.
  • You can explain why spot + checkpointing works, size a checkpoint interval qualitatively, and say what property a resumed run must preserve.
  • You can compare grid, random, and Bayesian search — including why random beats grid at equal budget in high dimensions.
  • You can describe a registry's four jobs (versions, live-pointer, lineage, approval) and both the stage and approval-status vocabularies.
  • You can write the target-tracking formula and explain both cooldowns and why scale-in is the dangerous direction.
  • You can walk a workload through real-time / MME / batch / serverless with the cost signature of each.
  • You can contrast shadow, canary, blue/green, and A/B — and give the composition order.
  • You can explain a pipeline cache key, what invalidates it, and the stale-cache foot-gun.
  • You can run the SageMaker-vs-Vertex-vs-Azure-ML decision framework (gravity: data, identity, GenAI dependency, team) and place Bedrock / Azure OpenAI on the map.
  • All three labs pass under both lab and solution (85 tests total).

19. Interview Q&A

Q: Your JD says "Vertex AI or Azure ML or SageMaker." How different are they really? A: The lifecycle is the same — managed training jobs, HPO, a model registry, real-time/batch/serverless serving, and a pipeline DAG engine — so skills transfer almost one-to-one. The differences that matter are gravitational: SageMaker has the broadest à-la-carte serving menu and the deepest AWS integration but keeps GenAI in a separate product (Bedrock); Vertex is the most integrated single surface with BigQuery data gravity and Gemini native inside the platform, and its pipelines are managed Kubeflow; Azure ML is MLflow-native with the strongest enterprise identity/compliance story and Azure OpenAI next door. I'd pick by where the data, identity, and GenAI dependency already live, not by feature checklists.

Q: What actually happens when you submit a managed training job? A: The control plane validates and queues it (Pending), acquires the requested instances, pulls your container or wraps your script in a framework container, stages the data channels from object storage, and runs your code with hyperparameters passed through; logs and metrics ship to the platform's monitoring; on exit 0 the artifact is uploaded and the job is Completed, on exception Failed, and spot interruption gives a distinct interrupted/stopped path. The platform owns that lifecycle — scheduling, provisioning, retry-on-infra-failure, artifact plumbing — while the learning itself is still your code's problem.

Q: How does spot/managed-spot training not corrupt your model? A: Checkpointing. The training loop writes model + optimizer state + progress counter to a synced path every N steps; an interruption loses only in-memory progress since the last checkpoint; on restart the platform re-supplies the checkpoint and the loop resumes from it. The correctness bar is that a resumed run produces the same final artifact as an uninterrupted one — if resume double-applies or skips an epoch, you've silently trained a different model. The economics: 60–90% discounts, paid for with redone-work-per-interruption of about half the checkpoint interval, which is how you size the interval.

Q: Grid, random, or Bayesian for hyperparameter tuning? A: Grid only for 2–3 discrete knobs — it's \(v^k\) trials and wastes budget re-testing unimportant dimensions. Random dominates grid at equal budget in higher dimensions because each important dimension gets many distinct values (Bergstra–Bengio). Bayesian fits a surrogate over completed trials and picks the next point by an acquisition function balancing exploitation and exploration — best sample efficiency when trials are expensive, but it's more sequential, so high parallelism erodes its advantage. And the objective must be a validation metric, or the tuner will faithfully select your most overfit trial.

Q: What is a model registry for, beyond storing artifacts? A: Four things: immutable versions grouped by model; a governed pointer to what's live — stage machines like None/Staging/Production/Archived in MLflow's vocabulary, or SageMaker's approval-status-plus-EventBridge-triggered-CD, or Vertex's aliases; lineage back to the exact job, hyperparameters, data, and code that produced each version; and the authorization point for promotion. It's the source of truth deployments read from — endpoints are just consumers of whatever version the registry designates.

Q: Design serving for 800 per-tenant models averaging 2 requests/minute each. A: Not 800 endpoints — at two instances each for availability that's a five-figure monthly bill for an idle fleet. This is the multi-model endpoint case: one shared autoscaled fleet, artifacts in object storage, lazy-load on first request per model, LRU eviction under memory pressure. I'd confirm the models share a framework/container, check tenant-isolation requirements (MME co-locates tenants in process space — a compliance question, not just a technical one), and carve out dedicated endpoints only for the few hot tenants with strict p99, since their first-hit cold-load tax is the pattern's one real cost.

Q: Contrast shadow, canary, and blue/green. When each? A: Shadow mirrors traffic to the candidate and never returns its responses — zero user risk, used to verify behavior on real traffic before exposure; its limit is it can't measure user reaction. Canary exposes a small real slice and promotes or rolls back on health signals — bounded blast radius, and the rollback path needs to be exercised, not assumed. Blue/green runs a full parallel environment with an atomic switch and instant total rollback — the strongest rollback story at the price of double capacity. Composition: shadow → canary → promote, with blue/green when instant total rollback is a hard requirement. A/B testing is the fourth idea and a different intent — measurement at fixed splits with statistical significance, not a safety progression.

Q: What makes autoscaling an ML endpoint harder than it looks? A: The formula is trivial — ceil(load / target-per-instance), clamped to min/max — the judgment is in the hysteresis and the floors. Cooldowns prevent flapping, and they're asymmetric on purpose: scaling out late costs latency; scaling in early costs an outage on the rebound, so scale-in cooldowns run longer. Min capacity is an availability decision (min=1 means one instance failure is 100% capacity loss), and model cold starts make every scaling mistake more expensive than it would be for a stateless web service.

Q: What does a pipeline's step cache key contain, and how does it burn you? A: Everything that should determine the step's output: step identity, resolved input artifact values, and parameters — production systems also key the container image and command. Same key skips execution and replays recorded outputs; that's why fixing the last step of a six-hour pipeline reruns in minutes. It burns you when the key misses a real input: a code change invisible to the key (KFP caches on the component spec), or a step reading external state — you get yesterday's outputs served as today's, silently. The fix is versioning images into the key and disabling caching on steps with side inputs.

Q: Where do Bedrock and Azure OpenAI fit if the ML platform already serves models? A: Different layer. The ML platform runs the lifecycle for models you train and own; Bedrock and Azure OpenAI serve frontier models you don't own — no training job, no artifact of yours, so the lifecycle collapses to choose/ground/govern, which is catalog-and-invocation machinery. AWS and Azure keep them as separate products beside SageMaker and Azure ML; Vertex is the exception, folding Gemini and Model Garden into the ML platform itself. The composition still uses this phase's discipline: prompt/model changes ride shadow/canary rollouts and evaluation-gated pipelines just like any model version.

Q: When would you argue against a managed ML platform? A: When the lifecycle is trivial (one model, one nightly batch job — a container on a scheduler is less machinery), when the economics invert at scale (a large, steady, well-staffed GPU fleet can beat managed pricing with Kubernetes + open tooling like Kubeflow/MLflow/Ray, at the cost of owning the platform team), or when portability is a hard requirement and the open stack is the strategy. The honest framing: managed platforms sell you leverage per engineer; at some scale-and-competence point the markup exceeds the leverage — but most teams claiming they're past that point are underestimating what the platform team costs.

20. References

  • Amazon SageMaker — Developer Guide (training jobs, endpoints, MME, pipelines, registry). https://docs.aws.amazon.com/sagemaker/
  • SageMaker — Managed Spot Training (checkpointing + interruption contract). https://docs.aws.amazon.com/sagemaker/latest/dg/model-managed-spot-training.html
  • SageMaker — Automatic Model Tuning (grid/random/Bayesian/Hyperband, early stopping). https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning.html
  • SageMaker — Model Registry (model package groups, approval status). https://docs.aws.amazon.com/sagemaker/latest/dg/model-registry.html
  • SageMaker — Multi-Model Endpoints (lazy loading, LRU eviction under memory pressure). https://docs.aws.amazon.com/sagemaker/latest/dg/multi-model-endpoints.html
  • SageMaker — Deployment guardrails (blue/green with all-at-once / canary / linear traffic shifting, auto-rollback on alarms). https://docs.aws.amazon.com/sagemaker/latest/dg/deployment-guardrails.html
  • SageMaker — Shadow tests. https://docs.aws.amazon.com/sagemaker/latest/dg/shadow-tests.html
  • SageMaker — Pipelines (steps, properties, caching, ConditionStep). https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines.html
  • SageMaker — Pricing (per-capability instance pricing). https://aws.amazon.com/sagemaker/pricing/
  • Google Cloud Vertex AI — documentation home (custom training, endpoints, registry, pipelines). https://cloud.google.com/vertex-ai/docs
  • Vertex AI — Pipelines introduction (managed KFP, execution caching, ML Metadata). https://cloud.google.com/vertex-ai/docs/pipelines/introduction
  • Vertex AI — Vizier overview (Bayesian hyperparameter optimization service). https://cloud.google.com/vertex-ai/docs/vizier/overview
  • Vertex AI — Model Registry (versions, aliases). https://cloud.google.com/vertex-ai/docs/model-registry/introduction
  • Azure Machine Learning — documentation home (jobs, sweep jobs, managed online endpoints, pipelines v2, MLflow integration). https://learn.microsoft.com/en-us/azure/machine-learning/
  • Azure ML — "Safe rollout for online endpoints" (traffic percentages, mirrored/shadow traffic) — in the Azure ML docs; title as named.
  • Azure OpenAI Service documentation (the Azure GenAI seam; now under Azure AI Foundry). https://learn.microsoft.com/en-us/azure/ai-services/openai/
  • Kubeflow Pipelines — documentation (component I/O, dsl.If, execution caching semantics). https://www.kubeflow.org/docs/components/pipelines/
  • MLflow — Model Registry (registered models, versions, stages; newer alias/tag workflow). https://mlflow.org/docs/latest/model-registry.html
  • Bergstra & Bengio, "Random Search for Hyper-Parameter Optimization," JMLR 2012 (why random beats grid at equal budget).
  • Amazon Bedrock — see Phase 24 of this track and its references for the GenAI-platform side of §13.