« Phase 25 README · Warmup · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 25 — Hitchhiker's Guide

The compressed practitioner tour. Read the WARMUP for the mechanism; this is the stuff you say in the meeting.

30-second mental model

SageMaker, Vertex AI, and Azure ML are the same platform wearing three uniforms: a managed training job (bring script/container + hyperparameters + instance request; the platform owns the Pending → InProgress → Completed/Failed lifecycle, spot capacity, and checkpoints), a tuning job (fan out trials, pick best by objective), a model registry (versions, stages or approval statuses, lineage — the source of truth deployments read from), a serving menu (real-time endpoint / multi-model endpoint / batch / serverless-async), deployment safety (shadow → canary → blue/green), and a pipeline DAG engine (artifacts, step caching, conditions). GenAI sits beside the platform on AWS (Bedrock, Phase 24) and Azure (Azure OpenAI), and inside it on GCP (Gemini in Vertex). The senior move: "pick the cloud by gravity — data, identity, existing commitment — because the lifecycle is the same everywhere."

The Rosetta Stone to tattoo on your arm

The distinctions that signal seniority

• Lifecycle platform vs GenAI platform → SageMaker/Vertex/Azure ML run models you own through train→register→deploy; Bedrock/Azure OpenAI serve someone else's frontier models. Different layers, they compose. Only Vertex merges them.
• Registry is the source of truth, not the endpoint → deployments read "latest approved/production version"; artifacts in a bucket with no stage machine is storage, not governance.
• Spot is a checkpointing discipline, not a checkbox → the resumed run must produce the identical artifact; sized checkpoint intervals are cost math (redone work ≈ half the interval per interruption).
• Throttle-shaped vs queue-shaped serving decisions → real-time pays 24/7 for latency; batch pays only while running; MME trades a cold-load tax for fleet consolidation; serverless trades cold starts for scale-to-zero.
• Canary vs A/B → canary is safety (small slice, promote-or-rollback, fast); A/B is measurement (fixed splits, significance, sticky users). Saying "we A/B-tested for safety" is a tell.
• Shadow's two invariants → the user always gets the primary's answer; a shadow crash changes nothing. If either fails, it isn't shadow.
• Cache key = correctness boundary → a pipeline step cache that doesn't key on the container image (or that feeds on external state) will serve stale models with a green checkmark.

The SDK one-liners

# SageMaker: train -> register -> deploy, the whole spine
from sagemaker.estimator import Estimator
est = Estimator(image_uri=img, role=role, instance_count=1, instance_type="ml.g5.2xlarge",
                use_spot_instances=True, checkpoint_s3_uri=ckpt,          # spot + checkpoints
                hyperparameters={"learning_rate": 0.1})
est.fit({"train": "s3://bucket/train/"})                                  # the training job
pkg = est.register(model_package_group_name="churn",                     # -> Model Registry
                   approval_status="PendingManualApproval")
predictor = est.deploy(initial_instance_count=2, endpoint_name="churn-prod")

# Vertex AI: same ideas, GCP dialect
from google.cloud import aiplatform
job = aiplatform.CustomContainerTrainingJob(display_name="churn", container_uri=img)
model = job.run(replica_count=1, machine_type="n1-standard-8")            # train + auto-register
endpoint = model.deploy(traffic_split={"0": 100}, machine_type="n1-standard-4")

# Azure ML: MLflow-native registration
from azure.ai.ml import command
job = ml_client.create_or_update(command(code="./src", command="python train.py",
                                         compute="gpu-cluster", environment=env))
ml_client.models.create_or_update(Model(path=job_output, name="churn", type="mlflow_model"))

War stories

• The endpoint bill nobody was watching. A team gave every experiment its own two-instance real-time endpoint "temporarily." Eleven months later: 40+ endpoints, most serving zero requests, ~$12k/month. The fix was boring — MME for the long tail, batch for the nightly ones, a deletion policy — and the lesson is the Warmup §14 division: always-on cost vs per-model traffic.
• The spot job that trained a different model. Resume logic re-applied the checkpoint epoch instead of starting from the next one. Metrics looked plausible; the artifact hash didn't match the on-demand rerun during an audit. One off-by-one, silently different weights for months — exactly the byte-identical assertion Lab 01 makes you write.
• The pipeline that kept shipping yesterday's model. A "fetch latest data" step read from an external table but had caching enabled — inputs unchanged as far as the key knew, so the step replayed March's output into April's retrain. Green pipeline, stale model, three weeks. Cache keys are a correctness boundary, not an optimization detail.
• The rollback that had never been rehearsed. A canary was wired to promote on healthy metrics; the rollback path had never fired in staging. The night it was needed, the rollback lambda had a stale endpoint-config ARN. Lab 02 tests promote AND rollback for a reason.

Vocabulary

training job · framework container / BYOC · data channels · managed spot / preemptible · checkpoint/resume · data parallel / model parallel / FSDP-ZeRO · tuning job (grid / random / Bayesian, early stopping, Hyperband) · model registry · model package group / approval status (SageMaker) · stages (MLflow: None/Staging/Production/Archived) · aliases (Vertex) · lineage · real-time endpoint · production variant / DeployedModel / deployment · target tracking · InvocationsPerInstance · cooldown · multi-model endpoint (lazy load, LRU eviction, TargetModel) · batch transform / batch prediction · async inference · serverless inference (scale-to-zero, cold start) · shadow test / traffic mirroring · canary / linear shifting · blue/green · trafficSplit · pipeline (SageMaker Pipelines / KFP / AzureML v2) · step properties · artifact · execution caching · ConditionStep / dsl.If · ML Metadata · feature store (offline/online, point-in-time correctness, training/serving skew) · AutoML / Autopilot · BigQuery ML · Bedrock / Azure OpenAI (the GenAI seam).

Beginner mistakes

1. Giving every model its own always-on endpoint — do the $/month math first; MME/batch/serverless exist for a reason.
2. Running spot without checkpoints (or with untested resume logic) — the discount isn't free, it's earned by the checkpoint discipline.
3. Tuning on training accuracy — the tuner will faithfully find your most overfit trial.
4. Treating the registry as a file-naming convention — no stages/approvals means no answer to "what's in prod and who approved it."
5. min_instances=1 on a customer-facing endpoint — one instance failure = 100% capacity loss while the autoscaler reacts.
6. Skipping the shadow/canary progression because "it's just a small model update" — the update size and the blast radius are unrelated.
7. Enabling pipeline caching on steps that read external state — green checkmark, stale output.
8. Saying "we use SageMaker for GenAI" when you mean Bedrock (or "Azure ML" when you mean Azure OpenAI) — naming the wrong layer is an instant seniority tell, the same way Bedrock-vs-AgentCore was in Phase 24.