« Phase 25 README · Warmup · Hitchhiker's · Deep Dive · Core Contributor · Staff Notes

Phase 25 — Principal Deep Dive: Cloud ML Platforms as Production Systems

The Deep Dive traced the mechanisms — the resumable training state machine, the autoscaling clamp, the LRU serving cache, the DAG engine. This doc is about the system those mechanisms live inside: where SageMaker / Vertex AI / Azure ML sit in a real enterprise ML platform, how each of the four capacity surfaces scales, what fails and how far the blast radius reaches, and which "looks wrong" decisions are load-bearing. The through-line: the value you buy is not any one job or endpoint — it is that you stop re-solving scheduling, capacity, governance, and orchestration once per model your organization ships, across a fleet where a fraud classifier, a forecaster, and an LLM all run the same lifecycle.

The control-plane / data-plane split is the architecture

The single most important structural fact — inherited straight from Phase 24's Bedrock split — is that job/endpoint/pipeline CRUD and the inference hot path are different planes with opposite operational profiles, and every capacity, cost, and blast-radius decision falls out of that boundary.

• Control plane (CreateTrainingJob, CreateEndpoint, CreateModelPackage, CreatePipeline) is slow-moving, IAM-heavy, asynchronous-by-design (it returns a resource in Creating/Pending and you poll the lifecycle forward), and driven by CI/CD or humans. Low volume, high privilege, latency-insensitive.
• Data plane (InvokeEndpoint, Vertex predict, Azure online scoring) is the per-request path that gets autoscaled, gets rate-limited, logs every inference dollar, and is where a p99 SLA either holds or doesn't.

Designing to this split matters because the two planes fail differently: a control-plane outage means you cannot deploy; a data-plane outage means you are down. Mature teams alarm on them separately and give CI a control-plane role that can register a version but never invoke, and give the serving identity a data-plane role scoped to exactly the endpoints it needs. Collapsing them into one over-broad role is the most common security-review finding on any of these platforms, and it is invisible until an audit.

The scaling envelope is four different capacity shapes

There is no single "platform throughput number." Your envelope is a composition of four surfaces, each with its own scaling law:

• Training capacity is a finite, interruptible pool. Managed compute is finite — real accounts hit instance quotas and ResourceLimitExceeded long before "the cloud is infinite" survives contact, which is exactly why Lab 01's TrainingCluster refuses placement at available() <= 0. Spot/preemptible capacity is 60–90% cheaper but revocable, so the scaling lever is checkpoint interval sized against interruption rate: expected wasted work per interruption is roughly half the interval, and under distribution one worker's preemption stalls the whole lockstep — checkpoint/resume is the mechanism that keeps a spot fleet economically viable rather than a lottery.
• Real-time endpoints scale by target tracking with hysteresis. desired = clamp(ceil(load / target_per_instance), min, max), asymmetric cooldowns, and — the fact the rest of the serving menu exists to mitigate — billed per instance-hour whether or not a request arrives.
• Multi-model endpoints trade a cold-load tax for fleet consolidation. One shared fleet, many models, memory-pressure LRU eviction; the scaling ceiling is aggregate memory, not request rate.
• Serverless/async scale to zero and trade a cold start for eliminating idle cost; batch removes latency from the equation for delay-tolerant work.

The capacity math you should be able to derive live: one modest instance at ~\$0.23/hour is ~\$166/month; two-instance-minimum for availability is ~\$330/month per model, before a single request. Ten models on ten endpoints is ~\$3,300/month; the same ten on one two-instance MME is still ~\$330. That one division — always-on fleet cost versus per-model traffic — is the entire architectural argument for MME, serverless, and batch, and it is the number to volunteer when someone asks "should every model get an endpoint?"

Failure modes and blast radius

Think in blast radius, because the expensive failures here are quiet correctness and cost regressions, not crashes.

• Endpoint sprawl. Every experiment gets a two-instance endpoint "temporarily"; a year later it is 40 idle endpoints and five figures a month. Blast radius: the budget, silently — no alarm fires because everything is healthy, just idle.
• The stale pipeline cache. A "fetch latest data" step reads an external table but has caching on; the key sees unchanged declared inputs and replays last month's output into this month's retrain. Green pipeline, stale model, indefinite duration. Blast radius: every consumer of that model, and the failure is invisible precisely because the run succeeded.
• The off-by-one resume. Spot resume replays or skips one epoch; the artifact hash diverges from the on-demand rerun. Blast radius: every prediction from a silently-different model with plausible metrics — which is why Lab 01 asserts byte-identity, not "resume doesn't crash."
• The unrehearsed rollback. A canary promotes on healthy metrics but the rollback path points at a stale endpoint-config ARN and has never fired in staging. Blast radius: the full canary exposure, extended by however long the broken rollback takes to fix under incident pressure.
• MME co-tenancy. A multi-model endpoint co-locates tenants in one process space — a real trust boundary, not just a technical one. Blast radius: cross-tenant if isolation was a compliance requirement someone assumed MME satisfied.

Cross-cutting concerns

Security. The trust boundary is architectural. IAM/service-accounts/Entra-ID gate every control-plane verb and every data-plane call; the platform-specific sharp edge is that a training job runs with its own role and can read your data lake with whatever breadth you granted — that role deserves the same least-privilege scrutiny as any service. VPC/PrivateLink keeps training and inference traffic off the public internet (where every regulated review starts); KMS/CMK encrypts artifacts, checkpoints, and registry entries; audit logs record who created which job, approved which version, flipped which traffic split.

Cost. Four capacity modes are four pricing shapes, not four numbers: training is instance-seconds (spot is the biggest lever, paid for in checkpoint discipline); real-time is the always-on tax; batch is zero-idle; serverless is per-invocation with a crossover into dedicated at sustained volume — the same crossover shape as Phase 24's On-Demand-vs-Provisioned-Throughput math, one layer down.

Observability. Data Capture logging endpoint requests/responses is exactly what you need for drift monitoring and a durable second copy of possibly-sensitive payloads to govern — the same double edge as Bedrock invocation logging.

Multi-tenancy. MME serves a fleet of per-tenant models from one endpoint; the LoRA-adapters- over-one-base-LLM pattern is the same economics applied to fine-tunes.

The "looks wrong but is intentional" decisions

• The injected pure-function seam. Every lab injects the one expensive, non-deterministic thing — training step, predict, step body — as a pure callable. This reads like a toy simplification and is actually the production discipline: reproducible training and record/replay are how you make an ML lifecycle testable. The lifecycle is what the platform owns; injecting the learning is what lets you prove the lifecycle is correct.
• Managed KFP. Vertex Pipelines is Kubeflow Pipelines — an open standard running inside a proprietary platform. That looks like Google not writing their own orchestrator; it is the portability hedge that lets the workflow layer outlive the vendor choice.
• The registry, not the endpoint, is the source of truth. Endpoints, batch, and serverless are interchangeable serving shapes for whatever version the registry blesses. Hardcoding an artifact path into deploy tooling instead of reading "latest production version" is the anti-pattern this inversion exists to prevent.
• Model-agnostic by construction. The same job that fits a gradient-boosted churn model fine-tunes a transformer; the same endpoint serves a pickle or an LLM. That generality is why one discipline covers the whole fleet — and why the GenAI seam (§13) is a layer distinction, not a competing product.

Where the platform fits the decision

The lifecycle is commoditized — all three train, tune, register, serve, and orchestrate competently — so platform selection is gravity, not features: where the data lives (BigQuery → Vertex, an S3 lake → SageMaker, Synapse/Fabric → Azure), where identity and compliance already sit (Entra ID → Azure), what the GenAI dependency looks like (OpenAI → Azure, model breadth under one AWS envelope → Bedrock+SageMaker, Gemini/multimodal → Vertex native), and what the team already knows. Answer those four and the platform picks itself; the feature table only settles residual ties. The architecture that actually ships is the same everywhere: a managed spot job trains, the registry governs, a shadow-then-canary rollout deploys, a cached DAG orchestrates the weekly retrain — and the GenAI layer rides the same registry-and-rollout discipline as every other model. Three vendors, one lifecycle, one set of boxes and lines.