Lab 02 — Real-Time Endpoints, Batch Transform & Deployment Safety

Phase 25 · Lab 02 · Phase README · Warmup

The problem

A registered model (Lab 01) is worth nothing until it serves predictions — and how it serves them is a menu of real architectural decisions every managed platform (SageMaker, Vertex AI, Azure ML) exposes and every interviewer probes:

1. Real-time endpoint — a synchronous HTTPS service in front of the model. It must autoscale: the platform watches a load signal (concurrency/invocations-per-instance) and target-tracks toward a desired instance count within your min/max, with cooldowns so it doesn't flap.
2. Multi-model endpoint — dozens/thousands of models sharing one fleet, routed by model name, lazy-loaded on first use and LRU-evicted under memory pressure. Great economics for many small models; a cold-load latency tax on first hit.
3. Batch transform / batch prediction — no persistent endpoint at all: spin up, map a dataset through the model offline, write results, tear down.
4. Deployment safety — you never yank the old model out and shove the new one in. You shadow (mirror traffic, compare, zero user impact), canary (give the candidate a small traffic slice, promote or roll back on a health signal), or blue/green (full parallel environment, atomic switch, instant rollback).

This lab builds all of it, deterministic and offline: the model is an injected pure predict callable, the clock is an injected tick, and the traffic split is deterministic by request index — so every scaling decision, routing choice, promotion, and rollback is a provable assertion.

What you build

File map

Run it

pytest test_lab.py -v                       # your lab.py (red until you implement)
LAB_MODULE=solution pytest test_lab.py -v   # the reference (green)
python solution.py                          # worked example

Success criteria

• Endpoint.observe scales out under load, scales in when load drops, clamps to [min_instances, max_instances], and holds during an active cooldown (both directions).
• MultiModelEndpoint routes by name, lazy-loads on first invoke, evicts the LRU model at the cap, and transparently reloads an evicted model on its next request.
• BatchTransform.run preserves input order deterministically; fail-fast raises BatchTransformError; continue_on_error captures per-record errors and keeps going.
• Canary: a 20% weight sends exactly 20 of every 100 requests to the candidate; a healthy signal promotes (weight 100, candidate active); an unhealthy signal rolls back (weight 0, baseline active).
• Blue/green: deploy_green moves no traffic; switch moves all of it atomically; rollback restores blue instantly; switching with no green raises.
• Shadow: the caller always receives the primary's output; mismatches are logged with both outputs; a shadow exception is counted but never reaches the caller.
• All 31 tests pass under both lab and solution.

How this maps to the real stack

• Real-time endpoints are SageMaker real-time inference endpoints (CreateEndpoint over an EndpointConfig of production variants), Vertex AI online prediction (Endpoint + DeployedModel), and Azure ML managed online endpoints (endpoint + deployments). Autoscaling in real SageMaker is Application Auto Scaling target tracking on SageMakerVariantInvocationsPerInstance with scale-out/scale-in cooldowns — exactly our desired = ceil(load / target) + cooldown hold. Vertex autoscaling targets (CPU/GPU utilization or QPS per replica) and Azure ML's scale rules are the same shape with different signals.
• Multi-model endpoints are SageMaker MME: thousands of models behind one endpoint, addressed by TargetModel, lazy-loaded from S3 into container memory on first invocation and evicted least-recently-used under memory pressure — our _ensure_loaded is that lifecycle verbatim. Vertex AI's analogue is co-hosting via a DeploymentResourcePool; Azure ML approximates it with multiple deployments behind one endpoint.
• Batch transform is SageMaker Batch Transform, Vertex AI batch prediction, and Azure ML batch endpoints/parallel jobs: no persistent endpoint, a job maps an input dataset (S3/GCS/Blob) through the model and writes outputs. Real batch runs distributed and unordered; ours is sequential and order-preserving so the tests can assert on it.
• Canary and blue/green are SageMaker deployment guardrails: blue/green with all-at-once, canary, or linear traffic shifting, auto-rollback wired to CloudWatch alarms — our injected healthy boolean is the alarm signal. Vertex AI does canary via trafficSplit percentages across deployed models on one endpoint; Azure ML mirrors this with traffic percentages across deployments on a managed online endpoint.
• Shadow is SageMaker shadow testing (shadow variants receive a copy of live traffic and their responses are logged, never returned) and Azure ML's traffic mirroring; on Vertex you'd build it with a duplicate-and-compare layer. The invariant our tests enforce — the user always gets the primary's answer and a shadow crash changes nothing — is the entire point of the pattern.

Limits of the miniature. Real autoscalers consume noisy, delayed metric streams (CloudWatch minutes, not ticks) and scale gradually; ours jumps straight to the target — the cooldown/hysteresis lesson survives, the metric plumbing doesn't. Real MME eviction is byte-based memory pressure, not a model-count cap. Real canary traffic split is random per-request at the load-balancer, not index-sliced — ours is deterministic specifically so the 20-of-100 assertion is exact. Real blue/green costs real double capacity during the window; here it's free.

Extensions (your own machine)

• Add serverless/async inference: a queue in front of the endpoint (injected tick clock), scale-to-zero after an idle TTL, and a cold-start latency penalty on the next request — then compare p50/p99 against the always-warm endpoint.
• Add linear traffic shifting to the canary: step the weight 10% → 25% → 50% → 100% with a health evaluation between steps, rolling back from any step.
• Give the shadow deployment a tolerance-aware comparator (inject compare(live, shadow) -> bool) for regression-testing models whose outputs are floats, not exact strings.
• Track per-variant invocation metrics (count, simulated latency histogram) and drive the canary health signal from them instead of an injected boolean.

Interview / resume signal

"Built the serving layer of a managed ML platform in miniature: a real-time endpoint with target-tracking autoscaling (cooldown hysteresis included), a multi-model endpoint with lazy-loading and LRU eviction, deterministic batch transform, and all three deployment-safety strategies — canary with promote/rollback on a health signal, blue/green with atomic switch and instant rollback, and shadow testing that provably never affects the live response."