« Phase 25 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor

Phase 25 — Staff Engineer Notes: Owning the ML Lifecycle Platform

Anyone can call est.fit() and model.deploy(). The gap between using SageMaker/Vertex/Azure ML and being trusted to own the lifecycle is judgment about four things nobody hands you a default for: which capacity mode each workload runs on, how governed the registry actually is, which deployment strategy a change earns, and — the one that gets escalated to you — which cloud the company standardizes on at all. This doc is about that judgment and the signal that proves you have it.

The decisions a staff engineer actually owns here

Capacity is a portfolio decision, not a per-service toggle. Training: spot with real checkpoint discipline for the bulk of runs (60–90% off, earned by resume that produces the identical artifact), on-demand reserved for the final convergence run where an interruption is expensive. Serving: real-time for steady interactive traffic with a p99; MME for the fleet of small, individually-idle models; batch for nightly scoring and offline eval; serverless for spiky, rare, or huge-payload work. The junior instinct is one mode everywhere; the staff move is to segment by shape and do the always-on-fleet-vs-per-model-traffic division out loud before anyone approves a 40-endpoint bill.

Registry governance is a discipline you enforce, not a feature you enable. "Artifacts in a bucket with good naming" is a confession, not a registry. You own that every production model got there through a versioned, lineage-tracked, authorized transition — so that the 2 a.m. question ("what exactly is running and how was it built?") is one query, not a grep through deploy logs and S3 timestamps. And you speak both vocabularies — MLflow stages and SageMaker approval statuses wired to CI/CD — because they are the same idea and someone will test whether you know it.

Deployment strategy is a risk decision you translate into traffic engineering. Shadow first (free learning, zero user risk), canary second (bounded blast radius, promote-or-rollback, both paths rehearsed), blue/green when instant total rollback is a hard requirement and you will pay double capacity for it. A/B is not on this ladder — it is measurement, not safety.

Platform selection is the escalation that lands on your desk. "SageMaker, Vertex, or Azure ML?" has no universal answer — only the gravity axis you optimize.

The decision framework, out loud

  • Already an AWS shop, want the broadest à-la-carte serving menu and deepest AWS operational integration → SageMaker (GenAI lives separately in Bedrock).
  • Data lives in BigQuery, want ML and GenAI on one surface, or Gemini/multimodal is a product requirement → Vertex AI (the one platform that folds GenAI in; pipelines are managed KFP, an open standard).
  • Identity, compliance, and procurement are already Microsoft, or OpenAI-day-one via Azure OpenAI anchors the strategy, or you want MLflow-native → Azure ML.
  • The lifecycle is trivial (one model, one nightly batch), or you have a large steady well-staffed GPU fleet and portability is the strategy → argue against a managed platform: a container on a scheduler, or Kubernetes + Kubeflow/MLflow/Ray, at the cost of owning the platform team.

Naming the gravity axis is the signal. "SageMaker is best" is the anti-signal. And keep the GenAI seam sharp: SageMaker is not AWS's GenAI answer (Bedrock is), Azure ML is not Microsoft's (Azure OpenAI is), Vertex is the only merge — naming the wrong layer is an instant seniority tell.

Code-review red flags

  • A new always-on endpoint per experiment. Ask for the \$/month math first; MME, batch, and serverless exist precisely for the idle-fleet case.
  • Spot with no checkpoint, or with untested resume logic. The discount is earned by a resume that produces the identical artifact — demand the byte-identity assertion, not "resume doesn't crash."
  • Tuning on training accuracy. The tuner will faithfully select the most overfit trial; the objective must be a validation metric.
  • Registry treated as a file-naming convention. No stages/approvals means no answer to "what's in prod and who approved it."
  • min_instances=1 on a customer-facing endpoint. One instance failure is 100% capacity loss while the autoscaler reacts; min≥2 across zones is the boring right answer.
  • Pipeline caching enabled on a step that reads external state. Green checkmark, stale model — the cache key doesn't know what you didn't tell it.
  • "We A/B-tested for safety." Canary is safety (small slice, promote-or-rollback, fast); A/B is measurement (fixed splits, significance, sticky users). Mixing them is a tell.
  • A canary with a rollback path that has never fired in staging. The night you need it is the worst time to discover the config it points at went stale.
  • README.md chapter links instead of index.md, or hardcoded artifact paths in deploy tooling instead of "latest production version" — small tells the author hasn't internalized the boundaries.

War stories worth carrying

  • The endpoint bill nobody watched. Every experiment got a two-instance endpoint "temporarily"; eleven months later, 40+ endpoints, most serving zero requests, ~\$12k/month. The fix was boring — MME for the long tail, batch for the nightly ones, a deletion policy — and the lesson is the always-on-cost-vs-per-model-traffic division.
  • The spot job that trained a different model. Resume re-applied the checkpoint epoch instead of starting from the next one; metrics looked plausible, the artifact hash didn't match the on-demand rerun during an audit. One off-by-one, silently different weights for months.
  • The pipeline shipping yesterday's model. A "fetch latest data" step read an external table but had caching on; inputs unchanged as far as the key knew, so it replayed March's output into April's retrain. Green pipeline, stale model, three weeks. Cache keys are a correctness boundary.
  • The rollback that had never been rehearsed. A canary wired to promote on healthy metrics; the rollback lambda had a stale endpoint-config ARN, discovered the night it was needed.

The interview / architecture-review signal

What a reviewer listens for: can you whiteboard the lifecycle once and translate it into all three vendors' vocabulary? Managed training job → HPO → registry → serving menu → deployment safety → pipeline DAG, same shape everywhere. Can you do the MME-vs-dedicated cost division live, explain why a resumed spot run must be byte-identical, name what invalidates a step-cache key, and place Bedrock / Azure OpenAI / Gemini-in-Vertex on the GenAI seam? Candidates who know one console are common; the person who draws the lifecycle and the lines between the layers is who gets handed "make this production-ready — and it has to run our fraud model and our LLM feature."

Closing takeaways

  1. Own the capacity portfolio, not a single mode. Segment training and serving by shape, and re-run the cost math when traffic moves.
  2. The registry is the source of truth; the endpoint is a consumer. Governance lives in versions, stages/approvals, and lineage — not in bucket naming.
  3. Deployment safety is a progression — shadow → canary → promote, blue/green when instant total rollback is worth double capacity — and the rollback path only exists if you've rehearsed it.
  4. The cache key is a correctness boundary. A key that misses a real input serves a stale model with a green checkmark.
  5. Pick the platform by gravity, not features, and keep the GenAI layer behind the same registry-and-rollout discipline as every other model. That is the staff signal.