Warmup Guide — MLOps

Zero-to-expert primer for Phase 07: the path from trained checkpoint to operated service — ONNX export, inference servers, containers, experiment tracking, and the monitoring that keeps a deployed model honest.

Table of Contents

Chapter 1: The Gap Between a Model and a Product

A checkpoint answers "what are the weights"; a product must answer: how do requests reach it, in what format, at what latency and cost, with what failure behavior, which version is running, how do we know it still works on this month's data, and how do we roll back? The recurring industry observation (the Hidden Technical Debt paper's famous figure): the model is a small box in a large system of data plumbing, serving, configuration, and monitoring — and the system, not the model, is where production incidents live. This phase builds the small honest version of each surrounding box; the discipline transfers to any scale.

Chapter 2: ONNX — Export as a Contract

ONNX is the interchange format that decouples training framework from serving runtime (PyTorch → onnxruntime/TensorRT/OpenVINO; the deep compiler view lives in the model-accuracy track's Phase 05 — here, the practitioner's contract):

  • Export (torch.onnx.export, dynamo-based in modern PyTorch) is a capture of your model — and capture has the standard pitfalls: data-dependent Python control flow gets baked, training-mode layers (dropout/BN) must be in eval, and dynamic axes (batch, image size) must be declared or the graph hard-codes your example shape.
  • The non-negotiable ritual: after export, run identical inputs through PyTorch and onnxruntime and assert numerical agreement (rtol ~1e-3 for FP32 CV models) — on several shapes if axes are dynamic. An unverified export is a rumor. Then benchmark: onnxruntime with graph optimizations enabled is often 1.5–3× PyTorch eager on CPU — free speed, but measured, not assumed.
  • What doesn't export: NMS and other post-processing sometimes does (as ONNX ops) and sometimes doesn't (Phase 05 Ch. 2's deployment surprise) — decide explicitly whether post-processing lives in-graph or in app code, and version them together.

Chapter 3: Inference Servers — FastAPI and Beyond

The lab's FastAPI service, and the concepts that outlive the framework choice:

  • Load once, serve many: the model loads at startup (lifespan handler), never per request. Sessions/models are process-level singletons.
  • Concurrency model: inference is CPU/GPU-bound — async alone doesn't help (the GIL and the compute both block); throughput comes from worker processes (uvicorn/gunicorn workers) sized to cores-per-model-copy, or a dedicated inference runtime behind the API.
  • Batching: GPUs (and even vectorized CPU) love batches; HTTP delivers requests one at a time. Dynamic batching — queue requests for up to N ms or K items, run one batch — trades a small latency floor for large throughput gains. Triton/ TorchServe provide it off the shelf; the lab implements a micro version to own the concept (the LLM track's Phase 09 is this idea at its most extreme).
  • API design for CV: accept common encodings (JPEG/PNG bytes, not raw arrays); validate sizes and formats before the model; return structured predictions with model-version metadata in every response (the debugging gift to your future self); health (/healthz) and readiness endpoints; explicit timeouts.
  • Know the landscape one level up: TorchServe/Triton/KServe/BentoML solve batching, multi-model, and GPU scheduling — the lab's hand-rolled server is for understanding, and saying so is the right production posture.

Chapter 4: Preprocessing Parity — the Silent Killer

The most common real-world CV serving bug, promoted to its own chapter: the serving path's preprocessing must be bit-faithful to training's — resize algorithm and library (PIL bilinear ≠ OpenCV bilinear ≠ torchvision antialias settings!), color order (Phase 01's BGR/RGB), normalization stats, dtype/range, crop policy. Each mismatch costs silent accuracy — the model serves, predictions look plausible, quality is just worse, and nothing errors.

Defenses, in order of strength: bake preprocessing into the exported graph where possible (resize+normalize as model layers — the Phase 04/TF Ch. 6 move, equally available in torch); a shared preprocessing module imported by both training and serving (one source of truth); and a golden-input test — N images with stored expected outputs, run in CI against the serving path, catching any drift in the dependency chain (a Pillow upgrade changing default resampling is a real incident class). The capstones inherit this test as a requirement.

Chapter 5: Containers — Reproducible Runtime

Docker for ML, the working subset:

  • Why: "works on my machine" is lethal when the machine includes CUDA versions, system codecs (libjpeg versions change pixel values!), and Python dependency trees. The image pins the entire runtime.
  • The craft: slim base images (python:3.11-slim; CUDA runtime images only when GPU-serving), layer ordering for cache efficiency (requirements before code — dependency layers rebuild rarely, code layers often), multi-stage builds (build tools out of the final image), .dockerignore (no datasets/checkpoints in context), non-root user, pinned dependency versions (pip install -r with hashes or a lock file).
  • Model weights don't belong in the image for anything beyond toys: mount or download-at-start from a registry/object store keyed by version (Ch. 6) — images ship code, registries ship models, and the two version axes stay independent.
  • The mental model: a container is the deployment-time analog of the experiment config (Ch. 6) — the run is reproducible because everything is pinned.

Chapter 6: Experiment Tracking and the Model Registry

MLflow (the lab's tool; the concepts are universal — W&B etc. are isomorphic):

  • Tracking: every training run logs params (full config), metrics (curves, not just finals), and artifacts (checkpoints, confusion matrices, sample-prediction grids) to a server. The discipline this enforces is the same ledger ethic every track's capstone demands: no untracked runs — an untracked result cannot be trusted, compared, or reproduced. Autologging gets you 80% for one line.
  • The registry: models get names, versions, and stage labels (staging/production/archived) with transition history — "which model is in prod" becomes a query, not a Slack thread. Promotion is a recorded decision with the evaluation evidence attached (Phase 02/08 ethics: the eval that justified promotion is part of the artifact).
  • Lineage: a production prediction should be traceable to model version → training run → config + data version → code commit. Each link is one logging call at the right moment; the chain is what incident response runs on.

Chapter 7: Monitoring and Drift

The model is now a service that degrades silently. Three monitoring layers:

  1. System: latency percentiles (p50/p99), throughput, error rates, GPU/CPU utilization — ordinary SRE, plus model-version tags on every metric.
  2. Input/data monitoring: the live input distribution vs training — image statistics (brightness, blur/sharpness scores, resolution mix), prediction distribution (class frequencies, confidence histograms). Drift detection is distribution comparison over windows (PSI/KL on binned statistics) — alarms that say "the world changed" before labels arrive. The canonical CV examples: a camera moved/dirtied, a season changed, an upstream app changed JPEG compression.
  3. Outcome monitoring: when ground truth arrives (delayed labels, human review, downstream corrections), score it — sampled human audit where labels never arrive. The flywheel that closes the loop: production failures get harvested into the eval set and the next training set (every track's evaluation phase preaches this; here is where it physically happens).

And the operational basics that make fixes shippable: shadow deployment (new model scores silently alongside prod), canary rollout (small traffic share + auto-rollback on metric regression), and the rollback path tested before it's needed.

Chapter 8: The CV Serving Performance Playbook

The latency budget of a CV service is usually not dominated by the model: decode (JPEG → pixels), resize, host↔device copies, and post-processing routinely exceed the forward pass. The playbook, in measurement-first order:

  1. Profile the whole request (timestamps per stage — decode/preprocess/infer/post) before optimizing anything (the universal lesson, again).
  2. Cheap wins: ONNX runtime/graph optimizations (Ch. 2), FP16 on GPU, right-sized input resolution (accuracy vs latency curve — Phase 05's operating point), turbojpeg/GPU decode when decode dominates.
  3. Throughput wins: dynamic batching (Ch. 3), worker scaling, pinned-memory async copies.
  4. Bigger guns, with eval gates: INT8 quantization (calibrated, accuracy-measured — the model-accuracy track's Phase 03 ladder), TensorRT/compiled backends, distilled or smaller architectures.
  5. Re-measure end-to-end after each step; keep the latency budget table in the repo.

Chapter 9: torch.compile and the Export Path — the 2026 PyTorch Deployment Story

Chapter 2 treated export as a contract with a foreign runtime. Since PyTorch 2 there is a second road — make PyTorch itself fast enough to serve — and its mechanics are worth owning because its failure modes are shape-shaped, and CV is the land of variable shapes.

  • Why eager mode leaves speed on the table: eager executes the model as ordinary Python — every op is an interpreter round-trip plus a separate kernel launch, and every intermediate tensor is written to and re-read from GPU global memory. For conv-heavy models the big conv/GEMM kernels dominate and the overhead is modest; but normalizations, activations, and glue ops are memory-bandwidth-bound, and an unfused chain of them can cost as much as the convs. A captured graph enables what per-op execution cannot: fusion (compile a pointwise chain into one kernel, so data stays in registers instead of touching DRAM three times), static memory planning (buffer reuse computed once), and whole-graph scheduling.
  • torch.compile mechanics, at working altitude: TorchDynamo hooks Python bytecode execution and traces the tensor operations into an FX graph, recording alongside it a set of guards — the assumptions under which that trace is valid (input shapes and dtypes, Python values that were branched on, module identities). Inductor then compiles the graph into fused Triton kernels (GPU) or vectorized C++ (CPU). On every later call the guards are checked cheaply: hold → run the compiled artifact; fail (a new input shape, a changed flag) → recompile, which costs seconds. Python constructs Dynamo can't trace cause graph breaks: the model still runs, split into compiled segments with eager gaps between — correct, silently slower.
  • Guards are the classic CV pitfall — be honest about them. Variable image sizes (aspect-preserving resize, dynamic crops, variable batch from an upstream batcher) mean each new shape is a guard failure → a recompile in the serving path → a p99 horror story while p50 looks great. Defenses: torch.compile(dynamic=True) (symbolic shapes — one graph covers a shape range, at the cost of some shape-specialized optimizations), or shape bucketing — pad/resize inputs to a small fixed set of shapes and warm each one up. And know the failure of the failure: past a recompile limit, Dynamo falls back to eager silently — your "compiled" service may not be. Log recompile counts as a first-class metric.
  • torch.export + AOTInductor vs ONNX export: torch.export is whole-graph, ahead-of-time capture — no eager fallback, so unsupported constructs fail loudly at export time instead of graph-breaking at runtime — and AOTInductor compiles the exported graph into a Python-free artifact (a shared library plus weights) loadable from C++ servers. When each wins in 2026: ONNX when the target is the cross-vendor ecosystem — onnxruntime execution providers, TensorRT, OpenVINO, NPU toolchains — or the serving stack must stay framework-neutral; torch.export/ AOTInductor when you stay inside the PyTorch world and want to skip opset-conversion breakage (the ops that "sometimes don't export" — NMS again, Ch. 2). Both are maintained first-class paths; Chapter 2's verification ritual applies identically to either — an unverified compiled artifact is still a rumor.
  • ExecuTorch, one paragraph: the mobile/edge leg of the same story — torch.export graphs lowered to a compact runtime whose delegates hand subgraphs to device accelerators (XNNPACK on CPU, Core ML, Qualcomm QNN). It is PyTorch's counterpart to the TFLite pipeline in Phase 04: same export-then-delegate shape, same quantize-and-verify obligations.
  • Practical serving guidance: compile in the serving process at startup (compiled state is process-local unless you configure a compile cache); send warmup requests covering every expected shape bucket before the readiness probe reports ready (Ch. 3's health-vs-readiness distinction earns its keep here); pin or bucket input shapes; and alert on the recompile counter — it is the smoking gun for the Q&A below.

The deep dive behind this chapter is the model-accuracy track's Phase 04 (Dynamo/FX/Inductor internals).

Chapter 10: Dedicated Inference Servers — Triton and the Dynamic-Batching Pattern

The Chapter 3 FastAPI loop is the right learning server and, unmodified, a throughput ceiling: one request per forward pass means the GPU runs at batch-1 — mostly idle — because HTTP has no notion of cross-request batching. The lab's micro batcher taught the concept; the production version needs per-model queues and schedulers, priorities, multiple model instances, versioned reloads, and metrics — undifferentiated heavy lifting, which is exactly what a dedicated inference server is.

  • Dynamic batching from first principles: the server keeps a queue per model; requests accumulate until max_batch_size is reached or max_queue_delay expires, whichever comes first; one forward pass serves the whole batch. The dial: longer delay → fuller batches → higher throughput, but every request pays the queue floor and tail-couples to its batchmates — the same trade Chapter 3 and its Q&A quantified, now as two config numbers. It is also the same queue-vs-utilization math the LLM track pushes to its extreme with continuous batching; the CV version is simpler because forward passes are uniform-length.
  • Triton Inference Server anatomy (NVIDIA's, and the current default answer):
    • Model repository: a directory tree — model-name/version/artifact — as the unit of deployment; models load/unload by polling or explicit API, so rollout is file placement plus a version bump.
    • Per-model config (config.pbtxt): declares inputs/outputs (names, dtypes, shapes), max_batch_size, and the dynamic_batching block (queue delay, preferred batch sizes) — batching is configuration, not code you maintain.
    • Instance groups: N execution instances of a model per GPU (or pinned across GPUs); concurrent instances overlap compute with transfers and let several small models — or several copies of one — keep a large GPU busy.
    • Ensembles and BLS: an ensemble is a server-side DAG of models (decode/preprocess → model → postprocess, with pre/post as Python-backend models) — which moves preprocessing parity (Ch. 4) into the versioned serving graph; BLS (Business Logic Scripting) is Python that calls models programmatically when the pipeline needs real control flow.
    • Multi-framework backends: ONNX Runtime, TensorRT, PyTorch/LibTorch, Python, OpenVINO behind one API — plus standardized Prometheus metrics and perf_analyzer for load-testing the exact config you wrote.
  • When Triton over FastAPI, honestly: it earns its complexity when GPUs must run hot at scale, when several models share hardware, or when you want fleet-standard metrics and deployment machinery. It is overkill for one small CPU model at modest traffic — a FastAPI + onnxruntime container (Labs 02–03) is simpler to debug, deploy, and staff for. Knowing which regime you are in, and saying so, is the production posture this guide keeps preaching.
  • TorchServe, one line of 2026 realism: it moved to limited-maintenance status in 2025 (no new features, minimal fixes) — betting new infrastructure on it now is a risk you would have to defend; the ecosystem consolidated on Triton, KServe, BentoML, and Ray Serve.

Chapter 11: Quantization for CV Serving

Chapter 8 stationed INT8 among the "bigger guns"; here is the mechanism, because you cannot eval-gate what you don't understand.

  • INT8 PTQ (post-training quantization) from first principles: represent FP32 values with 8-bit integers via an affine map \( x \approx s,(q - z) \) — scale \( s \), zero-point \( z \), 256 representable levels. Weights are known at conversion time and quantize directly (symmetric, \( z = 0 \)). Activations depend on inputs, so their ranges come from calibration: run a few hundred representative images through the FP32 model, record per-tensor activation statistics, and choose ranges — min/max, percentile, or entropy/KL-based clipping that sacrifices outliers to buy resolution where the distribution's mass lives. Calibration data quality is a silent accuracy input: calibrate on clean daytime images and deploy on night-time dashcam, and every range is wrong.
  • Per-tensor vs per-channel: one \( (s, z) \) pair for a whole tensor versus one per output channel. Conv weights need per-channel: filter magnitudes routinely span orders of magnitude across channels, and a single tensor-wide scale sized for the largest channel crushes the small ones toward zero. The standard settlement: per-channel weights, per-tensor activations.
  • What breaks, and how it hides: outlier activations stretch a range so the useful mass gets a handful of levels; fine spatial features and small objects degrade first and silently — a classifier's top-1 may drop 0.3 while a detector's AP_small drops 4. The non-negotiable rule: re-run the full evaluation suite — mAP with the size breakdown, per-class numbers, the golden-input set (Ch. 4) — before an INT8 artifact is promoted. A single headline metric is how quantization regressions reach production.
  • QAT (quantization-aware training) in one paragraph: insert fake-quantize ops (quantize→dequantize round trips) into the training graph so the forward pass experiences quantization error while gradients flow through via the straight-through estimator; the weights migrate to quantization-friendly minima. It costs a training run and the pipeline plumbing, and recovers most of what PTQ loses on sensitive models — which is why it sits after PTQ on the ladder, not before.
  • FP16/BF16 is the free first step on GPU: half the memory traffic, tensor-core throughput, and effectively zero accuracy change for CV inference (Ch. 8 already stations it before INT8); BF16 keeps FP32's exponent range, so overflow paranoia disappears. FP8 is the 2026 arrival on Hopper/Blackwell-class hardware — mainstream in LLM serving, reaching CV through TensorRT — same discipline: calibrate the scales, verify on the full suite.
  • The decision ladder: FP16/BF16 → INT8 PTQ → QAT → smaller/distilled model. Each rung costs more engineering; each is gated by the full eval; you step down only when the cheaper rung measurably misses your latency or accuracy bar — write the measurement into the decision record (Ch. 6's promotion-with-evidence ethic).

The deep dive behind this chapter is the model-accuracy track's Phase 03 (quantization and compression at full depth).

Lab Walkthrough Guidance

Order: labs 01→04 as numbered.

  • Lab 01 (ONNX export): export your Phase 05 detector or Phase 03 classifier with dynamic batch; run the verification ritual (multi-shape numerical parity) and the PyTorch-vs-onnxruntime benchmark table; decide and document where NMS lives.
  • Lab 02 (FastAPI service): lifespan-loaded model, bytes-in/structured-JSON-out, validation, version metadata, health endpoint; add the micro dynamic batcher and show its throughput/latency trade with a load test (locust/hey); write the golden-input test (Ch. 4).
  • Lab 03 (Docker): containerize lab 02 with multi-stage build + pinned deps; weights mounted, not baked; measure image size before/after the craft items; compose a two-service stack (API + tracking server).
  • Lab 04 (MLflow): instrument your Phase 03/05 training scripts; run a small sweep, compare in the UI; register the winner, promote staging→production with the eval evidence attached; have the serving container resolve "production" from the registry at startup — the full loop, closed.

Success Criteria

You are ready for Phase 08 when you can, from memory:

  1. Name the export pitfalls and recite the verification ritual; argue where post-processing should live.
  2. Explain why async ≠ throughput for inference, and what dynamic batching trades.
  3. List five preprocessing-parity failure modes and the three defenses.
  4. Apply the Docker craft list and justify weights-outside-images.
  5. Describe tracking/registry/lineage and the promotion-with-evidence rule.
  6. Sketch the three monitoring layers with one CV-specific drift example each.
  7. Order the performance playbook and defend measurement-first.
  8. Explain what a captured graph buys over eager execution, what a guard is, and why variable image sizes make torch.compile a p99 hazard — plus the two defenses.
  9. Sketch Triton's anatomy (repository, per-model config, instance groups, ensembles) and argue both when it beats a FastAPI container and when it's overkill.
  10. Walk INT8 PTQ end to end (scale/zero-point, calibration, per-channel weights), and defend the full-eval-suite gate and the FP16 → PTQ → QAT → smaller-model ladder.

Interview Q&A

Q: Accuracy in production is 8 points below your offline eval. Where do you look? In likelihood order: preprocessing parity (resize library/antialias, BGR/RGB, normalization — run the golden-input test first; it localizes this family instantly); distribution shift (compare live input stats vs training — camera, compression, season); eval set unrepresentativeness (offline set too clean — the gap was always there); version skew (is prod serving the model you think? — check the version-metadata field you wisely put in every response); label-definition drift in how production "correct" is being judged. The ordered list, with the golden-input test as step one, is the answer.

Q: Why is dynamic batching a bigger deal for GPU serving than CPU, and what's its cost? GPU kernels amortize launch overhead and saturate parallelism with batches — batch-8 often costs barely more wall-clock than batch-1, so batching multiplies throughput; CPU inference scales more linearly per item (vectorization helps but less dramatically). The cost: a queuing delay floor (max-wait window) added to every request's latency, plus tail coupling (your request waits on the batch's slowest companion). Tune window/size against the p99 SLO — and below certain traffic, the batch never fills and the feature is pure latency tax; turn it off.

Q: Walk me through deploying a retrained model safely. Registry promotion with attached eval (paired comparison vs current prod on the fixed eval set + the golden inputs); shadow deployment scoring live traffic silently — compare prediction distributions and disagreement rates; canary to ~5% with auto-rollback on system + quality metrics; ramp; keep the old version warm for instant rollback; harvest disagreement cases into the eval set. Each step exists because some team skipped it once — being able to say what each step catches is the senior version of the answer.

Q: Your p99 latency doubled after enabling torch.compile in prod — what happened? Recompilation — almost certainly guard failures on dynamic shapes. Each compiled graph is valid only under its guards (recorded input shapes, dtypes, branched-on values); production CV traffic with variable image sizes or variable batch sizes from an upstream batcher presents shapes the warmup never saw, each one failing guards and triggering a recompile whose seconds of compile time get charged to some unlucky request. The signature: p50 fine (hot shapes hit cached graphs), p99 wrecked (novel shapes pay compilation). Confirm it: enable Dynamo's recompile logging / export the recompile counter, and correlate p99 spikes with first-seen input shapes. Fix it: bucket or pad inputs to a small fixed shape set and send warmup requests through every bucket before the readiness probe passes; or dynamic=True and accept some kernel-quality loss. Then check the failure of the failure: past the recompile limit Dynamo silently falls back to eager — you may now be paying compile stalls without even keeping compiled speed. Secondary suspects worth naming to show breadth: warmup not re-run after a deploy/restart, and CUDA-graph capture interacting badly with variable batch sizes.

Q: Design the batching config for a GPU box serving 3 CV models with a 100 ms SLO. Start from the budget, not the config: 100 ms p99 end-to-end minus network, decode/ preprocess, and post-processing (Ch. 8 says these dominate — measure them) leaves maybe 50–60 ms for queue + forward. Per model, measure the latency-vs-batch-size curve (perf_analyzer), pick the largest batch whose forward time leaves headroom inside that budget, and set max_queue_delay ≈ budget − forward-at-max-batch − headroom — e.g., a model at 15 ms/batch-8 can afford a ~20–30 ms delay cap. Then the part single-model thinking misses: three models contend — kernels serialize and SMs are shared, so each model's tail lengthens under the mix; configure instance groups so small models overlap, and validate against the mixed production-like load, never per-model microbenchmarks. Respect the traffic floor: a low-QPS model's queue never fills, so its delay window is pure latency tax (Ch. 3's honest case) — set its delay near zero or disable batching for it. Finish like an operator: p99 alerting per model, the config in version control, and a load test that replays the real request mix before any of it ships.

References