Warmup Guide — Custom Models in Production: Fine-Tune → Package → Deploy

How to read this. Nothing here assumes you have deployed anything before. Every term — container, layer, image digest, readiness probe, cold start, concurrency target, adapter hot-swap, break-even volume — is built from first principles: what it is, why it exists, how it works underneath, and what it costs you in production. If you have shipped services before, chapters 2 and 3 will be fast; do not skip chapter 6 (cold starts) or chapter 9 (economics), which are where model deployment stops resembling web deployment.

The companion documents are meant to be open beside this one: DEPLOYMENT-COOKBOOK.md for the real commands and CUSTOMIZATION-SHOWCASES.md for six worked projects.

Table of Contents

• Chapter 1: What "Deploying a Custom Model" Actually Means
• Chapter 2: Containers, Images and Layers From First Principles
• Chapter 3: The Packaging Manifest, Field by Field
• Chapter 4: The Model Server Lifecycle — The Contract
• Chapter 5: What Happens Between push and a Live URL
• Chapter 6: Cold Starts, Dissected
• Chapter 7: Serving Custom Weights — Merged vs Adapters
• Chapter 8: Autoscaling Internals
• Chapter 9: The Economics — $/1M, Utilization, Break-Even
• Chapter 10: Choosing the Customization Lever
• Chapter 11: Edge and Hardware-Specific Builds
• Chapter 12: Operating a Custom Model
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: What "Deploying a Custom Model" Actually Means

The thing you have, and the thing you need

After Phase 05 you have a directory. It looks like this:

out/support-lora/
  adapter_config.json         # r=16, lora_alpha=32, target_modules, base model name
  adapter_model.safetensors   # 16–150 MB of A and B matrices
  tokenizer.json

Or, if you merged it, a much bigger directory of .safetensors shards holding the full weights. Either way, what you have is a file that computes a function — and what your product needs is an address that answers questions:

curl -X POST https://models.acme.com/support-triage/predict \
  -H "Authorization: Api-Key …" \
  -d '{"prompt": "card declined twice today"}'

Everything in this phase lives in the gap between those two things. The gap is wider than it looks, and it contains at least eight distinct problems:

Why this is not "just DevOps"

A conventional web service is megabytes of code with a database behind it. It starts in under a second, scales in seconds, and a bad deploy throws exceptions you can see.

A model service is twenty lines of code wrapped around sixteen gigabytes of weights. It takes 30–300 seconds to start. Scaling out means moving those gigabytes again. And — this is the one that catches people — a bad model deploy does not throw. Wrong adapter? No exception. Wrong quantization? No exception. Wrong tokenizer version? Often no exception, just subtly wrong tokens and quietly worse output.

That last property is why so much of this phase is about validation at build time and pinning. When there is no runtime signal, correctness has to be enforced earlier.

The five-step shape

Every platform — Baseten, Modal, Replicate, RunPod, SageMaker, BentoML, or a bare vllm serve on a box you rented — is these five steps with different syntax:

CUSTOMIZE → PACKAGE → DEPLOY → SERVE → OPERATE

Learn the shape and the vendor is a detail you can read in an afternoon. Chapters 2–9 are that shape, in order.

Misconception to kill now: "I'll just wrap it in FastAPI and Docker it." You can, and for a demo you should. But that gets you steps 2 and 3 only, with none of: readiness gating, admission control, batching, cold-start management, autoscaling, canary promotion, adapter routing, or cost accounting. Those are not decorations; they are the difference between a demo and a service. This phase is a list of them.

Chapter 2: Containers, Images and Layers From First Principles

Skip to chapter 3 if you can already explain why docker build prints CACHED.

What a container actually is

A container is not a virtual machine. There is no guest kernel, no emulated hardware. A container is a normal Linux process that the kernel has been told to lie to:

• namespaces — the process gets its own view of the filesystem (mount), process table (pid), network stack (net), hostname (uts), users (user), and IPC. It sees "its" / and "its" PID 1 because the kernel shows it a restricted view.
• cgroups (control groups) — the kernel caps how much CPU, memory, and I/O the process may consume. This is how "2 CPUs, 16 GiB" is enforced.
• a root filesystem — a directory tree the process treats as /, assembled from a container image.

For GPUs there is a fourth piece: the NVIDIA container runtime injects the host's driver libraries and the /dev/nvidia* device nodes into the container. The CUDA driver lives on the host; the CUDA runtime/toolkit lives in your image. This is why a driver too old for your image's CUDA version fails at load() with an unhelpful message — and why "works on my machine" is a real phenomenon even with containers.

What an image is: layers, and why they are ordered

A container image is a stack of read-only filesystem layers plus a JSON manifest. Each layer is a tarball of changes (files added, modified, deleted) relative to the layer below. At runtime a union filesystem (overlayfs) presents the stack as one tree, with a thin writable layer on top.

   ┌──────────────────────────┐  writable container layer (discarded on exit)
   ├──────────────────────────┤
   │ COPY model/  packages/   │  layer 5   ← your code: kilobytes, changes hourly
   ├──────────────────────────┤
   │ fetch weights            │  layer 4   ← 16 GB, changes when you retrain
   ├──────────────────────────┤
   │ pip install …            │  layer 3   ← 2–6 GB (torch!), changes monthly
   ├──────────────────────────┤
   │ apt-get install …        │  layer 2   ← megabytes, changes rarely
   ├──────────────────────────┤
   │ FROM cuda:12.1-python3.11│  layer 1   ← the base image, changes ~never
   └──────────────────────────┘

Every layer has a content digest (sha256:…) computed over its contents. Two consequences follow, and they are the whole of Lab 01's build-plan section:

1. Deduplication. If your registry already holds a layer with that digest, the push and the pull skip it entirely. This is why the second deploy of a 20 GB image can take 30 seconds.
2. Prefix-based cache invalidation. A build step's cache key includes the digest of the layer beneath it. Change layer 3 and layers 4 and 5 must be rebuilt, because their inputs changed. Layers 1 and 2 are untouched.

This is why layer order is a performance decision. Put the weight fetch after the code copy and every one-line edit to model.py re-downloads 16 GB. Put it before, and a code edit rebuilds one tiny layer.

# Lab 01: this is literally the cache policy, as data
LAYER_ORDER = ("base_image", "system_packages", "python_requirements",
               "model_weights", "user_code")     # least volatile ─────▶ most volatile

plan_cache_reuse(old_plan, new_plan)   # longest common PREFIX by digest — not a set

Common misconception: "the cache is a set — unchanged layers are always reused." No. It is a prefix. An unchanged layer 5 whose layer 3 changed is not reused, because layer 5's identity depends on everything below it. That is why bumping one pinned dependency can trigger a multi-gigabyte weight re-fetch, and why it surprises people every single time.

Reproducibility: what "pinned" means and why it is not pedantry

An image is reproducible if building it twice produces functionally identical software. Three things break that:

This is exactly why parse_config in Lab 01 is written to be hostile: it raises on an unpinned requirement and on a moving weight revision. In a normal service, an unpinned dependency causes a build failure you notice. In a model service, it causes a quality change you do not notice. Validation has to move earlier because the feedback signal is gone.

Under the hood detail worth knowing: BuildKit's cache key for a RUN step is a hash of the command string plus the parent layer's digest — not the resulting filesystem. So RUN pip install -r requirements.txt is cached on the text of the command; the content of requirements.txt only participates if it was COPY'd in a prior layer (which is why the COPY requirements.txt → RUN pip install two-step is a standard Dockerfile idiom). Lab 01 models the idealized content-addressed version; the real thing is slightly coarser, and knowing the difference is a good interview detail.

Chapter 3: The Packaging Manifest, Field by Field

A packaging framework (Truss, Cog, BentoML, a Helm chart) exists to turn a short declarative file plus a Python class into a container image and a serving process, so you do not hand-write a Dockerfile and a Deployment YAML for every model. Here is Truss's config.yaml, field by field, with the reasoning.

model_name: support-triage-8b        # identity: logs, dashboards, the model registry
python_version: py311                # selects the BASE IMAGE (layer 1)
requirements:                        # → layer 3. MUST be pinned.
  - vllm==0.6.3
  - transformers==4.45.2
system_packages:                     # → layer 2 (apt). Keep this list short.
  - git
resources:
  accelerator: A10G                  # the GPU SKU — see ch. 9 and Lab 03's pick_gpu
  use_gpu: true                      # DERIVE this; a hand-written lie here is a real bug
  cpu: "4"                           # tokenization, image decode, and the HTTP layer
  memory: 16Gi                       # host RAM ≠ VRAM. Weight loading needs host RAM too
runtime:
  predict_concurrency: 16            # the admission gate (ch. 4, 8)
model_cache:                         # → layer 4, the multi-GB one
  - repo_id: acme/llama-3.1-8b-support-merged
    revision: 8c22764a7e3675c50d4c7c9a4edb474456022b16     # PIN. IT.
    allow_patterns: ["*.safetensors", "*.json", "tokenizer*"]
secrets:
  hf_access_token: null              # a NAME. The value is injected at runtime.
environment_variables:
  VLLM_WORKER_MULTIPROC_METHOD: spawn

Field-by-field, the parts people get wrong:

resources.accelerator. Chosen by arithmetic, not by habit (ch. 9, Lab 03): weights

• KV pool + overhead → the cheapest SKU that fits. The common error is picking an A100 because "it's an LLM" when an int4 model fits an L4 at a third the price.

resources.memory. Host RAM, not VRAM. Loading .safetensors memory-maps the file and stages tensors through host memory; under-provisioning RAM produces an OOM-kill during load() that looks like a mysterious crash-loop with no Python traceback.

runtime.predict_concurrency. How many predict() calls may be in flight per replica. Set it to 1 and your GPU idles between requests (no batching). Set it to 200 and you queue inside the process where you cannot see it, blow past your VRAM budget, or get OOM-killed. For LLMs behind vLLM, 8–32 is the usual window — the engine batches underneath, so this is a ceiling on the queue you accept, not a throughput dial.

model_cache. Fetches weights at build time into the image (or into a platform-managed cache), so a cold start reads them locally instead of pulling from a model hub. This is the single biggest cold-start lever (ch. 6), and it also removes a third-party's uptime from your incident path. allow_patterns matters: many repos ship both .bin and .safetensors of the same weights, and fetching both doubles your image.

secrets. The config declares names. The values live in the platform's secret store and are injected as environment variables or a mounted file at container start.

Why this rule is absolute: image layers are immutable, content-addressed, cached, and replicated. A secret written into a layer is retrievable by anyone who can pull the image, and rotating the secret does not remove it from the layer. There is no undo. parse_config in Lab 01 rejects a secrets entry containing = or : for exactly this reason.

Unknown keys. Lab 01 rejects them, and this is not fussiness. A config with resourcses: (typo) silently falls back to defaults — CPU, one replica — and you discover it when your 8B model is running on a CPU at 2 tokens/second, or does not start at all. Loud beats convenient in deploy configs.

Chapter 4: The Model Server Lifecycle — The Contract

Inside the container, a small server (FastAPI/uvicorn or the framework's own) imports your class and drives it through a fixed lifecycle. Every framework has the same shape:

__init__ vs load() — why they are separate

class Model:
    def __init__(self, **kwargs):
        # Runs at IMPORT time. The health server is not up yet. The GPU may not be
        # usable yet. Any exception here is a crash-loop with a confusing traceback.
        self._config  = kwargs["config"]
        self._data_dir = kwargs["data_dir"]
        self._secrets = kwargs["secrets"]
        self._engine = None            # <- declare, do not build

    def load(self):
        # Runs ONCE, in the container, with the accelerator attached, while the
        # platform holds traffic back. This is where the 40 seconds go.
        self._engine = AsyncLLMEngine.from_engine_args(...)

Put the engine construction in __init__ and three things break: the container looks dead for longer (nothing is answering health checks), a load failure surfaces as an import error rather than a readiness failure, and any framework that imports your module for introspection (schema generation, CLI tooling) now tries to allocate 16 GB of VRAM.

The readiness contract, and why 503 is the correct answer

An orchestrator asks two different questions:

• Liveness: "is this process wedged? should I kill it?"
• Readiness: "may I send this replica traffic?"

They are different, and conflating them is a classic outage. A model that takes 5 minutes to load is alive but not ready. If your liveness probe fails during load, Kubernetes kills the pod, it restarts, and loads for 5 minutes again — an infinite crash-loop caused entirely by probe configuration. (Real fix: a startup probe, or a generous initialDelaySeconds.)

Until load() returns, requests must get 503 Service Unavailable — a retryable status that says "not me, not now". Not 500 (a bug), not a hang (a timeout everywhere upstream), and absolutely not a 200 with garbage.

def handle(self, request):
    if not self.ready:
        self.stats["rejected_not_ready"] += 1
        raise NotReadyError("model is not loaded yet (503)")

Admission control, and the finally that everyone forgets

predict_concurrency is a semaphore. When it is exhausted the correct answer is 429 Too Many Requests — backpressure — because the alternative is an unbounded internal queue, growing latency, and eventually an OOM kill that takes every in-flight request with it. Shedding one request is strictly better than losing fifty.

if not self.try_admit():
    raise OverCapacityError("over capacity (429)")
try:
    ...preprocess → predict → postprocess...
finally:
    self.release()          # ← the whole lesson

Without that finally, every exception permanently costs you one unit of capacity. Capacity decays monotonically with your error rate until someone restarts the pod, and the symptom — "throughput slowly degrades over hours, restart fixes it" — sends teams hunting for memory leaks for days.

Lab 01's test_slot_is_released_when_predict_raises is that bug, frozen.

Streaming, and eager validation

Streaming exists because time-to-first-token (TTFT) is what a user perceives. A 30-token/second model that starts in 200 ms feels faster than a 60-token/second model that starts in 2 seconds, even though the second finishes sooner.

Mechanically: the HTTP response is chunked (Server-Sent Events for OpenAI-compatible routes) and the server yields deltas as the engine produces them.

Two things go wrong:

1. Yield deltas, not cumulative text. If you yield the whole string each time, the client renders "The", "The cat", "The cat sat" — quadratic bytes and, in many clients, visibly duplicated output.
2. Validate eagerly. In Python, a function containing yield returns a generator and runs no code until the first next(). If handle_stream is itself a generator, your readiness check, your admission check, and your input validation all run after the HTTP layer has already committed a 200 response. The fix is a plain function that validates, then returns an inner generator:

def handle_stream(self, request):
    self._check_ready()                 # runs NOW
    if not self.try_admit():
        raise OverCapacityError(...)    # runs NOW

    def _gen():
        try:
            yield from ...
        finally:
            self.release()              # released even if the client disconnects
    return _gen()                       # the generator runs later

The finally inside _gen matters too: a client that disconnects mid-stream causes the generator to be closed (a GeneratorExit at the yield), and without the finally you leak the slot on every abandoned stream — which is most streams in a chat UI.

Speaking OpenAI

Whatever your model is, the cheapest interoperability decision you will ever make is exposing an OpenAI-compatible route:

{"id": "chatcmpl-…", "object": "chat.completion", "created": 1700000000,
 "model": "support-triage-8b",
 "choices": [{"index": 0, "message": {"role": "assistant", "content": "…"},
              "finish_reason": "stop"}],
 "usage": {"prompt_tokens": 31, "completion_tokens": 4, "total_tokens": 35}}

Then the OpenAI SDK, LangChain, LlamaIndex, your Phase 16 eval harness, and every internal tool already pointed at an API work against your model with a one-line base_url change. finish_reason deserves attention: a rising rate of "length" means you are truncating answers, which shows up as quality complaints long before anyone suspects max_tokens.

Chapter 5: What Happens Between push and a Live URL

This is the chapter people never read and then get asked about in interviews. Here is the whole pipeline, with the failure mode at each stage.

 (1) BUNDLE          your laptop tars config.yaml + model/ + packages/ and uploads it
        │            ✗ fails when: a giant file sneaks into the bundle (.git, a
        ▼               checkpoint you forgot); use the framework's ignore file
 (2) RENDER           the manifest becomes a Dockerfile:
        │               FROM <base for python_version + gpu>
        │               RUN apt-get install <system_packages>
        │               RUN pip install <requirements>
        │               RUN <fetch model_cache weights>
        │               COPY model/ packages/
        │            ✗ fails when: a dep needs a system package you did not declare
        ▼
 (3) BUILD            a build service (BuildKit, Kaniko, or the vendor's) executes it,
        │             consulting the layer cache; pushes new layers to a registry
        │            ✗ slow when: a dependency bump invalidated the weight layer
        ▼
 (4) RECORD           a Deployment object is created: id, image digest, config, status
        │             BUILDING → DEPLOYING → ACTIVE (or FAILED)
        ▼
 (5) SCHEDULE         the control plane finds a node with a free GPU of the requested
        │             type; if none exists it provisions one (minutes) or queues
        │            ✗ fails when: the SKU is out of capacity in that region — a real,
        ▼               frequent, and infuriating production condition
 (6) PULL             the node pulls image layers it does not already have
        │            ✗ slow when: the image is 20 GB and the node is cold
        ▼
 (7) START            the container runs; the server imports your module (`__init__`),
        │             then calls `load()`; `/health` reports not-ready throughout
        │            ✗ fails when: CUDA/driver mismatch, missing secret, OOM in load
        ▼
 (8) PROBE            readiness passes; the router adds this replica to the pool
        ▼
 (9) ROUTE            traffic flows:  POST /environments/production/predict
        │             the environment URL is stable; the deployment behind it is versioned
        ▼
(10) SCALE            the autoscaler watches concurrency/queue and adds or removes
                      replicas, each new one repeating (5)–(8)

The two ideas to take from this

The environment is an alias; the deployment is a version. …/environments/production/… points at whichever immutable deployment has been promoted. Promotion is an atomic pointer swap; rollback is swapping it back. Clients never hardcode a deployment id — that is what makes rollback a 2-second operation instead of a redeploy. This maps exactly onto Phase 17 Lab 03's blue_green_swap and rollback.

Development vs published deployments. A development deployment is mutable and supports live reload (truss watch syncs your model/ directory into the running container and restarts the server) — a 5-second edit loop instead of a 5-minute one. A published deployment is immutable: pushing again creates a new one. Never point production at a development deployment; the whole point of the distinction is that one of them can change under you.

Chapter 6: Cold Starts, Dissected

A cold start is the wall-clock time from "a replica is requested" to "that replica serves its first token". It is the single most important number in serverless GPU serving and the reason scale-to-zero is not free.

  cold start = node acquisition + image pull + weight fetch + model load + engine build
                    (0–300 s)      (10–180 s)    (5–400 s)     (5–120 s)   (0–3600 s)

Phase by phase, with the mechanism and the lever:

1. Node acquisition. If the cluster already has a warm node with a free GPU: ~0. If it must provision one from the cloud: 1–5 minutes, and it can fail — GPU capacity is genuinely scarce and region-specific. Lever: keep a warm pool (min_replicas), or reserve capacity.

2. Image pull. The node fetches layers it lacks. A CUDA + PyTorch + vLLM image is easily 8–20 GB. Levers: slim base images, no -devel image in production, prune build tools, and — the big one — pull fewer new layers by not invalidating them (chapter 2).

3. Weight fetch. 16 GB from a public model hub over the internet at ~300 MB/s is about 55 seconds, if nothing is throttling you. From a local cache or a network volume at 4 GB/s it is 4 seconds. Lever: bake weights into the image or a platform-managed cache. This is the single biggest one, and it is one config block.

# Lab 01 models exactly this branch:
cold_start_breakdown(image_gb=4, weights_gb=16)                       # ≈ 72 s
cold_start_breakdown(image_gb=4, weights_gb=16, weights_cached=True)  # ≈ 30 s

4. Model load. Reading .safetensors into host memory and copying to VRAM, building the tokenizer, allocating the KV pool. 5–60 s for a 7–8B model. Mechanism worth knowing: .safetensors is a flat, memory-mappable format with a JSON header — it exists specifically because the older pickle-based .bin format was both slower and a remote code execution vector. Levers: safetensors (not .bin), fewer/larger shards, and loading directly to the device where the framework supports it.

5. Engine build. vLLM captures CUDA graphs (recording the kernel launch sequence once so subsequent decodes skip launch overhead) — tens of seconds, disabled with --enforce-eager at the cost of slower steady-state decode. TensorRT-LLM compiles an engine, which takes minutes to hours and is specific to the GPU model, batch size, and sequence length. Lever: build the engine in CI, store the artifact, and load it at boot. Never compile on a cold start.

The trade you are actually making

The decision is not technical, it is product: is a user waiting? Interactive chat → keep one warm. Nightly batch scoring → scale to zero and enjoy the bill. Lab 03's test_warm_pool_trades_cost_for_latency is precisely this trade, made numeric.

Common misconception: "scale-to-zero drops requests." It does not — a correct platform queues them while the replica boots (Knative literally routes them through an "activator" component that buffers until a pod is ready). The cost is latency, not loss. Lab 03 asserts this as an invariant: arrivals == served + queued, always.

Chapter 7: Serving Custom Weights — Merged vs Adapters

You have a LoRA adapter. There are exactly two ways to serve it, and choosing correctly is the highest-leverage architectural decision in this phase.

Recap of the mechanism (Phase 05, in one paragraph)

LoRA freezes the base weight \(W \in \mathbb{R}^{d\times k}\) and learns a low-rank update:

$$\Delta W = \frac{\alpha}{r},A B,\qquad A \in \mathbb{R}^{d\times r},; B \in \mathbb{R}^{r\times k},; r \ll \min(d,k)$$

with B zero-initialized so \(\Delta W = 0\) at step 0. At inference you can either compute \(xW + \frac{\alpha}{r}(xA)B\) (keep them separate) or precompute \(W' = W + \Delta W\) (merge). The results are algebraically identical.

How big is an adapter, really

Each targeted \((d \times d)\) projection gains \(2rd\) parameters:

$$\text{params} = n_{\text{layers}} \cdot n_{\text{targets}} \cdot 2 \cdot r \cdot d$$

For Llama-3-8B (\(n_{\text{layers}}=32\), \(d=4096\)), r=16 on q,v, fp16:

$$32 \cdot 2 \cdot 2 \cdot 16 \cdot 4096 \cdot 2\ \text{bytes} = 16\ \text{MiB}$$

16 MiB against a 16 GB base — 0.1%. Internalize that ratio; it is the entire economic argument for multi-tenant serving, and it is adapter_vram_mb in Lab 02.

Option A: merge

merged = AutoPeftModelForCausalLM.from_pretrained("out/support-lora").merge_and_unload()

• ✅ Zero per-token overhead — it is an ordinary model afterwards.
• ✅ Works with every optimization: quantization, TensorRT compilation, anything.
• ❌ The artifact is the full model (16 GB), so each one needs its own replica.
• ❌ No hot-swap: changing behavior means a redeploy.

Use when: one flagship model, high QPS, latency-critical.

Option B: keep it dynamic (multi-LoRA)

One base resident in VRAM; adapters loaded into the spare VRAM and applied per request.

vllm serve BASE --enable-lora --lora-modules support=/a/support sql=/a/sql \
  --max-loras 8 --max-cpu-loras 64

curl … -d '{"model": "support", "messages": [...]}'    # per-request adapter selection

How it works under the hood. The naive implementation would batch only requests sharing an adapter, destroying throughput. The real one uses segmented gather matrix multiplication (SGMV) kernels — from the Punica and S-LoRA work — that let a single batched GEMM apply different A/B pairs to different rows of the batch. Requests for eight different adapters share one forward pass over the base weights; only the small low-rank multiply is per-row. That is why the overhead is a few percent rather than a few hundred.

There is also a two-tier cache: --max-loras bounds GPU-resident adapters, --max-cpu-loras bounds a host-RAM spill tier. When the working set exceeds the GPU tier, adapters are swapped over PCIe — fast (a 16 MiB copy is ~2 ms at 8 GB/s) but not free, and thrashing is the failure mode to watch.

• ✅ N tenants on one GPU; marginal tenant costs 16 MiB.
• ✅ Hot-swap in milliseconds; onboarding is a registry write.
• ❌ 3–8% per-token overhead, growing with the number of distinct adapters in a batch.
• ❌ All adapters must share one base checkpoint.

Use when: many models, low-to-moderate QPS each — the multi-tenant product shape.

The digest binding — the bug with no exception

An adapter is a delta against specific weights. Apply support-bot@1.0.0, trained against base commit 8c22764a, to base commit f39ab210 — same model name, different checkpoint — and:

• nothing raises;
• the shapes match, so no error is possible;
• the output is fluent, plausible, and worse;
• you discover it from an eval regression, or a customer.

adapter_config.json records base_model_name_or_path, which is a name, not a digest. That is precisely why this class of bug ships. The fix is a registry that binds adapters to a base digest and refuses the mismatch at registration:

if adapter.base_digest != self.base_digest:
    raise ValueError(f"{adapter.key} was trained against {adapter.base_digest!r}, "
                     f"registry serves {self.base_digest!r} — silent garbage")

Lab 02's test_registry_rejects_wrong_base_digest is that guard.

Sizing the adapter pool

The VRAM budget for adapters is what is left over:

adapter_pool = total_VRAM − base_weights − KV_cache_pool − runtime_overhead

On an 80 GB H100 serving an fp16 8B: 80 − 16 − (KV pool, say 50) − 2 ≈ 12 GB — room for ~750 r=16 adapters, far more than --max-loras will let you keep resident anyway. The binding constraint is usually the resident limit and the thrash rate, not raw capacity. Watch the cache hit rate; when p99 diverges from p50 as you add tenants, you are thrashing, and the fixes are: a bigger resident pool, a CPU spill tier, or tenant-affinity routing (hash the tenant to a replica so their adapter stays hot).

Chapter 8: Autoscaling Internals

Why concurrency, not CPU

Classic autoscaling watches CPU utilization. That is useless for GPU inference: the GPU is either ~100% busy in a kernel or idle, and host CPU tells you nothing. The right signal is concurrency — how many requests are in flight — or its close cousin, queue depth.

desired_replicas = ceil(in_flight / target_concurrency)      clamped to [min, max]

target_concurrency is "how many simultaneous requests should one replica handle". For LLM serving it interacts with continuous batching (Phase 09): a higher target means bigger batches, better GPU utilization, better tokens/s aggregate — and worse latency per request, plus more KV-cache pressure. It is the throughput/latency dial.

Observe the backlog, not the throughput

A subtle and important design point, and Lab 03 enforces it:

desired = self.desired_replicas(backlog)      # backlog = queue + arrivals
#                              ^^^^^^^ NOT `served`

If you scale on requests served, then when you are badly under-provisioned you serve few requests, so the signal falls, so you do not scale up. The metric goes blind exactly when you need it. Scale on demand (arrivals + queue), never on completions.

Up fast, down slow

if desired > total_replicas:            # scale UP: immediately
    launch(desired - total_replicas)
elif desired < ready:                   # scale DOWN: only after a delay
    idle_s += tick_s
    if idle_s >= scale_down_delay_s:
        ready = desired

The asymmetry is deliberate and is the anti-flapping design. Capacity you lack costs latency right now. Capacity you drop costs a full cold start if the load returns in thirty seconds. Since cold starts are 30 s – 6 min and scale-up is nearly free, the correct policy is aggressive up, conservative down.

Without the delay you get flapping: traffic dips → scale down → traffic returns → cold start → queue → scale up → dips → repeat, with every cycle paying a cold start of queued latency. The classic misconfiguration is a scale-down delay shorter than the cold start, which guarantees the pathology.

Real autoscalers (Knative, HPA) additionally smooth the metric over a stable window (30–60 s) and apply a stabilization window before scaling down. That makes them slower and more overshooting than Lab 03's idealized scaler — an extension worth building.

Scale-to-zero, and the queue that saves you

With min_replicas: 0, an idle deployment drops to zero replicas and costs nothing. The next request finds zero capacity — and is queued, not rejected, while a replica boots. (In Knative this is literally a separate component, the activator, that buffers requests and holds the connection open.)

So the price of scale-to-zero is a p99 spike of one full cold start on the first request after idle. Lab 03's conservation invariant — arrivals == served + queued — is the formal statement that nothing is lost, only delayed.

You are billed while booting

"billable_replicas": self.ready + self.starting

You pay from the moment a replica starts pulling an image, not from the moment it serves. A 4-minute cold start on a $3.20/hr H100 is ~$0.21 of pure idle — trivial once, and very much not trivial when a flapping autoscaler does it two hundred times a day.

Chapter 9: The Economics — $/1M, Utilization, Break-Even

Sizing: three terms, not one

VRAM_needed = weights + KV_cache_pool + runtime_overhead

• weights = params × bytes_per_param. 8B at fp16 → ~15 GiB; at int4 → ~3.7 GiB.
• KV pool = 2 · n_layers · n_kv_heads · head_dim · bytes · total_tokens. Note n_kv_heads, not n_heads — GQA/MQA divide this by the grouping factor, which is what makes long context affordable. Llama-3-8B: 128 KiB per token, so 100k cached tokens (say 50 concurrent sequences at 2k each) is 12.2 GiB.
• overhead = CUDA context, activations, fragmentation, the framework. 1–3 GiB. This is the term people omit right before they OOM at "98% free".

Worked example, straight out of Lab 03:

Quantization is a procurement decision. Same model, same task, −69% on the GPU bill, before you write a line of optimization code. (Then re-run your evals — a 4-bit model that lost 3 points on your task is a regression you paid for, not a saving.)

The unit cost

$$\frac{$}{1\text{M tokens}} = \frac{\text{GPU }$/\text{hr} \div 3600}{\text{tokens/s} \times \text{utilization}} \times 10^{6}$$

The term that decides everything is utilization — the fraction of rented time you are actually serving. You rent 730 hours a month; you serve traffic for maybe 250.

Nothing about the model changed. The bill quadrupled. Every serious cost optimization in inference is a utilization play: bigger batches, multi-tenancy (chapter 7), moving batch jobs onto the same replicas at night, scale-to-zero for spiky workloads.

Break-even, and the answer nobody expects

Naive framing: "at what volume does a dedicated GPU beat a per-token API?" Answer:

$$T^{*} = \frac{\text{GPU }$/\text{hr} \times 730}{\text{API }$/1\text{M}} \times 10^{6}$$

But a rented GPU has a capacity ceiling. If your self-hosted $/1M at full load already exceeds the API's price, then buying a second GPU raises cost and capacity in the same proportion — the lines are parallel and never cross. Volume cannot rescue a losing unit price.

feasible = self_host_usd_per_1m <= api_usd_per_1m
break_even_tokens_month = threshold if feasible else None

Two concrete cases from Lab 03's worked example, H100 @ $3.20/hr, 2,000 tok/s at 60% utilization ($0.74/1M):

• vs a hosted 8B at $0.60/1M → infeasible. Self-hosting never wins on token price. Someone else is running that model at higher utilization than you ever will, and they are passing some of it on.
• vs a frontier model at $5.00/1M → break-even at ~0.47B tokens/month. Above that, self-host.

This is the honest, senior answer, and it is why the four non-price reasons to self-host matter so much:

1. Latency — no network round trip, no shared-tenant queue; 400 ms instead of 2.5 s.
2. Privacy/residency — the data never leaves your VPC. Often not negotiable, and cost_compare does not get a vote.
3. Control — the model does not change under you on a Tuesday.
4. Capability — your fine-tuned model does something no API sells: your taxonomy, your domain, your robot's camera.

And the costs the arithmetic ignores: engineering time, on-call, storage, egress, and the eval/observability stack. At small scale those dominate everything above, which is why "start with the API" is usually correct and "we self-host on principle" usually is not.

Chapter 10: Choosing the Customization Lever

Deployment mechanics are useless if you customized the wrong way. The ladder, in order of cost — climb only as far as you must:

The three questions that resolve it in a meeting:

1. Can a careful human do the task from the prompt alone? No → the model lacks knowledge → RAG, not fine-tuning.
2. Does it know the answer but present it wrong? Yes → behavior → fine-tune.
3. Is quality fine and the problem is the bill or the p95? → distill or quantize something you already trust.

LoRA hyperparameters, decoded

Dataset beats hyperparameters, every time. 500 clean, deduplicated, correctly-split examples beat 100k scraped ones for a narrow task. The two dataset bugs that ruin projects: near-duplicates (inflate your eval, teach nothing) and entity leakage (splitting by row instead of by customer/session, so your eval score is a memorization score).

Chapter 11: Edge and Hardware-Specific Builds

When the model runs next to an actuator, network latency stops being a tuning parameter and becomes a safety property (Phase 15). Three runtimes matter:

llama.cpp / GGUF. A single-file format with the weights, the tokenizer, and the metadata together, plus a C++ runtime with no Python and no CUDA requirement. Runs on CPU, Apple Silicon, and Jetson. Q4_K_M (~4.5 bits/weight, mixed precision by tensor importance) is the standard quality/size point. llama-server exposes an OpenAI-compatible API, so your client code does not change.

ONNX Runtime. ONNX is a portable computation-graph format; ORT executes it with pluggable execution providers (CPU, CUDA, TensorRT, CoreML, DirectML). The portable choice when the target hardware is heterogeneous or unknown.

TensorRT-LLM. Maximum NVIDIA throughput, at the cost of an ahead-of-time compile that fuses kernels, selects tactics, and bakes in max_batch_size and max_input_len. Two consequences: the build takes minutes to hours, and the engine is specific to the GPU model. An engine compiled for an A100 does not run on an H100. Build it in CI, version the artifact, load it at boot — never compile during a cold start. (This is engine_build_s in Lab 01's cold-start model.)

Realistic targets:

The hybrid loop is the architecture, not a compromise: a small, fast, bounded-latency model on the robot for the control path, with a hard timeout and a tested safe fallback; the big model in the cloud for planning that tolerates a round trip.

Chapter 12: Operating a Custom Model

The gate

Never promote a model no script has judged. The gate lives in CI, runs your Phase 16 eval suite against a staging deployment, and exits non-zero on regression:

GATES = {
    "exact_match":   ("min", champion["exact_match"] - 0.01),   # no-regression
    "json_validity": ("min", 0.99),                             # absolute floor
    "p95_latency_ms":("max", 1500),
    "usd_per_1k":    ("max", 0.05),
}

Note both kinds of bar: absolute floors (validity, safety) and no-regression vs the current champion. "Passes thresholds" is weaker than "beats the incumbent", and Phase 17 Lab 03 implements exactly this EvalGate.

What to measure

Uptime dashboards lie about ML systems — the service returns 200s while the model gets quietly worse. The metrics that actually detect that:

Log the deployment id on every request. Without it, "quality dropped last Tuesday" is unanswerable; with it, it is a one-line group-by.

Multi-tenant security

Multi-LoRA puts forty customers' adapters in one process. Three rules:

1. Resolve the adapter from the authenticated session, never from a request field. A client-supplied adapter name is a cross-tenant data leak with extra steps.
2. Pin each tenant to an adapter version. "Latest" means a customer's assistant changes behavior because someone else's retrain finished.
3. Test the isolation path explicitly, the same way you would test authz.

And a property of weights that has no analogue in ordinary services: models memorize their training data. A PII record in a fine-tune set can be extracted from the weights later. Review training data for PII the way you would review a public API response, because in a real sense that is what it becomes.

Lab Walkthrough Guidance

Lab 01 — Truss-style packaging & the serving contract

Work top-to-bottom; the file order is the dependency order.

1. parse_config is the longest function and it is mostly raise ValueError. Write the validations one at a time and run the matching test after each. The two that teach the most: the unknown-key check (a typo in a deploy config must be loud) and the moving-ref check (revision: main is a reproducibility bug).
2. _is_pinned — remember the VCS case: a URL/git+ requirement carrying @ is pinned.
3. build_plan — sort the lists. test_build_plan_ignores_declaration_order_of_requirements fails otherwise, and it is right to: reordering lines in YAML must not bust a cache.
4. plan_cache_reuse — a zip + early break. If you are tempted to use a set intersection, re-read chapter 2.
5. ModelServer — do try_admit/release before handle. Then write handle's try/finally first and fill in the middle; that ordering makes the leak impossible to forget.
6. handle_stream — the trap is writing it as a generator function. It must be a plain function that validates and returns an inner generator.

Common stumbles: forgetting to reject bools where an int is required (isinstance(True, int) is True in Python); returning a set from build_plan instead of an ordered tuple; and incrementing requests before the model actually succeeded.

Lab 02 — Multi-LoRA adapter registry & hot-swap router

1. parse_version first. Everything depends on it, and the semver-vs-string test is the one to internalize.
2. adapter_vram_mb — check yourself against 16 MiB for (16, 4096, 32, 2) before moving on.
3. AdapterRegistry.register — the digest check is the point of the lab. Write it before the duplicate check so you internalize the ordering of error messages.
4. AdapterCache.load — use OrderedDict.move_to_end for a hit and popitem(last=False) for eviction. The while loop (not an if) matters: one big adapter can evict several small ones.
5. MultiLoRARouter.route — the merged branch must not touch the cache. If your test_route_merged_adapter_is_dedicated_with_zero_overhead sees cache.used_mb > 0, you have modeled the trade wrong.
6. fleet_plan — dedicated_gpus is len(adapters), not a bin-pack. That is the point: a merged model cannot share.

Lab 03 — Autoscaling, cold starts & deployment economics

1. Sizing functions first — they are pure arithmetic and give you quick greens.
2. Autoscaler.step — write it in the documented order and keep the order. Ageing booting replicas before computing capacity is what makes the cold-start test pass.
3. The scale-down branch is elif desired < self.ready, not else. If you use else, the "equal" case starts accumulating idle time and you scale down under steady load.
4. summarize — final_queue is the last record's queue, not the sum.
5. break_even_analysis — resist returning a number when feasible is False. Returning a break-even that can never be reached is worse than returning None; somebody will put it on a slide.

Debugging tip for all three labs: python solution.py prints a full worked example. Diff your output against it before you go line-hunting.

Success Criteria

You have finished this phase when you can:

• Explain what a container is (namespaces + cgroups + an image), what an image layer is, and why the layer cache is prefix-based rather than set-based.
• List the five build layers in volatility order and say what each invalidates.
• Name four things that must be pinned and the failure each one prevents.
• Explain why __init__ and load() are separate, and what breaks if you merge them.
• Map each request guard to its HTTP status: 503 not-ready, 429 over-capacity, 400 malformed, 500 model failure — and say why 503 (not 500) is correct before ready.
• Explain the concurrency-slot leak and write the three lines that prevent it.
• Explain why handle_stream must validate eagerly.
• Walk the ten stages from push to a live URL and name a failure mode at each.
• Decompose a cold start into its phases and name the biggest lever for each.
• Compute an adapter's VRAM from (r, d, layers, targets) and state the 0.1% ratio.
• Explain merged vs dynamic in both directions and pick correctly for two scenarios.
• Explain how one batched forward pass serves eight different adapters (SGMV).
• Explain the digest-binding bug and why there is no runtime signal for it.
• Explain why an autoscaler observes backlog rather than throughput.
• Explain up-fast/down-slow and describe flapping.
• Size a deployment end to end: KV pool → VRAM → SKU → replicas → $/1M.
• State when self-hosting is never cheaper on token price, and the four other reasons to do it anyway.
• Place a problem on the customization ladder and defend the rung.
• Get all three lab suites green against lab.py and LAB_MODULE=solution.

Interview Q&A

Q: Walk me through what happens when you deploy a fine-tuned model. Bundle → render a Dockerfile from the manifest → build with a layer cache → push layers to a registry → create a deployment record → the scheduler finds a node with the right GPU → the node pulls missing layers → the container starts, __init__ runs (cheap), load() pulls weights into VRAM while /health reports not-ready → readiness passes → the router adds the replica → the autoscaler manages replica count from concurrency. The environment URL is a stable alias over an immutable, versioned deployment, which is what makes promotion and rollback atomic pointer swaps.

Q: Why does the weight layer come before the code layer? Layer caching is prefix-based: a layer's identity depends on everything beneath it. Code changes hourly and weighs kilobytes; weights change per retrain and weigh gigabytes. Put weights last and every one-line edit re-fetches 16 GB. The corollary is the thing that bites people: bumping one pinned dependency invalidates the weight layer too, turning a 40-second deploy into a 9-minute one.

Q: A request arrives while your model is still loading. What do you return? 503, and it must be the readiness path, not the liveness path. 503 is retryable and tells the load balancer to route elsewhere. If liveness fails during load, the orchestrator kills the pod and you get an infinite crash-loop caused purely by probe configuration — use a startup probe or a generous initial delay.

Q: Your throughput slowly degrades over hours and a restart fixes it. What is it? Almost certainly a leaked concurrency slot: an error path that admits a request and never releases it, so capacity decays monotonically with your error rate. The fix is try/finally around the request body — including inside the streaming generator, so an abandoned stream also releases.

Q: How do you serve 200 customer-specific fine-tunes without 200 GPUs? Multi-LoRA. Keep one base resident and swap adapters — 16 MiB each at r=16 for an 8B, 0.1% of the base. vLLM applies different adapters to different rows of the same batch with SGMV/punica kernels, so throughput survives; the cost is 3–8% per-token overhead and the constraint that every adapter shares one base checkpoint. Bound the resident set with --max-loras, spill to host RAM with --max-cpu-loras, and watch the cache hit rate — thrash shows up as p99 diverging from p50.

Q: When do you merge an adapter instead? One flagship model, high QPS, latency-critical. Merging folds ΔW into W, so there is zero per-token overhead and every downstream optimization (quantization, TRT compilation) applies. The cost is a full-size artifact, its own replica, and no hot-swap. Density and peak throughput are different objectives; a mature platform serves both architectures.

Q: You apply a LoRA to the wrong base checkpoint. What happens? Nothing visible. Shapes match, no exception, no 5xx — the output is fluent and quietly worse. You find out from an eval or a customer. Since there is no runtime signal, you must catch it at registration by binding each adapter to the base's content digest. adapter_config.json only records a base name, which is exactly why this ships.

Q: How do you size the GPU for a 70B model at 32k context? Three terms. Weights: 70B × 2 bytes ≈ 130 GiB at fp16 → already more than one 80 GB card, so either shard with tensor parallelism or quantize (int4 → ~33 GiB). KV: 2·L·H_kv·d_head·bytes·tokens per sequence — compute it, do not guess, and remember GQA divides it. Overhead: 1–3 GiB. Then pick the cheapest SKU that fits, and if nothing does, that is the signal to quantize or shard, not to buy an H200 reflexively.

Q: Why does your autoscaler scale up faster than it scales down? Asymmetric costs. Missing capacity costs latency right now; dropped capacity costs a full cold start (30 s – 6 min) if load returns. So: scale up immediately, scale down only after a sustained idle window. If the scale-down delay is shorter than the cold start you get flapping, which pays a cold start on every cycle.

Q: What does scale-to-zero actually cost your users? One full cold start on the first request after idle — the requests are queued, not dropped (a platform like Knative buffers them in an activator until a pod is ready). So the price is a p99 spike, not lost traffic. Whether that is acceptable is a product question: interactive chat, no; nightly batch scoring, absolutely.

Q: At what volume should we stop calling the API and self-host? The naive answer is GPU $/month ÷ API $/1M. The real answer is a feasibility test first: compute your self-hosted $/1M at your actual utilization. If it already exceeds the API price, no volume ever wins, because a second GPU adds cost and capacity in the same proportion. Against a cheap hosted 8B, self-hosting usually loses on price; against a frontier model at $5/1M, break-even is around half a billion tokens a month. And the real reasons to self-host are often not price at all: latency, data residency, control over model changes, and a capability nobody sells you.

Q: Your $/1M doubled with no code change. What happened? Utilization halved. $/1M is inversely proportional to it, so the GPU sitting idle more of the month is the entire explanation. Traffic dropped, a warm pool got bigger, or a flapping autoscaler is buying replicas that never serve.

Q: Should we fine-tune? Only if the problem is behavior. If a careful human could not do the task from the prompt alone, the model lacks knowledge and you want RAG — fine-tuning a knowledge problem produces a model that is more confident about facts it still does not have, which is worse than where you started. If quality is fine and the problem is the bill or the p95, distill or quantize something you already trust. And the highest-ROI customization in most RAG systems is not the LLM at all — it is a fine-tuned embedding model: cheaper to train, improves every query forever, and halves the generator's prompt.

Q: How do you know a new model version is safe to promote? An eval gate in CI with two kinds of bar — absolute floors (schema validity, safety) and no-regression against the current champion — run against a staging deployment, exiting non-zero on failure. Then a canary: shift a small traffic slice, watch health, and abort by freezing at the last good percentage. Then promote, which is a pointer swap, so rollback is the same swap in reverse.

References

Containers and images

• OCI Image Specification — https://github.com/opencontainers/image-spec
• Docker/BuildKit build cache — https://docs.docker.com/build/cache/
• NVIDIA Container Toolkit — https://docs.nvidia.com/datacenter/cloud-native/

Packaging and hosting

• Truss — https://truss.baseten.co ; Baseten docs — https://docs.baseten.co
• Cog — https://cog.run/ ; BentoML — https://docs.bentoml.com
• Modal — https://modal.com/docs/guide ; RunPod — https://docs.runpod.io/serverless/overview
• SageMaker LMI — https://docs.aws.amazon.com/sagemaker/latest/dg/large-model-inference-container-docs.html
• Knative autoscaling (concurrency targets, scale-to-zero, the activator) — https://knative.dev/docs/serving/autoscaling/
• Kubernetes probes (liveness/readiness/startup) — https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

Serving engines

• Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention" (SOSP 2023) — https://arxiv.org/abs/2309.06180
• vLLM docs (LoRA adapters, engine args) — https://docs.vllm.ai
• Chen et al., "Punica: Multi-Tenant LoRA Serving" (2023) — https://arxiv.org/abs/2310.18547
• Sheng et al., "S-LoRA: Serving Thousands of Concurrent LoRA Adapters" (2023) — https://arxiv.org/abs/2311.03285
• TensorRT-LLM — https://nvidia.github.io/TensorRT-LLM/ ; llama.cpp — https://github.com/ggerganov/llama.cpp ; ONNX Runtime — https://onnxruntime.ai/docs/
• safetensors (format and rationale) — https://github.com/huggingface/safetensors

Customization

• Hu et al., "LoRA" (2021) — https://arxiv.org/abs/2106.09685
• Dettmers et al., "QLoRA" (2023) — https://arxiv.org/abs/2305.14314
• Rafailov et al., "Direct Preference Optimization" (2023) — https://arxiv.org/abs/2305.18290
• Hinton et al., "Distilling the Knowledge in a Neural Network" (2015) — https://arxiv.org/abs/1503.02531
• Lin et al., "AWQ" (2023) — https://arxiv.org/abs/2306.00978
• HF peft — https://huggingface.co/docs/peft ; trl — https://huggingface.co/docs/trl

Operations

• Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NeurIPS 2015)
• Google SRE Workbook, "Canarying Releases" — https://sre.google/workbook/canarying-releases/
• Carlini et al., "Extracting Training Data from Large Language Models" (2021) — https://arxiv.org/abs/2012.07805