Phase 20 — Custom Models in Production: Fine-Tune → Package → Deploy

The phase where your model stops being a checkpoint on a laptop and becomes a URL other people's software depends on. Phase 05 taught you to train a LoRA. Phase 09 taught you how a serving engine works inside. Phase 17 taught you the platform lifecycle. This phase is the seam between them — the part every "I fine-tuned a model" story omits: how a config.yaml and a load() method become a container on a GPU behind an autoscaler, why a one-line code edit can turn a 40-second deploy into a 9-minute one, how forty customer-specific models share one GPU, and how to answer "is this actually cheaper than the API?" with arithmetic instead of vibes.

Why this phase exists

There is a specific, common, expensive failure in AI engineering careers: an engineer who can fine-tune a model beautifully and cannot ship it. They produce an adapter_model.safetensors, a notebook of eval numbers, and a Slack message that says "it's ready" — and then someone else spends three weeks turning it into a service, and that someone else gets the staff-engineer title.

The JD for this track says it plainly: "Build scalable, low-latency inference systems for large models… optimize deployment for edge devices, GPUs, and cloud-based platforms." That is not the training half. That is this phase.

Three facts make custom-model deployment its own discipline rather than "just DevOps":

1. The artifact is enormous and the code is tiny. A normal service is megabytes of code. A model service is 20 lines of Python wrapped around 16 GB of weights. Every instinct from web deployment — rebuild fast, deploy often, scale instantly — breaks against a multi-gigabyte artifact and a 40-second CUDA load. Layer ordering, weight caching, and cold starts become first-class design concerns, not ops trivia.
2. A wrong deployment does not throw. Deploy the wrong adapter against the wrong base checkpoint and there is no stack trace, no 500, no alert. The model just gets quietly worse and you find out from a customer. Correctness here is enforced by pinning and validation at build time, because there is no runtime signal to catch.
3. Economics is a design input, not a postmortem. Whether you merge or hot-swap an adapter, whether you scale to zero, whether you quantize — these change the bill by 3–40× and the p99 by 5×. A senior engineer brings the number to the design review. Everyone else brings it to the retro.

This phase makes all three mechanical, then hands you a cookbook of real commands and six end-to-end showcases so the knowledge is usable the same week you learn it.

What to recall from earlier phases

• Phase 00 (cost math). 2N FLOPs per token, forever; the KV-cache is the memory wall. Every sizing and pricing decision in Lab 03 is that arithmetic, applied.
• Phase 05 (LoRA/QLoRA). ΔW = (α/r)·A·B with zero-init B, and merge() costing zero inference overhead. Lab 02 is what happens when you have forty of those deltas and one GPU.
• Phase 06 (quantization). int4 vs fp16 is not only a memory decision — in Lab 03 it moves an 8B replica from an A100 to an L4 and cuts the GPU bill by ~69%.
• Phase 08 (constrained decoding). Showcase 2 pairs a fine-tune with a grammar to take JSON validity from 91% to 100% by construction.
• Phase 09 (serving internals). Continuous batching and PagedAttention are what predict_concurrency is actually feeding. You are now configuring the scheduler you built.
• Phase 16 (evaluation). The eval gate in this phase's CI recipe is your harness.
• Phase 17 (MLOps). The registry, the canary, the rollback. This phase supplies the packaging and hosting layer those gates operate on.

Concept map — from checkpoint to URL

   CUSTOMIZE                    PACKAGE                     OPERATE
   ─────────                    ───────                     ───────
   data (dedupe, entity split)
        │
        ▼
   SFT / LoRA / QLoRA ──┐
   DPO (Phase 07)       │
   distillation         │
        │               │
        ▼               │
   adapter (~16 MiB) ───┼──▶ [merge?] ──▶ artifact + PINNED digest
        │               │        │
        │               │        ▼
        │               │   manifest: deps, GPU SKU, weights, concurrency, secrets
        │               │        │
        │               │        ▼
        │               │   ordered image layers ──▶ registry ──▶ scheduler ──▶ GPU node
        │               │   base ▸ apt ▸ pip ▸ WEIGHTS ▸ code        │
        │               │   (least volatile ─────────▶ most)         ▼
        │               │                                    __init__ → load() → ready
        │               │                                            │
        ▼               ▼                                            ▼
   ADAPTER REGISTRY ──────────────▶ multi-LoRA router ──────▶ OpenAI-compatible route
   (semver + base digest)          (base / dynamic / dedicated)      │
                                                                     ▼
                                            autoscaler ◀── concurrency, queue, cold start
                                                 │
                                                 ▼
                                     eval gate ▸ canary ▸ promote ▸ rollback (P17)
                                                 │
                                                 ▼
                                     $/1M tokens, utilization, p95 — the review slide

The spine of the phase in one sentence: pin everything, order layers by volatility, load once, gate admission, share the base, and know the number.

The labs

Two companion documents carry the hands-on material the labs deliberately cannot (they are offline and deterministic by design):

• DEPLOYMENT-COOKBOOK.md — real, copy-pasteable recipes: a complete Truss (config.yaml + model/model.py with vLLM inside), truss push through canary and rollback, the QLoRA/DPO/merge/quantize training path, vllm serve with multi-LoRA, and the same model deployed six ways (Baseten, Modal, Replicate/Cog, RunPod, SageMaker, BentoML) — plus CI eval gates, the observability table, and a failure-mode index.
• CUSTOMIZATION-SHOWCASES.md — six end-to-end projects with decision tests, commands, deploy configs, before/after tables, and the gotchas: support triage, structured extraction, a warehouse VLM, fine-tuned embeddings, 40 tenants on one GPU, and an 8B→1.5B distillation onto a robot.

Integrated scenario ideas

Do at least one of these end to end; they are what the interview conversation becomes.

1. The full loop, for real. Fine-tune a small model on a task you care about, merge it, package it as a Truss, deploy it with scale-to-zero, and put an eval gate in CI that fails the build on regression. Record the cold start, the p95, and the $/1M.
2. The multi-tenant product. Train three LoRAs on three different tasks, serve them from one vllm serve --enable-lora, and build the gateway that maps an authenticated tenant to an adapter version. Then break it deliberately: point one adapter at a different base revision and observe how silent the failure is.
3. The build-vs-buy memo. Take a real workload, measure tokens/s with vllm bench serve, compute $/1M at your real utilization, compare with two API prices, and write the one-page recommendation — including the case where the answer is "keep using the API."
4. The cold-start hunt. Deploy a 7B model naively, measure the cold start, then cut it in half three different ways (cached weights, smaller image, --enforce-eager) and report which lever paid.
5. The edge split. Distill your model to something that runs on a laptop CPU via GGUF, then build the hybrid loop: fast local call with a bounded timeout and a safe fallback, slow cloud call off the critical path.

Anti-patterns this phase kills

• "It works in my notebook." A notebook has no readiness contract, no admission control, no reproducible environment, and no rollback. It is not a deployment artifact; it is a claim.
• Unpinned everything. torch>=2.4, revision: main, :latest — three different ways to make "rebuild the same image" a lie.
• Heavy work in __init__. Loading weights at import time extends the window where the pod looks dead and breaks every health check assumption.
• Serving before ready. Answering a request while the weights are still loading is either a crash or, worse, a wrong answer with a 200 status.
• Leaking capacity on the error path. One missing finally and every failed request permanently costs you a concurrency slot.
• One GPU per customer. The default architecture for multi-tenant fine-tunes, and it is 40× too expensive.
• Secrets in the image. Immutable, cached, and un-rotatable. A secret in a layer is public to anyone who can pull the image, forever.
• Costing a deployment with GPU price alone. Utilization, not the sticker price, determines $/1M. A GPU that is busy 25% of the month costs 4× per token.
• Fine-tuning a knowledge problem. You get a model that is more confident about facts it still does not have — strictly worse than where you started.
• Deploying without a gate. If no script can fail the build, "we evaluated it" means "someone looked at ten examples."

Deliverables checklist

• All three lab suites green against your lab.py and LAB_MODULE=solution.
• You can explain, from memory, the five build layers in volatility order and what each one invalidates.
• You can trace a request through handle() and name the HTTP status each guard returns (503 / 429 / 400 / 500).
• You can compute an adapter's VRAM from (r, hidden, layers, targets) and say how many fit in 8 GB of spare VRAM.
• You can state the merged-vs-dynamic trade in both directions and pick correctly for a flagship model and for 300 per-customer adapters.
• You can size a deployment: KV pool → total VRAM → GPU SKU → replicas → $/1M.
• You can say when self-hosting is never cheaper on token price, and what the other four reasons to self-host are.
• One real deployment exists: a URL, a two-command curl in a README, an eval gate in CI, and a before/after table with your numbers.

Key takeaways

1. Packaging is a performance decision. Layer volatility order, pinned deps, cached weights — these decide whether your deploy loop is 40 seconds or 9 minutes, and whether your cold start is 30 seconds or 6 minutes.
2. load() once; refuse until ready; release always. Three lines of contract that separate a model server from a script with a web framework around it.
3. An adapter is a delta against exact weights. Bind it to a digest at registration, because the failure mode is silent quality loss, not an exception.
4. Share the base. One resident base plus 16 MiB adapters serves forty tenants on one GPU. The overhead is a few percent; the saving is 97%.
5. Up fast, down slow. Missing capacity costs latency now; dropped capacity costs a cold start later. Asymmetric scaling is the anti-flapping design.
6. Utilization is the hidden term in every cost model. Halve it and your $/1M doubles with no change to the model, the code, or the traffic.
7. Volume cannot rescue a losing unit price. If self-hosted $/1M at full load exceeds the API's, no amount of scale closes the gap — self-hosting then has to win on latency, privacy, control, or a model nobody sells you.
8. The best customization is often not the LLM. A fine-tuned embedding model is cheaper to train, improves every query forever, and halves the generator's prompt.
9. Climb the ladder in order. Prompt → few-shot → RAG → LoRA → full FT → distill. Most "we need a fine-tune" problems are on rung 3.
10. Ship it behind a gate. A model that no script can block from production is a model you do not control.