Hitchhiker's Guide — Custom Models in Production

The compressed practitioner tour. Read WARMUP.md for the why; this is what you keep in your head. Commands live in DEPLOYMENT-COOKBOOK.md.

The 30-second mental model

A deployment is a manifest plus a load() method, compiled into ordered image layers, scheduled onto a GPU, and gated by a readiness probe and an admission counter.

Five steps, always the same, on every platform:

CUSTOMIZE → PACKAGE → DEPLOY → SERVE → OPERATE
 (LoRA)     (layers)  (sched)  (batch) (scale, gate, $)

Six sentences that carry most of the phase:

1. Layers are prefix-cached — put weights before code or every edit re-downloads 16 GB.
2. __init__ is cheap, load() is once — and until it returns, the answer is 503.
3. Every request path needs finally: release() — or capacity decays until restart.
4. An adapter is 0.1% of the base — so one GPU serves forty tenants, not one.
5. Scale up now, down slowly — because the way back up costs a cold start.
6. $/1M is inversely proportional to utilization — which is the term everyone drops.

The numbers to tattoo on your arm

The commands to keep in muscle memory

# ── package & ship ────────────────────────────────────────────────────────────
truss init my-model                       # scaffold config.yaml + model/model.py
truss push                                # development deployment (live-reloadable)
truss push --publish                      # a new immutable production deployment
truss watch                               # hot-reload model/ into the dev deployment
truss predict -d '{"prompt":"hi"}'        # invoke
truss logs                                # build + runtime logs

# ── serve it yourself ─────────────────────────────────────────────────────────
vllm serve acme/support-merged --dtype bfloat16 --max-model-len 8192 \
  --gpu-memory-utilization 0.90 --port 8000

# multi-LoRA: one base, many adapters, per-request `model` selection
vllm serve BASE --enable-lora \
  --lora-modules a=/adapters/a b=/adapters/b \
  --max-lora-rank 32 --max-loras 8 --max-cpu-loras 64

# measure — never assume — the tokens/s that feeds your cost model
vllm bench serve --model M --dataset-name random --num-prompts 200 \
  --random-input-len 512 --random-output-len 128

# ── customize ─────────────────────────────────────────────────────────────────
# QLoRA SFT (trl) → merge → quantize → GGUF for the edge
python train_lora.py                      # r=16, alpha=32, lr=1e-4, 2 epochs
python -c "from peft import AutoPeftModelForCausalLM as M; \
  M.from_pretrained('out/lora').merge_and_unload().save_pretrained('out/merged')"
python convert_hf_to_gguf.py out/merged --outfile m-f16.gguf --outtype f16
./llama-quantize m-f16.gguf m-Q4_K_M.gguf Q4_K_M

# ── the client, unchanged ─────────────────────────────────────────────────────
curl -X POST "https://model-${ID}.api.baseten.co/environments/production/predict" \
  -H "Authorization: Api-Key ${BASETEN_API_KEY}" -d '{"prompt":"…"}'

# Your custom model, through the OpenAI SDK — the one-line integration
client = OpenAI(api_key=KEY, base_url=YOUR_ENDPOINT_V1)
client.chat.completions.create(model="support-triage-8b", messages=[...])

The tool cheat-table (what each is for)

War stories

The 9-minute deploy. A team bumped transformers by one patch version to pick up a bugfix. Deploy time went from 40 seconds to 9 minutes, every time, for a week, before anyone connected the two. The dependency layer sits above the weight layer, so invalidating it re-fetched 16 GB on every build. The fix was one line of layer ordering. The lesson is that layer order is a performance decision, and plan_cache_reuse in Lab 01 is the model of it.

The adapter that broke nothing. A base model was re-uploaded with a fixed tokenizer config; the repo's main moved. Nothing was pinned. Deploys kept succeeding, health checks stayed green, and the eval suite — which only ran weekly — showed a 4-point drop eleven days later. Eleven days of quietly worse answers, no alert, no exception. Pin the revision, bind the adapter to a digest, and run evals on every deploy.

Capacity that decayed. A service's throughput fell ~10% a day and a restart always fixed it, so it got restarted nightly and filed as "probably a memory leak." It was an error path in predict() that admitted a request and returned without releasing the semaphore. Every failed request cost one permanent unit of concurrency. Three lines of try/finally closed a bug that had been open for two months.

Flapping into a $4k month. Someone set the scale-down delay to 30 seconds and the cold start was 90 seconds. Traffic was spiky, so the deployment scaled down, immediately needed capacity again, paid a 90-second cold start, scaled up, dipped, repeated — all day. The bill tripled while serving fewer requests, because you are billed from the moment a replica starts booting. Scale-down delay must comfortably exceed the cold start.

The infinite crash-loop that was a YAML bug. A 5-minute model load with a liveness probe set to fail after 60 seconds. Kubernetes killed the pod mid-load, it restarted, loaded for 60 seconds, got killed. Forever. Nothing was wrong with the model. Readiness and liveness are different questions; use a startup probe for slow loads.

The tenant that saw another tenant's model. A gateway took the adapter name from a request field instead of the authenticated session. A customer with a debugging habit typed someone else's slug. Multi-LoRA makes tenant isolation an application concern — resolve the adapter from the session, always, and test that path like authz.

The break-even that wasn't. A team built a self-hosting business case against a $0.60/1M hosted 8B and shipped it. Their real utilization was ~20%, so their true cost was ~$2.20/1M. There was no volume that would have fixed it — more GPUs add cost and capacity in the same ratio. They should have argued latency and data residency, which were both true and both sufficient.

Vocabulary (rapid-fire)

• Image layer — a tarball of filesystem changes with a content digest; caching is prefix-based.
• Cold start — request-for-a-replica → first token. Node + pull + weights + load + engine build.
• Readiness vs liveness — "may I send traffic?" vs "should I kill this?". Conflating them causes crash-loops.
• predict_concurrency — in-flight request ceiling per replica; exceeding it is a 429, not a queue.
• Target concurrency — the autoscaler's per-replica setpoint; desired = ceil(backlog/target).
• Scale-to-zero — no replicas when idle; the next request is queued through a cold start, not dropped.
• Flapping — scale down → load returns → cold start → scale up → repeat. Caused by a scale-down delay shorter than the cold start.
• Merged adapter — W + ΔW folded into the weights: zero overhead, own replica, no hot-swap.
• Dynamic adapter (multi-LoRA) — A,B kept separate and applied per request via SGMV/punica kernels; a few % overhead, N tenants per GPU.
• SGMV — segmented gather matmul: one batched GEMM applying different adapters to different rows.
• Base digest binding — pinning an adapter to the exact checkpoint it was trained on. The only defense against a failure with no exception.
• Environment vs deployment — a stable alias over an immutable version; promotion and rollback are pointer swaps.
• Model cache / accelerated weight loading — weights fetched at build time so cold starts read locally.
• Utilization — fraction of rented GPU-time actually serving. The hidden term in every $/1M.
• Break-even feasibility — if self-hosted $/1M at full load exceeds the API's, no volume ever wins.
• GGUF / Q4_K_M — single-file edge weight format; ~4.5 bits/weight mixed precision.

Beginner mistakes

1. Loading the model in __init__ instead of load().
2. Returning 200 (or hanging) for a request that arrives before ready.
3. No finally: release() — the slow capacity leak.
4. Writing handle_stream as a generator, so nothing validates until the first chunk.
5. Yielding cumulative text instead of deltas while streaming.
6. torch>=2.4, revision: main, FROM cuda:latest — three ways to un-reproduce a build.
7. Weights fetched at request time instead of build time.
8. A secret in an image layer (immutable, cached, un-rotatable).
9. One GPU per fine-tuned customer model.
10. Scale-down delay shorter than the cold start.
11. Sizing VRAM as params × bytes and forgetting the KV pool and overhead.
12. Quoting $/1M at 100% utilization when you run at 25%.
13. Fine-tuning a knowledge problem (you needed RAG).
14. Splitting the dataset randomly by row instead of by entity — your eval is a memorization score.
15. Promoting a model no script could have blocked.

The one thing to take away

The model is the easy part. What separates "I fine-tuned a model" from "I run models in production" is a short, boring, unforgiving list: pin everything, order layers by volatility, load once and refuse until ready, always release the slot, share the base across tenants, scale up fast and down slow, and know your $/1M at your real utilization. Get those seven right and you can ship any model, anywhere, on any platform — and explain, in a design review, exactly what it will cost.