Deployment Cookbook — Custom Models, Real Commands

Everything here is meant to be typed, not read. Each recipe is a complete, runnable artifact: a real config.yaml, a real model/model.py, a real CLI invocation, a real client call. The labs in this phase are deterministic miniatures of these mechanisms; this file is the mechanisms themselves.

A standing caveat. Hosting platforms change endpoints, flags, and defaults faster than any document survives. The shapes below — a packaging manifest, a load/predict lifecycle, an OpenAI-compatible route, an autoscaling block — have been stable for years and are what you are actually learning. Before you paste a hostname, a price, or an exact flag name into production, check the vendor's current docs. Where a detail is especially prone to drift, it is marked [verify].

Table of Contents

• 0. The five-step shape of every custom-model deployment
• 1. Truss + Baseten — the reference path
  • 1.1 What truss push actually does
  • 1.2 A complete Truss for a merged fine-tune (vLLM inside)
  • 1.3 Deploy, invoke, stream
  • 1.4 Development vs production, environments, canary & rollback
  • 1.5 Autoscaling, scale-to-zero and concurrency
  • 1.6 Secrets, private weights and the model cache
  • 1.7 Async predictions and webhooks
  • 1.8 Multi-step pipelines (Truss Chains)
• 2. Customizing the model — the training half
  • 2.1 Build the dataset (the step that decides everything)
  • 2.2 LoRA / QLoRA SFT with trl + peft
  • 2.3 Preference tuning (DPO) when SFT is not enough
  • 2.4 Merge, quantize, and publish the artifact
  • 2.5 Distillation: make the model permanently smaller
• 3. vLLM on your own GPU
• 4. The same model, five other ways
  • 4.1 Modal
  • 4.2 Replicate (Cog)
  • 4.3 RunPod Serverless
  • 4.4 AWS SageMaker (LMI / DJL container)
  • 4.5 BentoML
  • 4.6 Choosing between them
• 5. Non-LLM custom models: embeddings, rerankers, VLMs
• 6. Edge and on-robot deployment
• 7. Production hardening
  • 7.1 Eval gates in CI
  • 7.2 Observability that is actually useful
  • 7.3 Load-testing your deployment
• 8. The failure modes, and what fixes each
• 9. References

0. The five-step shape of every custom-model deployment

Every platform in this document — Baseten, Modal, Replicate, RunPod, SageMaker, BentoML, or a bare vllm serve on a rented box — is the same five steps with different syntax. Learn the shape and the vendor is a detail.

  ┌───────────────────────────────────────────────────────────────────────────┐
  │ 1. CUSTOMIZE   data → SFT/LoRA/DPO/distill → adapter or merged checkpoint  │
  │                (Phase 05, 07)                          artifact + digest   │
  ├───────────────────────────────────────────────────────────────────────────┤
  │ 2. PACKAGE     manifest (deps, GPU, weights, concurrency) + a load/predict │
  │                class → a reproducible container image      (Lab 01)        │
  ├───────────────────────────────────────────────────────────────────────────┤
  │ 3. DEPLOY      push image → registry → scheduler puts it on a GPU node →   │
  │                readiness probe passes → traffic routed          (Lab 01)   │
  ├───────────────────────────────────────────────────────────────────────────┤
  │ 4. SERVE       OpenAI-compatible route, batching, streaming, adapters,     │
  │                admission control                            (Lab 01, 02)   │
  ├───────────────────────────────────────────────────────────────────────────┤
  │ 5. OPERATE     autoscale, cold starts, canary+rollback, evals, cost        │
  │                                                    (Lab 03, Phase 16, 17)  │
  └───────────────────────────────────────────────────────────────────────────┘

The two decisions that dominate everything downstream:

Lab 02 is exactly that trade, made numeric.

1. Truss + Baseten — the reference path

Truss is an open-source model-packaging format (Apache-2.0, by Baseten). It is the cleanest concrete example of the packaging layer, and it is not a lock-in: a Truss builds a Docker image you can run anywhere.

pip install --upgrade truss
truss init my-model          # scaffolds config.yaml + model/model.py

1.1 What truss push actually does

This is the part most people never look at, and it is the whole content of Lab 01.

  your laptop                     build service                    GPU cluster
  ───────────                     ─────────────                    ───────────
  config.yaml   ──┐
  model/model.py  ├─ bundle ──▶  1. render a Dockerfile from the manifest
  packages/     ──┘                 FROM  <cuda/python base image>
                                    RUN   apt-get install <system_packages>
                                    RUN   pip install    <requirements>
                                    RUN   <fetch model_cache weights>      ← multi-GB
                                    COPY  model/  packages/                ← bytes
                                 2. build with a layer cache; push to a registry
                                 3. create a Deployment record (id, version, status)
                                                │
                                                ▼
                                          4. scheduler picks a node with the requested
                                             accelerator; pulls the image
                                          5. container starts the model server:
                                               Model.__init__(config, data_dir, secrets)
                                               Model.load()          ← weights → VRAM
                                             /health starts returning ready
                                          6. router adds the replica to the pool;
                                             traffic flows to
                                             POST /environments/production/predict
                                          7. autoscaler watches concurrency and adds
                                             or removes replicas

Four consequences you must be able to state:

1. Layer order is a performance decision. Weights are fetched before your code is copied, so editing model.py reuses the cached weight layer. Bump a pinned dependency and everything after it — including the weight fetch — rebuilds. (plan_cache_reuse, Lab 01.)
2. __init__ runs before the accelerator is guaranteed usable and before the health server is answering. Heavy work belongs in load(). Anything slow in __init__ extends the window where the pod looks dead.
3. The readiness signal is the contract. Until load() returns, requests must be refused (503), not answered badly.
4. Cold start = image pull + weight fetch + load (+ engine build). Each has a different fix: smaller image, cached weights, faster loader, cached engine.

1.2 A complete Truss for a merged fine-tune (vLLM inside)

config.yaml:

model_name: support-triage-8b
description: Llama-3.1-8B LoRA-tuned on 18k support tickets, merged.

python_version: py311
requirements:
  - vllm==0.6.3
  - transformers==4.45.2
  - huggingface_hub==0.25.2
system_packages:
  - git

resources:
  accelerator: A10G          # see Lab 03 pick_gpu — int4 would let you drop to an L4
  use_gpu: true
  cpu: "4"
  memory: 16Gi

runtime:
  predict_concurrency: 16    # the admission gate; vLLM batches behind it

# Weights are fetched at BUILD time into the image cache, not at request time.
# Pin the revision — `main` makes two builds of the same commit serve different models.
model_cache:
  - repo_id: acme/llama-3.1-8b-support-merged
    revision: 8c22764a7e3675c50d4c7c9a4edb474456022b16
    allow_patterns:
      - "*.safetensors"
      - "*.json"
      - "tokenizer*"

secrets:
  hf_access_token: null      # NAME only — the value is injected at runtime

environment_variables:
  VLLM_WORKER_MULTIPROC_METHOD: spawn

model/model.py:

import os
from typing import Any, AsyncGenerator

from vllm import AsyncEngineArgs, AsyncLLMEngine, SamplingParams

MODEL_REPO = "acme/llama-3.1-8b-support-merged"
REVISION = "8c22764a7e3675c50d4c7c9a4edb474456022b16"


class Model:
    """Truss calls: __init__ -> load -> (preprocess -> predict -> postprocess)*"""

    def __init__(self, **kwargs: Any) -> None:
        # Runs at import time. NOTHING heavy here: no weights, no CUDA, no network.
        self._config = kwargs["config"]
        self._data_dir = kwargs["data_dir"]        # baked-in files ship here
        self._secrets = kwargs["secrets"]          # runtime-injected secret values
        self._engine: AsyncLLMEngine | None = None

    def load(self) -> None:
        # Runs ONCE, in the container, with the GPU attached. Everything expensive
        # goes here. Until it returns, the server reports "not ready" and the
        # platform withholds traffic.
        os.environ["HF_TOKEN"] = self._secrets["hf_access_token"]
        self._engine = AsyncLLMEngine.from_engine_args(
            AsyncEngineArgs(
                model=MODEL_REPO,
                revision=REVISION,
                dtype="bfloat16",
                max_model_len=8192,
                gpu_memory_utilization=0.90,   # leave room for the CUDA context
                enforce_eager=False,           # CUDA graphs: faster decode, slower load
            )
        )

    def preprocess(self, model_input: dict) -> dict:
        # Validate at the edge. A malformed request must fail here with a clear
        # error, not deep inside the engine.
        prompt = model_input.get("prompt")
        if not isinstance(prompt, str) or not prompt.strip():
            raise ValueError("`prompt` is required and must be a non-empty string")
        return {
            "prompt": prompt.strip(),
            "max_tokens": int(model_input.get("max_tokens", 256)),
            "temperature": float(model_input.get("temperature", 0.2)),
            "stream": bool(model_input.get("stream", False)),
        }

    async def predict(self, model_input: dict) -> Any:
        params = SamplingParams(
            max_tokens=model_input["max_tokens"],
            temperature=model_input["temperature"],
        )
        request_id = os.urandom(8).hex()
        results = self._engine.generate(model_input["prompt"], params, request_id)

        if not model_input["stream"]:
            final = None
            async for out in results:
                final = out
            return {"text": final.outputs[0].text,
                    "tokens": len(final.outputs[0].token_ids)}

        # Returning an async generator makes the endpoint stream. Yield DELTAS, not
        # the cumulative text, or every client renders the response N times over.
        async def deltas() -> AsyncGenerator[str, None]:
            sent = 0
            async for out in results:
                text = out.outputs[0].text
                yield text[sent:]
                sent = len(text)

        return deltas()

Local iteration before you ever push:

truss predict --target-directory . -d '{"prompt": "card declined twice today"}'

1.3 Deploy, invoke, stream

export BASETEN_API_KEY=...           # or: truss login

truss push                           # -> development deployment (live-reloadable)
truss push --publish                 # -> a new immutable production deployment
truss watch                          # hot-reload model/ into the dev deployment
truss logs                           # build + runtime logs

Invoke it — MODEL_ID comes from the push output or the dashboard:

curl -X POST \
  "https://model-${MODEL_ID}.api.baseten.co/environments/production/predict" \
  -H "Authorization: Api-Key ${BASETEN_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"prompt": "card declined twice today", "max_tokens": 64}'

[verify] The model-{id}.api.baseten.co/environments/{env}/predict shape is the current one; development deployments use /development/predict and a pinned deployment uses /deployment/{deployment_id}/predict.

Streaming from Python:

import os, httpx

url = f"https://model-{os.environ['MODEL_ID']}.api.baseten.co/environments/production/predict"
headers = {"Authorization": f"Api-Key {os.environ['BASETEN_API_KEY']}"}
payload = {"prompt": "summarize this ticket…", "max_tokens": 256, "stream": True}

with httpx.stream("POST", url, headers=headers, json=payload, timeout=None) as r:
    r.raise_for_status()
    for chunk in r.iter_text():
        print(chunk, end="", flush=True)

OpenAI-compatible clients. If your Truss runs an OpenAI-compatible server (vLLM's serve, TGI, or Baseten's built-in LLM server), point the OpenAI SDK at it and change nothing else — that is the entire benefit of to_openai_chat_completion in Lab 01:

from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://inference.baseten.co/v1",   # [verify] shared Model APIs
    # for a dedicated deployment the base_url is the deployment's OpenAI-compatible
    # route — check the deployment page for the exact path.
)

resp = client.chat.completions.create(
    model="support-triage-8b",
    messages=[{"role": "user", "content": "card declined twice today"}],
    max_tokens=64,
)
print(resp.choices[0].message.content)

Now every tool that speaks OpenAI — LangChain, LlamaIndex, your eval harness from Phase 16, an agent framework from Phase 12 — works against your model.

1.4 Development vs production, environments, canary & rollback

truss push                                   # development: mutable, live-reload, cheap
truss push --publish                         # production: immutable deployment version
truss push --publish --environment staging   # [verify] named environments

The mental model, which matches Phase 17 Lab 03 exactly:

Rule that survives every platform: the environment URL is stable; the deployment behind it is versioned. Clients never hardcode a deployment id.

1.5 Autoscaling, scale-to-zero and concurrency

# In the Truss config or the deployment settings [verify — often dashboard/API-managed]
runtime:
  predict_concurrency: 16      # concurrent predict() calls per replica (the 429 gate)

Deployment-level knobs, and what each one costs you (Lab 03 makes all of this numeric):

The three-line version: min_replicas: 0 is free until someone waits four minutes for the first token. Interactive product → keep one warm. Batch/back-office → scale to zero and enjoy the bill.

1.6 Secrets, private weights and the model cache

secrets:
  hf_access_token: null     # declare the NAME; set the VALUE in the platform's secret store

def load(self):
    token = self._secrets["hf_access_token"]     # injected at runtime, never in a layer

# Set the value out-of-band, never in git:
#   Baseten: dashboard → Secrets, or the API
#   Kubernetes: kubectl create secret generic hf --from-literal=token=...
#   Modal:  modal secret create hf-token HF_TOKEN=...

Three rules that are not negotiable:

1. A secret baked into an image layer is public forever to anyone who can pull the image — and rotating the secret does not un-bake it. Layers are immutable and cached.
2. Pin revision. main is not a version. (Lab 01 rejects it.)
3. Cache weights at build time, not on first request. Fetching 16 GB from a model hub on every cold start is minutes of latency and a hard dependency on someone else's uptime during your incident.

1.7 Async predictions and webhooks

For anything slower than a request timeout — batch scoring, long agent runs, video — use the async path instead of holding an HTTP connection open:

curl -X POST \
  "https://model-${MODEL_ID}.api.baseten.co/environments/production/async_predict" \
  -H "Authorization: Api-Key ${BASETEN_API_KEY}" \
  -d '{
        "model_input": {"prompt": "…", "max_tokens": 2048},
        "webhook_endpoint": "https://api.acme.com/hooks/inference"
      }'

You get a request id immediately; the result is POSTed to your webhook. [verify] the exact field names. Verify the webhook signature before trusting the payload, and make the handler idempotent — retries are guaranteed, exactly-once is not.

1.8 Multi-step pipelines (Truss Chains)

When one request needs several models with different hardware — say a small router on CPU, a VLM on an H100, and a reranker on an L4 — packaging them into one container means paying H100 prices for all three. Chains lets each step scale independently:

# [verify] the Chains API is younger than the core Truss config and moves faster.
import truss_chains as chains


class Transcribe(chains.ChainletBase):
    remote_config = chains.RemoteConfig(
        compute=chains.Compute(gpu="L4", predict_concurrency=4),
        docker_image=chains.DockerImage(pip_requirements=["faster-whisper==1.0.3"]),
    )

    def __init__(self):
        from faster_whisper import WhisperModel
        self._model = WhisperModel("large-v3", device="cuda", compute_type="float16")

    def run_remote(self, audio_url: str) -> str:
        segments, _ = self._model.transcribe(audio_url)
        return " ".join(s.text for s in segments)


@chains.mark_entrypoint
class SupportPipeline(chains.ChainletBase):
    def __init__(self, transcribe: Transcribe = chains.depends(Transcribe)):
        self._transcribe = transcribe

    def run_remote(self, audio_url: str) -> dict:
        text = self._transcribe.run_remote(audio_url)
        return {"transcript": text, "label": classify(text)}

truss chains push ./support_pipeline.py

The general lesson beats the specific API: do not co-locate steps with different accelerator needs or different scaling curves. A CPU pre-processing step pinned to a GPU replica wastes the most expensive resource you rent.

2. Customizing the model — the training half

Deployment is half the job. Here is the other half, end to end, with the commands.

2.1 Build the dataset (the step that decides everything)

Model quality is dataset quality. The single highest-leverage hour in a fine-tune project is spent on data, not hyperparameters.

# data/build.py — produce train.jsonl / eval.jsonl in chat format
import json, hashlib, random

def to_chat(ticket: dict) -> dict:
    return {
        "messages": [
            {"role": "system", "content": "You are a support triage assistant. "
                                          "Reply with JSON: {label, priority, team}."},
            {"role": "user", "content": ticket["body"][:4000]},
            {"role": "assistant", "content": json.dumps({
                "label": ticket["label"],
                "priority": ticket["priority"],
                "team": ticket["team"],
            })},
        ]
    }

rows = [to_chat(t) for t in load_tickets()]

# 1. DEDUPLICATE — near-duplicates inflate your eval score and teach nothing.
seen, deduped = set(), []
for r in rows:
    h = hashlib.sha256(r["messages"][1]["content"].encode()).hexdigest()
    if h not in seen:
        seen.add(h)
        deduped.append(r)

# 2. Split BY ENTITY (customer//session), never randomly by row — random splits leak
#    the same customer's phrasing into both sides and your eval lies to you.
random.Random(0).shuffle(deduped)
cut = int(0.9 * len(deduped))
for name, part in (("train", deduped[:cut]), ("eval", deduped[cut:])):
    with open(f"data/{name}.jsonl", "w") as f:
        for r in part:
            f.write(json.dumps(r) + "\n")

Checklist before you spend a GPU-hour:

• 500–5,000 high-quality examples beat 100k scraped ones for a narrow task.
• Deduplicated (exact and near-duplicate).
• Split by entity, with no leakage across the split.
• The output format is exactly what production will parse.
• A held-out set you never look at until the end.
• Failure cases from the current system are over-represented — that is the whole point of fine-tuning over prompting.
• PII reviewed. Weights memorize; a leaked training row is a leaked training row forever.

2.2 LoRA / QLoRA SFT with trl + peft

pip install "transformers==4.45.2" "trl==0.11.4" "peft==0.13.2" \
            "datasets==3.0.1" "accelerate==1.0.1" "bitsandbytes==0.44.1"

# train_lora.py — a single-GPU QLoRA SFT run
import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoTokenizer, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

BASE = "meta-llama/Llama-3.1-8B-Instruct"
REVISION = "0e9e39f249a16976918f6564b8830bc894c89659"   # PIN IT (Lab 01's lesson)

ds = load_dataset("json", data_files={"train": "data/train.jsonl",
                                      "eval": "data/eval.jsonl"})

tok = AutoTokenizer.from_pretrained(BASE, revision=REVISION)
tok.pad_token = tok.pad_token or tok.eos_token

# QLoRA: the frozen base lives in 4-bit NF4, the trainable adapters in bf16.
# This is what puts an 8B (or a 70B) fine-tune on one commodity GPU. See Phase 05.
quant = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

peft_config = LoraConfig(
    r=16,                      # rank: 8–16 for style/format, 32–64 for new knowledge
    lora_alpha=32,             # scaling is alpha/r — see Phase 05 Lab 01
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    # q,v is the classic minimum; adding k,o and the MLP projections helps harder tasks
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
)

trainer = SFTTrainer(
    model=BASE,
    args=SFTConfig(
        output_dir="out/support-lora",
        num_train_epochs=2,             # 1–3. More overfits a small set fast.
        per_device_train_batch_size=4,
        gradient_accumulation_steps=8,  # effective batch 32
        learning_rate=1e-4,             # LoRA likes 1e-4..2e-4; full FT wants ~1e-5
        lr_scheduler_type="cosine",
        warmup_ratio=0.03,
        bf16=True,
        gradient_checkpointing=True,    # trades ~20% speed for a lot of memory
        logging_steps=10,
        eval_strategy="steps",
        eval_steps=100,
        save_strategy="steps",
        save_steps=100,
        load_best_model_at_end=True,
        max_seq_length=2048,
        packing=True,                   # pack short samples — big throughput win
        seed=0,                         # reproducibility is not optional
        report_to="mlflow",             # Phase 17: every run tracked
    ),
    train_dataset=ds["train"],
    eval_dataset=ds["eval"],
    peft_config=peft_config,
    model_init_kwargs={"quantization_config": quant,
                       "torch_dtype": torch.bfloat16,
                       "revision": REVISION},
    processing_class=tok,
)

trainer.train()
trainer.save_model("out/support-lora")     # ~50–150 MB, not 16 GB

What lands in out/support-lora:

adapter_config.json         # r, lora_alpha, target_modules, base_model_name_or_path
adapter_model.safetensors   # A and B matrices only — this IS the adapter_vram_mb of Lab 02
tokenizer.json, tokenizer_config.json, special_tokens_map.json

Reality check on cost: a 2-epoch QLoRA on ~10k short examples is roughly 1–3 hours on one A100/H100, i.e. single-digit dollars. The expensive part of a fine-tune project is the dataset and the evaluation, not the GPU.

2.3 Preference tuning (DPO) when SFT is not enough

SFT teaches format and domain. When you need "this answer is better than that one" — tone, refusal behavior, terseness, citation discipline — you need preferences (Phase 07).

from trl import DPOConfig, DPOTrainer

# Rows: {"prompt": ..., "chosen": ..., "rejected": ...}
trainer = DPOTrainer(
    model="out/support-merged",     # start from your SFT'd model
    ref_model=None,                 # None = use the frozen PEFT base as reference
    args=DPOConfig(
        output_dir="out/support-dpo",
        beta=0.1,                   # KL strength: lower drifts further from the ref
        learning_rate=5e-7,         # DPO wants a MUCH smaller LR than SFT
        num_train_epochs=1,
        per_device_train_batch_size=2,
        gradient_accumulation_steps=16,
        bf16=True,
        seed=0,
    ),
    train_dataset=pref_ds,
    peft_config=peft_config,
    processing_class=tok,
)
trainer.train()

Order matters: SFT first, then DPO. DPO on a base model that cannot yet produce the format just makes it confidently malformed.

2.4 Merge, quantize, and publish the artifact

# merge.py — fold the adapter into the weights for dedicated serving
import torch
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

model = AutoPeftModelForCausalLM.from_pretrained(
    "out/support-lora", torch_dtype=torch.bfloat16, device_map="cpu",
)
merged = model.merge_and_unload()          # W  <-  W + (alpha/r)·A·B
merged.save_pretrained("out/support-merged", safe_serialization=True)
AutoTokenizer.from_pretrained("out/support-lora").save_pretrained("out/support-merged")

Merge or not? This is the Lab 02 decision:

Quantize the merged model to cut the SKU (Lab 03's pick_gpu in action):

pip install autoawq

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model = AutoAWQForCausalLM.from_pretrained("out/support-merged")
tok = AutoTokenizer.from_pretrained("out/support-merged")
model.quantize(tok, quant_config={"w_bit": 4, "q_group_size": 128,
                                  "zero_point": True, "version": "GEMM"},
               calib_data="your-domain-calibration-set")   # USE YOUR OWN DOMAIN DATA
model.save_quantized("out/support-merged-awq")
tok.save_pretrained("out/support-merged-awq")

Calibrate on your distribution, not wikitext. Then re-run your eval suite — a 4-bit model that lost 3 points of exact-match on your task is not a cost win, it is a regression you paid for.

Publish with a digest you can pin:

huggingface-cli upload acme/llama-3.1-8b-support-merged ./out/support-merged --private
git -C ./out/support-merged rev-parse HEAD     # -> the revision you pin in config.yaml

2.5 Distillation: make the model permanently smaller

When latency or edge constraints rule out the 8B entirely, train a small student on the big model's outputs (Phase 05).

# 1. Generate teacher outputs on your unlabeled production prompts (the "dark knowledge")
teacher_out = [{"messages": [...], "assistant": teacher(prompt)} for prompt in prompts]

# 2. SFT a 1.5B student on them — same SFTTrainer, smaller base:
#    BASE = "Qwen2.5-1.5B-Instruct"
# 3. Evaluate the student against the TEACHER, not against the ground truth alone:
#    agreement rate is the metric that tells you whether the distillation worked.

Typical outcome on a narrow task: a 1.5B student reaches 95–99% of the 8B's task accuracy at ~5× the throughput and a quarter of the VRAM — which is often the difference between a GPU and a Jetson.

3. vLLM on your own GPU

The reference open-source serving path, and the thing most managed platforms run internally. Everything here also works inside a Truss/Modal/SageMaker container.

pip install vllm==0.6.3

# A merged fine-tune, OpenAI-compatible, on one GPU:
vllm serve acme/llama-3.1-8b-support-merged \
  --revision 8c22764a7e3675c50d4c7c9a4edb474456022b16 \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --port 8000

Multi-LoRA — Lab 02, for real. One base, many adapters, per-request selection:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-lora \
  --lora-modules support=/adapters/support sql=/adapters/sql vision=/adapters/vision \
  --max-lora-rank 32 \
  --max-loras 8 \          # adapters resident on the GPU  → AdapterCache budget
  --max-cpu-loras 64 \     # host-resident spill tier      → the extension in Lab 02
  --max-model-len 8192 \
  --port 8000

# Select the adapter with the `model` field — exactly the router of Lab 02:
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "support", "messages": [{"role":"user","content":"card declined"}]}'

Quantized serving, and a smoke test of the throughput number Lab 03 needs:

vllm serve acme/support-merged-awq --quantization awq --dtype half --port 8000

vllm bench serve \
  --model acme/support-merged-awq \
  --dataset-name random --num-prompts 200 \
  --random-input-len 512 --random-output-len 128
# Feed the reported output tokens/s straight into usd_per_1m_tokens() from Lab 03.

Flags worth knowing by heart:

4. The same model, five other ways

4.1 Modal

# app.py
import modal

image = (
    modal.Image.debian_slim(python_version="3.11")
    .pip_install("vllm==0.6.3", "huggingface_hub==0.25.2")
    .env({"HF_HUB_ENABLE_HF_TRANSFER": "1"})
)
app = modal.App("support-triage", image=image)
volume = modal.Volume.from_name("hf-cache", create_if_missing=True)   # weight cache


@app.cls(
    gpu="A10G",
    volumes={"/root/.cache/huggingface": volume},
    secrets=[modal.Secret.from_name("hf-token")],
    scaledown_window=300,     # [verify] formerly container_idle_timeout
    min_containers=0,         # [verify] formerly keep_warm — 1 to avoid cold starts
)
class Triage:
    @modal.enter()            # the `load()` of Lab 01: runs once per container
    def load(self):
        from vllm import LLM
        self.llm = LLM("acme/llama-3.1-8b-support-merged", max_model_len=8192)

    @modal.method()
    def generate(self, prompt: str, max_tokens: int = 256) -> str:
        from vllm import SamplingParams
        out = self.llm.generate([prompt], SamplingParams(max_tokens=max_tokens))
        return out[0].outputs[0].text


@app.function()
@modal.fastapi_endpoint(method="POST")     # [verify] formerly web_endpoint
def web(item: dict):
    return {"text": Triage().generate.remote(item["prompt"])}

modal deploy app.py       # prints a public URL
modal run app.py          # one-off run

Modal's strength is that infrastructure is Python — the same file defines the image, the GPU, the volume, and the endpoint. Its @modal.enter() is Lab 01's load() and scaledown_window is Lab 03's scale_down_delay_s.

4.2 Replicate (Cog)

cog.yaml:

build:
  gpu: true
  cuda: "12.1"
  python_version: "3.11"
  python_packages:
    - "torch==2.4.0"
    - "transformers==4.45.2"
    - "accelerate==1.0.1"
predict: "predict.py:Predictor"

# predict.py
from cog import BasePredictor, Input
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer


class Predictor(BasePredictor):
    def setup(self):                       # == load()
        self.tok = AutoTokenizer.from_pretrained("/src/weights")
        self.model = AutoModelForCausalLM.from_pretrained(
            "/src/weights", torch_dtype=torch.bfloat16, device_map="cuda"
        )

    def predict(
        self,
        prompt: str = Input(description="The support ticket body"),
        max_tokens: int = Input(default=256, ge=1, le=2048),
    ) -> str:
        ids = self.tok(prompt, return_tensors="pt").to("cuda")
        out = self.model.generate(**ids, max_new_tokens=max_tokens)
        return self.tok.decode(out[0], skip_special_tokens=True)

cog predict -i prompt="card declined"     # local, in the real container
cog push r8.im/acme/support-triage        # publish

Cog's typed Input(...) signature is the API schema — the best ergonomics of the five for a public, demo-able model.

4.3 RunPod Serverless

# handler.py
import runpod
from vllm import LLM, SamplingParams

llm = LLM("acme/llama-3.1-8b-support-merged", max_model_len=8192)  # module scope = once


def handler(job):
    job_input = job["input"]
    prompt = job_input.get("prompt")
    if not prompt:
        return {"error": "prompt is required"}
    out = llm.generate([prompt], SamplingParams(
        max_tokens=job_input.get("max_tokens", 256)))
    return {"text": out[0].outputs[0].text}


runpod.serverless.start({"handler": handler})

Cheapest per GPU-hour of the managed options, thinnest platform. You own more of the operational surface — which is fine if you have already built the Lab 03 muscle.

4.4 AWS SageMaker (LMI / DJL container)

The path when the requirement is "it must be in our VPC/account".

import boto3, sagemaker
from sagemaker.djl_inference import DJLModel

role = sagemaker.get_execution_role()

model = DJLModel(
    model_id="s3://acme-models/llama-3.1-8b-support-merged/",
    role=role,
    env={
        "OPTION_ROLLING_BATCH": "vllm",          # vLLM behind DJL serving
        "OPTION_TENSOR_PARALLEL_DEGREE": "1",
        "OPTION_MAX_MODEL_LEN": "8192",
        "OPTION_DTYPE": "bf16",
    },
)

predictor = model.deploy(
    instance_type="ml.g5.2xlarge",     # A10G
    initial_instance_count=1,
    endpoint_name="support-triage",
    container_startup_health_check_timeout=900,   # cold starts are LONG here
)

print(predictor.predict({"inputs": "card declined twice today",
                         "parameters": {"max_new_tokens": 64}}))

Autoscaling is Application Auto Scaling on the endpoint variant; scale-to-zero on real-time endpoints is limited [verify] — use Serverless Inference or Async Inference when idle cost matters. Expect the deepest IAM/VPC integration and the slowest iteration loop of anything here.

4.5 BentoML

# service.py
import bentoml
from vllm import LLM, SamplingParams


@bentoml.service(
    resources={"gpu": 1, "gpu_type": "nvidia-a10g"},
    traffic={"timeout": 300, "concurrency": 16},   # == predict_concurrency
)
class SupportTriage:
    def __init__(self) -> None:                    # == load()
        self.llm = LLM("acme/llama-3.1-8b-support-merged", max_model_len=8192)

    @bentoml.api
    def generate(self, prompt: str, max_tokens: int = 256) -> str:
        out = self.llm.generate([prompt], SamplingParams(max_tokens=max_tokens))
        return out[0].outputs[0].text

bentoml serve service.py:SupportTriage     # local
bentoml build && bentoml deploy            # BentoCloud, or containerize and self-host

4.6 Choosing between them

Decision heuristic: start managed, measure, and only move to bare metal when the cost_compare from Lab 03 says the savings exceed the engineering time you would spend rebuilding the platform. That threshold is usually higher than people expect.

5. Non-LLM custom models: embeddings, rerankers, VLMs

A fine-tuned embedding model is the highest-ROI customization in most RAG systems — cheaper to train than an LLM, and it improves every query forever (Phase 11).

# train_embeddings.py — domain-tuned retriever with hard negatives
from sentence_transformers import SentenceTransformer, losses, InputExample
from torch.utils.data import DataLoader

model = SentenceTransformer("BAAI/bge-base-en-v1.5")

# Hard negatives are the whole trick: mine the top-k WRONG hits from your current
# retriever. Random negatives are too easy and teach the model almost nothing.
examples = [
    InputExample(texts=[q, positive_chunk, hard_negative_chunk])
    for q, positive_chunk, hard_negative_chunk in mined_triplets
]

model.fit(
    train_objectives=[(DataLoader(examples, shuffle=True, batch_size=32),
                       losses.MultipleNegativesRankingLoss(model))],
    epochs=2,
    warmup_steps=100,
    output_path="out/bge-support",
)

Serving it — the model is small (100–500 MB), so this is a cheap CPU/L4 deployment:

# model/model.py for an embedding Truss
class Model:
    def __init__(self, **kwargs):
        self._model = None

    def load(self):
        from sentence_transformers import SentenceTransformer
        self._model = SentenceTransformer("/app/data/bge-support", device="cuda")

    def predict(self, model_input: dict) -> dict:
        texts = model_input["texts"]
        if not isinstance(texts, list) or not texts:
            raise ValueError("`texts` must be a non-empty list")
        if len(texts) > 256:
            raise ValueError("batch too large; max 256")     # protect the replica
        vecs = self._model.encode(texts, normalize_embeddings=True, batch_size=64)
        return {"embeddings": [v.tolist() for v in vecs], "dim": int(vecs.shape[1])}

Dedicated embedding runtimes (Hugging Face TEI, Baseten's BEI [verify], Infinity) exist because a generic Python server wastes most of the throughput available on these tiny models. If embeddings are on your critical path, use one.

VLM fine-tune and deploy — the one that matters for a robotics JD (Phase 04/15):

# Fine-tune Qwen2-VL on grounded (image, instruction, answer) triples
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig

BASE = "Qwen/Qwen2-VL-7B-Instruct"
processor = AutoProcessor.from_pretrained(BASE)

peft_config = LoraConfig(
    r=32, lora_alpha=64, task_type="CAUSAL_LM",
    # Adapt the LANGUAGE tower; freezing the vision encoder is the usual starting
    # point — it already sees fine, it is the grounding that needs teaching.
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)
# ...SFTTrainer over a dataset of {"images": [...], "messages": [...]}

# Serve it with vLLM's multimodal support:
vllm serve acme/qwen2-vl-7b-warehouse \
  --max-model-len 8192 --limit-mm-per-prompt image=4 --port 8000

6. Edge and on-robot deployment

When the model has to run next to the actuator, network latency is not a tuning parameter — it is a safety property (Phase 15).

# llama.cpp / GGUF — CPU, Apple Silicon, Jetson
git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp && make
python convert_hf_to_gguf.py ../out/support-merged --outfile support-f16.gguf --outtype f16
./llama-quantize support-f16.gguf support-Q4_K_M.gguf Q4_K_M     # ~4.5 bits/weight
./llama-server -m support-Q4_K_M.gguf -c 4096 --port 8080        # OpenAI-compatible

# ONNX Runtime — the portable CPU/edge path
optimum-cli export onnx --model out/support-merged --task text-generation onnx/
python -c "
import onnxruntime as ort
sess = ort.InferenceSession('onnx/model.onnx', providers=['CUDAExecutionProvider'])
"

# TensorRT-LLM — maximum throughput on NVIDIA, at the cost of a build step
trtllm-build --checkpoint_dir ./ckpt --output_dir ./engine \
             --gemm_plugin bfloat16 --max_batch_size 8 --max_input_len 4096

The engine build is minutes-to-hours and is hardware-specific: an engine built for an A100 will not run on an H100. Cache the artifact and treat it as a build output (engine_build_s in Lab 01's cold-start model), never as something you do at boot.

The honest edge hierarchy:

Split the loop: run the small, fast, safety-critical policy on the robot; call the big model in the cloud for planning that tolerates a round trip. That hybrid is the standard embodied-AI architecture, not a compromise.

7. Production hardening

7.1 Eval gates in CI

Never promote a model a script has not judged (Phase 16 + Phase 17 Lab 03).

# .github/workflows/model-release.yml
name: model-release
on:
  push:
    tags: ["model-v*"]

jobs:
  evaluate-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - run: pip install -r requirements.txt

      # 1. Deploy the candidate to a NON-production environment first.
      - run: truss push --publish --environment staging
        env:
          BASETEN_API_KEY: ${{ secrets.BASETEN_API_KEY }}

      # 2. Run the eval suite against the staging URL. Exit non-zero on regression.
      - run: python evals/run.py --endpoint "$STAGING_URL" --baseline evals/champion.json
        env:
          STAGING_URL: ${{ vars.STAGING_URL }}

      # 3. Only now promote. The gate is the reason this job exists.
      - run: python scripts/promote.py --environment production
        env:
          BASETEN_API_KEY: ${{ secrets.BASETEN_API_KEY }}

# evals/run.py — the shape that matters, not the framework
import json, sys

results = run_eval_suite(endpoint=ENDPOINT)      # your Phase 16 harness
baseline = json.load(open("evals/champion.json"))

GATES = {
    "exact_match":    ("min", baseline["exact_match"] - 0.01),   # no regression
    "json_validity":  ("min", 0.99),                             # absolute floor
    "safety_refusal": ("min", 0.98),
    "p95_latency_ms": ("max", 1500),
    "usd_per_1k":     ("max", 0.05),
}

failed = [
    f"{k}: {results[k]} violates {how} {bound}"
    for k, (how, bound) in GATES.items()
    if (results[k] < bound if how == "min" else results[k] > bound)
]
if failed:
    print("EVAL GATE FAILED:\n  " + "\n  ".join(failed))
    sys.exit(1)
print("eval gate passed")

7.2 Observability that is actually useful

Uptime dashboards lie about ML systems: the service returns 200s while the model quietly gets worse. Instrument these instead.

# Structured request logs — the input to every offline analysis you will want later.
log.info(json.dumps({
    "request_id": request_id,
    "model": "support-triage-8b",
    "deployment_id": DEPLOYMENT_ID,       # exact version, so you can bisect a regression
    "adapter": adapter_key,               # multi-LoRA: which tenant
    "prompt_tokens": usage.prompt_tokens,
    "completion_tokens": usage.completion_tokens,
    "ttft_ms": ttft_ms,
    "total_ms": total_ms,
    "finish_reason": finish_reason,       # a spike in "length" = truncation bugs
    "json_valid": bool(parsed),
    "cache_hit": cache_hit,
}))

7.3 Load-testing your deployment

# Measure the numbers Lab 03 needs — never assume them.
vllm bench serve --model support-triage-8b --base-url https://your-endpoint \
  --dataset-name random --num-prompts 500 \
  --random-input-len 512 --random-output-len 128 --request-rate 10

Report exactly four numbers from every load test: p50 TTFT, p95 end-to-end, output tokens/s, and $/1M tokens at the measured utilization. Anything else is decoration.

8. The failure modes, and what fixes each

9. References

Packaging & platforms

• Truss docs — https://docs.baseten.co/development/model/overview and https://truss.baseten.co
• Baseten docs (deployments, environments, autoscaling, async) — https://docs.baseten.co
• Modal docs — https://modal.com/docs/guide
• Cog (Replicate) — https://cog.run/ and https://replicate.com/docs
• RunPod Serverless — https://docs.runpod.io/serverless/overview
• SageMaker LMI containers — https://docs.aws.amazon.com/sagemaker/latest/dg/large-model-inference-container-docs.html
• BentoML — https://docs.bentoml.com
• KServe / Knative autoscaling — https://knative.dev/docs/serving/autoscaling/

Serving engines

• vLLM — https://docs.vllm.ai (multi-LoRA: "LoRA Adapters"; PagedAttention: Kwon et al., SOSP 2023)
• TensorRT-LLM — https://nvidia.github.io/TensorRT-LLM/
• Text Generation Inference — https://huggingface.co/docs/text-generation-inference
• llama.cpp — https://github.com/ggerganov/llama.cpp
• Punica / S-LoRA (the multi-LoRA kernels) — Chen et al., "Punica: Multi-Tenant LoRA Serving" (2023); Sheng et al., "S-LoRA: Serving Thousands of Concurrent LoRA Adapters" (2023)

Customization

• Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models" (2021) — https://arxiv.org/abs/2106.09685
• Dettmers et al., "QLoRA: Efficient Finetuning of Quantized LLMs" (2023) — https://arxiv.org/abs/2305.14314
• Rafailov et al., "Direct Preference Optimization" (2023) — https://arxiv.org/abs/2305.18290
• Hinton et al., "Distilling the Knowledge in a Neural Network" (2015) — https://arxiv.org/abs/1503.02531
• Lin et al., "AWQ: Activation-aware Weight Quantization" (2023) — https://arxiv.org/abs/2306.00978
• HF peft — https://huggingface.co/docs/peft; trl — https://huggingface.co/docs/trl
• Sentence-Transformers training — https://sbert.net/docs/sentence_transformer/training_overview.html

Operations

• Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NeurIPS 2015)
• Google SRE Workbook, "Canarying Releases" — https://sre.google/workbook/canarying-releases/
• Argo Rollouts (canary/blue-green) — https://argo-rollouts.readthedocs.io