Customization Showcases — Six Projects, End to End

Six concrete projects, each one a complete path from "we have a problem" to "a custom model is serving production traffic behind a URL". Every showcase gives you the decision test, the exact recipe, the deployment config, the measured-shape result table, and the ways it goes wrong. Build any two of these and you can hold a Senior AI Engineer interview on custom models without hand-waving.

About the numbers. The result tables are worked examples, not audited measurements from a specific company. The cost and latency figures are computed from the arithmetic you implement in Lab 03 using publicly listed GPU and API prices; the quality deltas sit inside the ranges reported in the LoRA/QLoRA/distillation literature and in vendor case studies for narrow tasks. They are there so you can follow the reasoning and reproduce the arithmetic — not so you can quote them. Every showcase ends with the measurement you must run yourself.

Table of Contents

• 0. Should you customize at all? The ladder
• Showcase 1 — Support triage: replace a frontier API with a LoRA'd 8B
• Showcase 2 — Structured extraction: 100% parseable robot telemetry
• Showcase 3 — A domain VLM for warehouse grounding
• Showcase 4 — Fine-tuned embeddings: the cheapest RAG win there is
• Showcase 5 — 40 customer models on one GPU
• Showcase 6 — Distill 8B → 1.5B and put it on the robot
• 7. What fine-tuning cannot fix
• 8. The portfolio version of these projects

0. Should you customize at all? The ladder

Climb this ladder in order. Every rung is cheaper, faster to iterate, and easier to roll back than the one above it. Most teams that "need a fine-tune" actually need rung 2 or 3 — and the senior move is knowing which rung the problem is on.

Three tests that decide it in a meeting:

1. "Can a careful human do the task from the prompt alone?" If no → the model lacks knowledge → RAG, not fine-tuning.
2. "Does the model know the answer but present it wrong?" If yes → that is behavior → fine-tuning is exactly the right lever.
3. "Is quality already fine and the problem is the bill or the p95?" → distill or quantize a model you already trust; do not retrain for quality you already have.

And the constraint that overrides all three: if the data cannot leave your VPC, the ladder collapses to "self-host something." That is not a cost decision, and cost_compare from Lab 03 does not get a vote.

Showcase 1 — Support triage: replace a frontier API with a LoRA'd 8B

The problem

1.2M support tickets a month must be labeled {label, priority, team}. Today a frontier API does it with a 700-token few-shot prompt. It works, but: the bill grows linearly with the business, p95 latency is ~2.5 s (users see a spinner), and every prompt tweak silently changes behavior for every ticket.

Why this is rung 4

The model already knows how to classify a support ticket — it just needs your label taxonomy, your priority conventions, and your team routing, consistently, in your JSON shape. That is behavior, not knowledge. Fine-tune.

The recipe

# 1. Data: 18k historical tickets with human-confirmed labels.
#    Deduplicate; split BY CUSTOMER so one customer's phrasing cannot leak across.
python data/build.py --in tickets.parquet --out data/ --split-by customer_id

# 2. QLoRA SFT on Llama-3.1-8B-Instruct — ~2 h on one A100.
python train_lora.py                # cookbook §2.2, r=16, alpha=32, 2 epochs, lr=1e-4

# 3. Merge (single high-QPS model ⇒ dedicated serving ⇒ merge for zero overhead)
python merge.py                     # cookbook §2.4

# 4. Gate it before it goes anywhere near production.
python evals/run.py --model out/support-merged --baseline evals/api_champion.json

The system prompt shrinks from a 700-token few-shot block to ~40 tokens, because the examples now live in the weights. That alone is a 17× cut in prompt tokens — and it is the part people forget when they compare prices.

The deployment

model_name: support-triage-8b
python_version: py311
requirements: [vllm==0.6.3]
resources:
  accelerator: A10G
runtime:
  predict_concurrency: 32       # short outputs ⇒ batch aggressively
model_cache:
  - repo_id: acme/llama-3.1-8b-support-merged
    revision: 8c22764a7e3675c50d4c7c9a4edb474456022b16

min_replicas: 2 (HA, no cold start on the interactive path), max_replicas: 6.

The result

The real win is usually not the 53% cost cut. It is the 6× latency drop, plus the fact that the model stops changing underneath you.

Measure this yourself

Run the same 500-ticket eval set through both, and report: accuracy, JSON validity, p50 and p95 latency, and $/1k tickets at your measured utilization. If the fine-tune does not beat the API on at least two of those, ship the API — you just saved a quarter.

Gotchas

• Utilization is the whole cost story. At 1.2M tickets/month those two A10Gs are ~10% busy; the $1,533 is mostly idle GPU. Batch off-peak work onto the same replicas or drop to min_replicas: 1 and you halve it. (Lab 03: halving utilization doubles $/1M.)
• Ship a shadow deployment first: send 100% of traffic to both, log both answers, compare offline for a week. Nobody is harmed by a disagreement you find in a log.
• Keep the API path behind a feature flag for a month. Rollback is a flag flip, not a retraining.

Showcase 2 — Structured extraction: 100% parseable robot telemetry

The problem

Free-text maintenance notes and sensor logs from a robot fleet must become a strict schema — {component, fault_code, severity, action, confidence} — for a downstream scheduler that crashes on malformed JSON. A prompted model produces valid JSON about 91% of the time. At 40k documents a day that is 3,600 daily failures on a pipeline with no human in the loop.

Why this is rung 4 + Phase 08

Two independent fixes stack here, and using only one is the common mistake:

1. Fine-tune so the model's natural output is your schema.
2. Constrained decoding so malformed output is impossible, not merely unlikely — the grammar masks the logits so an invalid token can never be sampled (Phase 08).

Fine-tuning alone gets you to ~99.7%. The grammar gets you to 100% by construction. The fine-tune still matters because a grammar forces valid syntax, not correct content — an unconstrained-but-tuned model gets the fields right.

The recipe

# 1. Generate the schema from the source of truth, not by hand.
from pydantic import BaseModel, Field
from enum import Enum

class Severity(str, Enum):
    info = "info"; warn = "warn"; critical = "critical"

class Fault(BaseModel):
    component: str = Field(max_length=64)
    fault_code: str = Field(pattern=r"^[A-Z]{2}-\d{4}$")
    severity: Severity
    action: str
    confidence: float = Field(ge=0.0, le=1.0)

SCHEMA = Fault.model_json_schema()      # used for BOTH training targets and decoding

# 2. Build 6k (note -> validated JSON) pairs. Every target is Fault.model_validate()'d
#    at build time: a training set containing malformed targets teaches malformation.
python data/build_extraction.py --schema-check

# 3. QLoRA SFT, r=8 (format learning is low-rank; you are not teaching new knowledge)
python train_lora.py --rank 8 --epochs 3 --lr 1e-4

# 4. Serve with the grammar attached — vLLM's guided decoding
from vllm import SamplingParams
from vllm.sampling_params import GuidedDecodingParams

params = SamplingParams(
    max_tokens=256,
    temperature=0.0,                                  # extraction is not creative
    guided_decoding=GuidedDecodingParams(json=SCHEMA),
)

The result

Measure this yourself

Parse-success rate and per-field accuracy on 1,000 held-out documents, plus the latency cost of the grammar at your batch size. If the grammar costs more than ~10%, your schema is too permissive (unbounded strings, deep nesting) — tighten it.

Gotchas

• The grammar guarantees syntax, never semantics. fault_code: "AA-0000" is valid and wrong. Keep a field-level accuracy metric.
• Train on targets that were validated by the same schema object you decode with. Schema drift between training and serving is a silent quality bug.
• temperature=0 for extraction. Every "our JSON is sometimes wrong" incident starts with someone leaving the chat default at 0.7.

Showcase 3 — A domain VLM for warehouse grounding

The problem

A mobile manipulator must answer "which bin holds the blue calibration fixture, and is its lid open?" from an onboard camera. A general VLM describes the scene beautifully and grounds poorly: it does not know your bin labels, your fixture vocabulary, your lighting, or your camera's mounting angle. Grounding accuracy on your internal set: ~54%.

Why this is rung 4, on the multimodal stack

The vision encoder already sees fine — CLIP-style pretraining generalizes remarkably well. What fails is the mapping from your visual domain into your language of parts, bins, and states. That is the language tower and the projector, and it is exactly what a LoRA can move (Phase 04).

The recipe

# 1. Data: 4,000 (image, question, grounded answer) triples from the real robot camera.
#    Include the hard cases: glare, partial occlusion, an empty bin, a wrong-color decoy.
#    Label with a bounding box or a bin id — a free-text answer is not checkable.

# 2. LoRA on the language tower; freeze the vision encoder to start.
from peft import LoraConfig
peft_config = LoraConfig(
    r=32, lora_alpha=64, lora_dropout=0.05, task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)
# BASE = "Qwen/Qwen2-VL-7B-Instruct"; SFTTrainer over {"images": [...], "messages": [...]}
# If grounding is still weak after this, unfreeze the PROJECTOR next — not the encoder.

# 3. Serve with multimodal vLLM
vllm serve acme/qwen2-vl-7b-warehouse \
  --max-model-len 8192 --limit-mm-per-prompt image=2 \
  --gpu-memory-utilization 0.90 --port 8000

# The Truss: images are big, so cap the payload at the edge
resources:
  accelerator: H100          # 7B VLM + image tokens; an L40S/A100 also works
runtime:
  predict_concurrency: 8     # image prefill is expensive — batch less aggressively

def preprocess(self, model_input):
    img = model_input.get("image_b64")
    if not img:
        raise ValueError("image_b64 is required")
    if len(img) > 8_000_000:                  # ~6 MB decoded; reject at the edge
        raise ValueError("image too large; downscale client-side to 1024px")
    return model_input

The result

Measure this yourself

Grounding accuracy against box/bin-id labels, not string similarity, on a held-out day of footage taken under different lighting. And measure the decoy rate: how often it names a plausible-but-absent object. That is the number that decides whether the robot picks the wrong thing.

Gotchas

• Image tokens dominate the prefill. One 1024×1024 image can cost more tokens than the entire text prompt. Downscale client-side; it is the single biggest latency lever.
• Freeze the vision encoder first. Unfreezing it on 4k images is a fast way to destroy general visual competence you were getting for free.
• Collect the eval set on a different day from the training set. Same-session splits leak lighting and pose and flatter you by 10+ points.
• The robot's safety layer must not trust this model's confidence. Neurosymbolic verification (Phase 14) belongs between the VLM and the actuator.

Showcase 4 — Fine-tuned embeddings: the cheapest RAG win there is

The problem

A RAG assistant over 400k internal engineering documents answers well when the right chunk is retrieved — and recall@10 is 71%. Nearly a third of failures are retrieval failures, and no amount of prompting the generator fixes a chunk that was never retrieved.

Why this is the highest-ROI customization most teams skip

A generic embedding model has never seen your part numbers, your acronyms, or the fact that "TCU" and "thermal control unit" are the same thing. Teaching it that is a 2-hour job on one GPU with a 400 MB model, and it improves every query forever. Compare that to fine-tuning the generator: 10× the cost, 10× the risk, smaller effect.

The recipe

# 1. Mine training triples from what you already have:
#    - positives: (query, chunk) pairs from click/thumbs-up logs, or LLM-generated
#      questions for each chunk;
#    - HARD negatives: the top-k WRONG chunks your CURRENT retriever returns.
#      This is the whole trick. Random negatives are trivially separable and teach ~nothing.
python data/mine_triplets.py --hard-negatives-from-current-index --k 10

# 2. Train (cookbook §5) — 2 epochs, MultipleNegativesRankingLoss, ~2 h on one L4.
# 3. Re-embed the corpus with the new model. This is the operationally awkward part:
#    the index must be rebuilt ATOMICALLY. Query and document vectors from different
#    model versions are not comparable, and a half-migrated index is worse than either.
python reindex.py --model out/bge-support --target new_index --then-swap-alias

# 4. Deploy: it is a small model. Do NOT put it on an H100.
model_name: bge-support-embeddings
resources:
  accelerator: L4
runtime:
  predict_concurrency: 64      # tiny model, batch hard

The result

The second-order effect is the one to say out loud in an interview: better retrieval halves the generator's prompt, so a retrieval win is also a generation cost win.

Measure this yourself

recall@k and MRR@k on a held-out query set with human-labeled relevant chunks — before and after — plus end-to-end answer accuracy. If recall improves but answers do not, your bottleneck was never retrieval, and you have just learned something valuable for free.

Gotchas

• You must re-embed everything. Budget the reindex; do it behind an alias swap so you can roll back by pointing the alias back.
• Never mix vectors from two model versions in one index. Version the index, not the vectors.
• Add a reranker (cross-encoder, Phase 11) before fine-tuning the embedder if you have not already — it is often a bigger win with zero training.
• Hard negatives can be too hard: if a mined "negative" is actually relevant, you are training the model to be wrong. Sample from ranks 5–50, not 1–5.

Showcase 5 — 40 customer models on one GPU

The problem

A B2B product promises "an AI assistant trained on your data." Forty customers have signed. The naive architecture — one fine-tuned model per customer, each on its own replica — needs 40 GPUs, most of them idle most of the day, at roughly $93k/month on H100s. The unit economics do not survive contact with the fortieth customer.

Why multi-LoRA is the answer

A LoRA adapter for an 8B model at r=16 on q,v is 16 MiB — 0.1% of the 16 GB base. Forty of them are 640 MiB, which fits comfortably in the VRAM left over after the base weights and the KV pool. So: one base model resident, forty adapters swapped in the slack, per-request selection. That is Lab 02, and vLLM implements it with batched SGMV/punica kernels.

The recipe

# 1. One training job per customer, same base, same hyperparameters, pinned base revision.
for customer in $(cat customers.txt); do
  python train_lora.py \
    --data "data/${customer}/" \
    --base-revision 8c22764a7e3675c50d4c7c9a4edb474456022b16 \
    --rank 16 --out "adapters/${customer}"
done

# 2. Register each adapter against the base DIGEST — not the base NAME.
#    (Lab 02's soul test: the wrong checkpoint produces fluent garbage, silently.)
python registry/register.py --adapter adapters/acme --version 1.0.0 \
  --base-digest 8c22764a7e3675c50d4c7c9a4edb474456022b16

# 3. Serve: one base, many adapters, per-request routing.
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --revision 8c22764a7e3675c50d4c7c9a4edb474456022b16 \
  --enable-lora \
  --lora-modules acme=/adapters/acme globex=/adapters/globex … \
  --max-lora-rank 16 \
  --max-loras 16 \        # GPU-resident  (AdapterCache budget)
  --max-cpu-loras 64 \    # host spill tier
  --port 8000

# 4. The gateway maps the authenticated tenant to their adapter. NEVER let the client
#    pick the adapter name — that is a cross-tenant data leak with extra steps.
adapter = registry.resolve(f"{tenant.slug}@{tenant.pinned_version}")
resp = client.chat.completions.create(model=adapter.name, messages=msgs)

The result

97.5% cheaper, at the price of a few percent of latency. That trade is the entire design, and being able to state both halves of it is what makes it a senior answer.

Measure this yourself

Adapter cache hit rate and p99 latency as you add tenants, at your real traffic mix. The failure mode is thrash: when the working set of adapters exceeds the VRAM pool, every request pays a swap. Watch for p99 divergence from p50 — that is the tell.

Gotchas

• Tenant isolation is now an application concern. One process holds forty customers' adapters; a routing bug is a data-leak incident. Resolve the adapter from the authenticated session, never from a request field, and test that path explicitly.
• Pin each tenant to an adapter version. "Latest" means a customer's assistant changes behavior on a Tuesday because someone else's retrain finished.
• All adapters must share one base checkpoint. The day you want to upgrade the base, you retrain all forty — plan the migration (dual-serve both bases behind the router, move tenants in waves).
• A very high-QPS tenant should probably graduate to a merged, dedicated deployment. Density and peak throughput are different objectives; serve both architectures.

Showcase 6 — Distill 8B → 1.5B and put it on the robot

The problem

The grounding model from Showcase 3 works, but the robot's local planner needs a language call inside a 100 ms control budget, and the facility's Wi-Fi is not a safety-critical dependency. A cloud round trip to an 8B is 400–700 ms on a good day and unbounded on a bad one.

Why this is rung 6

Quality is already solved — Showcase 3 produced a model you trust. The problem is latency and locality, and the fix is a permanently smaller model, not a better one. Distillation transfers the teacher's behavior (including its soft, "dark knowledge" preferences among wrong answers) into a student small enough to run on the robot (Phase 05).

The recipe

# 1. Harvest teacher outputs on REAL production prompts — the distribution the student
#    will actually face. 200k prompts is cheap when the teacher is already deployed.
python distill/harvest.py --teacher acme/qwen2-vl-7b-warehouse \
  --prompts logs/prod_prompts.jsonl --out data/teacher_outputs.jsonl

# 2. SFT the student on the teacher's outputs (Qwen2.5-1.5B-Instruct).
python train_lora.py --base Qwen/Qwen2.5-1.5B-Instruct \
  --data data/teacher_outputs.jsonl --rank 32 --epochs 3 --out out/student

# 3. Merge, then quantize for the edge.
python merge.py --adapter out/student --out out/student-merged
python convert_hf_to_gguf.py out/student-merged --outfile student-f16.gguf --outtype f16
./llama-quantize student-f16.gguf student-Q4_K_M.gguf Q4_K_M

# 4. Evaluate the student against the TEACHER, not just the ground truth.
#    Agreement rate is the metric that says whether distillation worked.
python evals/agreement.py --student student-Q4_K_M.gguf --teacher acme/…-warehouse

# 5. On the robot (Jetson Orin): an OpenAI-compatible server on the local loopback.
./llama-server -m student-Q4_K_M.gguf -c 2048 --host 127.0.0.1 --port 8080

# 6. The hybrid loop — this is the architecture, not a compromise:
#    fast local model in the control path, big cloud model for planning that can wait.
try:
    action = local.chat(prompt, timeout=0.08)          # 80 ms budget, on-robot
except TimeoutError:
    action = SAFE_FALLBACK                             # never block the control loop
plan = cloud.chat(context, timeout=5.0)                # off the critical path

The result

Measure this yourself

Agreement rate with the teacher on a held-out set, and the p99 on the actual robot hardware under real thermal load — not on your laptop. Jetson throttles; a benchmark taken cold is a benchmark that lies.

Gotchas

• A student cannot exceed its teacher. Fix quality upstream first; distilling a mediocre teacher gives you a fast mediocre model.
• Distill on production prompts, not benchmark prompts. The student only learns the distribution you show it, and its failures off-distribution are ugly.
• Quantization compounds with distillation. Evaluate student-Q4_K_M, not student-merged — the thing you ship is the thing you measure.
• Keep the safe fallback and the timeout. A language model in a control loop must have a bounded, tested failure path (Phase 15).

7. What fine-tuning cannot fix

The most valuable thing a senior engineer contributes to a "let's fine-tune it" meeting is often the reason not to.

The two failure patterns that appear over and over:

1. Fine-tuning a knowledge problem. The model gets more confident about facts it still does not have. This is worse than the original problem.
2. Fine-tuning before evaluating. Without a held-out set and a baseline, "it feels better" is the only report you can write, and you will not be able to defend it.

8. The portfolio version of these projects

If you are building evidence for a Senior AI Engineer interview, this is the smallest set of artifacts that proves the whole stack — and it is genuinely buildable on a personal budget (a few dollars of rented GPU time).

Pick one showcase and go all the way through it. Depth beats breadth here; an interviewer will drill into whichever one you name.

1. A public repo with data/ (build + dedupe + entity split), train_lora.py, merge.py, evals/, truss/ (or modal/), and a Makefile that runs it end to end.
2. A before/after table with your numbers: accuracy, format validity, p50/p95, and $/1M tokens at your measured utilization. Include the run that did not work.
3. A deployed endpoint (scale-to-zero so it costs you nothing at rest) with a two-command curl in the README.
4. An eval gate in CI that fails the build on regression — the artifact that most separates "I fine-tuned a model" from "I ship models."
5. A one-page decision writeup: which rung of §0 you were on and why, what you did not do, and the trade you made. This is the document that reads as senior.

Resume bullets these support, in the language a hiring manager scans for:

• Replaced a frontier-API classification path with a QLoRA-tuned Llama-3.1-8B served on vLLM — 17× fewer prompt tokens, p95 2.5 s → 400 ms, −53% monthly cost — gated by a CI eval suite with a no-regression check against the incumbent.
• Built multi-tenant LoRA serving (one base, 40 adapters, per-request routing with a VRAM-budgeted LRU cache), collapsing a 40-GPU fleet to 1 (−97.5% serving cost) at ~5% latency overhead.
• Distilled a 7B VLM into a 1.5B on-robot student (96% teacher agreement, p99 2.1 s → 95 ms, network-independent) and shipped it as a quantized GGUF with a bounded-timeout safe fallback in the control loop.