Fine-Tuned Model Deployment

Phase 13 · Document 07 · Fine-Tuning and Adaptation Prev: 06 — Dataset Quality · Up: Phase 13 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

A trained model that lives in a checkpoint folder produces zero value — fine-tuning only pays off once the model is served, versioned, evaluated, and maintained in production. And this is where most fine-tuning projects actually fail: not in training, but in the operational burden — serving the weights/adapters efficiently, versioning so you can roll back, eval-gating every new fine-tune, and carrying the ongoing maintenance (re-tuning as the base model and data drift) that you signed up for in 00. This doc closes Phase 13 by connecting it to the serving stack (Phases 6/7) and the eval discipline (Phase 12): how to actually ship and live with a fine-tune.

2. Core Concept

Plain-English primer: shipping a fine-tune is an ops problem

Once you've trained an adapter or a full fine-tune, you have to serve it so requests hit your model instead of the base. The choices depend on how you trained (LoRA adapter vs full fine-tune) and where you run (managed API vs self-hosted vs local). Then — because a fine-tune is a living asset — you must version it, eval-gate it, monitor it, and re-tune it over time. The model is the easy part; the lifecycle is the work.

Two serving shapes: adapter vs merged

This is the central deployment decision, inherited from 02:

• Keep as a LoRA adapter — serve the frozen base + a small adapter loaded at runtime. Enables multi-LoRA serving: one base model in memory hosting many adapters (one per task/customer), hot-swapped or even batched together per request. Massive efficiency for many fine-tunes (50 tasks = 1 base + 50 tiny adapters, not 50 full models). vLLM supports this directly.
• Merge into the base — fold the adapter into the weights (W' = W + A·B, 02) to get a standalone model. Simpler single-model serving, no adapter-loading overhead, and you can export to GGUF for local serving (Phase 6.03). But you lose swappability — one model per fine-tune.

KEEP-ADAPTER: base + N small adapters → multi-LoRA serving (one base, many tasks, hot-swap/batch) — efficient for MANY fine-tunes
MERGE: W' = W + A·B → standalone model (simpler serving, GGUF-exportable [6.03]) — but ONE model per fine-tune

Three deployment surfaces

1. Managed/API fine-tuning (OpenAI, etc.) — you upload data, they train and host; you call a fine-tuned model ID. Easiest ops (no infra), but vendor-locked, opaque, and you pay their serving premium. Often the right first step.
2. Self-hosted open-weight — you serve the merged model or base+adapter on your own vLLM/TGI stack (Phase 7.01/7.02) — full control (multi-LoRA, quantization, routing), but you own the GPUs and the ops.
3. Local — merge → quantize (Phase 6.06) → GGUF (Phase 6.03) for on-device/edge serving (privacy, no per-token cost), at the cost of managing distribution and hardware (Phase 6).

The lifecycle: version, gate, monitor, re-tune

A fine-tune is not "train once." Treat it like any production model artifact:

• Versioning & registry — every fine-tune is a versioned artifact (model + adapter + the dataset version + training config + base-model version). You must be able to identify and roll back to any prior version. Reproducibility matters: which data + base produced this model?
• Eval gate (CI for models) — no fine-tune ships without passing the eval harness (Phase 12.08): quality up on target tasks, no regressions on general ability, safety gate passed (Phase 12.07). This is the regression gate from Phase 12 applied to your models.
• Rollout — canary / shadow / A-B the new fine-tune against the current one before full traffic (Phase 7.07); keep a fallback (previous version or base) for failures.
• Monitoring — track quality, latency, cost, and drift in production (Phase 7.08/7.09); capture errors for the data-centric loop (06).
• The maintenance burden (the part people underestimate) — the base model improves (a new base may beat your fine-tune for free, 00); your data distribution drifts; you must periodically re-tune, re-eval, re-deploy. Every fine-tune is a standing commitment, which is exactly why 00 says try prompting/RAG first.

Cost & efficiency at serving

• Multi-LoRA is the big win for serving many fine-tunes cheaply (shared base) — vs N full models eating N× the memory.
• Quantize the deployed model (Phase 6.06) for cheaper/faster serving (compose with distillation, 04).
• Right-size: a small fine-tuned/distilled model can replace an expensive base call (04) — track cost per resolved task (Phase 12.06).

When NOT to deploy a fine-tune

If eval shows the fine-tune doesn't beat a good prompt / RAG / a newer base model (00), don't ship it — you'd take on the maintenance burden for no gain. The eval gate is also a kill switch.

3. Mental Model

   a checkpoint = ZERO value until SERVED + VERSIONED + EVAL-GATED + MAINTAINED. Training is easy; the LIFECYCLE is the work.

   SERVING SHAPE [02]: KEEP-ADAPTER → multi-LoRA (one base + N adapters, hot-swap/batch) — efficient for MANY fine-tunes
                       MERGE (W+A·B) → standalone (simpler, GGUF [6.03]) — one model per fine-tune
   SURFACE: managed API (easy ops, vendor-lock) · self-host vLLM/TGI [7.01/7.02] (control + you own GPUs/ops) · local GGUF [6.03/6.06] (privacy/edge)

   LIFECYCLE: VERSION/registry (model+adapter+DATA version+base version → rollback/repro) →
              EVAL GATE [12.08] (quality↑, NO regressions, SAFETY [12.07]) → ROLLOUT canary/shadow/A-B + fallback [7.07] →
              MONITOR quality/latency/cost/DRIFT [7.08/7.09] → DATA-CENTRIC loop [06] → RE-TUNE (base improves, data drifts)
   ★ MAINTENANCE BURDEN: every fine-tune is a STANDING commitment (new base may beat it free [00]) → DON'T ship if it doesn't beat prompt/RAG/new base
   COST: multi-LoRA (many tunes cheap) · QUANTIZE deployed [6.06] · right-size/distill [04] · cost per resolved task [12.06]

Mnemonic: a fine-tune is worth nothing until it's served, versioned, eval-gated, and maintained. Keep-adapter (multi-LoRA, many tasks) vs merge (standalone/GGUF). The eval gate is the ship/kill switch — and the maintenance burden never ends.

4. Hitchhiker's Guide

What to look for first: adapter vs merged (do you serve many fine-tunes → multi-LoRA, or one → merge?) and which surface (managed/self-host/local). Then: does it pass the eval gate vs the current model and the base (Phase 12.08)?

What to ignore at first: premature self-hosting. Managed fine-tuning is the fastest path to ship; move to self-hosted (vLLM multi-LoRA) when control/cost/scale demand it.

What misleads beginners:

• "Training done = shipped." Serving, versioning, eval-gating, and maintenance are the real work.
• Merging when you needed swappability. Many fine-tunes → keep adapters (multi-LoRA); one → merge (02).
• Shipping without an eval gate. Always gate vs current + base; no regressions/safety failures (Phase 12.07/12.08).
• Ignoring the maintenance burden. The base improves and data drifts — you re-tune forever (00).
• No versioning/rollback. You must reproduce and roll back any fine-tune (track data + base versions).

How experts reason: they pick adapter vs merge by how many fine-tunes they serve, choose the simplest surface that meets control/cost needs (managed → self-host vLLM multi-LoRA → local GGUF), treat each fine-tune as a versioned artifact (model + data + base versions), eval-gate every deploy (Phase 12.08) with canary/fallback (Phase 7.07), monitor drift, run a data-centric re-tune loop (06), and kill fine-tunes that no longer beat prompting/RAG/a newer base (00).

What matters in production: serving efficiency (multi-LoRA/quantization), the eval gate (quality/regressions/safety), versioning/rollback, drift monitoring, and whether the fine-tune still earns its maintenance cost.

How to debug/verify: new fine-tune worse → eval gate catches it (don't ship); serving too costly → multi-LoRA + quantize (Phase 6.06); can't roll back → missing versioning; quality decayed over months → drift, re-tune; "base now beats it" → retire the fine-tune (00).

Questions to ask: adapter or merged? managed/self-host/local? does it pass the eval gate vs current + base? can I version/roll back? am I monitoring drift? does it still beat prompt/RAG/new base — or should I retire it?

What silently gets expensive/unreliable: N full models instead of multi-LoRA, shipping without eval gates (regressions/safety), no rollback, ignored drift, and maintenance debt from fine-tunes that no longer earn their keep.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• A fine-tune produces value only once served, versioned, eval-gated, and maintained — training is the easy part; the lifecycle is the work.
• Serving shape: keep-adapter (multi-LoRA — one base, many adapters, hot-swap/batch) vs merge (standalone, GGUF-exportable) (02/Phase 6.03).
• Surfaces: managed API (easy ops, vendor-lock), self-hosted vLLM/TGI (control + own GPUs), local GGUF (privacy/edge) (Phase 7/Phase 6).
• Version every fine-tune as an artifact (model + adapter + dataset version + config + base-model version) for rollback and reproducibility.
• Eval-gate every deploy (Phase 12.08): quality up, no regressions, safety passed (Phase 12.07) — the gate is also a kill switch.
• Roll out with canary/shadow/A-B + fallback (Phase 7.07); monitor quality/latency/cost/drift (Phase 7.08).
• The maintenance burden is the underestimated cost — bases improve (may beat your fine-tune free) and data drifts; you re-tune forever (00).
• Multi-LoRA + quantization (Phase 6.06) are the serving cost levers; retire a fine-tune that no longer beats prompt/RAG/a newer base.

9. Observations from Real Systems

• vLLM multi-LoRA / S-LoRA let platforms serve hundreds–thousands of adapters on a shared base — the economics that make per-customer fine-tunes viable (02).
• Managed fine-tuning (OpenAI et al.) is how most teams ship first — a fine-tuned model ID with no infra, traded against vendor lock-in and serving premium.
• Merging → GGUF is the standard path to ship a fine-tune for local/edge serving (Phase 6.03).
• The classic failure is shipping without an eval gate and discovering regressions/safety issues in production (Phase 12.08).
• Fine-tunes get orphaned — a newer base model quietly outperforms a months-old fine-tune, but nobody re-evaluated; the maintenance loop is what prevents this (00).

10. Common Misconceptions

11. Engineering Decision Framework

DEPLOY A FINE-TUNE:
 1. SHAPE [02]: serving MANY fine-tunes? → keep-adapter (multi-LoRA). Serving ONE / want simplest / local? → merge (→ GGUF [6.03]).
 2. SURFACE: fastest ship / no infra → managed API. Need control/cost/scale → self-host vLLM/TGI [7.01/7.02]. Privacy/edge → local GGUF [6.06].
 3. VERSION: register model + adapter + DATA version + config + BASE version (rollback/repro).
 4. EVAL GATE [12.08]: quality↑ on target, NO regressions, SAFETY pass [12.07] — else DON'T ship (kill switch).
 5. ROLLOUT: canary/shadow/A-B vs current + FALLBACK to prev/base [7.07].
 6. MONITOR quality/latency/cost/DRIFT [7.08/7.09]; capture errors → data-centric loop [06].
 7. RE-TUNE on drift; RETIRE if a prompt/RAG/newer base now beats it [00].

12. Hands-On Lab

Goal

Deploy a fine-tune end-to-end: choose adapter vs merge, serve it, eval-gate it vs the base, and set up versioning + a re-tune trigger — feeling the full lifecycle, not just training.

Prerequisites

• A trained LoRA adapter (02); a serving stack (vLLM locally or a managed FT endpoint); the eval harness (Phase 12.08); a clean held-out eval set (Phase 12.01).

Steps

1. Decide shape: if you'll serve multiple adapters → keep-adapter (multi-LoRA); if one → merge (02). Justify the choice.
2. Serve: load base + adapter in vLLM with multi-LoRA (or merge and serve standalone; or merge→GGUF for local, Phase 6.03). Confirm requests hit the fine-tune.
3. Multi-LoRA demo (if keep-adapter): load two adapters on one base; route requests to each — one base, many behaviors.
4. Eval gate: run the harness — fine-tune vs base vs (if any) current model: quality on target tasks, regression check, safety pass (Phase 12.07/12.08). Decide ship / don't ship.
5. Version: record the artifact (model/adapter + dataset version + config + base version); demonstrate rollback to the base.
6. Rollout + monitor: canary a fraction of traffic with a fallback (Phase 7.07); log quality/latency/cost (Phase 7.08). Define a drift/re-tune trigger and a retire-if-base-wins check (00).

Expected output

A served fine-tune (adapter or merged) that passed an eval gate vs the base, is versioned with rollback, runs behind a canary + fallback, and has a monitoring + re-tune/retire plan — the full deployment lifecycle.

Debugging tips

• Serving too costly → multi-LoRA + quantize (Phase 6.06).
• Fine-tune fails the gate → don't ship; back to data (06).

Extension task

Serve N adapters on one base (multi-LoRA) and measure memory vs N standalone models; compare merged-GGUF local serving vs hosted.

Production extension

Wire the data-centric loop (06): production errors → new examples → re-tune → eval-gate → canary; add a scheduled re-eval vs the latest base to catch "base now wins" (00).

What to measure

Quality vs base/current (gate), regression/safety pass, serving cost/latency (multi-LoRA vs standalone), rollback works, drift signal.

Deliverables

• A served fine-tune (adapter or merged) with an adapter-vs-merge justification.
• An eval-gate result (ship/don't-ship) vs base + safety.
• A versioned artifact + rollback demo.
• A rollout (canary/fallback) + monitoring + re-tune/retire plan.

13. Verification Questions

Basic

1. Why is a trained checkpoint worth nothing until deployed?
2. When do you keep an adapter (multi-LoRA) vs merge it?
3. What are the three deployment surfaces?

Applied 4. What must a versioned fine-tune artifact record for rollback/reproducibility? 5. Why is the eval gate also a kill switch?

Debugging 6. Serving 30 per-customer fine-tunes is too expensive. What do you change? 7. A months-old fine-tune underperforms. What two checks do you run?

System design 8. Design the deploy pipeline for fine-tunes: version → eval-gate → canary → monitor → re-tune.

Startup / product 9. How does multi-LoRA change the cost story for offering per-customer fine-tunes, and what ongoing burden must you budget for?

14. Takeaways

1. A fine-tune pays off only when served, versioned, eval-gated, and maintained — training is the easy part.
2. Keep-adapter (multi-LoRA, many fine-tunes) vs merge (standalone/GGUF, one) is the core serving choice (02/Phase 6.03).
3. Version every fine-tune (model + data + base versions) for rollback/reproducibility; eval-gate every deploy (Phase 12.08) — the gate is also a kill switch.
4. Roll out with canary + fallback; monitor drift; run a data-centric re-tune loop (06/Phase 7).
5. The maintenance burden never ends — re-tune as bases/data drift, and retire fine-tunes a newer base/prompt/RAG now beats (00).

15. Artifact Checklist

• A served fine-tune (adapter via multi-LoRA, or merged/GGUF) with a justified choice.
• An eval-gate result (vs base + safety) → ship/don't-ship.
• A versioned artifact (model + data + base versions) with rollback.
• A canary + fallback rollout and monitoring for drift.
• A re-tune / retire plan (data-centric loop + "base now wins" check).

Up: Phase 13 Index · Phase 13 complete → next phase: Phase 14 — Security and Governance