Lab 02 — Multi-LoRA Adapter Registry & Hot-Swap Router

Phase: 20 — Custom Models in Production Difficulty: ⭐⭐⭐⭐☆ (the LRU is easy; the digest binding and the merged-vs-dynamic trade are the senior content) Time: 4–5 hours

Phase 05 taught you to train a LoRA adapter. This lab is what happens next, and it is where the money is. A LoRA adapter for Llama-3-8B at r=16 on q,v is 16 MiB — 0.1% of the 16 GB base. So the naive deployment (merge each adapter and give it its own GPU) buys 40 tenants 40 GPUs at ~$93k/month, while keeping one base resident and swapping 16 MiB adapters in its spare VRAM serves the same 40 tenants on one GPU at ~$2.3k/month. You build that machine: a registry that binds every adapter to an exact base-checkpoint digest (apply a LoRA to the wrong checkpoint and you get fluent garbage with no error anywhere), a VRAM-budgeted LRU cache whose budget is a hard wall because overflow is a CUDA OOM that kills every in-flight request, and a router that turns model: "llama-3-8b:support-bot@2.1.0" into a concrete decision — base, dynamic, or dedicated — with an explicit latency model.

What you build

  • parse_version — semver into a comparable tuple, because "1.10.0" < "1.9.0" as a string and a string sort silently routes every unpinned request to a stale adapter.
  • adapter_vram_mb(rank, hidden, layers, targets, bytes) — the size arithmetic: layers · targets · 2 · r · d · bytes. This one number is the whole economic case.
  • Adapter — a frozen record: name, version, base_model, base_digest, rank, vram_mb, merged. Validation rejects a name containing : or @ (they are the ref grammar).
  • AdapterRegistry — register/resolve with semver resolution ("x" → latest, "x@1.2.0" → exact), immutable versions, and the digest binding that rejects an adapter trained against a different checkpoint of the "same" model.
  • AdapterCache(vram_budget_mb, transfer_mb_per_ms) — LRU with a hard budget, an eviction log, hit/miss counters, and a transfer-time model so a cache miss costs measurable milliseconds. An adapter bigger than the budget fails fast rather than thrashing forever.
  • MultiLoRARouter — parse_model_ref → base check → registry resolve → mode decision → RouteDecision(mode, cache_hit, evicted, adapter_load_ms, total_latency_ms). The decision record is your observability surface.
  • fleet_plan(adapters, …) — dedicated-per-tenant vs multi-LoRA: GPUs, monthly bills, savings, savings %.

Key concepts

ConceptWhat to understand
Adapter = delta, not modelΔW = (α/r)·A·B is meaningful only against the exact W it was trained on. Same model name, different checkpoint ⇒ silent quality collapse
Digest bindingthe only place you can catch that mismatch cheaply is registration. There is no runtime error to catch later — the model just gets worse
Semver, not strings1.10.0 vs 1.9.0; also why a bare name resolving to "latest" needs a deliberate policy (prod should pin)
Merged vs dynamicmerged = fold ΔW into W ⇒ zero per-token overhead, but a full model copy, its own replica, no hot-swap. Dynamic = keep A,B separate ⇒ a few % overhead via batched SGMV/punica kernels, but N tenants on one base
Spare VRAM is the budgettotal VRAM − base weights − KV-cache pool = what adapters may use. Sizing the KV pool and the adapter pool against each other is the real capacity decision
LRU vs FIFOevict least-recently-used: a hot adapter touched a thousand times must outlive one loaded earlier and never used again
OOM is not an exceptionexceeding the VRAM budget in production kills the process and every in-flight request. The invariant must hold at every point, not on average
Cache hit ⇒ zero transferthe miss cost is a host→device copy (~2 ms for 16 MiB at 8 GB/s). At 40 tenants and a small budget you can thrash; the fix is a bigger pool or tenant-affinity routing
Tenant affinityrouting a tenant's requests to the replica that already holds their adapter is the multi-LoRA equivalent of cache locality — the natural extension of this lab

Files

FilePurpose
lab.pyskeleton with # TODO markers — your implementation
solution.pycomplete reference; python solution.py runs a worked example
test_lab.pythe proof — run it red, make it green (62 tests)
requirements.txtpytest only (pure stdlib otherwise)

Run

pip install -r requirements.txt
pytest test_lab.py -v                       # against your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # against the reference
python solution.py                          # the worked example

Success criteria

  • All 62 tests pass against your implementation (and LAB_MODULE=solution).
  • You can explain why test_registry_rejects_wrong_base_digest is a soul test and describe exactly how the failure presents in production (no exception, no 5xx — just a model that quietly stops being good, discovered by an eval or a customer).
  • You can explain why test_cache_never_exceeds_budget asserts the invariant inside the loop rather than at the end.
  • You can explain test_cache_evicts_least_recently_used_not_oldest_inserted and why a FIFO would be wrong for this workload.
  • You can defend test_route_merged_adapter_is_dedicated_with_zero_overhead: state the trade (latency vs. density), and say which one you'd pick for (a) one high-QPS flagship model, (b) 300 low-QPS per-customer adapters.
  • You can compute, from memory, the VRAM of an r=32 adapter on a 70B model targeting q,k,v,o and say how many fit in 8 GB of spare VRAM.
  • You can quote the fleet_plan headline: 40 tenants, H100 @ $3.20/hr → 97.5% cheaper, and explain the two assumptions that could break it (adapter size, per-tenant QPS forcing more replicas anyway).

How this maps to the real stack

The miniatureThe production mechanismWhere to verify it
Adapter + adapter_vram_mba PEFT adapter directory: adapter_config.json (r, lora_alpha, target_modules, base_model_name_or_path) + adapter_model.safetensorsls a trained adapter; the file size is adapter_vram_mb
base_digest bindingadapter_config.json's base_model_name_or_path — which is a name, not a digest, and is exactly why this class of bug shipspin base weights by commit sha in your registry; see Lab 01's model_cache.revision
AdapterRegistryvLLM's --lora-modules name=path (static) or the runtime LoRA API; an MLflow/W&B model registry holding adapters as versioned artifactsvllm serve BASE --enable-lora --lora-modules support=/adapters/support
MODE_DYNAMICvLLM multi-LoRA: batched SGMV/punica kernels apply per-request adapters inside one batchvLLM --max-loras, --max-lora-rank, --max-cpu-loras; request model: "support"
AdapterCache budgetvLLM's --max-loras (GPU-resident) and --max-cpu-loras (host-resident spill)watch swap counts in vLLM's metrics under adapter churn
MODE_DEDICATEDpeft's merge_and_unload() then serve as an ordinary modelmerge, save, and serve — compare tokens/s against the dynamic path
RouteDecisionthe gateway that maps a request's model field to a backend deploymentBaseten deployment/environment routing; an OpenAI-compatible gateway; LiteLLM router
fleet_planthe capacity/cost review you present before building a multi-tenant productyour cloud bill vs. nvidia-smi utilization on the per-tenant fleet

Limits of the miniature (be honest in the interview): real multi-LoRA overhead is not a flat percentage — it depends on rank, how many distinct adapters appear in a batch, and the kernel (a batch with 8 different adapters costs more than 8 requests on one). We ignore CPU↔GPU adapter spill (--max-cpu-loras), so a real system has a second cache tier under this one. Merged vs. dynamic is modeled as a per-adapter flag; in practice it is a fleet-level decision with a serving-topology consequence. And fleet_plan assumes QPS never forces extra replicas — at high per-tenant load the dedicated fleet's GPUs are not idle and the savings shrink toward the utilization ratio.

Extensions (build these for real)

  • Do it for real: train two tiny LoRAs with peft, then vllm serve <base> --enable-lora --lora-modules a=/path/a b=/path/b and hit it with model: "a" and model: "b". Measure tokens/s vs. the merged model.
  • Add a CPU spill tier: a second, larger cache with a slower transfer rate, and measure how it changes p99 under adapter churn.
  • Add tenant-affinity routing across R replicas: hash the adapter key to a replica and show the cache hit rate versus round-robin.
  • Add a canary: route 5% of a tenant's traffic to support-bot@2.0.0 and the rest to 1.10.0, and reuse Phase 17 Lab 03's CanaryController to promote or abort.
  • Make the overhead model honest: charge more when a batch contains many distinct adapters, and see how it changes the merged-vs-dynamic break-even.
  • Add eviction-aware admission: refuse to evict an adapter with in-flight requests.

Interview / resume

  • Talking points: "How do you serve 200 customer-specific fine-tunes without 200 GPUs?" "When do you merge an adapter and when do you keep it dynamic?" "What happens if you apply a LoRA to the wrong base checkpoint, and where do you catch it?" "How do you size the adapter VRAM pool against the KV-cache pool?" "Your p99 spiked after onboarding customer #41 — what happened?"
  • Resume bullet: Designed and implemented a multi-tenant LoRA serving layer — a semver'd adapter registry with base-checkpoint digest binding, an LRU adapter cache with a hard VRAM budget and eviction telemetry, and a request router resolving base:adapter@version into merged/dynamic serving decisions with an explicit latency model — reducing a 40-tenant fleet from 40 GPUs to 1 (−97.5% serving cost); verified by a 62-test deterministic suite.