m07 — Multi-Adapter (LoRA) Serving

A fully worked design. Thousands of customer-fine-tuned adapters over one set of base weights, batched together in a single decode step.

The number that decides everything: an adapter is 0.05%–1.2% of the base model, so it can be paged in per request in ~2 ms where the base weights would take 2.2 seconds. That ratio is the entire reason this product can exist.

And the number that decides whether it works: at batch 128 with distinct adapters, an attention-only rank-8 adapter costs +6% decode bandwidth; an all-modules rank-64 adapter costs +151%. A research-side hyperparameter choice, made without reference to serving, changes the cost of the product by 25×.

Table of Contents

• The Prompt
• 1. Requirements and Scope
• 2. Scale Numbers
• 3. API Surface
• 4. Data Model
• 5. High-Level Architecture
• 6. Deep Dive A: Batching Heterogeneous Adapters
• 7. Deep Dive B: The Cold-Start Long Tail
• 8. Failure and Recovery
• 9. Bottlenecks and Evolution
• 10. Tradeoffs Explicitly Rejected
• The Hostile Critique
• The Revision
• References

The Prompt

"We want to let customers fine-tune the model on their own data and then serve it. We expect thousands of fine-tunes. We obviously can't give each one its own GPUs."

"We obviously can't give each one its own GPUs" is the requirement, and the arithmetic behind it is worth stating immediately: a 4×H100 replica is $88,000/year (m01 §1), and a customer fine-tune might serve ten requests a day. Dedicated serving is off by three orders of magnitude.

The thing that makes the product possible: a LoRA adapter is a low-rank delta, not a model. It is 65 MB to 1.7 GB against 140 GB of base weights (§2) — 0.05% to 1.2%.

That single ratio produces the design:

• Base weights load once and stay resident. 2.2 s to load, so never per-request.
• Adapters load in 1–26 ms, so they can be per-request.
• Therefore: one base model, many adapters, batched together in the same forward pass.

And the thing to flag in the first two minutes, because it is the design's real risk and it lives outside the design's control: the adapter's rank and target modules are chosen by whoever runs the fine-tune, and from §2 that choice moves serving cost by 25×. A serving design that does not constrain its inputs is not a design, it is a hope.

1. Requirements and Scope

Clarifying questions asked

"How many adapters, and what's the traffic distribution?" Assumed 5,000 adapters, with the usual brutal skew: top 50 adapters ≈ 80% of traffic; the bottom 3,000 see fewer than 10 requests/day. That skew is the design — the head is a caching problem and the tail is a cold-start problem, and they need different mechanisms (§6, §7).

"Do we control the fine-tuning, or do customers upload arbitrary adapters?" The most important question. Assumed we run the fine-tuning, which lets us constrain rank and target modules. If customers upload arbitrary adapters, §2's 25× cost spread becomes the customer's choice and the economics are unpredictable — worth saying explicitly, because the answer changes the design substantially and it is exactly the kind of constraint that gets discovered after launch.

"Same base model for everyone?" Assumed one base per model family and version. Adapters are bound to an exact base checkpoint — a LoRA trained against weights version A is not valid against version B, and applying it does not error, it just degrades quality silently. Same class of failure as m02's cache key.

"What's the SLO for a cold adapter?" Assumed TTFT p95 < 1 s warm, < 2 s cold. The cold path must be bounded, not fast — and the honest framing is that the tail's SLO is different from the head's, which the API should reflect rather than hide.

"Can adapters be merged into the base?" Yes, and it is the right answer for very high-traffic adapters — a merged model is a full-speed dedicated deployment. The design should support promotion to merged, since it is the natural end state for the head of the distribution.

Functional

1. Serve model=base:adapter_id with the same API as the base model.
2. Batch requests using different adapters in one forward pass.
3. Load adapters on demand; evict under memory pressure.
4. Version adapters; pin them to a base checkpoint.
5. Promote hot adapters to merged deployments.

Non-functional

Explicitly out of scope

• The fine-tuning process itself — different system; here we consume its artifact and constrain its shape (§1's key question).
• Full-weight fine-tunes. They are separate models, not adapters, and need their own replicas.
• The base serving platform — m01.
• Quality evaluation of adapters — m05.

2. Scale Numbers

Adapter size, computed rather than quoted. For a weight W of shape (d_out, d_in), LoRA adds B·A with A: (r, d_in) and B: (d_out, r) — so r × (d_in + d_out) parameters. For 70B (80 layers, d=8192, GQA kv-dim 1024, FFN 28672):

A 25× spread, entirely from configuration. Adding the MLP projections is the bigger factor — they are 3.2× the attention parameters at equal rank, because the FFN dimension is 3.5× the hidden dimension.

Load latency, which is the enabling fact:

Three orders of magnitude. The base model must be resident; adapters need not be. That asymmetry is the product.

Now the number that governs whether batching works. In a decode step, the base weights are read once and amortized across the whole batch. Adapters are not — B sequences using B distinct adapters require reading B adapters:

(Extra bytes read per decode step, as a percentage of the base weight read. Decode is memory-bound, so extra bytes ≈ extra time.)

Read the last row again. At batch 128 with all-modules rank-64 adapters, the adapters cost more bandwidth than the entire base model — the request is now majority-adapter, and the whole economic premise has inverted.

Against the <10% throughput-penalty requirement from §1, the viable operating envelope is essentially:

attention-only, rank ≤ 16, at batch ≤ 64.

That is a constraint on the fine-tuning team, discovered by the serving team, and it must be enforced in the platform — §3 does it in the API. This is the single most valuable thing to produce in this round: a serving-side constraint on a research-side parameter, derived from arithmetic.

HBM budget. Against the 172.5 GiB KV budget of a 4×H100 replica (m01 §2) — and adapters take that space from the KV cache, i.e. from batch capacity:

100 resident adapters, attn r=8   =  6.4 GB =  3.5% of KV budget
100 resident adapters, attn r=16  = 12.8 GB =  7.1%
100 resident adapters, all r=64   =  154 GB = 89.5%    <- destroys the batch

Every adapter you keep resident is batch capacity you gave up. The cache-vs-workload tension is identical to m02 — the cache is made of the resource it is caching for — and recognizing the same structure twice is worth saying.

3. API Surface

POST /v1/chat/completions   { model: "llama-70b-v3:acme-support-v2", ... }
                                     └── base ──┘ └──── adapter ────┘

POST /v1/adapters           { name, base_model, artifact_uri, rank, target_modules[] }
  -> 201 {adapter_id, status: "validating"}
  -> 400 {error: "rank 64 with target_modules including MLP exceeds the serving
                  envelope: projected +75.7% decode cost at batch 64.
                  Allowed: rank<=16 attention-only, or rank<=8 all-modules."}

GET  /v1/adapters/{id}      -> {status, base_model, size_bytes, projected_cost_pct,
                                traffic_7d, tier: cold|warm|hot|merged}
POST /v1/adapters/{id}/promote  -> merged dedicated deployment

Three decisions:

The 400 with an arithmetic explanation is the most important thing in this API. From §2, an unconstrained adapter shape can make serving 25× more expensive, and the person choosing the rank has no visibility into that. Rejecting it at registration, with the projected cost and the allowed envelope, moves the constraint to where it can be acted on. A platform that accepts any adapter and then struggles is a platform that has outsourced its economics to people who cannot see them.

The model string embeds the adapter, so every existing client works unchanged and routing has one field to parse. It also means the adapter is part of the cache key everywhere downstream — the prefix cache (m02), the metrics, the rate limits — which is correct, and falls out for free from putting it in the identifier rather than in a header.

tier is exposed. Customers on the cold tier get a different TTFT and should be told, not left to discover an inconsistent p95. Exposing the tier is what makes the two-SLO design honest rather than a hidden inconsistency.

4. Data Model

adapter    (adapter_id, tenant_id, name, version, base_model_digest,
            rank, target_modules[], artifact_uri, size_bytes, sha256,
            projected_cost_pct, status, created_at)
placement  (adapter_id, replica_id, tier: hbm|host|remote, loaded_at, last_used)
traffic    (adapter_id, hour, requests, tokens)         -- drives tiering and promotion
merged     (adapter_id, deployment_id, merged_at)       -- promoted adapters

base_model_digest is an exact artifact hash, not a name. An adapter trained against llama-70b-v3.1 applied to v3.2 produces degraded output with no error — the shapes match, the math runs, the quality quietly drops. This is the same silent-corruption class as m02's cache keys, and the same defence: exact digest, refuse to mix.

projected_cost_pct is computed at registration and stored. It is what the API rejects on, what capacity planning sums over, and what makes "which adapters are expensive" a query instead of an investigation.

placement is per-replica, per-tier. The scheduler needs to know which replicas already have an adapter resident, because routing to a warm replica saves the load entirely — this is m02's cache-aware routing again, with adapters instead of KV blocks.

traffic at hourly granularity drives tiering automatically, and per m02's R4 the tier must be inferred from observed use, never declared by the customer. Every customer believes their adapter is important.

5. High-Level Architecture

      request: model = "llama-70b-v3:acme-support-v2"
                    │
        ┌───────────▼─────────────────────────────────────────┐
        │ ROUTER                                               │
        │   parse base:adapter                                 │
        │   prefer replicas with the adapter ALREADY RESIDENT  │
        │   (adapter affinity, bounded by KV headroom)         │
        └───────────┬─────────────────────────────────────────┘
                    │
        ┌───────────▼─────────────────────────────────────────┐
        │ REPLICA (base weights resident, 140 GB)              │
        │                                                      │
        │  ADAPTER CACHE                                       │
        │    HBM   ~200 adapters   (LRU + frequency)           │
        │    host  ~5,000 adapters (DRAM, 2 ms to promote)     │
        │    remote  all           (object store, ~100 ms)     │
        │                                                      │
        │  SCHEDULER: continuous batching, but the batch is    │
        │    ADAPTER-AWARE -- see deep dive A                  │
        │                                                      │
        │  FORWARD PASS                                        │
        │    base GEMM  (shared by the whole batch)            │
        │    + BGMV/SGMV kernel: per-sequence adapter apply    │
        └──────────────────────────────────────────────────────┘

  SIDE: promotion job -- hot adapters merged into dedicated deployments

Five decisions:

1. Adapters are a three-tier cache, and the tiers are chosen by the same break-even logic as m02. Here it is trivially satisfied: a 65 MB adapter loads in 1 ms from host DRAM against a decode step of 24 ms. Host DRAM is essentially free; object storage at ~100 ms is not, and that is the cold path §7 is about.

2. Router prefers replicas that already hold the adapter. Turns a 100 ms cold load into 0 for most requests. Bounded by KV headroom so affinity cannot overload a replica — m02's R2 established that affinity is required for a cache to work at all, and the same applies here.

3. The batch scheduler is adapter-aware, not adapter-blind. §6 — this is where the 6%-vs-151% from §2 is actually won or lost.

4. A custom kernel (BGMV/SGMV) applies per-sequence adapters inside one batched forward. Without it, heterogeneous batching is impossible and you are back to one adapter per batch, which destroys throughput. The kernel is the enabling technology, and naming it specifically — rather than saying "we batch them" — is what shows you know how this actually works.

5. Promotion to merged for the head. The top 50 adapters are 80% of traffic; merging removes their per-step overhead entirely. The head and the tail get different architectures, and that is the correct response to an 80/20 distribution rather than one mechanism stretched over both.

6. Deep Dive A: Batching Heterogeneous Adapters

The mechanism

A LoRA forward is y = Wx + (B·A)x·(α/r). The base term is a big GEMM shared by everyone in the batch. The adapter term is per-sequence — different A, B per row.

naive: loop over sequences, apply each adapter separately
       -> B tiny GEMMs, terrible GPU utilization, kills continuous batching

right:  ONE batched kernel that gathers each sequence's adapter and applies it
        BGMV — batched gather matrix-vector (decode: one token per sequence)
        SGMV — segmented gather matrix-vector (prefill: variable-length segments)

The kernel takes the batch, an index vector mapping sequence → adapter slot, and a contiguous adapter buffer, and does the gather inside the kernel rather than by materializing per-sequence weights. This is the S-LoRA/Punica contribution and it is what makes the product feasible.

The cost is bandwidth, not compute

Compute added by LoRA is negligible and it is worth showing why, because the intuition points the wrong way:

base FLOPs per token per projection = 2 · d_in · d_out             = 2 · 8192 · 8192
LoRA FLOPs                          = 2 · r · (d_in + d_out)       = 2 · 16 · 16384
ratio = r(d_in + d_out) / (d_in · d_out) = 16 · 16384 / 8192²      = 0.39%

0.39% more compute. Irrelevant — and if you stop the analysis there, LoRA looks free.

The bandwidth is the whole cost, and it is entirely a function of distinctness. From §2:

base weights:  read once per step, amortized across the WHOLE batch
adapters:      read once per DISTINCT adapter in the batch, amortized across
               only the sequences using it

So the metric that matters is distinct_adapters in the batch, not batch_size. A batch of 64 sequences all using one adapter costs one adapter read; 64 sequences on 64 adapters costs 64. The scheduler's job is to make batches that share adapters.

The adapter-aware scheduler

def form_batch(queue, max_batch, max_distinct_adapters):
    # Group the queue by adapter and take groups whole where possible.
    by_adapter = group_by(queue, key=lambda r: r.adapter_id)
    order = sorted(by_adapter, key=lambda a: -len(by_adapter[a]))   # biggest groups first

    batch, distinct = [], 0
    for adapter in order:
        if distinct >= max_distinct_adapters and len(batch) >= MIN_BATCH:
            break                       # stop admitting NEW adapters, keep the batch
        take = by_adapter[adapter][: max_batch - len(batch)]
        batch += take
        distinct += 1
        if len(batch) >= max_batch:
            break
    return batch

max_distinct_adapters is the knob that makes §2's table actionable. Cap it at 32 and the worst case is bounded regardless of traffic shape:

Two mechanisms, deliberately layered: the API bounds the adapter shape, the scheduler bounds the adapter count. Either alone leaves a hole — a permissive API with a good scheduler still gets 38% overhead, and a strict API with a blind scheduler still degrades at high adapter diversity.

The fairness cost, which must be stated

Grouping by adapter is not FIFO. A request for a rare adapter can be passed over repeatedly in favour of large groups. Left alone, that is starvation for exactly the long-tail customers §7 is about.

The guard:

# Age-based override: a request older than a deadline is admitted regardless
# of grouping, and it BRINGS ITS ADAPTER with it.
urgent = [r for r in queue if r.waited_ms > MAX_QUEUE_MS]     # e.g. 200 ms
batch = urgent + fill_by_grouping(queue - urgent, ...)

Throughput optimization must always carry a fairness deadline, and the general form of the rule is worth stating: any scheduler that reorders for efficiency needs an age term, or the least efficient work never runs. Same structure as d05's reserved floors and m03's aging.

Prefill is different, and worse

Decode reads adapters once per step. Prefill reads them once per chunk of tokens, and a 4,000- token prefill using an adapter nobody else in the batch uses pays the full adapter read for one sequence.

So prefill batching should group by adapter even more aggressively than decode — and the combination with chunked prefill means the chunks of one prefill should stay together in adapter terms, which they naturally do.

7. Deep Dive B: The Cold-Start Long Tail

The distribution is the problem

From §1: 5,000 adapters, top 50 = 80% of traffic, bottom 3,000 = under 10 requests/day.

2% of traffic pays 100 ms. That is invisible in the mean and it is the entire p99 — and if the tail is where your newest customers live (it is: a new adapter starts cold), then the worst experience in the product is reserved for people evaluating it.

Why the cold path is 100 ms and not 2 ms

object storage GET, 65 MB at ~1 GB/s effective    =  65 ms
+ TLS, request overhead, first-byte latency        =  20 ms
+ HBM copy                                         =   1 ms
                                                    ~86-100 ms

Against a 1 s TTFT budget this is affordable — it is 10%, not a violation. The design decision is therefore not "eliminate the cold path" but "keep it bounded and off the critical path where possible."

The mechanisms, cheapest first

1. Host-DRAM tier holds far more than HBM.

host DRAM ~500 GB / 65 MB per adapter = ~7,600 adapters

Every adapter in the fleet fits in host DRAM at attention-only rank 8. The cold path then becomes a 1 ms PCIe copy rather than a 100 ms object-storage fetch — the entire long-tail problem dissolves at this adapter size.

This is the strongest argument for the §2 shape constraint, and it is a different argument from the bandwidth one: small adapters do not merely batch better, they make the tail disappear. At all-modules rank-64 (1.66 GB), host DRAM holds only ~300 and the cold tail is real. One configuration choice determines whether a whole class of problem exists.

2. Predictive preload on the first sign of traffic.

request arrives for a cold adapter
  -> start the load
  -> the request itself waits (~100 ms)
  -> BUT: also signal "this tenant is active"
  -> preload their OTHER adapters (customers usually have a few)

Cheap, and it converts a burst of cold starts into one.

3. Speculative load during queueing. A request that will queue 200 ms behind other work can load its adapter during the queue wait, for free. The load overlaps with work that was happening anyway — the classic move of putting latency where there is already latency.

4. Keep one warm replica per tenant for tenants above a traffic floor. Adapter affinity in the router (§5) does this naturally: route a tenant's traffic to the same 2–3 replicas and their adapters stay resident.

Eviction, and the trap in it

The adapter cache competes with the KV cache for HBM (§2: 100 adapters ≈ 3.5–89% of the KV budget).

A pure-LRU adapter cache is wrong here for the same reason as m02's TinyLFU argument: a one-shot cold adapter would evict a hot one it is 1,000× less valuable than.

value = requests_last_hour / size_bytes         # value per byte held
evict lowest value first, never evict an adapter with in-flight requests (ref-count)

And the trap that makes this different from an ordinary cache: an adapter cannot be evicted mid-generation. A sequence decoding with adapter X needs X present for every one of its steps — which can be minutes. So adapter eviction is ref-counted and deferred, and a replica serving many long generations on distinct adapters can find its adapter cache effectively pinned.

That failure mode has a name in this design — adapter cache pinning — and a bound:

if pinned_adapter_bytes > 0.5 x adapter_cache_budget:
    stop admitting NEW distinct adapters to this replica
    (router sends them elsewhere; existing sequences continue)

Admission control on the cache, not just on requests. A cache whose entries can be pinned by long-lived work needs a limit on how much of it can be pinned, or a slow leak becomes a hard stop.

8. Failure and Recovery

The base-model rollout row is the hardest operational problem in this design and it deserves the detail. When the base model is upgraded, every adapter is stale: the LoRA delta was trained against specific weights.

What must not happen: serving adapter_v1 against base_v2. It does not error. It produces degraded output that looks fine and shows up as a slow drift in customer-reported quality with no correlating event.

The mechanism:

1. adapters are keyed to base_model_digest (§4)
2. a new base version is a NEW SERVING POOL; adapters do not move automatically
3. per adapter, an offline quality check on the tenant's own eval set,
   old base vs new base   ([m05](m05-eval-harness.md))
4. migrate per adapter only when the check passes
5. adapters that fail need re-training -- and the customer must be told

This means base upgrades are gated by per-adapter validation, so the fleet runs two base versions for a migration window. State the cost honestly: 2× base weight memory during migration across the affected replicas, which is a real capacity requirement and the reason base upgrades on a fine-tuning platform are quarterly events rather than weekly ones.

And the deeper point: offering fine-tuning couples your model release cadence to your customers' retraining cadence. That is a product consequence of a serving design, and naming it is exactly the kind of second-order reasoning that separates a senior answer from a complete one.

9. Bottlenecks and Evolution

Now: HBM shared between adapters and KV cache, and adapter-distinctness in the batch (§6).

Interventions in order:

1. Enforce the shape envelope at registration (§3). Free, and it is the difference between +6% and +151%. Do this before anything else — every other optimization is smaller.
2. Merge the head. Top 50 adapters = 80% of traffic; merged deployments have zero adapter overhead and full base-model throughput. The cost is dedicated replicas, justified for exactly the adapters that can fill one. The head and the tail want different architectures, and serving both from one mechanism is a compromise neither needs.
3. Quantize adapters. Adapters tolerate int8 better than base weights (they are a small correction, so absolute error is small). Halves size, halves bandwidth, doubles residency. Needs an eval (m05) — but the risk is genuinely lower than base quantization, which is a useful thing to be able to argue rather than assert.
4. Adapter-aware prefix caching. The KV for a shared system prompt differs per adapter, because the adapter changes the K and V. So adapter_id must be in the prefix cache key (m02 §7) — which fragments the cache by adapter and lowers the hit rate. A real interaction between two designs, in the wrong direction, and worth surfacing rather than discovering.
5. Multi-adapter composition (apply two adapters to one request). Attractive for "domain + style" products; each additional adapter is another read and another kernel pass, and the quality behaviour of composed adapters is not well understood. Say the quality caveat, not just the cost one.

10. Tradeoffs Explicitly Rejected

Rejected: one replica per adapter. $88k/year against adapters serving ten requests/day. The premise of the question.

Rejected: one adapter per batch (homogeneous batching). Collapses continuous batching into adapter-serialized batching; throughput falls by roughly the number of distinct adapters in flight.

Rejected: merging every adapter into base weights. Correct for the top 50, absurd for 5,000 — each merge is a full 140 GB model.

Rejected: accepting arbitrary adapter shapes. §2 — a 25× cost spread chosen by someone with no visibility into it. Constrained at registration, with the arithmetic in the error message.

Rejected: pure-LRU adapter eviction. A one-shot cold adapter evicting a hot one. Value-per-byte, ref-counted.

Rejected: loading adapters from object storage on every request. ~100 ms each. Three tiers, with host DRAM doing the real work.

Rejected: automatic adapter migration across base versions. Silent quality degradation with no error. Per-adapter validation, and tell the customer.

Rejected: sharing a prefix cache entry across adapters. The adapter changes K and V; a shared entry is wrong output, not stale output.

Rejected: FIFO batching. Ignores adapter grouping and gives up most of the throughput — but grouping needs the age deadline from §6, or the tail starves.

The Hostile Critique

C1. "You cap max_distinct_adapters at 32 and admit the biggest groups first. Your traffic is 80% from 50 adapters — so those 50 always form the big groups and always get admitted. The long tail only ever enters via your 200 ms age override. What's the actual p99 TTFT for a long-tail adapter under load, and is it inside your 2 s SLO?"

C2. "Host DRAM holds all 5,000 adapters at 65 MB each — that's 325 GB of the host's 500 GB. What else is using host DRAM on that box? Page cache for model loading, the KV offload tier from m02, the CUDA context. Have you actually got 325 GB?"

C3. "Adapter eviction is deferred while a sequence is in flight. max_tokens is 4,096 at 40 ms per token — that's nearly three minutes. Your pinning guard stops admitting new adapters at 50%. On a replica serving 200 concurrent long generations across 200 adapters, how did it get to 200 distinct adapters in the first place, given max_distinct_adapters is 32?"

C4. "You reject rank-64 all-modules adapters at registration with a nice error message. The customer's fine-tune only works at rank 64 — that's why they chose it. Your API tells them no. What do they do, and what does your sales team do?"

C5. "Base upgrades require per-adapter validation against the tenant's own eval set. Most tenants don't have an eval set. They uploaded 500 examples and clicked fine-tune. What do you validate against, and what do you tell them when you migrate?"

C6. "adapter_id in the prefix cache key fragments the cache by adapter. You listed that as a §9 improvement with a caveat. But it's a correctness requirement — so it's true today. What is your prefix cache hit rate actually, on a fleet where every request has one of 5,000 adapters?"

The Revision

R1 — The long tail needs a reserved batch slot, not just an age override (answers C1)

The critique is right, and working the numbers shows the original guard is too weak. With the grouping rule, a long-tail request is admitted essentially only via the age override — so its TTFT is ≥ 200 ms of queue plus the cold load plus prefill, every time, and under load the override itself queues behind the batch-formation cycle.

Change: reserve slots for non-grouped requests, rather than relying on an override.

RESERVED_TAIL_SLOTS = 4          # of max_distinct_adapters = 32

def form_batch(queue, ...):
    tail = oldest_n(requests_with_group_size_1(queue), RESERVED_TAIL_SLOTS)
    rest = fill_by_grouping(queue - tail, max_distinct = 32 - len(tail))
    return tail + rest

A reserved floor, not a priority override — the same conclusion as m01's R2, d05 and m03. The fourth time this program arrives at reserved floors over priority, which is the point of the cross-cutting pattern map: priority schemes starve the bottom class, floors do not.

Now the p99 is computable rather than hoped for:

tail request TTFT = queue for a reserved slot (~1 batch cycle, ~25 ms)
                  + cold adapter load (~100 ms, or 1 ms from host DRAM)
                  + prefill
                  ≈ 200-400 ms   -- comfortably inside the 2 s cold SLO

Cost: 4 of 32 distinct-adapter slots are held for tail traffic that is 2% of volume, so the head's batches are slightly smaller. Roughly 1–2% throughput, for a bounded tail. State it, and state that it is measurable — tail_slot_utilization says directly whether 4 is the right number.

R2 — The host-DRAM budget must be shared explicitly, and it is contended (answers C2)

The critique is right and this was arithmetic never done — the same defect class that the taxonomy found in 10 of 12 Track C first drafts, now in mine.

The actual host DRAM budget on a 4×H100 node with ~500 GB:

110 GB, not 500 GB. At 65 MB per adapter that is ~1,700 adapters, not 7,600 — so at 5,000 adapters, roughly two-thirds still fall to the object-storage tier, and §7's claim that the long-tail problem dissolves was wrong as stated.

Change 1 — an explicit host-DRAM budget, allocated between the two caches and enforced.

host_dram_budget:
  kv_cache_t1:   300 GB     # m02
  adapters:      110 GB     # this design
  headroom:       50 GB
Both caches evict against their OWN budget. Neither can starve the other.

Two caches on one host, sized by decree rather than by competition — because a shared pool with LRU across both would let a burst in either evict the other, and the failure would look like a mysterious throughput drop in an unrelated subsystem.

Change 2 — the correct claim, which is narrower and still strong. Host DRAM covers ~1,700 adapters, and with adapter affinity in the router (§5) a replica only needs the adapters of the tenants routed to it. At 44 replicas with 3-way affinity, a replica is home to 5,000 × 3 / 44 ≈ 340 adapters — which fits comfortably in 110 GB.

So the original conclusion survives, but only because of affinity, not because host DRAM is large. That is the same correction as m02's R2 and it is worth noticing that the same missing premise appeared twice: a cache sized against the global working set is almost always wrong; size it against the routed working set.

R3 — The pinning guard was inconsistent, and the real bound is different (answers C3)

The critique catches a genuine contradiction and it is a good one: with max_distinct_adapters = 32 per batch, a replica cannot reach 200 distinct pinned adapters — unless batches change over time, which they do. Sequence A (adapter 1) starts at t=0 and runs 3 minutes; by t=30 s the batch has rotated through many adapter groups, each leaving a long-running sequence behind.

So the bound is not the per-batch cap; it is the arrival rate of long generations times their duration. Little's law, and it is the number the design should have had:

pinned_adapters ≈ arrival_rate_of_distinct_adapters x mean_generation_duration
                = 3 distinct adapters/s x 60 s
                = 180 adapters pinned in steady state

180 adapters × 65 MB = 11.7 GB of HBM pinned, against an adapter cache budget of maybe 12 GB (§2: ~7% of the KV budget). The guard fires almost immediately in steady state — meaning the original 50% threshold would trip constantly, not rarely, and the design would spend most of its time refusing new adapters.

Change 1 — size the adapter cache from Little's law, not from a guess.

adapter_cache_bytes >= 2 x arrival_rate_distinct x mean_duration x adapter_size
                    =  2 x 180 x 65 MB  ≈ 23 GB

The 2× is headroom for burstiness. 23 GB of HBM is 13% of the KV budget — a real cost, now justified by a derivation rather than asserted.

Change 2 — cap generation length for adapter requests, or account for it. A 4,096-token generation pins an adapter for ~3 minutes. Either:

• bound max_tokens on the multi-adapter fleet (e.g. 1,024 → 41 s → 3× fewer pinned), or
• route long-generation requests to the merged/dedicated deployments where pinning is irrelevant — which is better, and it uses the promotion path (§9) that already exists.

Change 3 — the guard's threshold now has a meaning. With the cache sized at 2× the steady-state pin, a 50% pinned ratio is the steady state and firing there is wrong. Set it at 80%, which now signals genuine anomaly (a burst of long generations on rare adapters) rather than normal operation.

And the lesson: when a resource is held for a duration, size it with Little's law before choosing a threshold. A threshold on an unsized resource is a random number, and it will either never fire or always fire.

R4 — There must be a path to yes, priced (answers C4)

The critique identifies a product failure hiding in an engineering rule. "No" is not an acceptable answer to a paying customer whose fine-tune genuinely needs rank 64, and a platform whose API says no will be overridden by a human, badly, under commercial pressure — which is worse than having no rule.

Change: the envelope becomes a tier, not a gate.

Every shape has a path to production; expensive shapes cost more. The rejection message becomes:

400 -> 200 with a tier assignment:
  "rank 64 all-modules: projected +75.7% decode cost at batch 64.
   Assigned tier: DEDICATED ($X/hour, ~90 s cold start on first use).
   To use the shared tier: rank<=16 attention-only.
   Estimated quality delta from our benchmarks: -0.4 to -1.2 pts."

The last line is what makes the choice informable. A customer choosing rank 64 for real quality reasons deserves the tradeoff in both currencies — quality and price — not a refusal.

Cost: three serving tiers to operate instead of one, and a pricing decision that requires the business to engage. That engagement is the point. The engineering constraint is real; the mistake was expressing it as a prohibition rather than a price. When a technical limit has a large cost gradient, expose the gradient rather than picking a point on it for the customer.

R5 — Validation needs a platform-provided fallback set, and honest disclosure (answers C5)

The critique is right that most tenants have no eval set, which makes the §8 migration gate unenforceable for the majority of adapters and therefore decorative.

Change: a three-level validation ladder, applied in order of availability.

1. Tenant's own eval set                    -> best. Rare.
2. HELD-OUT SLICE OF THEIR TRAINING DATA    -> platform-created, automatic.
   At fine-tune time, ALWAYS hold out 10% and keep it. The tenant does not
   have to do anything, and we have an eval set for every adapter forever.
3. Behavioural-drift check                  -> no labels needed:
   run ~200 stored prompts from the tenant's own traffic through
   (old base + adapter) and (new base + adapter); measure output
   divergence. Large divergence = migration risk, regardless of "correctness".

Level 2 is the fix and it costs nothing — hold out 10% at fine-tune time and every adapter has a validation set by construction. The original design assumed the eval set was the tenant's responsibility; making it a by-product of fine-tuning removes the dependency entirely.

Level 3 needs no labels at all, which matters because it works even for adapters trained before this policy existed. It cannot tell you the new output is worse — only that it is different, which is the actionable signal for a migration decision.

And what to tell the customer, which is the critique's real question:

"We're upgrading the base model on <date>. We tested your adapter on a
 held-out 10% of your training data:
    accuracy  87.1% -> 86.8%   (within noise; n=412, +/-3.2pp)
    output divergence on your recent traffic: 8% of responses differ materially
 We will migrate on <date>. To stay on the current base for 90 days, [opt out].
 To retrain on the new base (recommended, free), [retrain]."

Numbers, an interval, a date, and two buttons. The +/-3.2pp comes straight from m05 §2 — n=412 cannot resolve a 0.3 pp difference, and saying so is more trustworthy than reporting the delta alone.

R6 — Prefix cache fragmentation is real today, and the fix is to cache the shared part (answers C6)

The critique is correct that this was misfiled as a future improvement when it is a present correctness constraint, and the consequence is worse than the original text implied.

The measurement, which the design owed: with adapter_id in the key, each adapter has its own prefix cache. A long-tail adapter with 10 requests/day gets essentially zero reuse — its entries are evicted long before the next request. So:

head adapters (50, 80% of traffic):   near-normal hit rate  (~60%)
tail adapters (4,450, 2% of traffic): near-zero hit rate
weighted:                             ~50-55%, vs ~60% for a base-only fleet

A ~10% relative hit-rate loss — real, but far less catastrophic than "fragmented by 5,000", because traffic concentration means the cache is dominated by the head anyway. Skew rescues it, and the design should say so with the number rather than leaving the reader to fear the worst.

Change — and there is a genuine optimization available. LoRA modifies only the projections it targets. For an attention-only adapter, the MLP-path activations are identical to the base model's; only K and V differ, and they differ by the low-rank delta.

Cache TWO things:
  1. base K,V for the prefix           -> SHARED ACROSS ALL ADAPTERS
  2. per-adapter delta, recomputed     -> small: r x seq_len, not d x seq_len

At r=16 against d=8192, the per-adapter delta is 1/512th of the full KV. So the shared base KV is cached once for everyone, and each adapter recomputes a tiny correction.

Cost: a custom cache format and a fused kernel that applies the delta during attention. Real engineering, and it recovers most of the base-only hit rate across all 5,000 adapters. Worth scheduling once the simple version's hit rate is measured — which is the honest ordering: this is a substantial optimization justified by a number the design does not yet have, and saying that is better than proposing it as obviously worthwhile.

References

• m01-llm-api-platform.md — the base serving platform; the KV budget adapters compete for
• m02-kv-cache-tier.md — tiered caching, affinity, and the prefix-cache key interaction in R6
• m05-eval-harness.md — the validation ladder in R5, and the confidence intervals it reports
• m03-gpu-cluster-scheduler.md — where dedicated/merged deployments get their GPUs
• ../../systems-design/designs/d05-load-shedding.md — reserved floors, arrived at a fourth time in R1
• ../WARMUP.md#22-decode-is-memory-bandwidth-bound — why extra bytes are extra time
• Hu, E. et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022 — the parameterization §2 sizes
• Sheng, Y. et al. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. MLSys 2024 — the tiered cache and the batched kernel
• Chen, L. et al. Punica: Multi-Tenant LoRA Serving. MLSys 2024 — the BGMV/SGMV kernels in §6
• Dettmers, T. et al. QLoRA. NeurIPS 2023 — quantized adapters, §9's item 3