m01 — Multi-Tenant LLM API Platform

A fully worked design. The /v1/chat/completions product: many customers, one shared GPU fleet, per-tenant limits, streaming responses, and a latency SLO you have to hold while one customer sends a 128k-token prompt.

This is the platform around the engine. The engine internals — batching, PagedAttention, chunked prefill — are in ../WARMUP.md ch. 4, and ch. 7 walks the engine at two altitudes. Here the question is what wraps it: admission, fairness, quotas, accounting, and isolation.

Table of Contents

• The Prompt
• 1. Requirements and Scope
• 2. Scale Numbers
• 3. API Surface
• 4. Data Model
• 5. High-Level Architecture
• 6. Deep Dive A: The Unit of Fairness Is Not the Request
• 7. Deep Dive B: Admission Control on a Memory-Bound Resource
• 8. Failure and Recovery
• 9. Bottlenecks and Evolution
• 10. Tradeoffs Explicitly Rejected
• The Hostile Critique
• The Revision
• References

The Prompt

"We sell an LLM API. Thousands of customers on different plans share one GPU fleet. Design the serving platform — we need per-customer limits, a TTFT SLO, and we cannot let one customer's traffic degrade everyone else's."

The load-bearing clause is "share one GPU fleet." If every customer had their own GPUs this would be a routing problem. They do not, because a 4×H100 replica costs roughly $88,000/year and most customers cannot fill one. Sharing is the entire business model, and therefore isolation is the entire engineering problem.

The second load-bearing thing is unstated and you should say it: the resource being shared is not CPU and not requests — it is HBM bandwidth and KV-cache bytes. Every fairness mechanism you have used before was designed for a different resource, and most of them break here.

1. Requirements and Scope

Clarifying questions asked

"What is the latency contract — TTFT, TPOT, or end-to-end?" The three are in tension and you cannot hold all of them. Assumed: TTFT p95 < 1 s and TPOT p95 < 50 ms (≈ 20 tokens/s per stream, above comfortable reading speed). End-to-end is then a consequence of output length and is not an SLO — because a customer can make it arbitrarily bad by asking for 4,000 tokens, and an SLO the customer controls is not an SLO.

"Is this one model or many?" Assumed: three sizes (8B, 70B, and a frontier model), separately provisioned. Routing between them is by the customer's model parameter, not by us — so this is three fleets, and the interesting question is whether they share anything. Answer: they share the control plane and nothing on the data path.

"What does a plan actually buy?" The critical question. Assumed tiers: free (best-effort, sheddable), standard (rate-limited, no capacity guarantee), enterprise (a reserved token floor). The existence of a reserved floor is what makes this a capacity-allocation problem and not just a rate-limiting problem.

"Can we drop requests?" Assumed: yes for free tier, only with a 429 + Retry-After for standard, and enterprise gets its floor honoured before anyone else gets anything. Never a truncated stream — a half-written answer that the customer was billed for is worse than a refusal.

"Prompt-log retention?" Assumed: not retained by default, since it changes the data model substantially and is a compliance question, not a serving one. Stated and set aside.

Functional

1. POST /v1/chat/completions, streaming (SSE) and non-streaming.
2. Per-key rate limits expressed in requests/min and tokens/min, per model.
3. Plan-tier priority with an enterprise reserved floor.
4. Usage accounting exact enough to bill on.
5. Model versioning: a customer pinning model=x-2026-05 keeps getting that weights version.

Non-functional

Explicitly out of scope

• Fine-tuned/LoRA adapters — that is m07, and it changes the memory model.
• The inference engine's internals — ../WARMUP.md ch. 4–5.
• Content moderation in the path — noted in §9 because it is where the TTFT budget goes to die.
• Training and eval — m04, m05.

2. Scale Numbers

Do this arithmetic out loud. It selects the design, and skipping it is the most common failure in this round (the defect taxonomy found arithmetic never done in 10 of 12 first drafts).

Traffic. 5,000 req/min average = 83 req/s; peak 4× = 333 req/s.

Token shape. Prompt p50 800, p95 6,000. Output p50 300, p95 1,200. The p95/p50 ratio of 7.5× on prompts is the number that matters — the workload is heavy-tailed in both directions, and a mean-based capacity model will be wrong.

Per-replica throughput (70B FP16, TP=4 on H100, from ../gpu_math.py):

decode:   5,647 tok/s theoretical  →  ~2,259 out-tok/s at 40% realized
prefill:  4,096 tokens / 146 ms    →  ~28,000 tok/s
KV budget: 172.5 GiB per replica after weights + activations

Fleet size.

average:  83 rps x 300 out-tok  =  24,900 out-tok/s  ->  11 replicas  =   44 GPUs
peak:    333 rps x 300 out-tok  =  99,900 out-tok/s  ->  44 replicas  =  177 GPUs

And now the number that reframes the design. Prefill demand at average load is 83 × 800 = 66,400 tok/s, which is only 2.4 replicas' worth of compute — about 21% of the decode fleet's. So:

Prefill is cheap in aggregate and expensive in latency; decode is the reverse. Prefill is ~1/5 of the compute but owns 100% of TTFT. Decode is ~4/5 of the compute and owns TPOT.

That single sentence justifies chunked prefill, justifies separate SLOs, and sets up the disaggregation discussion in §9. It is worth 90 seconds.

The KV cache is the capacity unit. 70B with GQA-8: 320 KiB per token.

One customer's single 128k-context request occupies 23% of an entire 4-GPU replica for the whole duration of its decode. Four of them and the replica serves nothing else.

Say this explicitly: a request-per-minute limit does not bound this at all. One request/minute of 128k context is a quarter of a replica; 1,000 requests/minute of 200-token prompts is a rounding error. They differ by four orders of magnitude in cost and are identical to a request counter. That is deep dive A.

Cost. 44 GPUs at $2.50/hr ≈ $110/hr ≈ $963k/year at average load; provisioning for peak without autoscaling is 4× that. The gap between those two numbers is the entire argument for admission control instead of over-provisioning.

3. API Surface

POST /v1/chat/completions
  { model, messages[], max_tokens, stream, temperature, ... }
  ->  200 text/event-stream          (streaming)
  ->  200 application/json           (buffered)
  ->  429 + Retry-After + X-RateLimit-*
  ->  503 + Retry-After              (capacity, not quota — a different signal)

Response headers on every request:
  X-RateLimit-Limit-Requests / -Tokens
  X-RateLimit-Remaining-Requests / -Tokens
  X-RateLimit-Reset-Requests / -Tokens
  X-Request-Id                       (the join key for every later question)

Two rejection codes, deliberately. 429 means you exceeded your quota — the customer can fix it by slowing down. 503 means we are out of capacity — the customer cannot fix it and retrying immediately makes it worse. Collapsing them into one code is a real design error: it tells a well-behaved customer to back off for someone else's spike, and it hides our own capacity problem inside a metric that looks like customer misbehaviour.

Streaming is the default and it constrains everything downstream. Once the first SSE frame is sent the status code is committed. Anything that could fail — moderation, quota, capacity — must be decided before the first token, or the failure has to be expressed inside the stream:

data: {"choices":[{"delta":{"content":"..."}}]}
data: {"error":{"type":"server_error","message":"..."}}   # mid-stream failure
data: [DONE]

Clients handle a mid-stream error object badly in practice. So the design goal is that everything fallible happens pre-first-token, which is exactly the budget pressure that deep dive B is about.

Idempotency. Idempotency-Key on the request, retained 24 h with the response. Without it a client timeout at 59 s on a 60 s generation causes a retry that pays for the whole generation twice — and generations are not free, so this is a billing dispute, not a nicety.

4. Data Model

tenant     (tenant_id, plan, reserved_tps, created_at)
api_key    (key_hash, tenant_id, scopes, revoked_at)
limits     (tenant_id, model, rpm, tpm, max_context, max_output)
usage      (tenant_id, model, hour, prompt_tokens, completion_tokens, cached_tokens)
request_log(request_id, tenant_id, model, ts, prompt_tok, completion_tok,
            ttft_ms, tpot_ms, finish_reason, replica_id)

Three notes worth saying out loud:

max_context and max_output are per-tenant limits, not global constants. They are the only knobs that bound the 40 GiB request from §2. A free-tier key gets max_context = 8k; enterprise gets 128k because they are paying for the replica share it costs. A limit you cannot express per tenant is a limit you cannot sell.

usage is hourly, not per-request. Per-request billing rows at 83 rps is 7.2M rows/day, which is fine, but the aggregation is what billing reads and it should not scan. Write both: request_log for support and debugging (retain 30 days), usage for billing (retain forever).

cached_tokens is a separate column from the start. Prefix caching (§9) makes some prompt tokens ~free to serve, and if you bill them at full rate you are charging for compute you did not do. Adding the column later means a schema migration on your billing table — the worst table to migrate. Cost: nothing today. This is the cheapest correct decision in the design.

5. High-Level Architecture

                     ┌──────────────────────────────────────────────┐
   client ──────────►│ EDGE:  TLS · auth · schema · idempotency     │
                     │        rate limit (tokens AND requests)       │
                     └───────────────┬──────────────────────────────┘
                                     │  admitted
                     ┌───────────────▼──────────────────────────────┐
                     │ ROUTER:  model + version -> fleet             │
                     │          cost estimate -> queue class         │
                     │          replica choice on KV headroom        │
                     └───────────────┬──────────────────────────────┘
                                     │
        ┌────────────────────────────┼────────────────────────────┐
        │                            │                            │
  ┌─────▼──────┐              ┌──────▼─────┐              ┌───────▼────┐
  │ 8B fleet   │              │ 70B fleet  │              │ frontier   │
  │ TP1        │              │ TP4        │              │ TP8/PP2    │
  └─────┬──────┘              └──────┬─────┘              └───────┬────┘
        │  per replica: continuous batching + paged KV + chunked prefill
        └────────────────────────────┼────────────────────────────┘
                                     │  token stream back through router (SSE)
                     ┌───────────────▼──────────────────────────────┐
                     │ USAGE PIPELINE:  per-request events -> Kafka  │
                     │   -> hourly rollup -> billing (exactly-once)  │
                     └──────────────────────────────────────────────┘

  CONTROL PLANE (off the data path): limits, plans, model registry, rollout

The five decisions embedded here, each defensible:

1. The edge does auth and quota; the router does capacity. These are different questions with different answers — quota is about the contract, capacity is about the machine. A tenant can be within quota and still get a 503. Conflating them was the mistake in an earlier version of this design and it produced a system that shed enterprise traffic during a free-tier spike.

2. Model fleets are physically separate. Shared GPUs across model sizes sounds efficient and is not: swapping 140 GB of weights takes ~42 ms of pure HBM read at best, and in practice a cold model load from remote storage is 30–90 s. The unit of elasticity is a replica of one model, not a GPU.

3. The router is stateless but not blind. It needs per-replica KV occupancy to route, which means replicas push occupancy every ~250 ms. Stale-by-250 ms occupancy is fine because it is used as a hint; correctness comes from the replica's own admission check.

4. Streaming goes back through the router, not direct to client. It costs a hop (~1 ms), and it buys: connection draining on replica shutdown, mid-stream failover for the non-streamed case, and one place to count tokens for billing. You cannot bill accurately from the client side of a stream the client may abandon.

5. Usage is a Kafka pipeline, not a synchronous write. A synchronous billing write on the request path adds latency to the SLO you are trying to hold and makes billing an availability dependency of serving. At-least-once + dedupe on request_id (the outbox pattern).

6. Deep Dive A: The Unit of Fairness Is Not the Request

The mistake almost everyone makes

Rate limit by requests per minute. It is what every API does, the libraries exist, and it is wrong here by four orders of magnitude.

From §2: a 128k-context request holds 40 GiB of KV — 23% of a replica — for its whole decode. A 200-token request holds 64 MiB for a few hundred milliseconds. A counter that treats them identically is not a limiter, it is a random number generator.

What the resource actually is

Two distinct scarce things, and you must limit both:

The second one is the one nobody names, and naming it is most of the value of this deep dive.

KV occupancy is an integral, not a rate. A request holds context_tokens × 320 KiB for output_tokens × TPOT seconds. So its true cost is:

kv_cost_gib_seconds ≈ (prompt + output/2) × 320KiB × output × TPOT
                       └──── average residency ────┘   └─ duration ─┘

The output/2 is because the KV grows one token at a time during decode, so the average is roughly the midpoint. That factor is why output length appears squared: a request that generates twice as many tokens holds roughly twice the memory for twice as long.

Worked, at TPOT = 40 ms:

735×. A request limiter charges all three the same. A token limiter charges the third 3.5× the first. Only a KV·seconds accounting charges it what it costs.

The mechanism

Three-layer limiting at the edge, cheapest first:

# Layer 1 — requests/min. Cheap, catches runaway loops. Not a capacity control.
if not rpm_bucket.allow(key, 1):
    return 429("requests")

# Layer 2 — tokens/min. Charged on the ESTIMATE at admission, RECONCILED at completion.
est = prompt_tokens + min(max_tokens, tenant.max_output)
if not tpm_bucket.allow(key, est):
    return 429("tokens")

# Layer 3 — concurrent KV·seconds. The one that actually protects the fleet.
est_kv = (prompt_tokens + est_out / 2) * KV_PER_TOKEN * est_out * TPOT_TARGET
if tenant.inflight_kv_seconds + est_kv > tenant.kv_seconds_cap:
    return 429("concurrency")

Estimate-then-reconcile is the whole trick, and it is worth stating as a general pattern. You cannot know the output length in advance — the model decides. So:

• Charge max_tokens at admission. Pessimistic, so you never over-admit.
• Refund the difference at completion. A request that asked for 4,000 and stopped at 90 gets 3,910 tokens back into the bucket immediately.
• Bill the actual. Enforcement is pessimistic; accounting is exact. Different systems, different guarantees — the same split as d03's R5.

Without the refund, a client that always sets max_tokens=4096 "just in case" gets throttled at 7% of its real entitlement — a support ticket you will get, and a bug that looks like the limiter is broken because from the customer's side it is.

Where the state lives

Same problem as d03, same answer: lease from a shared store, enforce locally. But with one difference specific to this workload — because requests are long-lived (seconds to minutes, not milliseconds), the shared store also needs to know about in-flight work, not just completed work.

Redis, per tenant:
  tpm:{tenant}:{window}       counter, leased in blocks
  inflight:{tenant}           sorted set: request_id -> (est_kv, started_at, edge_id)

inflight is a sorted set scored by started_at so it is self-cleaning. An edge node that crashes mid-request leaves entries behind; a sweeper drops anything older than max_output × TPOT × 3. Without that sweep, one edge crash permanently reduces a tenant's concurrency allowance — a leak that shows up as "our limit got smaller" weeks later, with no event to correlate it to.

This is a lease with an expiry, which is the same primitive as d11. Say so. The interviewer is looking for whether you see it.

7. Deep Dive B: Admission Control on a Memory-Bound Resource

Why the usual answer fails

Standard admission control: measure utilization, shed above a threshold. What utilization?

• GPU "utilization" (nvidia-smi) is a lie for this workload. It reports the fraction of time at least one kernel was resident, not the fraction of the machine doing useful work. A decode step at batch 1 shows ~100% utilization while using 1/295th of the compute. Quoting nvidia-smi as a capacity signal is a tell that you have not run this in production.
• Request count is wrong for the reasons in deep dive A.
• Queue depth is directionally right but lags — by the time the queue is deep, TTFT has already blown.

The signal that works

KV-cache occupancy, because it is the actual binding constraint and it is predictive: it rises before latency does.

occupancy = allocated_kv_blocks / total_kv_blocks

Its behaviour is the useful part:

The non-linearity at ~95% is the thing to explain. When KV is exhausted, vLLM-style schedulers preempt a sequence: evict its blocks and later recompute its entire prefill. So a preemption does not cost a little latency — it costs the whole prompt's prefill again, ~146 ms for a 4k prompt. And the recompute needs KV, which triggers another preemption. That is a positive feedback loop, and it is why the curve is a cliff rather than a slope.

This is the same shape as the utilization knee from Track C, and worth naming as such: queueing systems degrade hyperbolically near saturation; this one degrades worse, because saturation destroys completed work.

The three-class scheduler

Admission is not one decision, it is a priority allocation with a floor:

def admit(req, replica):
    occ = replica.kv_occupancy
    cls = classify(req.tenant)                        # reserved | standard | best_effort

    if cls == "reserved":
        # Enterprise floor. Admitted until their OWN cap, regardless of global occupancy.
        # The floor is capacity we sold; honouring it under load is the product.
        return req.tenant.inflight_kv < req.tenant.reserved_kv

    if cls == "standard":
        return occ < 0.85

    return occ < 0.60                                 # best_effort / free

Reserved floors, not pure priority. Pure priority starves the bottom class completely under sustained load — free-tier customers who are also evaluating you before they buy. A floor gives enterprise what they paid for and leaves the rest genuinely shared. (Same conclusion as d05 and d12 — one primitive, three designs.)

The reserved floors must be over-subscribed deliberately and the ratio must be a written decision. If reserved floors sum to 100% of the fleet you have sold your entire capacity and have nothing for the standard tier. Sum them to ~60%: enterprise customers do not all peak together, and the 40% gap is what standard and free actually run on. Then measure the coincidence of enterprise peaks — if it rises, the over-subscription ratio must fall, and that is a capacity-planning input, not a scheduler parameter.

Protecting TTFT specifically

Even with correct admission, TTFT is threatened by prefill from other requests. From §2: an unchunked 4k prefill is 146 ms, against a 50 ms TPOT target — every active decode stream stalls for ~5 token-times when one lands. At 128k it is 4.7 seconds and the stall is catastrophic.

Chunked prefill (Sarathi) is the answer: split the prefill into fixed token budgets (say 512) and interleave chunks with decode steps in the same batch.

without chunking:  [====== prefill 146ms ======][dec][dec][dec]
with chunking:     [pf][dec][pf][dec][pf][dec][pf][dec]  ...
                    └ 18ms each; TPOT jitter bounded by one chunk

The cost is honest and you should state it: prefill throughput drops ~10–15% because the matmuls are smaller and less efficient. You are buying tail latency with throughput. Given that prefill is only 21% of the fleet's compute (§2) and 100% of its TTFT risk, this is a very good trade — and the arithmetic is why you can say that rather than assert it.

8. Failure and Recovery

The split fail-open/fail-closed decision is the interesting row and it deserves a sentence. When the limits store is down:

• TPM: fail open. Over-serving for a few minutes costs money and is recoverable through billing, which is the source of truth anyway. Refusing all traffic is a total outage.
• KV·seconds: fail closed, to a conservative local default. Because failing open here does not cost money, it destroys the fleet — unbounded concurrent long contexts drive occupancy past 95% and every tenant's TPOT collapses.

One store, two opposite policies, because the two limits protect different things: one protects revenue, one protects the machine. A design that applies one policy to both is wrong in one of the two directions, and it is worth saying which and why.

On in-flight streams during a replica death: they cannot be transparently failed over, because the KV cache — the entire state of the generation — lives in that replica's HBM. You can either (a) return an error and let the client retry, or (b) restart the generation on another replica, which re-prefills the prompt and produces different tokens from the point of failure. Option (b) looks better and is worse: a stream that silently changes its mind mid-answer is a correctness bug from the user's perspective. Choose (a), and make it cheap by keeping generations short enough that a retry is tolerable. For long generations, offer the batch API (§9) instead.

9. Bottlenecks and Evolution

Now: KV cache is the binding constraint on every replica. Everything else has headroom.

Order of interventions, cheapest first:

1. Prefix caching. System prompts are shared across a tenant's traffic and are often 500–2,000 tokens. Caching their KV skips that prefill entirely. Measure hit rate before promising anything — it is entirely workload-dependent, 5% for diverse chat and 80%+ for a RAG product with a fixed template. This is m02.
2. FP8 KV cache. Halves KV bytes → roughly doubles the batch → nearly doubles throughput per GPU. Quality impact is small but workload-specific and must be measured on your evals, not assumed from a paper. This is the single biggest throughput lever available and it is a quality decision, not an infra decision — so it needs m05 to land first.
3. Autoscaling on occupancy. Not on QPS. The scale-up ramp is 30–90 s (weight loading), so the trigger must lead demand by that much — which means a predictive component on top of the reactive one, because a purely reactive scaler that takes 90 s to act is not a scaler, it is a post-mortem. Keep a small warm pool for the reactive gap.
4. Disaggregated prefill/decode. Separate pools, KV transferred over the interconnect. Prefill is compute-bound and decode memory-bound (§2), so they want different hardware ratios and scale on different signals. The cost is a 40 GiB KV transfer for a 128k request — over 400 Gb/s that is 800 ms, which annihilates the TTFT budget. So: worth it for short-context high-volume traffic, actively harmful for long-context. Route by context length, or do not do it.
5. A batch/async API at a discount. Moves the long-output, latency-insensitive traffic off the interactive fleet entirely, which is a better answer to the 735× request from §6 than any scheduler tweak. Making the expensive workload a different product is often better than making the scheduler smarter.

Where moderation goes. A classifier in the request path costs 20–50 ms of the 1 s TTFT budget — acceptable. Output moderation is the hard one: you have already streamed tokens when the classifier fires. Options are (a) buffer N tokens before emitting (adds N × TPOT to TTFT), (b) stream and retract (clients handle it badly), (c) run the classifier on a sliding window and cut the stream on trigger (leaks a few tokens). There is no free option, and the honest answer is (c) plus a small buffer, sized from the classifier's latency, with the leak accepted and measured.

10. Tradeoffs Explicitly Rejected

Rejected: per-tenant dedicated replicas. Perfect isolation, trivially. Rejected on arithmetic: a 4×H100 replica is $88k/year and the median tenant uses <2% of one. Dedicated replicas for the top ~20 tenants who can fill one — that is worth doing, and it is the natural evolution of the reserved floor into physical isolation. For everyone else, sharing plus enforced caps.

Rejected: request-count rate limiting alone. §6. Off by 735× on real traffic.

Rejected: a single global queue with priorities. Attractive, and it fails on the KV constraint: a queue orders time, but the binding resource here is space. Two short requests and one 128k request may be admissible in either order by time and only one order by memory. Admission must be memory-aware, which means it happens at the replica that has the memory.

Rejected: routing by round-robin or least-connections. Both ignore the actual constraint. Least-connections sends the 128k request to the replica with fewest streams, which may be the one with least KV headroom (it is serving three long contexts). Route on KV headroom.

Rejected: strict priority without floors. Starves free tier to zero under sustained load. Free-tier users are prospective customers, and an evaluation that 503s is a lost sale. Floors.

Rejected: synchronous billing writes. Adds a store write to the TTFT path and makes billing an availability dependency of serving. At-least-once through Kafka with idempotent rollup.

Rejected: nvidia-smi utilization as the autoscaling signal. It reads ~100% during a batch-1 decode that uses 1/295th of the machine. Wrong by two orders of magnitude, and reaching for it signals inexperience with this workload specifically.

The Hostile Critique

C1. "Your KV·seconds estimate multiplies by TPOT_TARGET. That's the TPOT you want, not the TPOT you have. When the fleet is loaded TPOT rises — that's what loaded means. So your estimate of how long a request holds memory shrinks exactly when requests are holding memory longest. Walk me through what your admission controller does as the fleet degrades."

C2. "Enterprise gets a reserved floor 'regardless of global occupancy'. So at 99% occupancy, with the scheduler thrashing on preemption, you keep admitting enterprise traffic into a replica that is destroying itself. You've guaranteed them admission, not latency. What exactly did you sell them?"

C3. "Prefix caching: you put it first because it's cheapest. Two tenants send the same system prompt. Do they share a cache entry? If yes, tell me why that isn't a cross-tenant information leak. If no, tell me your hit rate on a fleet where the same 200 templates account for most traffic."

C4. "You refund unused tokens at completion. A client sets max_tokens=4096 and aborts the HTTP connection after 50 tokens. Who refunds? And your inflight sorted set has an entry scored by started_at with a sweep at max_output × TPOT × 3 — for max_output=4096 at 40 ms that's eight minutes. So an aborted request holds that tenant's concurrency budget for eight minutes. Is that what you intended?"

C5. "You route on KV headroom, pushed every 250 ms. At 333 rps that's 83 routing decisions per occupancy update. All of them see the same stale value, so they all pick the same emptiest replica. Describe what happens to that replica."

C6. "You said a dead replica's streams 'return an error and the client retries'. At peak you have 44 replicas each holding ~137 sequences. One dies: 6,000 clients retry at once, into a fleet that just lost 2% of its capacity. What does your Retry-After say, and what happens if every client honours it exactly?"

The Revision

R1 — Admission must use measured TPOT, and the feedback sign matters (answers C1)

The critique identifies a real inversion, and it is the most dangerous kind of bug: the control loop has the wrong sign under load. Using TPOT_TARGET = 40 ms in the KV·seconds estimate means that when actual TPOT rises to 120 ms — a loaded fleet — every request actually holds memory 3× longer than estimated, while the controller keeps admitting as if nothing changed.

Change: estimate from the measured TPOT, and make the estimate conservative in the right direction.

# Fleet-wide p95 TPOT over the last 30 s, floored at target so the estimate is
# never optimistic, and clamped so one pathological replica cannot freeze admission.
tpot_est = clamp(measured_tpot_p95, TPOT_TARGET, 4 * TPOT_TARGET)
est_kv   = (prompt + est_out / 2) * KV_PER_TOKEN * est_out * tpot_est

Now the loop is negative-feedback: rising TPOT raises the estimated cost of every request, which tightens admission, which lowers TPOT. The clamp at 4× prevents a single stuck replica from driving the estimate to infinity and shutting the platform down — a failure mode that a naive "just use the measurement" fix introduces.

Cost: admission becomes coupled to a fleet-wide measurement, so a bad metrics pipeline now degrades admission. Mitigation: the measurement is a hint with a safe default; if it is stale by more than 60 s, fall back to TPOT_TARGET × 2 — pessimistic, which is the safe direction.

And the general lesson: when a controller's input is a target rather than a measurement, check its behaviour at the point where target and measurement diverge. That is exactly where it will have to work, and exactly where it has never been tested.

R2 — A floor must guarantee latency, not admission (answers C2)

The critique is correct and it is a product bug, not just an engineering one. "You will always be admitted" is worthless if admission is into a thrashing replica. What enterprise bought was a latency SLO, and the design delivered a queue position.

Change: the reserved floor becomes a capacity reservation, enforced by keeping the replicas that serve it out of the thrash zone.

if cls == "reserved":
    if replica.kv_occupancy > 0.92:
        # Do not admit into a replica that cannot honour the latency contract.
        # Try another replica; if the whole fleet is there, this is a capacity
        # incident and enterprise is told the truth rather than served badly.
        return TRY_ANOTHER_REPLICA
    return req.tenant.inflight_kv < req.tenant.reserved_kv

And, structurally: the fleet holds back a reserve of replicas that only reserved traffic may enter, sized to the sum of enterprise floors × the measured coincidence factor. Standard and free traffic never enters them, so their occupancy is controlled by construction rather than by hope.

Cost: real money. Reserve replicas idle when enterprise is quiet, which is most of the time. That is what the enterprise tier is for — the price should carry it. This is the honest version of the tradeoff: you cannot sell a latency guarantee on shared capacity without holding capacity back, and any design that claims to is deferring the cost to an incident.

R3 — Prefix cache entries are tenant-scoped, with one deliberate exception (answers C3)

The critique names a genuine risk, and the naive answer (share everything, it's just KV) is a cross-tenant leak: KV cache hits are timing-observable, so a shared cache lets tenant A detect that tenant B has sent a particular prefix. That is a real side channel, it has been demonstrated against production LLM APIs, and "it's only cached compute" is not a defence.

Change: cache key includes the tenant.

key = H(tenant_id, model_version, token_ids[0:n])

The one exception, made explicitly: prefixes we ourselves inject — the platform system prompt, tool-definition preambles we generate — are ours, identical for everyone, and carry no tenant information. Those may be globally shared because the attacker learns nothing from a hit on a string we publish in our own documentation.

On the hit-rate cost the critique correctly anticipates: tenant-scoping does lower the hit rate, and the honest answer is that it lowers it less than it appears, because prefix reuse is overwhelmingly within a tenant — the same customer's application sends the same template thousands of times. Cross-tenant sharing mostly duplicates hits that intra-tenant sharing already gets. I would measure both and be prepared to be wrong, but I would not turn on cross-tenant sharing to find out, because the experiment itself is the leak.

Cost: more cache memory for the same hit rate, since popular prefixes are stored per tenant. Bounded by evicting on (tenant, LRU) with a per-tenant cache quota — otherwise one tenant's churn evicts everyone else's entries, which is the same noisy-neighbour problem one level down.

R4 — Client disconnect is a first-class event, and the sweep was two orders of magnitude off (answers C4)

Both halves of the critique are right, and the second is the worse bug.

Change 1 — disconnect handling. Abort is not an edge case; it is normal (users close tabs).

• The edge detects the closed connection and cancels the generation at the replica, freeing KV immediately. Continuing to generate for a client that left is pure waste — at peak this is measurably several percent of the fleet.
• The refund happens on cancellation, same path as completion. There is exactly one terminal handler for a request and every ending goes through it: complete, error, cancel, timeout.
• Billing charges tokens actually generated before cancel. The customer got them (streamed), so this is defensible; and it means cancel is not a free way to get compute.

Change 2 — the sweep interval. The critique's arithmetic is correct: 4096 × 40 ms × 3 = 492 seconds. A crashed edge would hold a tenant's concurrency for over eight minutes.

The fix is not a shorter timeout — it is a lease with a heartbeat, which is the primitive this should have been from the start:

Edge renews each in-flight entry every 5 s.
Sweeper reclaims anything unrenewed for 15 s.

Now reclaim is bounded by 15 s regardless of max_output, and a live long request is never reclaimed because it keeps renewing. This is d11's session lease, and I should have reached for it directly rather than inventing a timeout.

Cost: renewal traffic — one pipelined Redis call per edge per 5 s covering all its in-flight requests, which is negligible. And the standard lease caveat applies: an edge partitioned from Redis but still serving will have its entries reclaimed while it is still using the capacity. That over-admits by one edge's share, which is bounded and acceptable — and it is the right direction to be wrong, since the alternative leaks capacity permanently.

R5 — Routing needs load-aware randomization, not "pick the emptiest" (answers C5)

The critique describes a herd, and it is a classic: 83 decisions against one stale observation, all choosing the same target. The emptiest replica receives 83 requests, becomes the fullest, and 250 ms later the herd stampedes elsewhere. The fleet oscillates and every replica alternates between starved and thrashing.

Change: power-of-two-choices with in-flight accounting.

a, b = random.sample(healthy_replicas, 2)
# Effective load = last pushed occupancy + what this router has sent since,
# which is the term that makes the stale observation safe.
pick = min(a, b, key=lambda r: r.pushed_occupancy + r.optimistic_inflight_kv / r.total_kv)

Two changes, both necessary:

1. Choosing between two random replicas instead of the global minimum caps the herd at the fraction that samples the same pair. This is the standard result: the max load goes from Θ(log n / log log n) to Θ(log log n) — exponentially better for one extra random draw.
2. optimistic_inflight_kv — the router's own record of what it has dispatched since the last push — closes the 250 ms blind spot, which is the actual cause. Without it, power-of-two still herds, just into two replicas instead of one.

Cost: routers now hold per-replica state, so they are no longer trivially stateless. This is fine: the state is soft, per-router, and self-correcting on the next push. It never needs to be replicated or persisted. Say that explicitly — "stateful" is a word interviewers probe, and the answer "soft state with a 250 ms half-life" ends the probe.

R6 — Retry-After must be jittered and the fleet must be able to say "no" (answers C6)

The critique's number is right — 44 replicas × 137 sequences ≈ 6,000 concurrent streams, and one replica's death releases ~137 of them into a fleet that just shrank. The general form is worse than the specific: a correlated failure produces a correlated retry, which is the thundering herd.

Change 1 — never send a bare Retry-After.

Retry-After: 7        # base 5 s + uniform jitter in [0, 5 s], computed per response

If every client honours an unjittered Retry-After: 5 exactly, they retry simultaneously. The header creates the herd it was meant to prevent. Jitter is not an optimization here; it is the entire mechanism.

Change 2 — a retry budget at the edge, not just backoff. Backoff spreads the herd in time; it does not reduce total load. Under a correlated failure the fleet needs to shed more, not later:

# If retries exceed 20% of admitted traffic, the fleet is in a retry storm.
# Shed retries preferentially over first attempts: a first attempt is a user
# waiting, a retry is a client library.
if retry_ratio_1min > 0.20 and req.is_retry and cls != "reserved":
    return 503(retry_after=jitter(30, 60))

Identified by the Idempotency-Key already in the API (§3) — which is a nice property to point out: the idempotency mechanism added for billing correctness turns out to be what makes retry identification possible. That is not luck, it is what happens when requests carry identity.

Change 3 — capacity headroom for exactly this. Run at ≤ 85% of peak-provisioned capacity so that losing one replica of 44 (2.3%) does not push the rest past the preemption cliff. The headroom is not waste; it is the thing that makes a single failure survivable rather than correlated. Static stability, same as d08.

And the general lesson worth stating: every mechanism that tells clients what to do — status codes, Retry-After, backoff hints — is a fleet-wide broadcast. Design it as one. The question is never "what should this client do", it is "what should ten thousand clients do at the same instant, and what happens if they all comply perfectly."

References

• ../WARMUP.md — the roofline, KV budgets, batching, chunked prefill, from zero
• ../WARMUP.md#chapter-7-worked-answer--design-chatgpt — the engine at two altitudes; this design is the platform around it
• ../gpu_math.py — every number in §2, reproducible
• m02-kv-cache-tier.md — prefix caching as its own design
• m07-lora-serving.md — what changes when tenants bring their own weights
• ../../systems-design/designs/d03-rate-limiter.md — leasing, estimate-then-reconcile, fail-open/closed
• ../../systems-design/designs/d05-load-shedding.md — shed classes and reserved floors
• ../../systems-design/designs/d11-lock-service.md — the session lease used in R4
• Yu, G.-I. et al. Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022 — continuous batching
• Kwon, W. et al. Efficient Memory Management for LLM Serving with PagedAttention. SOSP 2023 — the preemption/recompute behaviour behind the 95% cliff
• Agrawal, A. et al. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024 — chunked prefill and its throughput cost
• Mitzenmacher, M. The Power of Two Choices in Randomized Load Balancing. — the R5 result
• Amazon Builders' Library. Timeouts, retries, and backoff with jitter. — R6