Knowledge 04 — Serving Frameworks

The JD bullet: "Build, test, and deploy scalable inference pipelines using serving frameworks such as vLLM, TGI, and Triton." A serving framework is what turns an optimized model into a production service that handles many concurrent users at a target SLO. This module explains the algorithms inside these servers — continuous batching, PagedAttention, scheduling, chunked prefill — so you can tune them, not just launch them.

Table of Contents

• 1. What a serving layer must do
• 2. Static batching and why it's bad
• 3. Continuous (in-flight) batching — the key idea
• 4. PagedAttention and KV-cache memory management
• 5. Prefix caching, chunked prefill, and disaggregation
• 6. The scheduler: the brain of the server
• 7. The frameworks compared: vLLM, TGI, TensorRT-LLM, Triton, others
• 8. The knobs you actually tune
• 9. Triton Inference Server: the general-purpose host
• 10. A tuning playbook for a target SLO
• 11. References

1. What a serving layer must do

A naive server (one request → one model.generate() → response) wastes the GPU catastrophically: while it decodes one user's tokens at batch 1, the tensor cores are ~0.3% utilized. A production serving layer must:

1. Multiplex many requests onto the GPU concurrently to amortize weight reads (the batching math).
2. Manage KV-cache memory efficiently across requests of different and growing lengths.
3. Schedule prefill and decode work to meet latency SLOs while maximizing throughput.
4. Handle the queue: admission, preemption, priorities, timeouts.
5. Expose an API (usually OpenAI-compatible) with streaming.

Each numbered item is an algorithm with real cleverness. The frameworks differ mostly in how well they do 1–3.

2. Static batching and why it's bad

The obvious approach: collect N requests, run them as a batch, return all N when all finish. Two fatal flaws for LLMs:

• Ragged completion: in a batch of 8, one request generates 500 tokens and seven generate 20. The batch runs for 500 steps; the 7 short requests finished at step 20 but their GPU slots sit idle for 480 steps, padded and wasted. Utilization tanks.
• Head-of-line blocking: a request arriving just after a batch starts waits for the entire batch to finish before it's even considered. TTFT explodes under load.

Static batching is fine for fixed-size, equal-length workloads (some CV/embedding jobs) and terrible for variable-length autoregressive generation. This is the problem continuous batching solves.

3. Continuous (in-flight) batching — the key idea

Continuous batching (a.k.a. in-flight batching; Orca, 2022) operates at the granularity of a single decode step, not a whole request. The scheduler maintains a running batch and, after every decode step:

• Any sequence that emitted its end-of-sequence token (or hit max length) leaves the batch and returns immediately.
• A waiting request immediately takes the freed slot — it gets prefilled and joins the in-flight batch mid-stream.

So the batch is continuously refilled; no slot waits for the slowest sequence; new arrivals join within a step or two instead of waiting for a batch boundary.

Static batch:   [A....finishes at 500][B done@20  idle........][C done@30 idle.......]   ← wasted slots
Continuous:     [A.....................500]
                [B done@20 → D joins → D....][...]
                [C done@30 → E joins → E.........]                                       ← slots always full

Impact: 5–20× higher throughput than static batching at the same latency, because the GPU stays full. This single idea is why vLLM/TGI/TensorRT-LLM exist and why you should never hand-roll a server. It's the first thing to verify is enabled when a customer's GPU utilization is low.

Whiteboard: "Static batching wastes slots on early-finishing sequences. Continuous batching swaps a finished sequence out and a waiting one in every step, so the GPU never idles on padding. It's the difference between 30% and 90% utilization."

4. PagedAttention and KV-cache memory management

Continuous batching creates a memory problem: sequences have different, unpredictable, growing lengths, so their KV caches are different sizes and grow over time. Naively you'd reserve max_seq_len of contiguous KV memory per sequence — but most sequences never reach max length, so you massively over-allocate and fragment. Studies found naive serving wasted 60–80% of KV memory to fragmentation and over-reservation.

PagedAttention (vLLM, SOSP 2023) borrows the OS virtual-memory idea:

• KV cache is split into fixed-size blocks (e.g., 16 tokens each).
• A sequence's KV is a list of (possibly non-contiguous) physical blocks, tracked by a block table (like an OS page table).
• Blocks are allocated on demand as the sequence grows, and freed instantly when it finishes — near-zero fragmentation, no over-reservation.
• The attention kernel is modified to gather K/V from these scattered blocks.

Payoff: you fit far more concurrent sequences in the same VRAM → bigger effective batch → more throughput. And it enables copy-on-write block sharing for prefix caching (§5).

The analogy that lands: "PagedAttention is virtual memory for the KV cache. Instead of giving every sequence one big contiguous reservation that mostly goes unused, it pages KV in fixed blocks on demand. Same trick the OS uses to let many processes share RAM without fragmentation — applied to attention." Knowing this by the OS analogy signals real systems depth.

5. Prefix caching, chunked prefill, and disaggregation

Three more techniques that separate a tuned server from a default one:

Prefix caching (automatic prefix caching)

Many requests share a prefix — a long system prompt, few-shot examples, a RAG context reused across questions. Prefix caching detects the shared prefix and reuses its KV blocks across requests (via the block-sharing PagedAttention enables) instead of re-prefilling it every time. For agent/RAG workloads with a 2k-token fixed system prompt, this can slash TTFT and prefill cost dramatically. (vLLM enable_prefix_caching.)

Chunked prefill

A long prompt's prefill is one big compute-bound chunk that stalls everyone else's decode (a 4k-token prefill blocks the batch for many ms → decode TPOT spikes). Chunked prefill (Sarathi-Serve) splits a long prefill into small chunks and interleaves them with ongoing decode steps, so prefill and decode share the GPU smoothly. Result: much better TPOT tail latency under mixed load, at a small TTFT cost. The knob trades prefill latency for decode smoothness.

Prefill/decode disaggregation

Prefill (compute-bound) and decode (memory-bound) have opposite hardware needs and interfere when co-located. Disaggregated serving (DistServe, Splitwise, 2024) runs prefill on one pool of GPUs and decode on another, streaming the KV cache between them. This lets each pool be sized and optimized independently and hit tighter SLOs for both TTFT and TPOT — at the cost of KV transfer and orchestration complexity. The current frontier for large-scale serving. (What actually shipped 2024–2026 — the KV-transfer arithmetic, Mooncake, NVIDIA Dynamo, and the disaggregate-vs-chunked-prefill decision rule — is Knowledge 09 §3.)

Customer mapping: agent/RAG workload with huge shared prompts → prefix caching is the first win. Mixed interactive load with latency-spiky long prompts → chunked prefill. Massive scale with strict separate TTFT/TPOT SLOs → consider disaggregation. Matching technique to workload is the JD's "trade-off analysis."

6. The scheduler: the brain of the server

Every step, the scheduler decides: which waiting requests to admit (prefill), which running requests to continue (decode), and what to do when KV memory is full. Policies:

• Admission: how many new sequences to prefill this step (more = better throughput, worse TPOT for existing users — the throughput↔latency dial again).
• Preemption / swapping: when KV memory is exhausted, evict (recompute or swap to CPU) lower-priority sequences to make room — then resume them later. Prevents OOM crashes under load spikes.
• Priority / fairness: premium vs free tiers, latency-sensitive vs batch jobs.
• Max batch / max tokens: hard caps that bound memory and tail latency.

A principal understands that the scheduler config is the SLO policy. "We're hitting p99 TPOT violations under burst" → look at admission rate, chunked-prefill setting, and preemption behavior, not the model.

7. The frameworks compared: vLLM, TGI, TensorRT-LLM, Triton, others

The relationship to know (a common confusion): Triton Inference Server is a general serving host for any model/framework (CV, classifiers, ensembles) with dynamic batching — it is not LLM-specialized. TensorRT-LLM is the LLM engine; you can serve it via Triton's TRT-LLM backend, or via vLLM/TGI for other engines. JD2 names both because the role spans LLM and CV/multi-model serving (§9, Knowledge 07).

Recommendation a principal gives: "Open-source, multi-vendor, fast-moving → vLLM. Locked into NVIDIA and chasing the last 20% of latency → TensorRT-LLM (optionally behind Triton). Mixed CV + LLM multi-model pipeline → Triton as the host. Edge/air-gapped → llama.cpp/Ollama." Defend it on their workload and hardware, not fashion.

8. The knobs you actually tune

The settings that move the SLO (names vary by framework; concepts are universal):

The mistake juniors make is tuning for max throughput and reporting a big tokens/s number while quietly blowing the latency SLO. Tune for goodput — throughput subject to the p95/p99 latency constraint. That's the honest, customer-aligned objective.

9. Triton Inference Server: the general-purpose host

Because JD2 spans CV and multi-model pipelines, Triton (the server, not the kernel language) deserves its own note. It hosts models from any backend (TensorRT, ONNX Runtime, PyTorch, TF, Python, vLLM, TRT-LLM) behind one HTTP/gRPC endpoint and provides:

• Dynamic batching: automatically groups incoming requests into batches up to a configured size/delay — the CV/embedding analog of continuous batching.
• Model ensembles / Business Logic Scripting (BLS): chain models server-side (e.g., detector → tracker → recognizer for video pipelines) without round-trips to the client.
• Concurrent model execution: multiple models / instances on one GPU, with instance groups controlling parallelism.
• Model management: load/unload, versioning, A/B.
• Metrics: Prometheus endpoints for utilization, queue time, batch size.

When you'd reach for Triton: a customer's pipeline is multi-model and multi-modal (vision detection + an LLM + a re-ranker) and they want one serving plane with server-side orchestration and per-model batching. That's Triton's sweet spot, and it's squarely in JD2's "multi-model workflows" and "end-to-end pipeline" duties.

10. A tuning playbook for a target SLO

The repeatable procedure you'd run in an engagement (and in Lab 03):

1. Pin the SLO as numbers: p95 TTFT, p95 TPOT, target concurrency/QPS, accuracy floor.
2. Characterize the workload: input/output length distribution, request arrival pattern (steady vs bursty), prefix reuse.
3. Establish a baseline: default server config, measure throughput + latency percentiles under a realistic load generator (not single-request timing). Tools: vLLM's benchmark_serving, locust, or a custom async client; see Knowledge 06.
4. Find the binding constraint: KV-memory-bound (raise gpu_memory_utilization, quantize KV, GQA model)? Compute-bound prefill (chunked prefill, FP8)? Decode-bound (batch, spec decode)?
5. Sweep one knob at a time, plotting throughput vs p95 latency — you're mapping the goodput Pareto frontier.
6. Pick the operating point that maximizes throughput while meeting the SLO. That's your recommended config.
7. Stress + soak test: burst load, long-running stability, memory leaks, preemption behavior.
8. Document the config, the curve, and the headroom — the customer-ready runbook the JD wants.

The deliverable is a curve and a chosen point with justification, not a single number. "At this config we serve 240 concurrent users at p95 TTFT 380 ms / p95 TPOT 32 ms; pushing batch higher gains 15% throughput but breaks the TPOT SLO, so we hold here." That sentence is the job.

11. References

• Yu et al., "Orca: A Distributed Serving System for Transformer-Based Generative Models" (OSDI 2022) — continuous/in-flight batching.
• Kwon et al., "Efficient Memory Management for LLM Serving with PagedAttention" (SOSP 2023) — vLLM.
• Agrawal et al., "Sarathi-Serve" (OSDI 2024) — chunked prefill. Zhong et al., "DistServe" and Patel et al., "Splitwise" (2024) — prefill/decode disaggregation.
• Zheng et al., "SGLang / RadixAttention" (2024) — advanced prefix sharing.
• vLLM, Hugging Face TGI, NVIDIA TensorRT-LLM, NVIDIA Triton Inference Server official documentation.
• NVIDIA, "LLM Inference Benchmarking" and Triton Model Analyzer / Performance Analyzer docs.

➡ Next: Knowledge 05 — Distributed Inference — when one chip isn't enough.