Serving Terms — Throughput, Latency, and the Inference Engine Vocabulary

Phase 1 · Document 05 · LLM Vocabulary and Mental Models Prev: 04 — Model Capabilities · Next: 06 — Local Model Terms

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

The moment you move past "call the API and print the answer," you enter the world of serving: throughput, latency percentiles, batching, KV cache, prefill vs decode. This vocabulary is what vLLM dashboards, SLO documents, capacity plans, and provider status pages are written in. Misreading it leads to the two classic production failures: a system that's fast in a demo but collapses under concurrency (KV cache exhaustion), and a cost model that's wrong by 5× because you confused throughput with per-user latency. This document gives you the words; Phase 7 builds the systems.

2. Core Concept

The two phases of every request (the root of all serving behavior)

• Prefill: the engine processes your entire prompt in one parallel forward pass and builds the KV cache. It is compute-bound and scales with prompt length. It determines TTFT (time to first token).
• Decode: the engine generates output one token at a time, each step reading the whole KV cache. It is memory-bandwidth-bound and serial. It determines TPOT (time per output token).

total_latency ≈ TTFT + (TPOT × output_tokens)
TTFT  ← prefill (prompt length, batch, prefix cache hits)
TPOT  ← decode  (model size, hardware bandwidth, batch, speculative decoding)

Latency vs throughput (different, often opposed, metrics)

• Latency = time experienced by one request (TTFT, TPOT, end-to-end). What a user feels.
• Throughput = total work across all requests (tokens/sec, requests/sec). What your bill and capacity track.

They trade off: bigger batches raise throughput (better GPU utilization) but can raise individual latency. Serving is the art of maximizing throughput while keeping latency within SLO.

Batching

• Static batching: wait, group N requests, run together. Simple but wastes time padding/waiting.
• Continuous (in-flight) batching: the engine adds and removes requests from the running batch every decode step, so a finished sequence frees its slot immediately. This is the single biggest throughput win in modern serving (vLLM, TGI, SGLang).
• Chunked prefill: split a long prompt's prefill into chunks interleaved with ongoing decodes, so one huge prompt doesn't stall everyone else.

KV cache and PagedAttention (the real concurrency limit)

The KV cache stores attention keys/values for every token so decode skips recomputation. It grows linearly with (context length × concurrent sequences) and lives in GPU memory alongside the weights. KV cache — not the weights — usually caps how many requests you can serve at once. PagedAttention (vLLM) manages KV memory in fixed pages like an OS virtual-memory system, eliminating fragmentation and enabling far higher concurrency and prefix caching (sharing the KV of a common prompt prefix across requests).

The headline serving metrics

3. Mental Model

A serving engine is a KITCHEN:

PREFILL   = reading the whole order            (compute-bound, sets TTFT)
DECODE    = plating dishes one at a time        (bandwidth-bound, sets TPOT)
KV CACHE  = counter space holding each table's in-progress order
            → run out of counter space → can't seat new tables (concurrency cap)
CONTINUOUS BATCHING = chefs pick up the next ticket the instant a plate goes out
PAGEDATTENTION      = organizing counter space into tidy trays (no wasted gaps)
PREFIX CACHING      = reuse the prep for a shared appetizer across tables

THROUGHPUT = total plates/hour (the restaurant's capacity & your bill)
LATENCY    = how long ONE table waits (what the diner feels)
            → bigger batches feed more tables but each may wait a bit longer

4. Hitchhiker's Guide

What to look for first: TTFT and TPOT targets (your latency SLOs), and the KV-cache-limited concurrency (how many users you can serve at once). These drive both UX and cost.

What to ignore at first: kernel-level details (FlashAttention internals, CUDA graphs) — know they speed up prefill/decode; tune them in Phase 7.

What misleads beginners:

• Quoting average latency — tails (p95/p99) are what break SLOs under load.
• "It fits in memory, so it's fine" — weights fit, but KV cache for many concurrent long contexts may not.
• Confusing per-request tokens/sec with aggregate throughput.
• Assuming bigger batch = strictly better — it raises throughput but can blow latency SLOs.

How experts reason: they separate prefill vs decode and latency vs throughput in every conversation, size KV-cache memory for the concurrency × context they actually expect, and tune batch size to the point where throughput is high and p95 latency still meets SLO.

What matters in production: p95/p99 latency, KV-cache headroom under peak concurrency, graceful queueing/backpressure, and cost-per-token at realistic utilization (idle GPUs are pure loss).

Debug/verify: load-test at target concurrency; watch TTFT/TPOT and KV-cache utilization together; if TTFT spikes, suspect prefill queueing; if you get OOM under load (not at startup), suspect KV cache.

Questions to ask providers: What are your p50/p95 TTFT and TPOT? Do you do continuous batching and prefix caching? Is there per-request latency variance under load? What's the rate limit / max concurrency?

What silently gets expensive/unreliable: long contexts under high concurrency (KV blow-up), low GPU utilization (paying for idle), and tail latency that only appears at peak traffic.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Every request has two phases: prefill (compute-bound, sets TTFT) and decode (bandwidth-bound, sets TPOT).
• KV cache, not weights, usually caps concurrency; it grows with context length × concurrent sequences.
• Continuous batching is the largest single throughput improvement in modern serving.
• Latency and throughput trade off: larger batches raise throughput but can raise per-request latency.
• Averages hide tail latency — always track p95/p99 under realistic load.
• Prefix caching makes long, stable system prompts cheap and lowers TTFT.
• Streaming improves perceived latency (first token sooner) but not total time.
• Idle GPU time is wasted money — cost-per-token depends heavily on utilization.

9. Observations from Real Systems

• vLLM popularized PagedAttention + continuous batching; its metrics expose TTFT, TPOT, and KV-cache utilization directly.
• TGI (Hugging Face) and SGLang also do continuous batching; SGLang's RadixAttention is a powerful prefix-cache for shared-prefix workloads (agents, few-shot).
• OpenAI / Anthropic publish latency characteristics and support prompt caching to discount and speed up repeated prefixes — the managed version of prefix caching.
• OpenRouter surfaces per-provider latency/throughput so you can route to the fastest endpoint for a given model.
• Cursor and chat UIs lean on streaming so users see tokens immediately, masking total generation time.

10. Common Misconceptions

11. Engineering Decision Framework

Set SLOs first:
  Chat UI?        → optimize TTFT (prompt caching, smaller/cached prefix).
  Long generation? → optimize TPOT (speculative decoding, smaller model).
  Batch/offline?   → optimize THROUGHPUT (big batches, accept higher latency).

Capacity planning:
  expected_concurrency × avg_context → KV memory needed.
  KV memory + weights + overhead  ≤  GPU memory?  (see Phase 6 calculator)
  If not → shorter context limit, fewer concurrent slots, more/ bigger GPUs, or quantize.

Tuning batch size:
  Raise batch until throughput plateaus OR p95 latency hits SLO — stop at the binding one.

Buy vs self-host:
  Spiky/low volume → managed API (no idle GPU cost).
  Steady/high volume + data control → self-host vLLM, tune utilization (Phase 5).

12. Hands-On Lab

Goal

Measure TTFT, TPOT, and throughput against any OpenAI-compatible endpoint, and watch latency change with concurrency.

Prerequisites

• Python 3.10+; an OpenAI-compatible endpoint (a hosted API, or a local vLLM/llama.cpp server from Phase 6).

Setup

pip install openai
export OPENAI_BASE_URL=...   # your endpoint
export OPENAI_API_KEY=...

Steps

import time, statistics, concurrent.futures as cf
from openai import OpenAI
client = OpenAI()

def timed_call(prompt="Write a 200-word explanation of KV cache."):
    t0 = time.perf_counter(); first = None; n = 0
    stream = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role":"user","content":prompt}],
        max_tokens=256, stream=True)
    for chunk in stream:
        if chunk.choices[0].delta.content:
            if first is None: first = time.perf_counter()
            n += 1
    end = time.perf_counter()
    ttft = first - t0
    tpot = (end - first) / max(n-1, 1)
    return ttft, tpot, n/(end - t0)  # last = output tokens/sec

# Single-request metrics
ttft, tpot, tps = timed_call()
print(f"TTFT={ttft*1000:.0f}ms  TPOT={tpot*1000:.1f}ms  tok/s={tps:.1f}")

# Concurrency sweep: watch latency rise
for c in (1, 4, 16):
    with cf.ThreadPoolExecutor(max_workers=c) as ex:
        res = list(ex.map(lambda _: timed_call(), range(c)))
    ttfts = [r[0]*1000 for r in res]
    print(f"concurrency={c:>2}  p50 TTFT={statistics.median(ttfts):.0f}ms  max={max(ttfts):.0f}ms")

Expected output

• A clear TTFT (prefill) and TPOT (decode) split.
• TTFT/tail latency rising as concurrency increases.

Debugging tips

• TTFT ≈ TPOT and tiny? You may not actually be streaming — confirm stream=True.
• Errors at high concurrency on a local server → KV cache / max_num_seqs limit reached.

Extension task

Repeat with a short prompt vs a 2,000-token prompt; show how prompt length raises TTFT but barely affects TPOT.

Production extension

Log TTFT/TPOT per request to compute p50/p95/p99 over a load test, and chart latency vs concurrency to find your KV-cache-bound max.

What to measure

TTFT, TPOT, output tokens/sec; p50/p95 TTFT across concurrency 1/4/16; prompt-length effect on TTFT.

Deliverables

• A latency-vs-concurrency chart.
• A note: where does p95 TTFT exceed your chosen SLO?

13. Verification Questions

Basic

1. Define prefill and decode, and which latency metric each drives.
2. What is the KV cache and why does it limit concurrency?
3. Latency vs throughput — what's the difference and why do they trade off?

Applied 4. A request has 2,000 input and 300 output tokens. Which phase dominates latency, and which metric (TTFT/TPOT) does each token count map to? 5. Why does p95 latency rise with batch size even as throughput improves?

Debugging 6. Your service OOMs only under load, never at startup. What's the likely cause and fix? 7. TTFT is fine alone but terrible under concurrency. What's happening?

System design 8. Plan KV-cache memory for 50 concurrent users at 8K context. What inputs do you need, and what do you do if it doesn't fit?

Startup / product 9. Your unit economics depend on cost-per-token. Explain how continuous batching and GPU utilization affect that number, and one lever to improve margin.

14. Takeaways

1. Every request is prefill (→TTFT) then decode (→TPOT); optimize them differently.
2. KV cache, not weights, usually caps concurrency — size it for context × concurrent users.
3. Continuous batching + PagedAttention are why modern serving is fast and cheap.
4. Latency and throughput trade off; tune batch to the binding SLO.
5. Track p95/p99, not averages — tails break SLOs under load.
6. Prefix/prompt caching cuts TTFT and cost for stable prompts.

15. Artifact Checklist

• Code: TTFT/TPOT/throughput measurement script.
• Benchmark result: latency-vs-concurrency chart with p50/p95.
• Notes: the kitchen mental model and prefill/decode split.
• Capacity estimate: KV-cache memory for a target concurrency × context.
• SLO doc: TTFT/TPOT targets for one product surface.
• Cheat card: symptom → cause → lever table.

Next: 06 — Local Model Terms