Prefix and Prompt Caching

Phase 7 · Document 05 · Production Serving Prev: 04 — PagedAttention · Up: Phase 7 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Prefix/prompt caching is the highest-ROI optimization in LLM serving: for any workload that reuses a long prefix — a system prompt, few-shot examples, a CLAUDE.md, a RAG document set, a multi-turn chat history — it skips re-prefilling the shared part, cutting TTFT and cost dramatically (often 50–90% on cached tokens) for no quality change. It's the server-side realization of the prompt caching you met in the lifecycle doc, made possible by PagedAttention's block sharing (04). Knowing when it helps (and when it silently doesn't) is the difference between a chatbot that's cheap and one that re-pays for its 2,000-token system prompt on every single turn.

2. Core Concept

Plain-English primer: don't recompute what you already computed

Prefill turns the prompt into a KV cache — the keys/values for every token at every layer (Phase 2.06) — and that's the compute-heavy part of a request (Phase 2.07). If two requests start with the same tokens (e.g., the same system prompt), their KV for that shared prefix is identical. So why compute it twice? Prefix caching keeps the KV for already-seen prefixes and reuses it, so a new request with a matching prefix skips prefill for the shared part and starts decoding almost immediately.

This is exactly the §2 prompt caching mechanism, viewed from the server: PagedAttention stores KV in blocks (04); a cached prefix is just a set of physical blocks that a new sequence's block table can point at instead of recomputing (copy-on-write).

Why it must be an exact prefix

Because of attention, the KV at position i depends on every token 0..i (what-happens §2 internals). So you can only reuse an unbroken prefix from the start: match blocks token-for-token until the first difference, then prefill the rest. Change one token near the front and everything after it must be recomputed. This is physics, not policy — and it's why prompt ordering matters: put the stable content (system prompt, tools, fixed context) first and the volatile content (the user's new turn) last (what-happens §4).

How it's implemented

The server hashes the token sequence block by block (each block's key includes all preceding tokens), and keeps a lookup (hash map / radix tree) from prefix-hash → physical KV blocks. On a new request it walks blocks from the start, reusing cached blocks until the first miss (what-happens §2).

• vLLM: --enable-prefix-caching — automatic: hashes everything, shares across all requests with a common prefix, including different users with the same system prompt.
• SGLang: RadixAttention — organizes cached prefixes as a radix tree, maximizing reuse across branching prefixes (few-shot variants, agent trees) (02).
• Managed APIs: Anthropic uses explicit cache_control breakpoints (you mark the cacheable prefix; ~5-min TTL; billed as cache-write then cheap cache-reads); OpenAI/Google do automatic prompt caching above a token threshold. Pricing/automatic-vs-explicit differs by provider (Phase 4.04).

Hit rate is everything

The benefit is governed by cache hit rate — the fraction of prefill tokens served from cache:

• High hit rate (fixed long system prompt, shared few-shot, multi-turn chat where history is a growing stable prefix): huge TTFT + cost wins.
• Low/zero hit rate (every request has a unique prefix; or requests arrive slower than the TTL so the cache goes cold; or the volatile content is at the front): little to no benefit, and a tiny overhead.

So: maximize hit rate by stable-prefix-first ordering, keeping volatile content (timestamps, IDs, the user turn) out of the prefix, and keeping the cache warm.

3. Mental Model

   request A: [SYSTEM PROMPT ........][user A]   → prefill all, cache the system-prompt blocks
   request B: [SYSTEM PROMPT ........][user B]   → REUSE cached blocks, prefill only [user B]
                └──────── shared exact prefix ───────┘        ↑ TTFT ↓↓, cost ↓↓ (no quality change)

   RULE: reuse only an UNBROKEN prefix from the start (KV@i depends on tokens 0..i)
         → put STABLE first (system/tools/docs), VOLATILE last (user turn, timestamps)
   WIN governed by HIT RATE:  high (fixed prompt / chat history) → big; unique prefixes → ~0
   built on PagedAttention block sharing [04]; = server view of what-happens §2

Mnemonic: same start = computed once. Stable-prefix-first maximizes hit rate; a changed early token busts everything after it.

4. Hitchhiker's Guide

What to look for first: does your workload have a long, repeated prefix? If yes, enabling prefix caching is almost free money. Then measure hit rate and TTFT with it on vs off.

What to ignore at first: micro-optimizing block size or TTL. Get ordering right and turn it on; tune later.

What misleads beginners:

• Putting volatile content first. A timestamp/user-ID/"current date" at the front busts the cache every call — the cache only works on the unbroken prefix (what-happens §4).
• Expecting fuzzy matching. It's exact-prefix, not semantic — "almost the same" prompt is a miss.
• Confusing it with response caching. Prefix caching reuses KV (compute), still generates fresh output; response caching returns a stored answer for an identical request (different tool, different risks).
• Forgetting TTL/eviction. Cache is finite and time-limited; cold caches give no benefit. Anthropic's is ~5 min, refreshed on use.

How experts reason: they design prompts stable-prefix-first, isolate volatile bits to the tail, enable prefix caching by default, and track hit rate as a first-class metric. On managed APIs they place cache_control breakpoints (Anthropic) deliberately or structure prompts to trip automatic caching (OpenAI/Google), and they reconcile savings against the usage fields (cache_read_input_tokens, what-happens §10).

What matters in production: hit rate, TTFT improvement, cost reduction on cached tokens, and not letting cache-busting content creep into the prefix (the most common regression). For multi-tenant correctness, ensure cross-user prefix sharing is acceptable (it's KV reuse of identical public prefixes — usually fine; verify for sensitive system prompts).

How to debug/verify: compare TTFT and cache_read/gpu_cache_usage with caching on vs off on a repeated prefix; if hit rate is low, inspect prompt ordering for early-position volatility.

Questions to ask providers: automatic or explicit caching? TTL? cache-write vs cache-read pricing? minimum cacheable length? per-org isolation? (Phase 4.04)

What silently gets expensive/unreliable: volatile-first prompts (0% hit rate while you think it's on), cold caches under low traffic, and conflating prefix caching with response caching (stale answers).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Prefix caching reuses the KV of a shared prefix, skipping its prefill → lower TTFT and cost, no quality change.
• It's the server view of what-happens §2, enabled by PagedAttention block sharing (04).
• It's exact-prefix, not fuzzy — match from the start until the first differing token.
• Ordering rule: stable content first, volatile last — a changed early token busts everything after it.
• The win is governed by hit rate — high for fixed system prompts / multi-turn chat; ~0 for unique prefixes or cold caches.
• vLLM = automatic; SGLang = RadixAttention; Anthropic = explicit cache_control; OpenAI/Google = automatic (with thresholds).
• TTL matters (Anthropic ~5 min, refreshed on use) — low traffic = cold cache = no benefit.
• Prefix caching ≠ response caching — the former reuses compute, the latter returns a stored answer.

9. Observations from Real Systems

• Claude Code and other agents rely on prefix caching so the big stable prefix (system + tools + CLAUDE.md + history) is near-free each turn — a major reason long sessions stay affordable (what-happens §2/§4).
• Chatbots with long system prompts see 20–40% cost cuts and lower TTFT just by enabling it (Phase 4.04).
• RAG systems cache a stable retrieved-document prefix across follow-up questions; agent frameworks cache the tool definitions + few-shot (Phase 9, Phase 10).
• OpenRouter / LiteLLM pass provider prompt-caching through and expose the cache usage fields (what-happens §3.5, Phase 8).
• SGLang's RadixAttention shines for agent trees and few-shot where many requests share branching prefixes (02).

10. Common Misconceptions

11. Engineering Decision Framework

EXPLOIT PREFIX CACHING:
 1. Does the workload reuse a long prefix (system prompt / few-shot / docs / chat history)?
      NO  → little to gain; skip (tiny overhead). 
      YES ↓
 2. ORDER the prompt: stable first (system, tools, fixed context), volatile last (user turn, IDs, time).
 3. ENABLE it:
      self-host vLLM    → --enable-prefix-caching     (automatic)
      self-host SGLang  → RadixAttention              (automatic, branching-friendly)
      Anthropic         → set cache_control breakpoints (explicit) 
      OpenAI/Google     → structure prompt to trip automatic caching / cached content
 4. MEASURE hit rate, TTFT, and cost (cache_read tokens) on vs off.
 5. KEEP IT WARM (TTL) for low-traffic paths; KEEP VOLATILE OUT of the prefix.

12. Hands-On Lab

Goal

Measure prefix caching's effect on TTFT and cost as a function of hit rate, and prove that prompt ordering controls it.

Prerequisites

• A vLLM endpoint (01) and/or a managed API key (Anthropic/OpenAI). pip install openai httpx.

Setup

# Self-host: caching ON
vllm serve Qwen/Qwen2.5-1.5B-Instruct --max-model-len 8192 --enable-prefix-caching --port 8000
# Compare against a second run WITHOUT --enable-prefix-caching

Steps

1. Build a shared prefix: a ~1,500-token system prompt + a short varying user turn.
2. On vs off (self-host): send 50 requests (same system prompt, different user turns) with caching on vs off; record p50 TTFT for each. Expect a large TTFT drop with caching on (prefill skips the system prompt).
3. Ordering test: move a timestamp to the front of the prompt; re-run with caching on. TTFT should regress to ~uncached (every request now has a unique prefix) — proving stable-prefix-first matters.
4. Managed API cost: on Anthropic, send the same long prefix twice with a cache_control breakpoint; read usage.cache_creation_input_tokens (first call) then usage.cache_read_input_tokens (second) and compute the cost delta (what-happens §10). On OpenAI, observe cached_tokens in usage.
5. Hit-rate sweep: vary the fraction of requests sharing the prefix (10%→100%) and plot average TTFT/cost — the benefit scales with hit rate.

Expected output

A table: caching on/off and ordering → p50 TTFT; a managed-API cost comparison (cache-write vs cache-read); and a hit-rate-vs-savings curve.

Debugging tips

• No TTFT improvement with caching on → prefix isn't actually shared (volatile content at front, or prompt below the min cacheable length).
• Managed: no cache_read tokens → breakpoint misplaced or below threshold / cache expired (TTL).

Extension task

On SGLang, test branching prefixes (same few-shot, different final question vs different few-shot) and compare RadixAttention reuse to vLLM's linear prefix cache (02).

Production extension

Add hit-rate and cache-read-token metrics to your dashboard; alert if hit rate drops (a sign volatile content crept into the prefix) (08).

What to measure

p50 TTFT (on/off, good/bad ordering); cache-write vs cache-read tokens + cost; hit-rate-vs-savings curve.

Deliverables

• A TTFT on/off table and an ordering before/after.
• A managed-API cost comparison (cache-write vs read).
• A hit-rate-vs-savings curve + a prompt-ordering guideline for your app.

13. Verification Questions

Basic

1. What does prefix caching reuse, and what does it save?
2. Why must it be an exact prefix from the start?
3. What's the difference between prefix caching and response caching?

Applied 4. Where do you place a timestamp in a prompt, and why? 5. Anthropic explicit vs OpenAI automatic caching — how does your prompt design differ?

Debugging 6. Caching is "on" but TTFT didn't improve. Three likely causes. 7. Hit rate dropped overnight after a prompt change. What probably happened?

System design 8. Design prompt structure + caching for a RAG chatbot to maximize hit rate across follow-ups.

Startup / product 9. Quantify how prefix caching improves gross margin for a high-volume bot with a fixed 2k-token system prompt.

14. Takeaways

1. Prefix caching reuses a shared prefix's KV, cutting TTFT and cost with no quality change.
2. It's the server view of §2 prompt caching, built on PagedAttention block sharing (04).
3. Exact-prefix → order stable-first, volatile-last; a changed early token busts everything after.
4. Hit rate governs the win — huge for fixed prompts/chat, ~0 for unique prefixes or cold caches.
5. vLLM automatic · SGLang RadixAttention · Anthropic explicit · OpenAI/Google automatic — and it's not response caching.

15. Artifact Checklist

• A TTFT on/off comparison on a shared prefix.
• An ordering before/after (volatile-first busts the cache).
• A managed-API cost comparison (cache-write vs cache-read tokens).
• A hit-rate-vs-savings curve.
• A prompt-ordering guideline + a hit-rate alert for your app.

Up: Phase 7 Index · Next: 06 — Streaming