What Happens When You Prompt an LLM Agent (e.g. Claude Code)

Reference deep-dive (a sequential, nitty-gritty walkthrough — not a 15-section concept doc), modeled on alex/what-happens-when but for LLM agents. Up: References · Builds on: Phase 1 Vocabulary · Phase 2 Transformers · Phase 10 Agents · Phase 11 Coding Platforms

This document traces, end to end, what actually happens when you type a prompt into an agentic coding tool like Claude Code and hit enter — from your keystroke to tokens on a GPU and back. It answers the subtle questions: how context is built and reduced, why the same command can give different output, what "resume" does, what happens when you switch models mid-session, and what makes static files efficient. It also opens up the inference server itself — how the model actually computes a response (the forward pass), why prefill is fast and decode is slow, how one model serves thousands of users at once, and how it's distributed across many GPUs and machines. Every term is explained from zero, then taken to the nitty-gritty with code.

Accuracy note. The concepts (stateless model, context assembly, prefill/decode, the agent loop, caching, non-determinism, KV-cache paging) are universal and primary-sourced. Specific harness behaviors (Claude Code's CLAUDE.md, /compact, --resume, on-disk transcript layout) are described per Anthropic's public docs, standard agent design, and behavior observable on disk — but exact proprietary internals and file formats can change without notice. Treat the JSONL examples as illustrative of the observable shape, not a stable schema; verify details in the official docs (the curriculum's "go to primary sources" rule, Phase 0.02).

Contents

The one big idea: the model is stateless

Burn this in first, because every answer below follows from it:

The LLM has no memory. It is a pure function: tokens_in → next_token. Everything that feels like "memory," "session," or "the agent knowing things" is the harness (Claude Code) re-sending a reconstructed block of text — the context — on every single call.

The model is a stateless transformer that, given a sequence of tokens, predicts the next one (Phase 1.00). It does not remember your last message, your repo, or that it just read a file. The client/harness — the program you're talking to — owns all state and rebuilds the full input every turn. "Claude Code" is mostly context-management software wrapped around a stateless next-token function plus a tool-execution loop.

0. First: what is "context"?

Context is the exact, complete sequence of tokens the model sees in one forward pass. For a chat/agent model it's assembled from typed messages with roles:

  • system — the agent's standing instructions (persona, rules, how to use tools). Sent first, every turn.
  • user — your messages.
  • assistant — the model's prior replies (including its tool requests).
  • tool (a.k.a. tool_result) — the outputs your harness fed back after running a tool.

Plus, for an agent, two more things ride along in the context:

  • tool definitions — JSON schemas describing each tool the model may call (Read, Edit, Bash, Grep…). (Phase 1.04)
  • injected content — loaded files (CLAUDE.md), file contents you've read, and <system-reminder>-style notes the harness adds.

A chat template renders all of this — roles, tool schemas, content — into one flat token sequence with special delimiter tokens, then it's tokenized into integer token IDs (Phase 1.01). That flat sequence is the context.

[system prompt] [tool definitions] [history: user/assistant/tool_result …] [latest user message]
└──────────────────────────── one flat token sequence ────────────────────────────┘
                          ▲ everything the model "knows" this turn

Context window = the maximum number of tokens that sequence may contain (Phase 1.01). If it's not in the context, the model does not have it — Law 1 of Phase 1.00. Context is related to almost everything: it sets cost (you pay per token in it, Phase 1.08), latency (longer context → longer prefill/TTFT, Phase 2.07), memory (the KV cache grows with it), and quality (too much context → "lost in the middle", Phase 2.01).

What context actually is, concretely

To the model, "context" is not messages or roles — it's one flat array of integers. The harness keeps a list of dicts:

messages = [
  {"role": "system", "content": "You are Claude Code. Use tools to edit files…"},
  {"role": "user",   "content": "Add input validation to the login endpoint"},
  {"role": "assistant", "content": [
     {"type": "text", "text": "I'll find it."},
     {"type": "tool_use", "id": "toolu_01", "name": "Grep", "input": {"pattern": "login"}}]},
  {"role": "user", "content": [
     {"type": "tool_result", "tool_use_id": "toolu_01", "content": "auth/login.py:42:def login(req):"}]},
]

A chat template flattens that into a single string with special delimiter tokens that mark role boundaries (each model family has its own format):

<|system|>You are Claude Code…<|end|>
<|user|>Add input validation to the login endpoint<|end|>
<|assistant|>I'll find it.<tool_use>{"name":"Grep",…}</tool_use><|end|>
<|user|><tool_result>auth/login.py:42:def login(req):</tool_result><|end|>
<|assistant|>            ← generation starts HERE; the model predicts what comes next

…and the tokenizer turns that into integers — the only thing the GPU ever sees:

ids = [27, 91, 8948, 91, 29, 2675, 553, 14653, …]   # e.g. ~1,200 ints for the above
# THIS is the context. Not "a conversation" — a flat sequence of token IDs.

Three things follow that people miss:

  • Roles are not magic. <|user|> / <|assistant|> are just learned tokens; the model acts "like an assistant after <|assistant|>" only because it was trained on millions of examples in this exact format. Feed a malformed template and behavior degrades.
  • Tool definitions and tool_results live in the same sequence as your words — the model "calls a tool" by generating a tool_use block as text, and "sees a result" only because the harness pasted a tool_result block into the next prompt (Law 6). Nothing is out-of-band.
  • The context window is a hard cap on len(ids) for input and output together. Hit it and the harness must reduce — there is no "extra" memory anywhere. A 200k-token window literally means len(ids) ≤ 200,000.

1. The lifecycle: keystroke → tokens → answer

Here's the whole pipeline when you type a prompt in Claude Code and press enter.

Step 1 — You type; the harness captures input

The CLI/IDE captures your text. Nothing is sent yet.

Step 2 — The harness assembles the context (the request body)

This is the heart of an agent. The harness builds a messages array:

  1. System prompt — the agent's instructions (+ environment info, available tools policy).
  2. Tool definitions — JSON schemas for every tool (Read/Write/Edit/Bash/Grep/…).
  3. Loaded project context — e.g. CLAUDE.md, relevant memory, and any <system-reminder> notes.
  4. Conversation history — all prior user/assistant/tool_result messages this session.
  5. Your new message.

It also computes the token budget: sum the tokens, compare to the model's context window, reserve room for the output. If it's too big, it triggers reduction/compaction.

Step 3 — Caching markers and serialization

The harness marks the stable prefix (system prompt + tools + early history) for prompt caching (§2), serializes everything into the provider's request shape (Anthropic Messages API), and sets parameters (max_tokens, temperature, optionally thinking, tool config).

Step 4 — HTTPS request to the provider

The body is sent over TLS to the API endpoint. Server-side: authentication (API key), rate limiting (RPM/TPM, Phase 1.08), request validation, and routing to a model server.

Step 5 — Tokenization

The server renders the chat template and tokenizes: text → token IDs via the model's vocabulary (Phase 1.01). Roles become special tokens. This exact sequence is what the model computes on.

Step 6 — Prefill (reading the prompt)

The model processes the entire prompt in parallel in one forward pass, building the KV cache (attention keys/values for every token) (Phase 2.06, Phase 2.07). This is compute-bound and sets TTFT (time to first token). If a cache prefix hits (§2), the KV for that prefix is reused — prefill skips it → much faster + cheaper first token.

Step 7 — Decode (writing the answer)

The model generates one token at a time (Phase 2.05): produce logits → sample a token → append → repeat. This is memory-bandwidth-bound and sets TPOT (time per output token). Tokens stream back as Server-Sent Events (§10).

Step 8 — Tool use: the model requests an action

For an agent, the model often emits a tool_use block instead of (or before) a final answer — a structured request like Read({"file_path": "src/app.py"}). The model does not run anything — it just emits text shaped like a tool call (Phase 1.00 Law 6, Phase 10).

Step 9 — The agent loop (this is what makes it an "agent")

The harness:

  1. Parses the tool_use block and validates the arguments against the schema.
  2. Executes the tool itself (reads the file, runs the bash command) — with permissions/approval gates.
  3. Captures the result and appends it as a tool_result message.
  4. Calls the model again — with the now-larger context (your turn + the model's tool request + the tool result).
  5. Repeats until the model emits a final answer (no more tool calls) or a stop condition fires.
you → [model: "I'll read app.py" + tool_use(Read)] → harness runs Read → tool_result(file contents)
    → [model: "now edit line 42" + tool_use(Edit)] → harness runs Edit → tool_result(ok)
    → [model: "done, here's what I changed"]  ← final answer (end_turn)

Each arrow is a separate, stateless API call carrying the whole growing context. This is why a multi-step task makes many calls and accumulates tokens (cost grows per step — Phase 5.06).

Step 10 — Stop conditions

Generation/loop ends on: end_turn (model is done), max_tokens (hit the output cap), a stop_sequence, tool_use handed back to you for approval, or the harness's step limit. The stop_reason is returned.

Step 11 — Render + account

The final text is rendered to you. Usage is tallied: input tokens, output tokens, and cache tokens (creation/read) — your ground truth for cost (Phase 1.08).

The eleven steps above are the what. Because you asked for the systems view — how the model actually responds, how a server handles many users, how it's distributed — the rest of this section takes the middle of that pipeline (steps 5–9) down to the metal, with code.

1.A — The request body, concretely

When the harness "sends a prompt," this is the literal HTTP body (Anthropic Messages API shape). Everything from §0 gets serialized into it:

POST https://api.anthropic.com/v1/messages
{
  "model": "claude-…",
  "max_tokens": 4096,
  "temperature": 1.0,
  "system": [
    {"type": "text", "text": "You are Claude Code…",
     "cache_control": {"type": "ephemeral"}}     // ← marks the end of the cacheable prefix (§2)
  ],
  "tools": [
    {"name": "Read", "description": "Read a file from disk",
     "input_schema": {"type": "object",
        "properties": {"file_path": {"type": "string"}},
        "required": ["file_path"]}},
    {"name": "Bash", "description": "Run a shell command", "input_schema": { /* … */ }}
  ],
  "messages": [
    {"role": "user", "content": "Add input validation to the login endpoint"},
    {"role": "assistant", "content": [
       {"type": "tool_use", "id": "toolu_01", "name": "Grep", "input": {"pattern": "login"}}]},
    {"role": "user", "content": [
       {"type": "tool_result", "tool_use_id": "toolu_01",
        "content": "auth/login.py:42:def login(req):"}]}
  ]
}

Things worth internalizing here:

  • Tools are JSON Schemas, not functions. The model was post-trained to emit tool_use blocks whose input conforms to these schemas. It cannot run them — it only proposes a call (Law 6).
  • A tool_result rides inside a user message. That's the protocol: the harness reports tool output back as if the user said it, paired to the request by tool_use_id.
  • cache_control is where you draw the cache breakpoint (§2). Everything before it is the reusable prefix.
  • This whole JSON is about to become one token sequence — the server concatenates system + tools + messages via the chat template and tokenizes it.

1.B — Tokenization up close

Tokenization turns text into the integer IDs the model computes on. Modern LLMs use Byte-Pair Encoding (BPE): start from raw bytes, then greedily apply a table of "merge the most frequent adjacent pair" rules learned at training time, until the text is covered by ~100k–200k known sub-word tokens (Phase 1.01).

enc.encode("Hello, world!")              # → [9906, 11, 1917, 0]            (4 tokens)
enc.encode(" login")                     # → [5224]                         (1 token: common)
enc.encode("def login(req):")            # code packs tight — keywords/punct are single tokens
enc.encode("anticonstitutionnellement")  # rare word → splits into many sub-tokens
enc.encode("速度")                        # CJK → often ~1 token per character or more

Rules of thumb you can budget with:

  • ~4 characters ≈ 1 token, or ~0.75 words ≈ 1 token, for English prose.
  • Code is dense (language keywords, brackets, common identifiers are single tokens) → cheap per "idea". This is part of why §5 prefers code/compact forms.
  • Rare words and non-English fragment into many tokens → 1.5–3× more tokens for the same meaning → more cost and more window used.
  • Whitespace and indentation are tokens too — pretty-printed JSON and deep indentation waste budget.
  • token ≠ word ≠ character. "Counting words" is the wrong mental model; only token counts predict cost, latency, and whether you fit the window.

The chat template also injects special tokens the tokenizer reserves: a begin-of-sequence marker, role delimiters, and an end-of-turn token. You never type these; the template adds them, and the model's behavior depends on them being exactly right.

1.C — How the model "responds": one forward pass, step by step

The model is just weight matrices in GPU memory (the "parameters"). "Responding" is computing a function over the token IDs. For the whole sequence, one forward pass is:

def forward(token_ids, W):                       # W = the loaded weights (in VRAM)
    x = W.embedding[token_ids]                   # (seq, d): each ID → a vector (a lookup)
    for layer in W.layers:                        # e.g. 32–80 transformer blocks, in order
        # ---- attention sublayer: mix information ACROSS positions ----
        h = rms_norm(x, layer.norm1)
        q, k, v = h @ layer.Wq, h @ layer.Wk, h @ layer.Wv   # linear projections
        q, k = apply_rope(q, k)                   # inject token positions (RoPE)
        scores = (q @ k.transpose(-1, -2)) / sqrt(d_head)
        scores = causal_mask(scores)              # a token may not attend to the future
        attn = softmax(scores) @ v                # weighted sum of past tokens' values
        x = x + attn @ layer.Wo                   # residual add (note the "x +")
        # ---- feed-forward sublayer: transform EACH position ----
        h = rms_norm(x, layer.norm2)
        x = x + swiglu(h, layer.W_gate, layer.W_up, layer.W_down)   # residual add
    x = rms_norm(x, W.final_norm)
    logits = x @ W.unembedding.T                  # (seq, vocab): a score for EVERY token
    return logits[-1]                             # the distribution for the NEXT token

So "the model responding" is: embedding lookup → a stack of (attention + feed-forward) blocks that repeatedly mix and transform the vectors → a final projection to one logit per vocabulary token (a vector of ~150,000 numbers). The sampler (§1 step 7, §6) turns those logits into one chosen token. Then that token is appended and the whole thing runs again for the next one. There is no hidden state between calls, no "thinking" stored anywhere — just this pure matmul pipeline re-run per token. (Each sublayer maps to Phase 2: embeddings 2.01, attention 2.02, FFN 2.03, norms/residuals 2.04.)

Cost intuition you can carry everywhere: a forward pass costs ≈ 2 × (number of parameters) FLOPs per token (one multiply + one add per weight). A 70B model ≈ 140 GFLOP per token. That single fact explains why bigger models are slower and pricier — it is literally more arithmetic per token.

1.D — Prefill vs decode at the metal (compute-bound vs memory-bound)

These two phases (Phase 2.07) feel different because they bottleneck on different hardware limits:

  • Prefill runs the forward pass over all prompt tokens at once. The matmuls become large (seq × d) · (d × d) — exactly what GPU tensor cores devour. It's compute-bound: the limiter is FLOPs. Its product is the KV cache (every token's K and V at every layer) plus the first output token. Time grows with prompt length → this is your TTFT. A cache hit (§2) lets prefill skip the cached prefix.
  • Decode generates token N+1 using the cached KV: only one new token flows through the network. The matmuls are tiny (1 × d), but to do them the GPU must read every weight from VRAM. For a 70B fp16 model that's ~140 GB streamed per token. So decode is memory-bandwidth-bound: the limiter is moving weights, not math. Time per token ≈ model_bytes ÷ HBM_bandwidth. At ~3 TB/s, 140 GB ≈ 47 ms → ~21 tokens/sec for a single stream. This is your TPOT, and why long answers stream out steadily instead of appearing instantly.
PREFILL  (compute-bound)            DECODE  (memory-bandwidth-bound)
 ┌──────────────────────┐           ┌───┐┌───┐┌───┐┌───┐  one token each step
 │  all prompt tokens    │   →       │ t │→│ t │→│ t │→ …   reads ALL weights per step
 │  in ONE big matmul    │           └───┘└───┘└───┘└───┘
 └──────────────────────┘
 builds KV cache, sets TTFT          appends to KV cache, sets TPOT

The punchline that explains all of LLM serving: since decode reads all the weights regardless of how many tokens it computes, you can compute many users' next tokens in the same weight sweep by batching them. One 47 ms read can produce 1 token for 1 user or 1 token each for 100 users — ~100× the throughput for nearly the same wall-clock. Batching is the economic engine of inference.

1.E — Inside the inference server: how one model serves thousands of users at once

A production endpoint (vLLM, TGI, TensorRT-LLM, or a provider's in-house server) is far more than "load model, call model":

  • Weights load once, into VRAM, read-only. Every request shares the same weights. What's private per request is its KV cache and its position in generation.
  • A scheduler + request queue. Requests arrive continuously and wait in a queue. Each decode step, the scheduler decides which sequences are in the running batch.
  • Continuous (in-flight) batching. Instead of a fixed batch that waits to fill, the server adds new requests (they prefill, then join) and drops finished ones every single step. This keeps the GPU saturated — and is the concrete reason your request shares a batch with strangers' requests.
  • The KV cache is the scarce resource — not compute. It grows by layers × 2 × heads × d_head values per token per request. PagedAttention (§3.5) allocates it in fixed blocks; when VRAM for KV runs out the scheduler queues, preempts, or evicts sequences. How many users fit is usually a KV-memory limit.
  • Chunked prefill. A giant prompt's prefill is split into chunks interleaved with others' decode steps, so one 100k-token request doesn't freeze everyone (latency fairness).
  • Multi-tenancy consequences: your latency depends on current load (noisy neighbors); your numerics depend on batch composition (the deep cause of §7); and the operator tunes batch size to trade throughput (big batch) against per-request latency (small batch).
requests →  [ QUEUE ] → SCHEDULER ──► running batch on the GPU(s)
                          │            ┌──────────────────────────────┐
                          │            │ weights (shared, in VRAM)     │
   new req → prefill ─────┘            │ + KV cache per seq (paged)    │
   finished seq ◄── drop from batch    │ continuous batching each step │
                                       └──────────────────────────────┘

1.F — How the model is distributed across GPUs and machines

When a model is too big for one GPU, or one replica can't serve enough users, providers split and replicate it. Four orthogonal axes (Phase 6, Phase 7 go deeper):

  • Tensor parallelism (TP) — split each matrix across GPUs. Every GPU holds a slice of every layer's weights, computes part of each matmul, and they combine results with an all-reduce (NCCL over NVLink). This splits a single forward pass across (typically) 2–8 GPUs in one node. Cuts per-GPU memory and speeds each token, at the cost of inter-GPU communication every layer.
  • Pipeline parallelism (PP) — split the layers across GPUs/nodes. GPU 0 runs layers 1–20, GPU 1 runs 21–40, … tokens flow down the pipeline like an assembly line; micro-batches keep every stage busy. Lets a model span multiple machines.
  • Expert parallelism (EP) — for MoE models (Phase 2.08): place different experts on different GPUs; each token is routed to the GPU(s) holding its chosen experts.
  • Data parallelism / replicas — full copies behind a load balancer. This scales users/throughput (not model size). A provider serving millions runs many replicas, each itself TP×PP across several GPUs.
                       ┌── replica A ── [GPU0│GPU1│GPU2│GPU3]  (TP within node)
your request → GATEWAY  ┤   (auth, rate                     …  (PP across nodes)
 (auth/rate/route) → LB │── replica B ── [GPU0│GPU1│GPU2│GPU3]
                       └── replica C ── …                     (autoscaled up/down)

The topology you actually hit: request → gateway (auth, rate-limit, route to the right model pool) → load balancer → one replica (TP across GPUs in a node, maybe PP across nodes) → your tokens stream back. Autoscaling adds/removes replicas with demand. Crucially, different replicas may run different builds, quantization, or versions → output drift between otherwise-identical calls; system_fingerprint is the breadcrumb. This is the mechanism underneath provider variance (§9, Phase 5.10).

1.G — The agent loop, in code

Strip away the UI and the entire "agent" is this loop — each iteration a fresh, stateless call carrying the whole growing messages:

messages = [{"role": "user", "content": user_prompt}]
while True:
    resp = client.messages.create(model=MODEL, system=SYSTEM, tools=TOOLS,
                                  messages=messages, max_tokens=4096)
    messages.append({"role": "assistant", "content": resp.content})   # record the model's turn

    tool_uses = [b for b in resp.content if b.type == "tool_use"]
    if resp.stop_reason != "tool_use" or not tool_uses:
        break                                       # final answer → exit

    results = []
    for tu in tool_uses:                            # may be several (parallel tool calls, §10)
        output = run_tool(tu.name, tu.input)        # ← THE HARNESS executes it, with permission gates
        results.append({"type": "tool_result", "tool_use_id": tu.id, "content": output})
    messages.append({"role": "user", "content": results})   # feed results back as a user turn
    # loop again — the next call re-sends the ENTIRE messages array (statelessness in action)

Every concept in this document hangs off that loop: caching (§2) makes the unchanged front of messages cheap; reduction (§3) keeps it under the window; persistence (§4) writes each appended turn to disk; resume (§8) reloads messages and re-enters the loop.

2. Prompt caching: how the cost is "reduced"

The biggest "reduction" lever isn't shrinking context — it's not re-paying for the stable part of it.

The problem: in an agent session, the system prompt + tool definitions + early history are identical on every call and can be many thousands of tokens. Re-prefilling them every turn is wasteful.

Prompt caching (a.k.a. prefix caching, Phase 2.06): the provider stores the KV cache for a prompt prefix. On the next call, if the prefix matches exactly, the cached KV is reused — the model skips prefilling it.

  • Cost: cached input tokens are ~90% cheaper (read price) than fresh input (Phase 1.08, Phase 4.04). Anthropic bills a one-time cache-write (slightly more than normal input) then cheap cache-reads.
  • Latency: TTFT drops sharply because prefill skips the cached prefix.
  • Mechanics: caching is prefix-based and exact-match — it works from the start of the prompt up to a cache breakpoint. It has a short TTL (~5 minutes on Anthropic; refreshed on use). Anything that changes the prefix invalidates everything after it.

This is why ordering matters (and connects to §5): put the stable stuff first (system prompt, tools, CLAUDE.md, unchanging history) and the volatile stuff last (the new user turn, fresh tool results). A single changed token early in the prompt busts the cache for the whole rest. Claude Code leans heavily on prompt caching, which is a big reason long sessions stay affordable.

[ STABLE PREFIX — cached: system + tools + CLAUDE.md + old turns ]  ← reused, ~90% cheaper, fast
[ VOLATILE TAIL — fresh: new user msg + latest tool_result ]        ← re-prefilled each turn
   change anything in the stable prefix → cache MISS for everything after it

How prefix caching is actually implemented

The provider doesn't cache "your prompt" as text — it caches KV blocks keyed by the tokens that produced them:

  • The token sequence is chopped into fixed blocks (e.g. 16 tokens). Each block gets a hash that includes all preceding tokens, so block k's key = hash(tokens[0 : (k+1)·16]).
  • A lookup structure — a hash map or radix tree of blocks — maps those keys → physical KV blocks already computed and sitting in GPU memory (§1.E).
  • On a new request the server walks blocks from the start: while a block's key is found, it reuses that KV (skips recompute); at the first miss it stops reusing and prefills the rest. "Stop at the first difference" is the exact-prefix rule.

Why only a prefix, never the middle: because of attention, the KV at position i depends on every token 0..i (§1.C). Change token 3 and the KV for tokens 4..N is now wrong — so everything after an edit must be recomputed. That's the deep reason the rule above is physics, not policy.

Two flavors you'll meet:

  • Explicit (Anthropic cache_control) — you mark the breakpoint (§1.A); the prefix up to it is cached with a short TTL (~5 min) and LRU eviction, billed as one cache-write then cheap cache-reads.
  • Automatic (vLLM enable_prefix_caching) — the server hashes blocks automatically and shares them across all requests with a common prefix, including different users who happen to share a system prompt. Same mechanism, no breakpoint needed.

This is why §4's stable system + tools + CLAUDE.md front is near-free on every turn, and why a single edited byte near the top of CLAUDE.md re-prices the whole session's prefix.

3. Context reduction: compaction when it fills up

As a session runs, the context grows (every turn + every tool result is appended) toward the context-window limit. The harness reduces it to stay under budget. The toolkit, roughly in order of preference:

TechniqueWhat it actually doesLossy?Cost to do itCache impactLives in
Prompt caching (§2)Reuse KV for an unchanged prefixNo~0 (saves money)Is the cacheProvider / inference server
Compaction / summarizationReplace a long span of old turns with a model-written summaryYes (detail)An extra model callBusts everything after the cutHarness (client)
Sliding window / truncationDrop oldest turns to fit a budgetYes (hard drop)~0 (just slicing)Busts prefix at the cutHarness, proxy, or model arch
Tool-result trimming / elisionShorten or remove large tool outputsYes (that output)~0Busts from edit pointHarness
Retrieval instead of stuffingKeep the corpus outside context; fetch only the relevant slice per queryNo (nothing lost — it's re-fetchable)An embed/search (or a tool call)Keeps prefix small & stableHarness + a tool / vector store
SubagentsPush a sub-task into its own context window; return only a summaryNo (isolated)A whole sub-sessionSeparate cacheHarness (orchestration)

The rest of this section explains the four interesting ones — compaction, the two senses of "sliding window," retrieval-vs-stuffing, and subagents — mechanically, with code, and then from the serving/proxy angle (vLLM, LiteLLM, OpenRouter), because that's where most of the real "seasoning" lives.

Before the mechanics, the key distinction. There are two different things people sloppily call "context," and reduction touches both:

  1. The messages array — a list of dicts in the harness's RAM (and mirrored on disk). This is what you and the harness edit when you "compact" or "truncate." It's just data structures.
  2. The KV cache — large GPU tensors on the inference server holding one (key, value) pair per token per layer. This is what makes long context expensive in memory and compute. You never touch it directly; it's a consequence of how many tokens you sent.

Shrinking #1 (fewer tokens sent) is what shrinks #2 (smaller KV cache, faster prefill). Caching (§2) is the one lever that touches #2 without shrinking #1 — it reuses KV you already paid to build.

3.1 Compaction / summarization — what really happens

The plain idea: when the transcript gets long, you can't keep every word. So you ask the model to write a summary of the old part, then throw the old part away and keep the summary. The summary is far smaller but preserves the gist.

What mechanically happens (this is the part a glossary skips). Compaction is itself a model call:

def compact(messages, model, keep_recent=6):
    """Summarize everything except the last `keep_recent` turns, then
       replace that old span with ONE synthetic message."""
    head = messages[:-keep_recent]          # the OLD span we will discard
    tail = messages[-keep_recent:]           # recent turns we keep verbatim

    # 1) An EXTRA inference call whose only job is to compress `head`.
    summary = model.complete([
        {"role": "system", "content":
            "Summarize the conversation so far for handoff to a fresh context. "
            "PRESERVE: decisions made, file paths touched, exact identifiers, open "
            "TODOs, user constraints, and the current goal. DROP: chit-chat, "
            "resolved dead-ends, raw tool output already acted on. Be terse."},
        *head,
    ])

    # 2) Rebuild the array: summary stands in for the entire head.
    return [
        {"role": "user", "content": f"[Summary of earlier conversation]\n{summary}"},
        *tail,
    ]

Three consequences fall straight out of that code:

  1. It costs tokens to save tokens. You pay to read the whole head (input) plus write the summary (output) once, in exchange for every future call being smaller. It only pays off if the session continues for a while afterward — which is why you compact at a boundary, not mid-thought.
  2. It is lossy and irreversible (in-context). Whatever didn't make it into the summary is gone from the model's view. A constraint you stated 40 turns ago ("never touch legacy/") survives only if the summarizer judged it important. This is exactly why a freshly-compacted session can "forget" a nuance — best practice: restate hard constraints right after a compaction.
  3. It nukes the cache. Remember caching is prefix-exact (§2). Compaction rewrites the front of the array, so the cached KV for the old prefix is now worthless. The first post-compaction call re-pays a cache-write on the new (shorter) prefix. Net effect: a cost spike at compaction, then cheaper calls afterward.

In Claude Code specifically: there's /compact (you trigger it, optionally with an instruction like /compact focus on the auth refactor) and auto-compaction that fires when the context approaches the window limit. The summary becomes the new opening of the working context, and — crucially for §4 — it's also recorded in the on-disk transcript so --resume can rebuild from it.

BEFORE:  [sys][tools][CLAUDE.md] | turn1 turn2 … turn38 turn39 turn40
                                   └──────── head (38 turns) ────────┘ keep tail →
COMPACT (one model call summarizes the head)
AFTER:   [sys][tools][CLAUDE.md] | [SUMMARY of turns 1–38] turn39 turn40
                                    └ ~300 tokens instead of ~30,000 ┘

3.2 Sliding window & truncation — two different things with the same name

This term is overloaded, and conflating the two is a classic interview trip-wire. Pull them apart:

(a) Conversation-level sliding window (a harness/proxy choice). "Keep the last N tokens (or messages); drop the oldest." It's a deque with a max length. No model call, no summary — just slicing. Cheaper than compaction but bluntly lossy: dropped turns are simply gone, with no summary standing in for them.

def trim_to_budget(messages, max_tokens, count_tokens):
    """Keep the system message + as many of the NEWEST turns as fit.
       This is a sliding window over the conversation (a.k.a. truncation)."""
    system = [m for m in messages if m["role"] == "system"]
    rest   = [m for m in messages if m["role"] != "system"]
    budget = max_tokens - sum(count_tokens(m) for m in system)

    kept = []
    for m in reversed(rest):              # walk newest → oldest
        c = count_tokens(m)
        if budget - c < 0:
            break                         # the rest falls off the window
        budget -= c
        kept.append(m)
    return system + list(reversed(kept))

This is essentially what LiteLLM's trim_messages and OpenRouter's middle-out transform do at the proxy layer (see §3.5). Notice it keeps the system message and the newest turns and drops the middle/oldest — because models attend best to the edges of the context ("lost in the middle", Phase 2.01).

(b) Sliding-Window Attention, SWA (a model architecture property). This is a completely different thing, baked into models like Mistral 7B, Mixtral, Gemma, and several "OSS" architectures. Normally token i attends to all previous tokens (cost grows O(n²) in sequence length, and the KV cache grows O(n)). With SWA, token i only attends to the previous W tokens (e.g. W=4096). That bounds per-token attention cost to O(W) and lets the server keep only the last W tokens' KV — a rolling buffer — instead of all of them.

You implement SWA with a banded attention mask (causal and windowed):

import torch

def sliding_window_mask(seq_len, window):
    # token i may attend to tokens (i-window, i]  →  causal AND within the window
    i = torch.arange(seq_len)[:, None]    # (seq, 1)
    j = torch.arange(seq_len)[None, :]    # (1, seq)
    causal = j <= i                       # can't look at the future
    within = j > (i - window)             # can't look further back than W
    return causal & within                # True = this (query i, key j) pair is allowed

And the memory payoff shows up as a rolling (circular) KV cache — you overwrite the oldest slot instead of growing forever:

class RollingKVCache:
    """SWA only needs the last W tokens' K/V → memory is capped at O(W), not O(n)."""
    def __init__(self, window, d):
        self.W = window
        self.K = torch.zeros(window, d)
        self.V = torch.zeros(window, d)
        self.pos = 0                       # next write slot (wraps around)
        self.count = 0
    def append(self, k, v):                # k, v: shape (d,)
        self.K[self.pos] = k
        self.V[self.pos] = v
        self.pos = (self.pos + 1) % self.W # circular overwrite of the OLDEST token
        self.count = min(self.count + 1, self.W)

Why you must keep these straight: (a) is your decision in the harness/proxy — it's about which messages to send. (b) is the model's fixed behavior — it's about what the model can physically attend to regardless of what you send. If you send 100k tokens to a SWA model with W=4096, the model literally cannot attend across the whole thing; tokens far apart never "see" each other directly. That's a serving-fidelity subtlety (cousin of Phase 5.10): two providers serving the "same" weights but one with a smaller served window will behave differently on long inputs.

3.3 Tool-result trimming / elision

Tool outputs are often the biggest, least durable thing in an agent context: a 3,000-line file you Read, a giant npm test log, a git diff. Once the model has acted on that output, keeping the full text wastes budget every subsequent turn.

So the harness elides old tool results — replaces the body with a stub like [output truncated — 2,910 lines omitted] while keeping the structural tool_result record (so the tool_use/tool_result id pairing stays valid — §10). The model still knows that it read the file and roughly what happened, but the raw bytes are gone.

def elide_old_tool_results(messages, keep_last=3, head=200, tail=200):
    """Keep recent tool outputs verbatim; shrink older ones to head+tail slices."""
    tool_idxs = [i for i, m in enumerate(messages) if m.get("role") == "tool"]
    for i in tool_idxs[:-keep_last]:                 # all but the most recent few
        body = messages[i]["content"]
        if len(body) > head + tail:
            omitted = len(body) - head - tail
            messages[i]["content"] = (
                body[:head] + f"\n…[{omitted} chars elided]…\n" + body[-tail:])
    return messages

This is "reduction" with the best ROI per lost-information: huge token savings, and the dropped content (a file, a log) is usually re-fetchable on demand — which is exactly the next idea.

3.4 Retrieval instead of stuffing — what this really means

This is the most important architectural idea in the whole section, so let's not gloss it.

"Stuffing" = the naive approach: take everything that might be relevant and paste it into the context. All 40 files, the whole wiki, the entire schema. It "works" until it doesn't: you blow the context window, you pay for tens of thousands of tokens every call, and quality drops because the needle is buried in a haystack ("lost in the middle").

"Retrieval" = keep the big corpus outside the context, in a store you can query. At question time, find the few most relevant chunks and inject only those. The context stays small, cheap, and focused; the corpus can be arbitrarily large.

There are two flavors, and a coding agent like Claude Code leans on the second:

Flavor 1 — Embedding-based retrieval (classic RAG, Phase 9). Pre-compute a vector ("embedding") for every chunk; at query time embed the question and grab the nearest chunks by cosine similarity.

import torch
import torch.nn.functional as F

def embed(texts):           # placeholder: a real call to an embedding model
    ...                     # returns a (N, d) float tensor, one row per text

# Done ONCE, offline. The corpus lives in a vector index, NOT in the context.
corpus      = load_all_chunks()        # could be millions of tokens total
corpus_emb  = embed(corpus)            # (N, d), stored in a vector DB

def retrieve(query, k=4):
    q    = embed([query])                          # (1, d)
    sims = F.cosine_similarity(q, corpus_emb)      # (N,) similarity to every chunk
    top  = sims.topk(k).indices                    # the k nearest chunks
    return [corpus[i] for i in top]                # ONLY these k go into the prompt

# The prompt becomes: system + question + retrieve(question)   ← small & focused

The win is quantitative: stuffing a 2,000-chunk corpus might be 1,000,000 tokens (impossible/expensive); retrieving k=4 chunks is ~1,200 tokens. Same answer quality or better, ~1000× cheaper context.

Flavor 2 — Agentic / tool-based retrieval (what coding agents mostly do). Claude Code usually doesn't maintain a vector DB of your repo. Instead the model decides what to fetch using tools, and the harness returns just that slice:

model → tool_use: Grep({"pattern": "def login", "glob": "**/*.py"})
harness runs ripgrep → tool_result: "auth/login.py:42:def login(req):  …"   ← 5 lines, not the repo
model → tool_use: Read({"file_path": "auth/login.py", "offset": 30, "limit": 60})
harness → tool_result: <only lines 30–90 of that one file>                  ← a retrieved slice
model → now it has exactly what it needs, and nothing else

That's retrieval too — the "index" is your filesystem, and Grep/Glob/Read are the query engine. The principle is identical: don't carry the haystack in context; fetch the needle on demand. It's also why Read takes offset/limit — so the agent pulls a slice, not a whole 5,000-line file.

The trade-off vs stuffing: retrieval can miss (if the search/embedding doesn't surface the right chunk, the model never sees it — a silent Law-1 failure). Stuffing never misses but is expensive and noisy. Mature agents blend them: cheap retrieval first, with the ability to fetch more if the first slice was insufficient.

3.5 Another perspective: where this happens in the serving stack

Everything above lives in the harness (client). But you asked the sharp question: what about a proxy or inference server — vLLM, LiteLLM, OpenRouter? These sit between your harness and the GPU, and each touches these mechanisms differently. This is exactly the kind of thing that separates someone who's read about LLMs from someone who's run them.

vLLM — the actual inference server (the GPU side). vLLM does not do conversation compaction or summarization — it serves whatever token sequence you hand it. What vLLM does own is the KV cache in GPU memory, and its headline contribution is PagedAttention: instead of one giant contiguous KV tensor per request, the KV cache is split into fixed-size blocks (e.g. 16 tokens each), managed like OS memory pages. A per-sequence block table maps logical token positions to physical blocks.

// PagedAttention (vLLM), conceptually. KV is NOT one contiguous tensor.
constexpr int BLOCK = 16;                       // tokens per block (like a memory page)
struct KVBlock { half K[BLOCK][HEADS][HEAD_DIM];
                 half V[BLOCK][HEADS][HEAD_DIM]; };

KVBlock kv_pool[NUM_PHYSICAL_BLOCKS];           // one big pool of physical blocks
int32_t block_table[NUM_SEQS][MAX_BLOCKS];      // per-seq: logical block → physical id

// Two requests that SHARE A PREFIX can point their first blocks at the SAME
// physical blocks (copy-on-write). That is server-side PREFIX CACHING:
// the shared system prompt / CLAUDE.md is prefilled once, reused by all.

The attention kernel then gathers KV block by block via that indirection table:

// Simplified paged-attention inner loop for one query token.
for (int b = 0; b < num_blocks_for_seq[seq]; ++b) {
    int phys = block_table[seq][b];             // indirection → physical block
    const KVBlock* blk = &kv_pool[phys];
    // accumulate dot(query, blk->K[...]) → scores; softmax; weighted sum of blk->V[...]
}

Two payoffs map straight back to this section: (1) no fragmentation — blocks pack tightly, so you fit more concurrent sequences (higher throughput); (2) automatic prefix caching (enable_prefix_caching) — sequences sharing a prefix share physical blocks, which is the server-side realization of the prompt caching from §2. vLLM also implements SWA (§3.2) as a rolling block buffer, and continuous batching (new requests slot into a running batch each step) — which is also one source of the non-determinism in §7, because your request's batch-mates change the floating-point reduction order.

LiteLLM — the unifying proxy/SDK (the translation side). LiteLLM's main job is to give you one API shape (usually OpenAI-style) across 100+ providers, mapping your call to Anthropic/Bedrock/vLLM/etc. and normalizing responses. Relevant to reduction:

  • It ships a trim_messages(messages, model) helper — literally the conversation-level sliding window from §3.2: it trims to fit the target model's window, keeping the system message and most-recent turns. So trimming can happen at the proxy, not just in your harness.
  • It has a caching layer (in-memory/Redis) for whole-response caching, and it can pass through provider prompt-caching headers — but it does not summarize/compact for you.
  • It does fallbacks and routing across models/keys. The catch: if it fails over to a model with a different tokenizer or smaller window, your carefully-budgeted context math changes underneath you (§9).

OpenRouter — the aggregator/router (the marketplace side). OpenRouter forwards your request to one of several upstream providers for a given model ID. Two features intersect this section:

  • Transforms — middle-out. When your prompt exceeds the chosen model's context window, OpenRouter can compress the middle of the message list (drop/condense middle messages, keep the edges) so it fits — an automatic sliding-window/truncation transform applied at the proxy, often without you noticing. Great for "just make it fit," dangerous if you assumed all your context arrived intact.
  • Provider selection decides which backend (and thus possibly which quantization / served window) handles your "same" model ID — the direct tie-in to Phase 5.10 — Provider Variance & Serving Fidelity. Prompt caching only applies if the chosen upstream supports it, so caching economics can vary call to call.

The mental model across the stack:

YOUR HARNESS            PROXY (LiteLLM / OpenRouter)        INFERENCE SERVER (vLLM / provider)
─────────────           ───────────────────────────        ──────────────────────────────────
builds messages   →     may TRIM (trim_messages /      →    tokenizes; PREFILL builds KV cache
compaction/summary      middle-out) to fit window           (PagedAttention blocks);
retrieval (tools)       routes/falls back to a model        PREFIX CACHING shares prefix blocks;
elides tool results     passes through cache headers        SWA rolling buffer; continuous batch
                        picks provider (quant/window!)      DECODE streams tokens back

Lesson: "reduction" is not one thing in one place. Your harness summarizes and retrieves; a proxy may silently trim or compress to fit; the inference server pages and shares KV blocks. When output looks wrong or context seems "forgotten," ask at which layer something got dropped.

3.6 Subagents — reduction by isolation

The cleverest reduction is to never put the tokens in the main context at all. A subagent is a spawned, separate agent loop with its own fresh context window. The parent hands it a task ("audit every file in src/ for SQL injection"), the subagent burns thousands of tokens reading 50 files in its own context, and returns only a short summary to the parent. The parent's context grows by a paragraph, not by 50 files.

PARENT context (stays small)
   └─ spawn subagent ──► SUBAGENT context (its own window)
                            reads 50 files, runs greps, reasons … (20k tokens here)
                            returns: "Found 2 issues: auth/login.py:42, api/users.py:88"
   ◄── parent appends ONLY that 1-line summary

It's the same principle as retrieval and elision — keep the bulky work out of the durable context — applied at the orchestration level. (More on subagents in §10 and Phase 10.)

3.7 The trade-off, restated

Compaction loses information; truncation loses it bluntly; retrieval can miss; SWA can't see beyond its window. Every reduction technique trades some fidelity for budget. That's why:

  • Compact at natural boundaries, and restate hard constraints afterward.
  • Prefer retrieval/elision (re-fetchable) over summary (lossy) over truncation (bluntly lossy) when you have the choice.
  • Remember reduction protects all three of cost, latency (less prefill), and quality (less "lost in the middle") — not just cost.

4. Use case: a real prompt's journey through Claude Code

Now let's make it concrete with Claude Code as the tool under study: you type one prompt — where does it go, what gets written to disk, and how does the loop unfold? This section is grounded in the observable on-disk layout (paths you can actually inspect on your machine); treat the exact JSONL fields as illustrative (per the accuracy note up top).

4.1 Where Claude Code keeps things on disk

Claude Code is a local program, so "your session" is files on your machine, not state on a server. The layout you can observe:

~/.claude/
├── settings.json                 # global settings (model, permissions, etc.)
├── CLAUDE.md                     # GLOBAL instructions (ride every session, all projects)
└── projects/
    └── <cwd-with-slashes-as-dashes>/      # one folder per project directory
        ├── <session-uuid>.jsonl           # THE TRANSCRIPT — append-only event log
        ├── <another-session-uuid>.jsonl   # each resume/new session = its own file
        └── …
<your repo>/
├── CLAUDE.md                     # PROJECT instructions (ride every call in this repo)
└── …your code…

The project folder name is the working directory with / replaced by -. Concretely, a session started in /Users/s0x/src/10xdev/ai-engineer is stored under:

~/.claude/projects/-Users-s0x-src-10xdev-ai-engineer/<session-uuid>.jsonl

That sanitized-path naming is exactly why this very project's persistent data lives at ~/.claude/projects/-Users-s0x-src-10xdev-ai-enigneer/. Each session is one JSONL file (JSON Lines: one JSON object per line) named by a UUID. JSONL is used because it's append-only — every event (your message, the model's reply, each tool call, each tool result, a compaction summary) is written as a new line the instant it happens, so an abrupt Ctrl-C or crash still leaves a valid, replayable log up to the last completed line. That property is the entire foundation of --resume (§8).

4.2 The walkthrough: one prompt, end to end

Say you're in your repo and type:

"Add input validation to the login endpoint and run the tests."

Here is what happens to those words, in order, and where each artifact lands:

① Keystroke → in-memory messages → immediately appended to the JSONL. The harness adds your turn to its in-RAM messages list and writes a line to ~/.claude/projects/<proj>/<session>.jsonl:

{"type":"user","uuid":"a1","timestamp":"2026-06-13T10:00:00Z",
 "cwd":"/Users/s0x/src/.../repo",
 "message":{"role":"user","content":"Add input validation to the login endpoint and run the tests"}}

Your prompt text now physically exists in that file. (It is not persisted by the provider beyond processing-time, subject to their data policy — the durable copy is your local JSONL.)

② Context assembly (the request body). The harness builds the full messages array for the API call — none of this is your typed text alone:

[ system prompt + tool schemas ]        ← stable → marked for prompt caching (§2)
[ global ~/.claude/CLAUDE.md ]          ← stable
[ project ./CLAUDE.md ]                 ← stable (this is why CLAUDE.md "just works")
[ prior turns this session ]            ← stable-ish
[ your new message: "Add input validation…" ]   ← VOLATILE tail (re-prefilled)

Your prompt sits in the volatile tail, so it's (re)prefilled fresh; the big stable prefix above it is a cache read (§2).

③ The agent loop turns one prompt into many API calls. The model can't "see" your repo, so it retrieves via tools (§3.4). Each step appends new line(s) to the same JSONL:

{"type":"assistant","uuid":"a2","parentUuid":"a1","message":{"role":"assistant","content":[
   {"type":"text","text":"I'll locate the login endpoint."},
   {"type":"tool_use","id":"toolu_01","name":"Grep","input":{"pattern":"def login","glob":"**/*.py"}}]}}
{"type":"tool_result","uuid":"a3","parentUuid":"a2","toolUseID":"toolu_01",
   "content":"auth/login.py:42:def login(req):"}
{"type":"assistant","uuid":"a4","parentUuid":"a3","message":{"role":"assistant","content":[
   {"type":"tool_use","id":"toolu_02","name":"Read","input":{"file_path":"auth/login.py","offset":30,"limit":60}}]}}
{"type":"tool_result","uuid":"a5","parentUuid":"a4","toolUseID":"toolu_02","content":"<lines 30-90 of auth/login.py>"}
{"type":"assistant","uuid":"a6","parentUuid":"a5","message":{"role":"assistant","content":[
   {"type":"tool_use","id":"toolu_03","name":"Edit",
    "input":{"file_path":"auth/login.py","old_string":"def login(req):","new_string":"def login(req):\n    validate(req)"}}]}}
{"type":"tool_result","uuid":"a7","parentUuid":"a6","toolUseID":"toolu_03","content":"edit applied"}
{"type":"assistant","uuid":"a8","parentUuid":"a7","message":{"role":"assistant","content":[
   {"type":"tool_use","id":"toolu_04","name":"Bash","input":{"command":"pytest -q tests/test_auth.py"}}]}}
{"type":"tool_result","uuid":"a9","parentUuid":"a8","toolUseID":"toolu_04","content":"5 passed in 1.2s"}
{"type":"assistant","uuid":"a10","parentUuid":"a9","message":{"role":"assistant","content":[
   {"type":"text","text":"Added validate(req) to login() and confirmed the 5 auth tests pass."}]}}

Walk the chain: each tool_use has an id; the matching tool_result carries the same id (toolUseID) so the pairing is unambiguous; parentUuid links each record to the previous one, forming the conversation tree the harness replays. The model requested Grep/Read/Edit/Bash; the harness ran them (with your permission gates) and fed results back. That's ~5 separate stateless API calls for "one prompt," each carrying the whole growing context — and each appending lines to the JSONL.

④ Render + account. The final text is shown to you; usage (input/output/cache tokens) is recorded. The JSONL now contains the complete, replayable story of this exchange.

4.3 What a /compact looks like in the transcript

Fast-forward: the session is long, and you run /compact (or auto-compaction fires, §3.1). The harness makes the extra summarization call, then writes a summary record and continues from it:

{"type":"summary","uuid":"s1","leafUuid":"a10",
 "summary":"Goal: harden auth. DONE: added validate(req) in auth/login.py:42; 5 auth tests pass. "
           "CONSTRAINTS: do not touch legacy/. OPEN: add rate-limiting to /reset."}

From here on, the rebuilt working context is [system][tools][CLAUDE.md] + [that summary] + recent turns instead of the full 40-turn history. Note the two consequences from §3.1 in action: the original detailed lines are still in the file (history isn't erased from disk) but they're no longer sent to the model — and the next call re-pays a cache-write because the prefix changed. If "do not touch legacy/" hadn't made it into that summary, the model would lose it — which is why you restate critical constraints post-compaction.

4.4 What --resume reads back

You close your laptop. Tomorrow you run claude --resume (or --continue). The harness:

  1. Finds the project folder (~/.claude/projects/<cwd-as-dashes>/) for your current directory.
  2. Reads the chosen <session>.jsonl line by line, reconstructing the messages tree (following parentUuid, honoring any summary record as the new starting point).
  3. Rebuilds the context — re-attaches the system prompt + tool defs + CLAUDE.md + the reconstructed history/summary — and resumes the agent loop.

The model still "remembers" nothing — continuity is entirely that replayed file. Caveats from §8 apply: the prompt cache has expired (so the first call re-pays cache-write), and if the environment changed overnight (you edited auth/login.py by hand, switched branches), the agent's next Read/Grep sees the new reality even though the transcript describes the old one.

The whole point of this section: "your conversation with Claude" is a sequence of edits to a local JSONL file plus a series of stateless API calls. Knowing that, every earlier concept becomes operational — caching is about not re-sending the file's stable front; compaction is rewriting the file's front into a summary; resume is replaying the file; "it forgot" is almost always the file no longer contains that detail (compacted away) or the environment changed under it.

5. What static files are more efficient?

Two different notions of "efficient" — fewer tokens and more cache-friendly — and you want both.

Token efficiency (smaller = cheaper, faster, better recall)

  • Be concise. Every token in a context file (e.g. CLAUDE.md) is sent on every call. A 2,000-token bloated guide taxes the whole session.
  • Code tokenizes efficiently; prose and whitespace less so. Minified/structured beats verbose. (Phase 1.01)
  • Avoid token-wasters: pretty-printed JSON with deep indentation, giant tables, repeated boilerplate, pasted logs, base64 blobs, huge auto-generated files. Compact JSON over pretty JSON for the same data.
  • Prefer references over dumps. Point to a file/path or a short excerpt instead of pasting an entire file; let the agent Read it only when needed (this is retrieval-vs-stuffing, §3.4).
  • Structured, scannable Markdown (short headers, bullets) is both token-light and easy for the model to use.
  • Non-English text costs 1.5–3× more tokens — keep instruction files in the model's strongest language where possible.

Cache efficiency (stable = cache hits = cheap + fast)

  • Stable files placed in the prefix get cached (§2). A CLAUDE.md that rarely changes sits in the cached prefix → near-free on repeat calls.
  • Volatility kills caching. A context file containing a timestamp, random ID, or frequently-edited content changes the prefix every call → cache miss → you re-pay for the whole prefix. Keep dynamic content out of stable context files (or place it at the very end).
  • Order matters: most-stable → least-stable. The system prompt and tool defs are most stable; CLAUDE.md next; conversation last.

So the "most efficient static file" is: small, structured, in code/compact form, free of volatile content, and stable across calls — so it's both token-light and a reliable cache hit. A sprawling, frequently-edited CLAUDE.md full of logs and timestamps is the opposite: expensive and cache-busting.

6. How the prompt changes the results

The model computes a probability distribution over the next token, conditioned on the entire context (Phase 1.00). Change the conditioning and you change the distribution — so everything in the prompt steers the output:

  • Wording & specificity. "Fix the bug" vs "Fix the off-by-one in paginate() at line 42; keep the signature; add a test." The second narrows the distribution toward the answer you want.
  • System prompt. Persona, constraints, and tool policy shift behavior globally (and refusals).
  • Examples (few-shot). Showing 1–3 input→output examples conditions the model toward that format/style far more reliably than describing it.
  • Format instructions / structure. Asking for a unified diff, JSON schema, or "answer in one sentence" constrains the output (Phase 5.07).
  • Ordering & placement. Put the most important instructions and the question at the edges (start/end), not buried in the middle of a long context ("lost in the middle", Phase 2.01).
  • What context you include. Relevant files/errors in context → grounded answers; missing them → guesses/hallucination (Law 1).
  • Sampling parameters. Lower temperature → more focused/deterministic; higher → more varied.

Practical upshot: prompt engineering is just choosing the conditioning that makes the desired tokens most probable. Be specific, show examples, constrain the format, and put the key bits at the edges.

Seeing the "conditioning" concretely

"The prompt changes the distribution" isn't a metaphor — it's the literal output of §1.C. The model emits logits (one score per vocab token); the sampler converts them to probabilities and picks one:

import torch
import torch.nn.functional as F

def sample(logits, temperature=1.0, top_p=1.0):
    if temperature == 0:                              # greedy / argmax → most deterministic
        return int(logits.argmax())
    probs = F.softmax(logits / temperature, dim=-1)   # temperature reshapes the curve
    # (top_p / top_k would truncate the low-probability tail here)
    return int(torch.multinomial(probs, 1))           # DRAW a token from the distribution

Change the conditioning → the logits change → the whole curve moves:

prompt: "The capital of France is"               → P(" Paris")=0.92  P(" a")=0.01 …
prompt: "In Klingon, the capital of France is"   → the mass spreads; " Paris" drops sharply

And temperature reshapes that same distribution without changing the model or the prompt:

  • T → 0: take the argmax every time (focused, repeatable).
  • T < 1: sharpen — likely tokens get more likely (rich get richer).
  • T > 1: flatten — rare tokens get a real chance (more variety, more risk).

So prompt design (§5 decisions: wording, examples, format, edge-placement) and sampling settings are two knobs on the same object — the next-token distribution.

7. Why the same command twice can differ

Even typing the exact same thing can yield different output. There are two families of reasons — the input wasn't actually identical, and generation is non-deterministic.

A. The context differed even though your text didn't

  • The environment changed. Files, git state, directory contents, timestamps, command output — the agent's tools return current reality. "Run the tests" reflects the tests now. (Phase 1.00)
  • The history differed. Earlier turns / prior tool results in the session changed the context.
  • Injected content changed. A <system-reminder>, date, or memory note differs.
  • Caching/compaction state differed. A /compact between runs changed what the model sees (§3).

So "the same command" often isn't the same input — and different input → different output, deterministically.

B. Generation is genuinely non-deterministic

Even with identical input:

  • Sampling. If temperature > 0, the next token is drawn from the distribution → different draws → different text (Phase 1.03).
  • Floating-point non-associativity on GPUs. (a+b)+c ≠ a+(b+c) in floating point; parallel GPU reductions sum in nondeterministic order, so even temperature = 0 (greedy) can occasionally pick a different top token when two logits are near-tied.
  • Batching / kernel nondeterminism. Your request is batched with others (vLLM continuous batching, §3.5); batch composition changes the numerics slightly.
  • MoE routing variance. In mixture-of-experts models (Phase 2.08), batch-dependent expert routing can shift outputs.
  • Backend / version drift. The provider may have re-deployed, re-quantized, or rolled an alias to a new version (Phase 5.10); system_fingerprint flags this.
  • Seed not honored. Even seed doesn't fully guarantee reproducibility across all providers/backends.

To maximize reproducibility: temperature = 0 + fixed seed (where honored) + pinned model version + pinned provider + identical context. You'll get near-deterministic, rarely bit-identical. For agents, also freeze the environment (same repo state) — usually the bigger factor.

Float non-associativity, demonstrated

The claim "temp 0 still isn't always bit-identical" is concrete and reproducible:

import torch
a = torch.tensor([1e8, 1.0, -1e8, 1.0])
print(a.sum())              # ideal = 2.0, but FP rounding depends on summation ORDER
print(a.flip(0).sum())      # summing the other way can give a slightly different result
# (a + b) + c  ≠  a + (b + c)  in floating point — addition is NOT associative

On a GPU, the order in which thousands of partial sums are reduced depends on how work was tiled across cores, which depends on batch shape, which depends on who else is in your batch this millisecond (§1.E). When two top logits are nearly tied, a sub-ulp difference can flip which token is the argmax → a different greedy token → a divergent continuation. Layer on MoE routing (batch-dependent, Phase 2.08) and version/quantization drift across replicas (§1.F), and "identical input ⇒ identical output" is simply not guaranteed in a shared, distributed serving system — even at temperature 0.

8. What "resume after interruption" does

Because the model is stateless and the harness owns the session, "resume" is a harness operation, not a model one.

  • What a session actually is: the harness persists the transcript — the full messages array (system prompt reference, your turns, the model's turns including tool calls, and tool results) — to local storage as you go, as the append-only JSONL described in §4.1.
  • Interruption (you hit Ctrl-C, close the terminal, a call times out): in-flight streaming output may be discarded or saved as a partial assistant turn; the saved transcript is the source of truth (and because JSONL is append-only, lines completed before the interrupt survive).
  • Resume (claude --resume / --continue): the harness reloads the saved transcript, rebuilds the context (re-attaches the system prompt + tool defs + history, possibly a summary if it was compacted), and continues the agent loop from there. The next call sends that reconstructed context to the model (§4.4).
  • The model "remembers" nothing — continuity is entirely the replayed message history. (This is the same mechanism as normal multi-turn chat; "resume" just persists it across process restarts.)
  • Caveats: the prompt cache has likely expired (short TTL), so the first post-resume call re-pays cache-write; if the session was compacted, fine early details may be gone (only the summary survives); and if the environment changed while you were away (files edited, branch switched), the agent's tools will see the new reality.
running session ──persist──► transcript on disk (messages + tool calls/results [+ summary])
        ⨯ interrupt
   --resume ──load──► rebuild context (system + tools + history) ──► next stateless call continues the loop

9. Changing the model mid-session — how context carries over

You can switch models within a session and "keep the conversation." How? Same reason as resume: the context lives in the harness's messages array, not in any model. Switching models just sends that same reconstructed history to a different model ID. The new model reads the whole transcript as its context and continues — it was never "in" the old model.

But there are real gotchas (this is where seasoning shows):

  • Different tokenizer → different token counts. The same transcript is a different number of tokens for a different model family (Phase 1.01); your budget math changes.
  • Different context window. If you switch to a smaller-context model, the history may no longer fit → forced compaction/truncation on the spot (§3).
  • Prompt cache is per-model (and often per-provider). Switching models invalidates the cache — the first call on the new model re-pays full prefill / cache-write (§2, Phase 5.10).
  • Behavioral discontinuity. The new model has different style, refusal behavior, and tool-calling reliability (Phase 5.06); the conversation may "feel" different even though the history is identical.
  • Tool-format / capability differences. Tool-calling formats and feature support vary; the harness normalizes most of this, but edge behaviors differ. A proxy like LiteLLM also normalizes formats here (§3.5).

This is exactly why agentic tools route by sub-task (a strong/reasoning model for planning, a fast model for edits, Phase 5.04/5.00): they swap the model while reusing the harness-held context — paying the cache reset as the cost of the switch.

10. More nitty-gritty (streaming, usage, stop reasons, thinking, subagents)

  • Streaming (SSE). Responses stream as Server-Sent Events: message_start → repeated content_block_start / content_block_delta (token chunks) → content_block_stop → message_delta (with stop_reason, usage) → message_stop. Streaming improves perceived latency only (Phase 2.05); the harness assembles deltas into the message.
  • Usage accounting. Each response reports input_tokens, output_tokens, and cache fields (cache_creation_input_tokens, cache_read_input_tokens). This is your real cost signal — reconcile estimates against it (Phase 4.04).
  • Stop reasons. end_turn (done), max_tokens (hit output cap — output truncated!), stop_sequence, tool_use (paused for the harness to run a tool/approval). Watching stop_reason is how the harness drives the loop.
  • Thinking / reasoning tokens. With extended thinking on, the model emits hidden reasoning tokens before the answer — billed, latency-heavy, often not shown (Phase 2.09). They consume context/output budget.
  • Tool-use protocol details. Each tool_use has an id; the matching tool_result references that id (you saw the toolUseID pairing on disk in §4.2). Models can emit parallel tool calls (several at once) — the harness runs them and returns multiple tool_results. (Phase 10)
  • System reminders / injected context. The harness can insert <system-reminder>-style notes (e.g. "todo list is empty", file-state hints). These are harness-authored context, not your words — they still steer the model (and count as tokens).
  • Subagents. A spawned subagent runs in its own fresh context window, does a task, and returns only a summary to the parent — a deliberate context-isolation technique to keep the main context clean and within budget (§3.6, Phase 10).
  • Why long sessions get slower/pricier. Context (and the KV cache) grows with every turn → more prefill, more memory, higher per-call cost — until compaction resets it. The fix is the reduction toolkit.

What streaming actually looks like on the wire

Server-Sent Events arrive as event: / data: line pairs; the harness stitches the deltas into the message:

event: message_start
data: {"message":{"role":"assistant","usage":{"input_tokens":1203,"output_tokens":1}}}

event: content_block_delta
data: {"delta":{"type":"text_delta","text":"I'll"}}

event: content_block_delta
data: {"delta":{"type":"text_delta","text":" read app.py"}}

event: message_delta
data: {"delta":{"stop_reason":"tool_use"},"usage":{"output_tokens":37}}

event: message_stop

Streaming only improves perceived latency — the tokens are still produced one decode step at a time (§1.D); you just see each as it lands instead of waiting for the whole message.

Reading the bill (worked example)

Each response's usage is the ground truth. Suppose a turn reports:

input_tokens=1203, cache_read_input_tokens=11500, cache_creation_input_tokens=0, output_tokens=420

At illustrative prices ($3 / 1M input, $0.30 / 1M cache-read, $15 / 1M output):

fresh input :  1,203 × $3    /1e6 = $0.0036
cache read  : 11,500 × $0.30 /1e6 = $0.0035   ← 11.5k prefix tokens for a third of a cent (§2 paying off)
output      :    420 × $15   /1e6 = $0.0063
                                    ─────────
                             total ≈ $0.0134 for this turn

Without caching, those 11,500 prefix tokens would cost ~$0.0345 (not $0.0035) — and you'd re-pay it on every turn. Multiply by the dozens of calls in §1.G's loop and caching is the line between "cheap" and "unaffordable."

Handling max_tokens (truncated output)

If stop_reason == "max_tokens", the answer was cut off mid-stream, not finished. The harness must raise the cap or continue — append the partial assistant turn and call again so the model resumes from where it stopped. Silently treating a truncated answer as complete (e.g. parsing half a JSON or half a diff) is a classic agent bug.

Thinking tokens are real budget

With extended thinking on (Phase 2.09), the model emits hidden reasoning tokens before the visible answer. They are decoded one-by-one just like output (§1.D), so they cost latency and money and consume the output/context budget even though you usually don't see them. Reserve thinking for genuinely hard steps — it is not free.

11. Best practices (consolidated)

Context & files

  • Keep CLAUDE.md/context files small, structured, and stable — they ride every call and (if stable) get cached (§5).
  • Keep volatile content (timestamps, IDs, logs) out of the stable prefix — it busts the cache.
  • Prefer references/excerpts over pasting whole files; let the agent Read on demand (§3.4).

Cost & latency

  • Lean on prompt caching: stable prefix first, volatile last (§2).
  • /compact at natural boundaries; re-state critical constraints after a big compaction (§3).
  • Cap max_tokens; use subagents to isolate big sub-tasks (§10).
  • Reserve reasoning/thinking for genuinely hard steps (Phase 5.04).

Prompting

  • Be specific, give examples, constrain the format, and put key instructions/questions at the edges (§6).
  • Put the right files/errors in context — the model can't use what isn't there (Law 1).

Reproducibility & model choice

  • For repeatable runs: temperature 0 + seed + pinned version + pinned provider + frozen environment (§7).
  • When switching models mid-session, expect a cache reset and possible recompaction; switch deliberately (§9).
  • Verify quality per endpoint/provider, not just per model (Phase 5.10) — and know that a proxy (OpenRouter middle-out, LiteLLM trim_messages) may silently reshape your context to fit (§3.5).

📚 References

The inspiration

  • alex/what-happens-when — https://github.com/alex/what-happens-when

Anthropic API & Claude Code (primary sources)

  • Messages API — https://docs.anthropic.com/en/api/messages
  • Tool use — https://docs.anthropic.com/en/docs/build-with-claude/tool-use
  • Prompt caching — https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
  • Streaming (SSE) — https://docs.anthropic.com/en/docs/build-with-claude/streaming
  • Extended thinking — https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking
  • Context windows — https://docs.anthropic.com/en/docs/build-with-claude/context-windows
  • Claude Code docs (CLAUDE.md, sessions/resume, subagents) — https://docs.anthropic.com/en/docs/claude-code

Serving stack & proxies (for §3.5)

  • vLLM — PagedAttention (intro blog) — https://blog.vllm.ai/2023/06/20/vllm.html
  • vLLM docs — automatic prefix caching, sliding window, continuous batching — https://docs.vllm.ai
  • LiteLLM docs — unified API, trim_messages, fallbacks, caching — https://docs.litellm.ai
  • OpenRouter — message transforms (middle-out) — https://openrouter.ai/docs/transforms
  • OpenRouter — provider routing/selection — https://openrouter.ai/docs/provider-routing

Mechanism deep-dives in this curriculum

Up: References · Companion: Agent Memory and Context · Related snapshot: Trending LLMs Landscape