Tokenization and Context Windows

Phase 1 · Document 01 · LLM Vocabulary and Mental Models Prev: 00 — Core Mental Model · Next: 02 — Parameters, Weights, and Checkpoints

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Every LLM API call begins with tokenization and is bounded by the context window. Together these two facts decide whether your input fits at all, what it costs, what the model can see when generating, and how you must design prompts, chunking, caching, and retrieval. You will use this knowledge dozens of times a week: estimating cost before shipping a feature, sizing a RAG chunk, diagnosing a "context length exceeded" error, or explaining to a PM why a 200-page PDF won't "just fit."

Get tokenization wrong and your cost estimates are off by 2–3× for non-English or JSON-heavy workloads. Get the context window wrong and you ship a feature that fails on real documents in production.

2. Core Concept

Tokenization (plain English, then technical)

Tokenization splits text into tokens — the model's fundamental input/output unit. A tokenizer maps a string to a list of integer token IDs, and back. The full set of tokens a model knows is its vocabulary.

"Hello, world!"  →  [9906, 11, 1917, 0]   (GPT-4 family tokenizer)

Modern tokenizers are subword tokenizers — usually Byte-Pair Encoding (BPE) (GPT, Llama) or SentencePiece/Unigram (Gemma, T5). BPE starts from bytes and repeatedly merges the most frequent adjacent pair into a new token, learning a vocabulary where common words are single tokens and rare words split into pieces. This is why token counts are unintuitive:

Rules of thumb: English ≈ 0.75 words/token (≈ 4 chars/token). Code ≈ 3–4 chars/token. Non-English prose can use 1.5–3× more tokens per word — a direct, often-overlooked cost multiplier.

Counting tokens:

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
print(len(enc.encode("The context window limits what the model sees.")))  # ~9

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
print(len(tok.encode("Hello world")))

Context window

The context window (a.k.a. context length / max_model_len) is the maximum number of tokens the model can process in a single forward pass — typically input + output combined, though some providers expose separate max input and max output limits.

context_window ≈ input_tokens + output_tokens     (exact semantics vary by model/provider)

Critical caveat: the context window is what the model will accept, not what it can reason over reliably. Long-context models often suffer "lost in the middle" — facts placed in the middle of a long context are recalled worse than facts at the start or end. Acceptance ≠ comprehension. Always evaluate long-context recall for your task instead of trusting the headline number.

3. Mental Model

"Context window" = a fixed-size glass window.
  • The model sees ONLY what is inside the window.
  • Anything outside the window does not exist to the model.
  • The window is measured in tokens, not words or characters.
  • Everything you want used — system prompt, history, docs, the question —
    must fit inside, AND leave room for the answer.

budget = context_window − reserved_output_tokens − system_prompt − history
         ↑ what's left is your usable space for retrieved docs / the user turn

4. Hitchhiker's Guide

What to look for first: the model's context window and its separate max-output limit; whether pricing is per input/output token; whether cached input is discounted.

What to ignore at first: the exact BPE merge rules. You need to count tokens accurately, not implement a tokenizer.

What misleads beginners:

• "50,000 words fits in 128K" — count tokens, not words; and reserve output space.
• "Long context = better understanding" — measure mid-context recall; don't assume.
• "The context handles memory" — it does not persist across calls; you re-send history.

How experts reason: they treat the context window as a budget to allocate — reserve output, subtract a stable (cacheable) system prompt, then fill the remainder with the highest-value retrieved tokens. They count tokens with the model's own tokenizer, never a word count.

What matters in production: token count drives cost; prompt length drives prefill latency (TTFT); context length drives KV-cache memory (the real concurrency limiter when self-hosting).

Debug/verify: before sending, log the token count of the fully rendered prompt; alert at >80% of the window. On a "context exceeded" error, log per-message token counts to find the bloat.

Questions to ask providers: Is the window input-only or input+output? Is there prefix/prompt caching, and at what discount? How are image/audio tokens counted? What happens on overflow — error or silent truncation?

What silently gets expensive: unbounded chat history; verbose system prompts re-sent each call; pretty-printed JSON (whitespace is tokens); non-English text; high-res images.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Tokens are the unit of pricing for all major providers.
• Most providers bill input and output tokens separately; output is often 2–5× the input price.
• The same text costs different token counts on different models (different vocabularies).
• Code tokenizes more efficiently than prose; pretty-printed JSON wastes tokens on whitespace.
• Non-English text can cost 1.5–3× more tokens per word.
• A single image typically costs ~500–2000 tokens depending on resolution and model.
• Cached input tokens may be 75–90% cheaper where prompt caching is supported.
• Reasoning/thinking tokens are billed (usually at output rates) even though hidden.
• Context window is a trained property; exceeding the trained length (e.g. via --ctx-size) degrades quality.

9. Observations from Real Systems

• OpenRouter / provider pricing pages list input/output $/1M tokens; some rows show a separate cached-input price — your first stop for cost math.
• vLLM rejects or truncates requests over max_model_len; the KV cache for long context is the practical concurrency limit, not the weights.
• llama.cpp --ctx-size and Ollama num_ctx set the window; setting them beyond the trained length produces degraded, repetitive output.
• Cursor / VS Code Copilot continuously trim and prioritize file/repo context to stay under the window — a live demonstration of context budgeting.
• Anthropic / OpenAI prompt caching make a long, stable system prompt cheap on repeat calls, changing the economics of verbose instructions.

10. Common Misconceptions

11. Engineering Decision Framework

Sizing a request:
  reserved_output  = max answer you need (tokens)
  fixed_overhead   = system prompt + tool schemas (cache these if possible)
  usable           = context_window − reserved_output − fixed_overhead
  fill `usable` with the highest-value retrieved/ history tokens, newest/most-relevant first.

Choosing a context strategy:
  Input > window?            → chunk + retrieve (RAG), don't widen the model blindly.
  Repeated long system prompt? → enable prompt/prefix caching.
  Need facts deep in a long doc? → rerank to the edges; verify recall; consider summary-then-detail.
  Non-English / heavy JSON?  → recount tokens; budgets from English will be wrong.

12. Hands-On Lab

Goal

Build a token counter + cost estimator + context-fit checker, and observe tokenization differences across content types and models.

Prerequisites

• Python 3.10+; optional Hugging Face access for the Llama tokenizer.

Setup

pip install tiktoken transformers

Steps

import tiktoken

def count_tokens(text: str, model: str = "gpt-4o") -> int:
    return len(tiktoken.encoding_for_model(model).encode(text))

def estimate_cost(text, output_tokens, in_price, out_price, model="gpt-4o"):
    n_in = count_tokens(text, model)
    return {
        "input_tokens": n_in,
        "output_tokens": output_tokens,
        "total_cost": n_in/1e6*in_price + output_tokens/1e6*out_price,
    }

samples = {
    "question": "What is the capital of France?",
    "code": "def fib(n): return n if n<=1 else fib(n-1)+fib(n-2)",
    "prose": "The context window bounds what the model can see. " * 100,
    "json_pretty": '{\n  "name": "Alice",\n  "age": 30\n}\n' * 50,
    "json_compact": '{"name":"Alice","age":30}' * 50,
}
for name, txt in samples.items():
    r = estimate_cost(txt, 200, 2.50, 10.00)  # gpt-4o prices
    print(f"{name:12} {r['input_tokens']:>6} tok  ${r['total_cost']:.6f}")

Expected output

• json_pretty costs noticeably more than json_compact for the same data (whitespace tokens).
• code is token-efficient relative to its character count.

Debugging tips

• API prompt_tokens higher than your count? You omitted role/message formatting overhead.
• HF tokenizer download fails? Gate access on the model page or use a public tokenizer.

Extension task

Add check_fits(messages, model, window, reserved_output) that sums all message tokens, subtracts reserved output, and warns above 80% usage.

Production extension

Compare the same 2,000-word document tokenized by gpt-4o (tiktoken) vs Llama-3 (HF). Produce a cost-delta table — this is exactly the analysis you'd run before choosing a model for a high-volume pipeline.

What to measure

Tokens per content type; cost per type; pretty vs compact JSON delta; GPT vs Llama token delta for identical text.

Deliverables

• A token + cost table for 5 content types.
• The check_fits function.
• A one-line finding: which content type is most expensive per word, and why.

13. Verification Questions

Basic

1. Roughly how many tokens is "the quick brown fox"?
2. What's the difference between max input and max output tokens?
3. Why do two models report different token counts for the same string?

Applied 4. A RAG step retrieves 5 chunks of 500 words each. Approximate the context tokens. 5. 10,000 calls/day at ~500 input + ~200 output tokens, $3/1M in, $15/1M out — monthly cost?

Debugging 6. You hit "context length exceeded." Give three distinct fixes. 7. Your French-language feature costs 2× the English estimate. Why, and how do you confirm?

System design 8. Design a pipeline to process a 200-page PDF through a 32K-context model. Address chunking, retrieval, and output assembly.

Startup / product 9. Your product pastes the full user document into every call. Margins are thin. Propose two token-level changes (caching, retrieval) and estimate their effect on gross margin.

14. Takeaways

1. Tokens are the atomic unit of LLM pricing and limits.
2. English ≈ 0.75 words/token; always count with the model's own tokenizer.
3. The context window is a hard budget: reserve output, cache the stable prefix, fill the rest with the best tokens.
4. Long context increases cost, latency, and KV memory — and doesn't guarantee good recall ("lost in the middle").
5. JSON whitespace and non-English text are silent cost multipliers.
6. Cache repeated input where supported to cut cost dramatically.

15. Artifact Checklist

• Code: token counter + cost estimator + check_fits.
• Benchmark result: token/cost table across 5 content types.
• Comparison report: GPT vs Llama token counts for one real document.
• Cost estimate: monthly cost for a realistic call volume.
• Notes: the context-budget formula in your own words.
• Diagram: the "glass window" budget allocation.

Next: 02 — Parameters, Weights, and Checkpoints