Core Mental Model for LLMs

Phase 1 · Document 00 · LLM Vocabulary and Mental Models Prev: Phase 1 Index · Next: 01 — Tokenization and Context

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Before you learn a single specific term — KV cache, temperature, RAG, speculative decoding — you need one durable mental model that everything else hangs on. Engineers who lack this model memorize disconnected facts and make expensive mistakes: they assume the model "remembers" past chats, that bigger context is free, or that tool calling is something the model "does." Engineers who have this model can reason about a brand-new technique they've never seen, because they can place it in the pipeline.

This single model drives real decisions: how you budget tokens, where latency comes from, why a request is expensive, who is responsible for safety, and what "the model" actually controls versus what your application controls. If you internalize only one document in this entire curriculum, make it this one.

2. Core Concept

Plain-English primer (the words this page will use before they're fully explained)

This is the very first concept document, so a few terms appear here that get their own deep treatment later. One plain line each now — so nothing below is mysterious — with a pointer to where you'll master it:

• Token — a chunk of text (≈ ¾ of a word) that the model reads/writes one at a time. → 1.01
• Tokenizer / vocabulary — the tool that splits text into tokens, using a fixed list (vocabulary) of all tokens it knows. → 1.01
• Embedding — turning a token into a list of numbers (a "vector") the model can compute on. → 2.01
• Transformer / attention / FFN — the model's internal machinery: attention lets each word look at earlier words; the feed-forward network (FFN) then "thinks" about each word. You don't need the details yet. → Phase 2
• Logits → softmax → sampler — the model outputs a raw score (logit) for every possible next token; softmax turns those into probabilities; the sampler picks one (this is where temperature acts). → 1.03
• Prefill vs decode — prefill = the model reading your whole prompt at once (fast); decode = writing the answer one token at a time (slower). They explain most latency. → 2.07
• KV cache — a memory of the conversation so far that the model keeps so it doesn't re-read everything each step; it's the main thing that limits how many users you can serve at once. → 2.06
• Context window — the maximum number of tokens the model can consider at once. → 1.01
• TTFT / TPOT — time to first token (how long until the answer starts) and time per output token (how fast it then types). → 1.05

You can read this whole document with just these one-liners; each gets a full, from-zero treatment in its own doc.

Plain English

An LLM is a next-token prediction machine. You give it a sequence of tokens (text broken into pieces); it predicts the most likely next token; it appends that token and repeats. That loop, run fast on a GPU, produces everything you see — essays, code, JSON, "reasoning." There is no hidden database, no live internet, no memory between calls. There is only: given these tokens, what token comes next?

Technical depth — the inference pipeline

Every generation, on every model, follows the same path:

1. Tokenize. Input text is split into tokens (subword units) by a tokenizer. Each token maps to an integer ID via a fixed vocabulary table.
2. Embed. Each token ID indexes into an embedding matrix, producing a high-dimensional vector. Positional information (e.g. RoPE) is mixed in so the model knows token order.
3. Transform. The vectors flow through N transformer layers, each containing self-attention (every token looks at every previous token) and a feed-forward network, wrapped in residual connections and normalization.
4. Project to logits. The final hidden state is multiplied by an output matrix to produce a logit (raw score) for every token in the vocabulary.
5. Softmax + sample. Logits become a probability distribution. The sampler (temperature, top-p, top-k, min-p) picks one token. Greedy decoding picks the argmax.
6. Append and repeat. The chosen token is appended to the sequence and steps 2–5 repeat. This is autoregressive generation.
7. Stop. The loop ends at an end-of-sequence token, a stop sequence, or max_tokens.

The first pass over your whole prompt is the prefill phase (compute-bound, parallelizable). Each subsequent single-token step is the decode phase (memory-bandwidth-bound, serial). The KV cache stores attention keys/values from prior tokens so decode does not recompute the whole prompt each step. These three terms — prefill, decode, KV cache — explain almost all of LLM performance and cost behavior, and we return to them throughout the curriculum.

Everything else is application logic

Tool calling, structured output, RAG, agents, and chain-of-thought are not new model abilities bolted on. They are prompt formatting + application control loops sitting on top of the same next-token loop:

• Tool calling: the prompt describes tools as text; the model emits text shaped like a tool request; your code parses it, executes the tool, and feeds the result back as more tokens.
• RAG: your code retrieves relevant text and pastes it into the prompt before generation.
• Agents: a loop that calls the model, runs a tool, appends the result, and calls again until a stop condition.
• Reasoning: the model is trained/prompted to emit intermediate "thinking" tokens before its answer.

Seeing these as application patterns over one primitive is the entire point of this document.

3. Mental Model

Your Text
    ↓
Tokenizer ───────────────► token IDs        (Phase 1.01)
    ↓
Embedding + Positions ───► vectors           (Phase 2.01)
    ↓
Transformer Layers ×N ───► hidden state      (Phase 2.02–2.04)
  (Attention + FFN)
    ↓
Linear + Softmax ────────► probability over vocabulary
    ↓
Sampler ─────────────────► one token ID      (Phase 1.03)
  (temp, top-p, top-k)
    ↓
Detokenize → append → REPEAT until stop

Six Laws of LLM Engineering (the load-bearing consequences of the loop above):

4. Hitchhiker's Guide

What to understand first: the token loop, the context window, and sampling parameters. These three touch every API call you will ever make.

What to ignore at first: attention math, CUDA kernels, RoPE derivations. Learn the behavior before the implementation — you can ship production systems knowing behavior, and the math becomes easy later once you know why it matters.

What usually misleads beginners:

• "The model knows X." It was trained on X. At inference it knows only what is in the current context.
• "Temperature makes it smarter." It changes randomness of sampling, nothing else.
• "Bigger context is better." More context = more cost, more prefill latency, more KV memory, and often worse mid-context recall.
• "Tool calling is built into the model." The model emits text; your code executes.

How experienced engineers reason: they trace every feature back to "which tokens end up in context, and who acts on the output tokens." When a system misbehaves, they ask what was actually in the context window and what did the application do with the output — not "is the model broken."

What matters in production: token-budget management, TTFT vs TPOT targets, cost per request, output validation, and graceful fallback when the model errors or refuses.

How to debug/verify: log the exact token-rendered prompt sent to the model and the exact raw output. 90% of "the model is dumb" bugs are actually "the context didn't contain what you thought" or "your parser mangled the output."

Questions to ask a provider: Do you bill input and output separately? Are cached input tokens discounted? Are reasoning tokens billed? Does the listed context window include output? Is there a separate max-output limit?

What silently gets expensive/unreliable: unbounded chat history (every turn re-sends all prior tokens), verbose system prompts sent on every call, reasoning tokens you didn't budget for, and retries that double cost on transient errors.

5. Warmup Readings

6. Deep Readings and External References

Prefer these primary sources over blog summaries when a detail matters for a cost or capacity decision.

7. Key Terms

8. Important Facts

• Tokens are not words: on average 1 token ≈ 0.75 English words (≈ 4 characters).
• A "1M-token context window" is roughly 750,000 words — several books.
• A 70B-parameter model holds 70 billion numbers; in FP16 that's ≈ 140 GB, and ~35 GB at 4-bit.
• The KV cache grows linearly with context length × batch size — it, not the weights, often caps concurrency.
• Most providers bill input and output tokens separately; output is commonly 2–5× more expensive.
• Reasoning tokens (hidden chain-of-thought) are usually billed at output rates even though you never see them.
• Cached input tokens can cost 75–90% less when prompt caching is supported.
• The model has no memory between calls — continuity exists only because you re-send prior turns as tokens.
• temperature = 0 is near-deterministic, not guaranteed bit-identical, due to floating-point and backend nondeterminism.

9. Observations from Real Systems

• OpenAI / Anthropic APIs return a usage object with input, output, and (sometimes) cached/reasoning token counts — this is your ground truth for cost, not your own estimate.
• OpenRouter normalizes many providers behind one /chat/completions shape; the same messages array routes to dozens of models — direct proof that "the model" is an interchangeable next-token engine behind a uniform interface.
• vLLM exposes max_model_len (the context window) and pages the KV cache (PagedAttention) precisely because KV memory — not weights — is the scaling bottleneck described in Law 1/Fact above.
• llama.cpp / Ollama let you set --ctx-size / num_ctx; pushing it beyond the trained window degrades output, demonstrating that context window is a trained property, not a free dial.
• Cursor / VS Code Copilot spend most of their engineering effort assembling the context window (open files, repo index, diffs) — concrete evidence that getting the right tokens into context (Law 1) is the real product.

10. Common Misconceptions

11. Engineering Decision Framework

Use this when you are unsure how to handle any new LLM feature or problem:

Q1: Is the needed information in the context window?
    No  → add it via RAG / tools / prompt; the model cannot invent it.
    Yes → continue.

Q2: Who must act on the model's output?
    A tool/system → treat output as a PROPOSAL: validate, permission, execute, log.
    A human       → format for review; never auto-apply high-impact actions.

Q3: What is the cost driver?
    Input tokens  → compress context, cache prefixes, retrieve less.
    Output tokens → cap max_tokens, lower verbosity, avoid needless reasoning.

Q4: What is the latency driver?
    TTFT high     → shrink/cached prompt, smaller model, prefill optimization.
    TPOT high     → speculative decoding, smaller/faster model, less output.

Q5: Is quality actually the bottleneck?
    Measure first → only then reach for a bigger model / reasoning / fine-tuning.

12. Hands-On Lab

Goal

Observe the token-generation loop directly and prove the Six Laws to yourself with real numbers.

Prerequisites

• Python 3.10+
• An OpenAI API key in OPENAI_API_KEY (or adapt to Anthropic/any OpenAI-compatible endpoint)

Setup

pip install openai tiktoken
export OPENAI_API_KEY=sk-...

Steps

1. Count tokens (the unit of everything):

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
text = "Explain the KV cache to a software engineer in 3 sentences."
tokens = enc.encode(text)
print(f"Token count: {len(tokens)}")
print(f"First IDs: {tokens[:20]}")

2. Call the model and read ground-truth usage:

from openai import OpenAI
client = OpenAI()
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is a transformer in 2 sentences?"},
    ],
    max_tokens=100,
)
print(resp.choices[0].message.content)
u = resp.usage
print(f"input={u.prompt_tokens} output={u.completion_tokens} total={u.total_tokens}")

3. Compute cost manually (Law 2):

in_price, out_price = 0.15, 0.60  # gpt-4o-mini $/1M tokens
cost = u.prompt_tokens/1e6*in_price + u.completion_tokens/1e6*out_price
print(f"Estimated cost: ${cost:.6f}")

4. Test determinism (Law 4 / sampling): run the same prompt 3× at temperature=0, then 3× at temperature=1.0. Record whether outputs are identical.

5. Prove "no memory" (Law 1): make one call asking the model to remember a secret word; in a second separate call (no history) ask for the word. Observe it cannot.

Expected output

• Token counts in the 10–20 range for short prompts.
• usage numbers that match your tiktoken estimate closely.
• temperature=0 outputs near-identical; temperature=1.0 visibly varied.
• The model fails to recall the secret across separate calls.

Debugging tips

• tiktoken count ≠ API prompt_tokens? You forgot the system/role formatting overhead — the API counts the full rendered message structure.
• Auth errors → check OPENAI_API_KEY. 429 → you hit a rate limit; back off.

Extension task

Write check_fits(messages, model, ctx_limit) that sums tokens across all messages and warns at >80% of the context window.

Production extension

Wrap calls in a function that logs model, input_tokens, output_tokens, cost, latency_ms to a CSV/DB row — the seed of a real usage meter (Phase 9, Lab 4).

What to measure

Token count per prompt, cost per call, output variance by temperature, TTFT (time to first streamed token if you enable stream=True).

Deliverables

• A token+cost table for 5 different prompts.
• A note answering: did temperature=0 give identical outputs? Did memory persist across calls?

13. Verification Questions

Basic

1. What is a token, and why isn't it the same as a word?
2. What does "autoregressive" mean for generation?
3. What does the context window bound?
4. What does temperature control — and what does it not control?

Applied 5. A 128K-context model is given a 200-page doc (~100K words). Does it fit? Show your token math. 6. A request uses 1,000 input and 500 output tokens at $3/1M in, $15/1M out. What is the cost?

Debugging 7. The same question yields inconsistent answers. Name two causes and how to test each. 8. Costs are 4× your estimate. List three things to check first.

System design 9. You must process 10,000 documents/day. How do you reason about cost, batching, and parallelism using the Six Laws?

Startup / product 10. An investor asks why your AI feature's gross margin will improve at scale. Using Laws 1–2, give a token-based answer (hint: caching, routing, retrieval).

14. Takeaways

1. An LLM predicts one token at a time — every feature is built on that loop.
2. The context window is the model's only runtime memory; if it's not in context, the model doesn't have it.
3. Cost = tokens × price; latency = TTFT + TPOT × output.
4. Prefill, decode, KV cache explain nearly all performance and cost behavior.
5. The model proposes; the application decides — and the application owns safety.
6. Bigger is not automatically better — measure before scaling model, context, or reasoning.

15. Artifact Checklist

Produce and keep these in your repo:

• Notes: one-page summary of the token loop and the Six Laws in your own words.
• Diagram: redraw the inference pipeline (Section 3) from memory.
• Code: the token counter + cost estimator from the lab.
• Benchmark result: a 5-prompt token+cost table.
• Observation log: determinism and "no memory" findings.
• Cost estimate: monthly cost for a 10,000-call/day workload.
• Cheat card: the Six Laws taped above your desk (literally or in your notes).

Next: 01 — Tokenization and Context Windows