Warmup Guide — Tokenization & the Multimodal Input Pipeline

Zero-to-senior primer for Phase 01. The model is a function from a sequence of integer IDs to a probability distribution over the next ID. Everything between the human-readable input and that first integer is tokenization — and it is where the most expensive, most silent bugs in the entire LLM stack live. We start from "why can't the model just read characters?" and end at "an image is just more tokens." Along the way: the BPE training algorithm step by step, byte-level encoding and why it kills <unk>, the special/control tokens that make or break chat, the chat template you must reproduce byte-for-byte, the vocabulary-size tradeoff, the famous pitfalls (arithmetic, whitespace, multilingual fairness, glitch tokens), and the bridge to multimodal input.

Table of Contents

• Chapter 1: The Model Speaks Integers, Not Text
• Chapter 2: Char vs Word vs Subword — Why Tokenization Exists
• Chapter 3: The BPE Training Algorithm, Step by Step
• Chapter 4: Merge Priority & the Deterministic Tie-Break
• Chapter 5: Byte-Level BPE & the Byte Fallback (No <unk>)
• Chapter 6: Special & Control Tokens — the #1 Chat Bug
• Chapter 7: Chat Templates (ChatML) & Byte-for-Byte Matching
• Chapter 8: Vocabulary-Size Tradeoffs — Compression vs the Embedding Table
• Chapter 9: Tokenization Pitfalls — Numbers, Whitespace, Fairness, Glitch Tokens
• Chapter 10: The Bridge to Multimodal — Image Patches as Tokens
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: The Model Speaks Integers, Not Text

From zero. A transformer's first layer is an embedding lookup: a table with one row per vocabulary entry, where row i is a learned d_model-dimensional vector. To use that table, the input must already be a list of integer IDs, one per position. The model has no notion of "letters" or "words" — it has a fixed set of V symbols numbered 0..V-1, and it manipulates the vectors those numbers point to. The entire job of the tokenizer is the two functions that bracket the model:

   "the cat"  ──encode──►  [1037, 4937]  ──►  [embedding lookup] ──► transformer ──► logits
                                                                                        │
   "the cat"  ◄──decode──  [1037, 4937]  ◄──────────────── argmax / sampling ◄──────────┘

Why this framing matters. Because the model only ever sees IDs, three properties of the tokenizer become correctness-critical, and none of them are about the model:

• It must be reversible. decode(encode(x)) should return x exactly, or you have silently corrupted the user's input before the model ever ran.
• It must be closed. Every possible input must map to some sequence of IDs — there can be no "I don't have a symbol for this" hole that loses data.
• It must match training. The model learned statistics over a specific tokenization. Feed it a different one at serving time and you are off-distribution, with no error to tell you so.

Production significance. Every "the model got dumber overnight" incident, every "it can't count the letters in a word" complaint, every "non-English users are billed triple" ticket traces back to a tokenizer property, not a model weight. The tokenizer is the cheapest component to get wrong and the most expensive to debug, because it fails silently.

Common misconception. "Tokenization is preprocessing — boring plumbing I can ignore." It is the plumbing, and the plumbing is where the leaks are. Senior engineers treat the tokenizer as a first-class part of the model contract, versioned and tested like one.

Chapter 2: Char vs Word vs Subword — Why Tokenization Exists

The design question. We need to chop text into a sequence of symbols drawn from a fixed vocabulary. What should the symbols be? There are three natural choices, and tokenization exists because the obvious two are both bad.

Option A — characters (or bytes). Vocabulary is tiny (~100 characters, or exactly 256 bytes). Closed by construction (every string is a sequence of characters). But sequences are long: "internationalization" is 20 character-tokens. Since inference cost is ~2N FLOPs per token (Phase 00) and attention is O(T²), long sequences are directly expensive, and the model must learn to compose meaning from tiny pieces over a long range — harder to learn.

Option B — words. Sequences are short (one token per word) and each token is meaning-rich. But the vocabulary is enormous and open: natural language has a long tail of rare words, names, typos, and morphology, so you either keep a million-entry vocab (a giant embedding table) or you cap it and emit <unk> for everything outside — which is lossy (you can't decode <unk> back) and brittle (the model is blind to exactly the rare, information-dense words you most need it to read).

Option C — subwords (the winner). Keep whole common words as single tokens, but break rare words into reusable sub-word pieces. "tokenization" might be token + ization; an unseen name breaks into smaller fragments that still exist in the vocab. You get short-ish sequences and a bounded vocabulary and, with a byte-level base alphabet (Chapter 5), no <unk> at all. The data decides where the boundaries are, via a learned algorithm — Byte-Pair Encoding (BPE).

How it works under the hood (the intuition). BPE starts from the smallest pieces (bytes) and greedily merges the most frequent adjacent pair over and over. Frequent sequences — common words, common suffixes — get merged into single tokens because they show up a lot; rare sequences stay broken into smaller pieces. The vocabulary self-organizes around the statistics of the corpus.

Production significance. Compression (characters per token) directly sets your effective context length and your per-request cost. A good subword vocab gets English to ~4 characters per token (≈ 0.75 words/token) — that's the number to remember.

Common misconception. "Tokens are words." They are frequency-driven fragments. " the" (with a leading space) is often one token; "tokenization" might be two or three; a rare name might be five. The model reasons over these fragments, which is exactly why it struggles to count letters or do digit arithmetic (Chapter 9).

Chapter 3: The BPE Training Algorithm, Step by Step

What BPE training produces. Two artifacts: an ordered list of merges (each a pair of symbols that fuse into one) and a vocabulary (every token string → integer ID). The ordered merge list is the heart of it — its order is the encoding priority (Chapter 4).

The algorithm. Given a corpus and a target number of merges:

1. Pre-tokenize the corpus into "words." (Our lab splits on \S+|\s+, so whitespace runs are their own words — this is what makes spaces survive the round-trip; Chapter 9. Real GPT-2/GPT-4 use a richer regex.) Represent each word as a tuple of its base symbols (in our lab, its UTF-8 bytes). Count how many times each distinct word-form occurs.
2. Repeat num_merges times: a. Count every adjacent symbol pair across all words, weighted by word frequency. b. Pick the most frequent pair (tie-break: lexicographically smallest — Chapter 4). c. Append it to the merge list and rewrite every word, fusing that pair into one symbol. d. Stop early if there are no pairs left.
3. Assemble the vocab: reserved specials first (stable IDs), then the base alphabet, then each merged token in merge order.

A fully worked example. Take the toy corpus and pretend the base symbols are the characters themselves (the lab uses bytes, but the counting is identical):

corpus words (with counts):   low ×5   lower ×2   newest ×6   widest ×3

Represent each as characters and count adjacent pairs across all words, weighted by the word counts. The pair e+s appears in newest (×6) and widest (×3) → count 9. The pair s+t appears in the same two words → count 9 too. (Here e+s wins on the tie-break since it's lexicographically smaller.) So:

merge 1:  e + s  ->  es        # newest -> n e w es t,  widest -> w i d es t

Recount. Now es+t appears in both → count 9, the new winner:

merge 2:  es + t -> est        # newest -> n e w est,   widest -> w i d est

Recount. Now l+o appears in low (×5) and lower (×2) → count 7:

merge 3:  l + o  -> lo
merge 4:  lo + w -> low        # "low" is now a single token; "lower" -> low e r

The merge count is the literal answer to "how many times did we run the loop" — four merges produced the tokens es, est, lo, low. After k merges the vocabulary is (base alphabet) + k (minus any key collisions). This is exactly what train_bpe(corpus, num_merges) does in the lab; its first merge on the lab corpus is ('a','t') (frequency 8: cat×3, sat×2, mat, rat, ate), which beats ('t','h') at frequency 6 (the×6) — count the actual most-frequent pair, not the eye-catching one.

Why frequency-greedy. The original Sennrich et al. (2016) insight was that you don't need a clever objective: greedily merging the most common pair already discovers morphologically sensible units (stems, suffixes, common words) because those are the frequent sequences. It's simple, fast, and good enough — which is why it's everywhere.

Production significance. This is the exact loop inside HuggingFace tokenizers BpeTrainer and SentencePiece's BPE mode. The only production differences are scale (millions of merges, parallelized counting), a richer pre-tokenizer, and stopping at a target vocab size rather than a fixed merge count (Chapter 8).

Common misconception. "BPE finds the linguistically correct morphemes." It finds frequency units. They often look morphological (because frequent things often are), but BPE will happily merge " the" or "ization" or an artifact of your corpus's quirks — it has no grammar, only counts.

Chapter 4: Merge Priority & the Deterministic Tie-Break

The problem. Two facts make BPE subtle. First, the merge list isn't just a set — it is an ordered list, and that order is reused at encoding time. Second, ties are common (many pairs share the same frequency), and if you resolve them by "whatever the dict happened to yield first," your tokenizer becomes non-deterministic across Python versions, machines, and corpus orderings — a nightmare, because two runs produce different vocabularies and the IDs no longer mean the same thing.

How merge order becomes encoding priority. When you encode a new word, you don't recount frequencies — you apply the learned merges in rank order. The merge learned first has rank 0 (highest priority); to tokenize a word you repeatedly find the present adjacent pair with the lowest rank and fuse it, until no learned pair remains. In the lab this is merge_ranks (a pair -> rank dict) and the _bpe method:

candidate = min(
    (p for p in pairs if p in self.merge_ranks),  # only learned pairs
    key=lambda p: self.merge_ranks[p],            # lowest rank = applied first
    default=None,
)

This is why the order matters: applying es+t before e+s would give a different segmentation. Encoding must replay training's priority exactly.

The deterministic tie-break. During training, when two pairs tie on frequency, pick the lexicographically smallest pair. This single rule makes the whole algorithm a pure function of (corpus contents, num_merges) — independent of dict iteration order or the order lines appear in the corpus. The lab proves this with test_training_is_order_independent_of_dict_iteration: reversing the corpus lines yields byte-identical merges. (The lab implements "smallest wins" by inverting the key inside a max; the effect is the same.)

Why tattoo this on your arm. A tokenizer is a contract. Its IDs are baked into a trained model. If the same corpus can produce two different vocabularies, you cannot reproduce a model, cannot share a tokenizer across a team, and cannot debug a mismatch. Determinism is engineered, via the tie-break — never assumed.

Production significance. Real trainers (HF, SentencePiece) pin determinism the same way (a defined tie-break, fixed iteration). It's why a merges.txt is a versioned, frozen artifact shipped with the model.

Common misconception. "Ties are rare, so the tie-break doesn't matter." On real corpora, ties are everywhere (huge numbers of pairs share low frequencies), and an unstable tie-break is exactly the kind of bug that passes your tests on one machine and corrupts a model on another.

Chapter 5: Byte-Level BPE & the Byte Fallback (No <unk>)

The problem it solves. If your base alphabet is "characters seen in the training corpus," then any unseen character — an emoji, a CJK glyph, a rare diacritic, a raw control byte — has no base token and must become <unk>. That is lossy (you can't decode it back) and the model is blind to it. The fix is to make the base alphabet the 256 byte values themselves. Since every string, in any language, is ultimately a sequence of UTF-8 bytes, every possible input decomposes into base byte-tokens. There is nothing left to be out-of-vocabulary.

How it works under the hood. Encode the text to UTF-8 bytes; each byte is a base symbol; BPE merges sit on top of the byte alphabet exactly as before. A 4-byte emoji with no learned merges simply becomes its 4 byte-tokens — test_byte_fallback_for_unseen_unicode asserts exactly len(ids) == len("🦩".encode("utf-8")). Decoding maps each byte-token back to its byte and UTF-8-decodes the byte stream. The codec is closed, and decode(encode(x)) == x holds for any round-trippable input — the "soul test."

The GPT-2 byte↔unicode trick (why the lab has BYTE_ENCODER). Some byte values are control or whitespace bytes that are ugly or ambiguous as dictionary keys. GPT-2 (and the lab) map each of the 256 bytes to a distinct printable unicode character via a fixed bijection, so every base token is a clean single-character string with no raw control bytes inside vocab keys. It is perfectly invertible (BYTE_DECODER is the inverse), so it changes nothing semantically — it just makes the implementation tidy. This is why you'll see funny characters like Ġ (a space) in a GPT-2 vocab dump.

Production significance. GPT-2, GPT-3, GPT-4 (cl100k_base/o200k_base via tiktoken), Llama, and friends are all byte-level. The practical payoff: no <unk> token in normal operation, exact round-trips, and graceful handling of any language or binary-ish input — a whole category of data-loss bugs simply cannot occur.

Common misconception. "Byte-level means it tokenizes every character as a byte, so it's slow and long." No — the base is bytes, but the merges fuse common byte sequences back into whole-word tokens. Byte-level BPE on English still hits ~4 chars/token; bytes only show up un-merged for the rare, unseen stuff. You get closedness for free without sacrificing compression on common text.

Chapter 6: Special & Control Tokens — the #1 Chat Bug

What they are. Special tokens are reserved vocabulary entries that carry structure, not text: <bos> (beginning of sequence), <eos> (end of sequence — the stop signal), <pad> (filler to make a batch rectangular), <unk> (the unknown token we engineered away in Chapter 5 but still reserve an ID for), and the chat control tokens <|im_start|> / <|im_end|> that mark role boundaries (Chapter 7). In the lab these are SPECIAL_TOKENS = ("<pad>", "<unk>", "<bos>", "<eos>") plus IM_START / IM_END.

Why they exist. The model needs to know where a sequence starts and stops, where one turn ends and the next begins, and which positions are real vs padding (so attention can mask the pad). These are structural signals the model is trained to recognize as dedicated symbols — they get their own embedding rows and the model attends to them as hard boundaries.

How they work under the hood — and the trap. The critical property: a special token is added structurally (e.g., encode(text, add_bos=True) prepends bos_id), and it must never be produced by tokenizing user text that merely contains the same characters. If a user types the literal string <bos> and your tokenizer maps it to the reserved bos_id, you have let user input forge a control signal — a prompt-injection-flavored bug. The byte path makes this safe: the literal <bos> decodes into ordinary byte-tokens, so test_special_token_string_is_not_split_into_pieces asserts bos_id not in encode("<bos>") and that it still round-trips. Reserved IDs are added by the framework, never minted from the content.

  encode("<bos>")            -> byte-tokens for '<','b','o','s','>'   (NOT bos_id)  ✅ safe
  encode("hi", add_bos=True) -> [bos_id, ...byte-tokens for 'hi']     (structural)  ✅ correct

Production significance. This is the #1 chat bug, in two flavors. (1) Stop-token mishandling: if you don't recognize <eos> / <|im_end|> as a stop, generation runs on past the turn into hallucinated "user" messages; if you accidentally split it, the model never stops. (2) Injection: if content can mint role/control tokens, a user can fake a system turn. Decode must skip specials (they carry no source text — the lab does if tok in specials: continue), and encode must add them structurally.

Common misconception. "<eos> is just text the model emits." <eos> is a reserved ID and the serving loop's stop condition. Treating it as ordinary text — or forgetting to register it as a stop — is why a model "won't shut up" or bleeds into a fake next turn.

Chapter 7: Chat Templates (ChatML) & Byte-for-Byte Matching

What a chat template is. A base language model just continues text. To make it a chat model, you fine-tune it on conversations rendered into a specific string format with role markers. The template is the function that turns a list of {"role", "content"} messages into that exact string. The dominant format is ChatML (OpenAI's "Chat Markup Language"), which the lab implements:

<|im_start|>system
You are concise.<|im_end|>
<|im_start|>user
Hi!<|im_end|>
<|im_start|>assistant

Each message becomes <|im_start|>{role}\n{content}<|im_end|>\n, and when you want the model to respond, you append an open assistant turn — <|im_start|>assistant\n with no closing tag — so the model's job is literally "continue this string." That trailing open turn is the add_generation_prompt=True flag (test_chat_template_exact_string pins the exact bytes).

How it works under the hood. The control tokens (<|im_start|>, <|im_end|>) are special tokens (Chapter 6) the model learned as role boundaries. The model was trained to emit <|im_end|> to end its turn — that's the stop signal the serving loop watches for. Generation is "complete the open assistant turn until you produce <|im_end|>."

Why byte-for-byte matching is non-negotiable. The model learned the exact boundary bytes, including the newline after the role and the newline after <|im_end|>. If your serving renderer adds a space (<|im_start|>user instead of <|im_start|>user\n), drops the trailing newline, or uses a different role word than training did, you are now feeding the model a string it never saw during training. There is no error — the model just produces measurably worse completions. This is the single most common, most maddening silent quality regression in chat systems.

  TRAINING:  ...<|im_start|>user\nHi!<|im_end|>\n<|im_start|>assistant\n
  SERVING :  ...<|im_start|>user Hi!<|im_end|> <|im_start|>assistant       ← extra spaces, no \n
                              ▲                ▲                 ▲
                       off-distribution: no exception, just worse answers

Production significance. Every model ships its own template (tokenizer_config.json carries a Jinja chat_template); tokenizer.apply_chat_template(messages, add_generation_prompt=True) renders it. Using the wrong model's template, hand-rolling the format, or a framework that "helpfully" normalizes whitespace are classic causes of a model that "got dumber" with no code change to the model.

Common misconception. "Any reasonable chat format works; the model is smart enough." It is not a robustness question — the model has memorized the exact format. Match it byte-for-byte or eat a silent quality hit. Use the model's own template; never improvise.

Chapter 8: Vocabulary-Size Tradeoffs — Compression vs the Embedding Table

The lever. Vocabulary size V is a hyperparameter you choose (via the number of merges / a target size). Typical modern values are 32k–128k (GPT-2: 50k; Llama-2: 32k; GPT-4 cl100k: ~100k; o200k: ~200k). Bigger isn't free in either direction — it's a genuine tradeoff.

The benefit of a bigger vocab — compression. More merges means more whole words/phrases are single tokens, so a given text becomes fewer tokens. Fewer tokens means: lower 2N/token compute cost (Phase 00), more text fitting in a fixed context window, and (often) better quality because the model reasons over meaning-rich units. Roughly, more vocab → fewer tokens per character (higher compression).

The cost of a bigger vocab — the embedding table. The embedding matrix and the output (LM-head) matrix are each V × d_model. Doubling V doubles those two matrices' parameters and FLOPs (and the final softmax is over V logits). For a model with d_model = 4096, going from 32k → 128k vocab adds (128k - 32k) × 4096 × 2 (in + out) ≈ 0.8B parameters — pure tax, plus the softmax cost on every step. There are also diminishing compression returns and a rare-token problem: a huge vocab has many tokens seen too rarely in training to learn good embeddings (the glitch-token risk, Chapter 9).

$$ \text{embedding+head params} = 2 \times V \times d_{\text{model}}. $$

How to decide (carry the Phase 00 habit). Score the dials: compression (→ context + $/token), embedding/softmax cost, multilingual coverage, and rare-token risk; weight them for your workload; name the deciding dial. A multilingual product weights coverage and compression heavily (favoring a bigger, balanced vocab); a single-language, latency-tight product may favor a smaller one.

Production significance. The trend is upward (GPT-4 → o200k) because larger, well-balanced vocabs improve multilingual fairness and compression, and the V·d tax is small relative to a big model. But for a small model, the embedding table can be a large fraction of total params — there the tradeoff bites hard.

Common misconception. "Bigger vocab is strictly better compression, so maximize it." Compression returns diminish, the V·d cost is real, and an oversized vocab manufactures rarely-trained tokens. There is a knee; find it for your corpus and model size.

Chapter 9: Tokenization Pitfalls — Numbers, Whitespace, Fairness, Glitch Tokens

Tokenization causes a cluster of famous, non-obvious model failures. Knowing their root cause is a senior signal.

Numbers and arithmetic. BPE merges digits by frequency, not by place value, so "327" might be one token while "328" is two, and "1234567" fragments unpredictably. The model sees inconsistent, non-positional chunks, which is a major reason LLMs are shaky at multi-digit arithmetic and digit manipulation. The fix some models adopt: split every digit into its own token (a pre-tokenizer rule), giving a consistent positional representation. When someone asks "why is GPT bad at math," the first-order answer is tokenization, not reasoning.

Whitespace. Spaces are part of tokens (often a leading-space token like " the"), and exact whitespace must survive the round-trip — otherwise the model can't reproduce code indentation, can't count spaces, and decode(encode(x)) != x. The lab guarantees this by splitting on \S+|\s+ so whitespace runs are their own words (test_whitespace_only, the multi-space and \n/\t round-trip cases). Whitespace bugs are why a model mangles Python indentation or "loses" blank lines.

Multilingual fairness. If the BPE corpus is English-heavy, English compresses to ~4 chars/token but a low-resource language may compress to ~1–2 chars/token — its text becomes 2–3× more tokens. Consequences: those users pay 2–3× more (cost is per token), hit the context limit sooner, and the model has less effective capacity for their text. This fertility gap is a real fairness and economics problem, and it's a direct artifact of whose data trained the tokenizer. Balanced multilingual corpora and bigger vocabs (Chapter 8) narrow it.

Glitch / "SolidGoldMagikarp" tokens. A token can land in the vocabulary (it appeared in the BPE corpus — e.g., a Reddit username or a scraping artifact) but be rare or absent in the model's training data. Its embedding row is then barely trained — essentially random — so when that token appears at inference the model behaves bizarrely: refuses, hallucinates, repeats, or emits gibberish. The infamous SolidGoldMagikarp was one such token. Root cause: a mismatch between the tokenizer's corpus and the model's training corpus. Mitigations: train them on aligned data, filter pathological tokens, and monitor rare-token activations.

Production significance. Each of these is a recurring support ticket: "bad at math," "broke my code's indentation," "non-English costs more," "weird output on this one string." A senior names the tokenizer cause immediately instead of blaming the model.

Common misconception. "These are model intelligence limits." They are representation artifacts of how text was chopped before the model ever saw it. Change the tokenization and several of them improve — which is why digit-splitting and balanced multilingual vocabs exist.

Chapter 10: The Bridge to Multimodal — Image Patches as Tokens

The unifying idea. A transformer doesn't care that its tokens came from text. It consumes a sequence of d_model vectors and does attention over them. So to make a model see, you only need to turn an image into a sequence of d_model vectors — i.e., into tokens. Everything you learned about sequences, special tokens, and templates carries over unchanged.

How an image becomes tokens (the ViT recipe). Cut the image into a grid of fixed-size patches (e.g., 16×16 pixels). Flatten each patch and pass it through a small linear projection to get one d_model vector — a patch embedding, the image's analog of a token embedding. A 224×224 image at 16×16 patches yields a 14×14 grid = 196 patch tokens. Add positional information (so the model knows the 2D layout), and you now have a sequence the transformer can attend over, exactly like text.

   image ──► [16×16 patches] ──► linear proj ──► 196 patch vectors  (each d_model)
                                                        │
   text  ──► [BPE tokens] ───────► embed ────► text vectors (each d_model)
                                                        │
                         ┌──────────────────────────────┘
                         ▼
   "What is in <image> ?"  →  [..., <image-placeholder expands to 196 patch vectors>, ...]
                                                        │
                                                  one mixed sequence → transformer

The interface trick (why this connects to Chapter 6). A multimodal model reserves an <image> special token. At input assembly, that one placeholder is expanded into k patch slots, and the patch embeddings are spliced into those positions in the sequence. The text tokens and image tokens live in the same sequence, the same attention, the same template — tokens are the universal interface. Audio is the same story (frames/spectrogram patches → tokens).

Production significance. This is how GPT-4V, Llava, Qwen-VL, and friends work: a vision encoder produces patch tokens, a projector maps them into the language model's embedding space, and they're inserted at the <image> placeholder. The full mechanism — patch embedding, the vision encoder, cross-modal alignment — is Phase 04. For now, internalize the one sentence: an image is just more tokens.

Common misconception. "Multimodal models have a totally separate 'vision brain.'" There's a vision encoder up front, but the heavy lifting happens in the same transformer over a single token sequence. Once you see "everything is tokens," multimodality stops being magic and becomes plumbing — the same plumbing this phase is about.

Lab Walkthrough Guidance

The lab (lab-01-bpe-tokenizer) turns these chapters into code. Suggested order (matches the file):

1. The byte bijection (BYTE_ENCODER / BYTE_DECODER) — Chapter 5. Given; understand that it maps each of the 256 bytes to a clean printable char and is invertible.
2. train_bpe — Chapters 3–4. Pre-tokenize on \S+|\s+, count frequency-weighted pairs, pick the most frequent with the lexicographic tie-break, rewrite the words, repeat. Assemble the vocab: specials, then 256 bytes, then merges. test_most_frequent_pair_merged_first and test_tie_break_is_lexicographically_smallest are the correctness anchors.
3. BPETokenizer.__post_init__ — Chapters 5–6. Validate the codec is closed (all specials + all 256 bytes present, unique IDs); build id_to_token and merge_ranks.
4. _bpe + encode — Chapter 4. Apply merges in rank order (lowest rank first). Add <bos> / <eos> structurally.
5. decode — Chapters 5–6. Skip specials, map byte-tokens back to bytes, UTF-8 decode. This is the soul test: test_decode_encode_roundtrip across ASCII/spaces/newlines/accents/CJK/emoji.
6. render_chat — Chapter 7. Emit <|im_start|>{role}\n{content}<|im_end|>\n per message and the open assistant turn; test_chat_template_exact_string pins it byte-for-byte.

Run red → green: pytest test_lab.py -v, then LAB_MODULE=solution pytest test_lab.py -v, then python solution.py for the worked example.

Success Criteria

• All tests pass against your lab.py and against LAB_MODULE=solution.
• You can run one BPE merge step by hand (count pairs, pick the most frequent, apply the lexicographic tie-break, rewrite) and explain why the lab's first merge is ('a','t'), not ('t','h').
• You can explain why the merge order is the encoding priority and why the tie-break makes training a pure function of (corpus, num_merges).
• You can explain why byte-level BPE has no <unk> and why decode(encode(x)) == x holds for emoji and CJK — and demonstrate it.
• You can explain why encode("<bos>") must not contain bos_id, and why that is a security property.
• You can write a ChatML turn byte-for-byte and name the failure mode of a whitespace mismatch.
• You can state the vocab-size tradeoff (compression vs 2 · V · d_model) and the deciding dial for a given workload.
• You can name three tokenization pitfalls and their root cause, and explain "an image is just more tokens."

Interview Q&A

• "Char, word, or subword — and why did subword win?" — Char: tiny closed vocab but very long sequences (expensive 2N/token, O(T²) attention). Word: short sequences but unbounded vocab and lossy <unk>. Subword (BPE) balances both, and byte-level makes it closed. ~4 chars/token for English is the number.
• "Walk me through one BPE merge step." — Count all adjacent symbol pairs weighted by word frequency; merge the most frequent (tie-break: lexicographically smallest pair); rewrite every word fusing that pair; repeat. The ordered merge list is the encoding priority.
• "Why does byte-level BPE have no <unk>?" — The base alphabet is the 256 byte values, so every possible input decomposes into base byte-tokens. Nothing is out-of-vocabulary; the codec is closed and decode(encode(x)) == x exactly.
• "How do you guarantee a deterministic tokenizer?" — A defined tie-break (lexicographically smallest pair on a frequency tie) makes the merge list a pure function of (corpus, num_merges), independent of dict order or corpus line order. Determinism is engineered, not assumed.
• "What's the #1 chat bug you've seen?" — A chat-template mismatch: the served format differs from training (a stray space, a missing newline, the wrong role word, or the wrong model's template). No error, just silently worse answers. Always use the model's own template, byte-for-byte.
• "Why must encoding the literal text <bos> not yield the <bos> id?" — Special tokens are structural IDs added by the framework, not minted from content. If user text could forge a control token, that's a prompt-injection vector. The byte path makes <bos> decode into ordinary byte-tokens.
• "Why are LLMs bad at arithmetic?" — Largely tokenization: BPE chunks digits by frequency, not place value, so numbers fragment inconsistently. Digit-splitting pre-tokenizers fix much of it. It's a representation artifact, not (only) a reasoning limit.
• "What is a glitch token like SolidGoldMagikarp?" — A token that's in the vocabulary (it appeared in the BPE corpus) but barely or never in the model's training data, so its embedding is essentially random and the model behaves bizarrely when it appears. Root cause: tokenizer-corpus vs training-corpus mismatch.
• "How does vocab size trade off?" — Bigger vocab → better compression (fewer tokens → cheaper, more context, often better) but a larger 2 · V · d_model embedding+head cost and more rarely-trained tokens. 32k–128k is the usual range; name the deciding dial for the workload.
• "How does multimodal input reuse this?" — An image is cut into patches; each patch is linearly projected to a d_model vector (a patch token) and spliced into the sequence at a reserved <image> placeholder. Same sequence, same attention, same template — tokens are the universal interface (mechanism in Phase 04).

References

• Sennrich, Haddow, Birch, Neural Machine Translation of Rare Words with Subword Units (2016) — the paper that introduced BPE to NLP.
• Radford et al., Language Models are Unsupervised Multitask Learners (GPT-2, 2019) — byte-level BPE and the bytes_to_unicode trick.
• OpenAI tiktoken — the production byte-level BPE tokenizer and cl100k_base / o200k_base encodings: https://github.com/openai/tiktoken.
• HuggingFace tokenizers docs — BPE, BpeTrainer, pre-tokenizers, and apply_chat_template: https://huggingface.co/docs/tokenizers and the Chat Templates guide.
• Kudo, Richardson, SentencePiece: A simple and language independent subword tokenizer (2018) — and Kudo, Subword Regularization (the unigram-LM alternative to BPE).
• Andrej Karpathy, minBPE and his "Let's build the GPT Tokenizer" walkthrough — the clearest from- scratch BPE build: https://github.com/karpathy/minbpe.
• OpenAI ChatML documentation — the chat markup format and its control tokens.
• Rumbelow & Watkins, SolidGoldMagikarp (LessWrong, 2023) — the glitch-token investigation.
• Dosovitskiy et al., An Image is Worth 16×16 Words (ViT, 2020) — patches as tokens (Phase 04 preview).