Phase 01 — Tokenization & the Multimodal Input Pipeline

The phase that builds the boundary between the human world and the tensor world. A model never sees your text, your image, or your audio — it sees a sequence of integer token IDs, and it emits integer IDs back. Everything upstream of the first matmul is tokenization, and it is the most under-respected, highest-leverage component in the stack: get it wrong and your round-trips break, your chat quality silently collapses, your multilingual users pay 3× the cost, and a "harmless" string of text reserves a control token it was never supposed to. Phase 00 taught you the model is a physical system; this phase teaches you what you actually feed it.

Why this phase exists

Every later phase — attention, fine-tuning, RAG, serving, agents — assumes a sequence of token IDs already exists. Where do they come from, and why are they the right ones? The answer is a learned subword vocabulary (BPE), a byte-level base alphabet that makes the codec closed (no <unk>, a true round-trip), a set of reserved special/control tokens the model attends to for structure, and a chat template that must match the training format byte-for-byte. Get these wrong and the failures are silent — no exception, no crash, just degraded quality that takes a week to diagnose. That makes tokenization a senior-level topic disguised as a beginner one.

The five things you will own after this phase:

1. Why subword. Char-level makes sequences too long (and 2N/token cost too high); word-level makes the vocab huge and chokes on out-of-vocabulary words with <unk>. BPE is the data-driven compromise: learn a merge table from a corpus.
2. Byte-level + byte fallback. Start the alphabet from the 256 byte values so every possible input — any language, any emoji, any control byte — decomposes into base tokens. The codec is closed: no <unk> for normal text, and decode(encode(x)) == x exactly.
3. Special/control tokens. <bos>, <eos>, <pad>, and the ChatML <|im_start|> / <|im_end|> are reserved IDs added structurally — never produced by tokenizing user text that happens to contain those strings. Mishandling them is the #1 chat bug.
4. The chat template. Models are trained on an exact string format. Serving must reproduce it byte-for-byte; a stray space or missing newline degrades quality with no error.
5. The bridge to multimodal. An image is just more tokens: cut it into patches, embed each patch, and splice the patch embeddings into the same sequence behind an <image> placeholder. Tokens are the universal interface (full mechanism in Phase 04).

Concept map

                ┌──────────────────────────────────────────────┐
                │  text / image / audio  →  INTEGER TOKEN IDs   │
                └──────────────────────────────────────────────┘
                                     │
          ┌──────────────────────────┼──────────────────────────┐
          ▼                          ▼                          ▼
   GRANULARITY               BASE ALPHABET               STRUCTURE
   char  → seqs too long     bytes (256)  ──────────►    special tokens
   word  → huge vocab, <unk> closed vocab               <bos> <eos> <pad>
   SUBWORD (BPE) ◄───────────no <unk>, round-trips      <|im_start|>/<|im_end|>
          │                          │                          │
          ▼                          ▼                          ▼
   merge table (learned)      byte fallback              CHAT TEMPLATE (ChatML)
   freq-greedy + tie-break    decode(encode(x))==x       byte-for-byte = training
          │                          │                          │
          └──────────────┬───────────┴──────────────┬───────────┘
                         ▼                          ▼
                  VOCAB-SIZE TRADEOFF        PITFALLS
                  compression ↑ vs           numbers/arithmetic, whitespace,
                  embedding table ↑          multilingual fairness, glitch tokens
                         │
                         ▼
                  MULTIMODAL: image patches as tokens  (→ Phase 04)

The lab

The lab is a runnable, test-verified miniature — see the lab standard. Run it red (pytest test_lab.py), make it green, then read solution.py. The soul test is the round-trip: if decode(encode(x)) ever differs from x, the codec is broken — and you will be able to say exactly which invariant failed.

Integrated scenario ideas

• Diagnose a "the model got dumber" report. A chat app's quality dropped after a tokenizer upgrade. Show that the served chat template no longer matches the training format (an extra space after the role), and that this is a byte-for-byte mismatch the model never saw — no exception, just degraded completions.
• Defend a vocab size in a design review. Given a target language mix and an embedding budget, argue 32k vs 128k: bigger vocab compresses sequences (fewer tokens → cheaper 2N/token and longer effective context) but inflates the embedding + LM-head matrix (vocab × d_model). Name the deciding dial (carry the Phase 00 tradeoff habit forward).
• Explain a multilingual cost complaint. A user writing in a low-resource language is billed 3× for the "same" text. Show the fertility gap (tokens per word) and tie it to who the BPE corpus was trained on — a fairness and a cost problem.
• Hunt a glitch token. Reproduce the SolidGoldMagikarp failure mode: a token that exists in the vocab (it appeared in the BPE corpus) but was rarely or never seen in training, so its embedding is essentially random and the model behaves bizarrely when it appears.
• Sketch the multimodal splice. Reserve an <image> token, expand it into k patch slots, and describe how patch embeddings replace those slots in the sequence — the same interface, more tokens (preview of Phase 04).

Deliverables checklist

• lab.py passes pytest test_lab.py -v (and LAB_MODULE=solution does too).
• You can explain char vs word vs subword tradeoffs and why BPE won.
• You can run one BPE merge step by hand: count pairs, pick the most frequent, apply the lexicographic tie-break, rewrite the words.
• You can explain why byte-level BPE has no <unk> and why decode(encode(x)) == x for any input including emoji and CJK.
• You can explain why encoding the literal text <bos> must not yield the reserved bos_id.
• You can state, byte-for-byte, what a ChatML turn looks like and what breaks on a mismatch.
• You can articulate the vocab-size tradeoff (compression vs embedding-table size) with the deciding dial.
• You can name at least three tokenization pitfalls (numbers/arithmetic, multilingual fairness, glitch tokens) and their root cause.

Key takeaways

• The model never sees text — it sees integer IDs. The tokenizer is the entire interface, and every silent quality bug at the edges of an LLM app starts here. Respect it accordingly.
• Byte-level BPE makes the codec closed. Starting from the 256 byte values means every possible input decomposes into base tokens, so there is no <unk> for normal text and the round-trip is exact. This one design choice removes a whole class of data-loss bugs.
• Determinism comes from the tie-break, not from luck. When two pairs tie on frequency, the lexicographically smallest wins — so the merge table is identical regardless of corpus ordering or dict iteration. Determinism is a feature you engineer, not one you hope for.
• Special tokens are structural, not textual. They are reserved IDs added by the framework, and the literal angle-bracket strings in user text must never collide with them — that boundary is both a correctness and a security property.
• The chat template must match training byte-for-byte. This is the most common and most expensive silent failure in chat systems: no error, just worse answers. Always serve the exact format the model was trained on.
• Tokens are the universal interface. Text subwords, image patches, audio frames — all become entries in the same sequence. Once you internalize "everything is tokens," multimodal stops being mysterious (Phase 04 fills in the mechanism).

Next: Phase 02 — The Transformer From Scratch.