Hitchhiker's Guide — Tokenization & the Multimodal Input Pipeline

The compressed practitioner tour. If WARMUP.md is the professor, this is the senior who leans over and says "here's what you actually need to remember about the thing that turns text into numbers."

The 30-second mental model

The model speaks integers, not text. The tokenizer is the whole bridge: encode text → IDs, decode IDs → text. Use subword units (between char-level, which is too long, and word-level, which is too big and <unk>-lossy), learned by BPE = "greedily merge the most frequent adjacent pair, repeat." Make the base alphabet the 256 bytes so the codec is closed — no <unk>, decode(encode(x)) == x for anything (emoji, CJK, binary). Special tokens (<bos>, <eos>, <|im_start|>) are structural IDs, never minted from user text. The chat template must match training byte-for-byte or quality silently dies. And remember: an image is just more tokens (patches → vectors → spliced into the sequence).

The numbers to tattoo on your arm

The framework one-liners (where these live in real tools)

# OpenAI / tiktoken — byte-level BPE, closed vocab, no <unk>
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
ids = enc.encode("the cat");  enc.decode(ids)            # exact round-trip
len(enc.encode(text))                                    # token count == your bill

# HuggingFace — train, encode, and (critically) the chat template
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
tok.bos_token_id, tok.eos_token_id, tok.pad_token_id     # the structural IDs
prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
# ^ ALWAYS use this. Never hand-roll the chat string.

# SentencePiece — BPE or unigram, language-independent
# spm_train --model_type=bpe --vocab_size=32000 --input=corpus.txt

War stories

• The chat template that ate 8 points of quality. A team hand-built the prompt string with an extra space after the role (<|im_start|>user instead of \n). No crash. Evals quietly dropped; it took a week to find. Fix was one line: apply_chat_template. The model had memorized the exact bytes.
• The <eos> that wouldn't stop. The serving loop didn't register the model's end token as a stop, so generations ran past the turn and hallucinated a fake user reply. Classic special-token mishandling — <eos>/<|im_end|> is a stop condition, not just text.
• The triple-billed customer. A non-English user complained their "same length" messages cost 3× and truncated early. The English-trained tokenizer had high fertility on their language — 2–3 bytes/token vs ~4 chars/token for English. Not a bug in the meter; a bias in the vocab.
• The cursed string. One specific username made the model emit gibberish on sight. A glitch token (SolidGoldMagikarp-style): in the vocab, but unseen in training → random embedding → chaos.
• "Why is it bad at math?" Asked to add 7-digit numbers, the model flailed. Root cause wasn't reasoning — BPE had chunked the digits non-positionally. Digit-splitting tokenizers exist for exactly this.

Vocabulary (rapid-fire)

• BPE — Byte-Pair Encoding; greedily merge the most frequent adjacent pair, repeat.
• Merge / merges.txt — the ordered, frozen list of learned merges; its order is the encoding priority.
• Byte-level / byte fallback — base alphabet is the 256 bytes; makes the codec closed (no <unk>).
• Special / control token — reserved structural ID: <bos>, <eos>, <pad>, <|im_start|>.
• Chat template / ChatML — the exact string format the chat model was trained on.
• Fertility — tokens per word; the multilingual fairness/cost knob.
• Compression — chars per token; sets context length and $/token.
• Glitch token — in the vocab but undertrained → bizarre behavior.
• Patch token — an image patch projected to a d_model vector; multimodal's "token."
• add_generation_prompt — append the open <|im_start|>assistant\n turn for the model to complete.

Beginner mistakes

• Hand-rolling the chat string instead of apply_chat_template (the silent quality killer).
• Treating <eos>/<|im_end|> as text instead of a registered stop.
• Letting user text mint control tokens (encoding <bos> to bos_id) — an injection vector.
• Assuming decode(encode(x)) == x without testing spaces, newlines, CJK, and emoji.
• Quoting "tokens ≈ words" — they're frequency fragments; the, ization, and a 5-piece name all happen.
• Blaming the model for arithmetic / indentation / multilingual-cost issues that are tokenization artifacts.
• Maxing out vocab size for "better compression" while ignoring the 2 · V · d_model tax and rare- token risk.

The one thing to take away

The tokenizer is the contract between text and the model, and its failures are silent. Before you blame the model, check the boundary: is the round-trip exact, are the special tokens structural, and does the chat template match training byte-for-byte? If yes, it's the model. If you didn't check, you're guessing.