« Phase 30 README · Warmup · Hitchhiker's · Principal Deep Dive · Core Contributor · Staff Notes

Phase 30 — Deep Dive: Hugging Face

The load-bearing idea under this entire phase is smaller than the API surface suggests: everything Hugging Face does at inference time is a deterministic chain of transforms over two representations — an integer sequence (input_ids) and a float tensor (logits/hidden states). The tokenizer is the string-to-int boundary, the model is the int-to-float map, decoding is a float-to-int loop, and the tokenizer runs again backwards for int-to-string. Once you see the four labs as different segments of that one pipeline, the "magic" dissolves into data structures you can trace by hand. This doc traces the mechanisms the three labs implement, at the level of the actual algorithm.

BPE: the merge table is the whole artifact

A trained BPE tokenizer is not code with parameters — it is an ordered list of merge rules plus a vocabulary, and Lab 02 makes you build both. Training is a greedy agglomerative loop:

1. Seed every word as a sequence of base symbols (bytes, in byte-level BPE, which is why there are no unknown tokens — every string is representable).
2. Count the frequency of every adjacent symbol pair across the corpus.
3. Take the most frequent pair, emit a merge rule (a, b) -> ab, and rewrite every occurrence of that adjacency in the corpus.
4. Repeat until the vocabulary reaches its budget.

The output is the merge list in creation order. That ordering is load-bearing: encoding new text replays the merges in the same order they were learned, because a later merge may depend on a symbol only a prior merge could have produced. The invariant Lab 02 tests: encoding is a function of (merge_list, text) and nothing else — deterministic, position-independent, corpus-free at inference time.

Worked trace on the corpus ["low", "low", "lower", "newest", "newest", "newest"]. Start: l o w, l o w, l o w e r, n e w e s t (×3). Pair counts: ("e","s") appears 3 times, ("s","t") 3, ("l","o") 3, ("o","w") 3, ("n","e") 3, ("e","w") 3, ("w","e") 4... The most frequent is ("w","e") at 4 (it occurs in both lower and each newest). Merge it: rule ("w","e") -> we. Re-count on the rewritten corpus, take the new max, emit the next rule. The tie-break matters: when two pairs share the top count, Lab 02 breaks ties lexicographically so the merge table is reproducible and the tests can assert it byte-for-byte. Real BPE trainers break ties by insertion order or frequency-then-order; the point of the deterministic rule is that tokenization must be a pure function, and a nondeterministic trainer would make the model's own vocabulary unstable.

Encoding complexity: a naive "scan for any applicable merge, apply, repeat" is O(n^2) per word in the worst case; production tokenizers keep a priority structure over merge ranks so each word encodes in roughly O(n log n). The mechanism-level reason the naive alternative — a fixed word dictionary with an UNK fallback — fails is coverage: natural text has an unbounded vocabulary (names, code, morphology, typos), so any fixed word list drowns in UNK tokens and the model loses the information. BPE's subword decomposition means a never-before-seen word still lands on frequent pieces the embedding matrix has seen, which is why tokenization splits cleanly into token + ization and stays meaningful.

Padding, the attention mask, and why the side matters

Batching requires rectangular tensors, so short sequences are padded to the batch's longest. The attention mask is a parallel 0/1 tensor — 1 for real tokens, 0 for pad — and inside attention it is added as -inf (or a large negative) to the pre-softmax scores at masked positions, so softmax drives their weight to zero. Drop the mask and pad embeddings get non-zero attention weight: the output degrades silently, no exception, which is the whole danger.

The subtle invariant Lab 02 forces you to respect: decoder-only generation must be left-padded. Generation appends to the end of the sequence. If you right-pad, the "end" is padding, and the model's next-token prediction is conditioned on pad positions — it continues from garbage. Left-pad and every sequence's real final token sits adjacent to the generation frontier, with the mask marking the leading pads as ignorable. This is not a style preference; it is a consequence of where the autoregressive frontier lives.

Offset mapping: the token-to-char bridge

Fast tokenizers return, per token, a (start, end) character span into the original string. This is a genuine data structure carried alongside input_ids, and Labs 01 and 02 both build it because every "map a token-level decision back to the user's text" task depends on it. In Lab 01's NER post-processor, the model emits a label per subtoken; aggregation walks the offsets to stitch ["New", "##Y", "##ork"] back into the single span (0, 7) over the source string. Without offsets you can only report token indices, which are meaningless to a user and unusable for highlighting, redaction, or citation.

generate(): a loop of ordered logit transforms

model.generate() is the mechanism most people hand-wave, and Lab 02 dismantles it into exactly what it is: a loop, one model forward per generated token, where each step transforms the logits in a fixed order and then selects. The order Lab 02 enforces (matching HF's LogitsProcessor chain):

1. Repetition penalty — for every token id already in the sequence, divide its logit by the penalty if the logit is positive, multiply if negative. Both push seen tokens down. Applied first because it operates on raw logits.
2. Temperature — divide all logits by T. T below 1 sharpens toward the top tokens; above 1 flattens; T -> 0 degenerates to argmax. It rescales magnitudes but never reorders the ranking.
3. Top-k — keep the k highest logits, set the rest to -inf. A static cut: k candidates whether the model is confident or lost.
4. Top-p (nucleus) — sort by probability, keep the smallest prefix whose cumulative mass reaches p, mask the rest. Adaptive: a peaked distribution yields few candidates, a flat one many.
5. Softmax then sample (or argmax if do_sample=False). Append the chosen id, check stopping (eos_token_id emitted, max_new_tokens reached), loop.

Worked single step: logits [2.0, 1.0, 0.5, -1.0] for tokens [A, B, C, D], with A already in the sequence and repetition_penalty=1.2. Step 1: A's logit is positive, so 2.0 / 1.2 = 1.67; logits become [1.67, 1.0, 0.5, -1.0]. Step 2 at T=0.8: divide by 0.8 -> [2.08, 1.25, 0.63, -1.25]. Step 3 top_k=3: mask D -> [2.08, 1.25, 0.63, -inf]. Step 4 top_p=0.9: softmax the survivors, keep the nucleus, renormalize. Step 5: sample. Order is not cosmetic — applying temperature after top-k would rescale an already-truncated set, and applying repetition penalty after temperature would penalize on already-scaled values, changing the effective strength. Lab 02's tests pin the order precisely for this reason.

The naive alternative — greedy argmax every step — fails on open-ended text at the mechanism level: argmax is a fixed point of the model's own bias toward high-probability continuations, so it collapses into repetition loops (the the the). Sampling exists to inject the controlled entropy that breaks those loops without wandering into incoherence — which is exactly what the transform stack tunes.

The KV cache: turning quadratic decode into incremental

Attention at position n needs the keys and values of all prior positions. Recomputing them every step makes generating n tokens cost O(n^2) in attention work. The KV cache stores each layer's past keys/values, so step n computes K/V only for the single new token and reuses the rest — each decode step becomes O(n) and the whole generation O(n^2) total instead of O(n^3). Three consequences a mechanism-level understanding gives you: decode is memory-bandwidth-bound (you stream the whole cache per step, not compute-bound), the cache grows linearly in context per layer per head (which is what eats VRAM), and prefill (the whole prompt in one parallel pass) is a different cost regime from decode (one token at a time) — which is why time-to-first-token and inter-token latency are separate metrics. Lab 02's loop is unbatched and uncached for clarity, but it makes the shape explicit: one forward per token is the thing the cache optimizes.

LoRA: the low-rank delta and the merge identity

Lab 03 implements the mechanism that changed adaptation economics. Freeze the pretrained weight W (shape d × k) and add a factored update: W' = W + (alpha/r) · A · B, where A is d × r, B is r × k, and r is far below min(d, k). Only A and B train. For a 4096 × 4096 projection at r = 8, that is ~65K trainable parameters against 16.8M — under 1% — which collapses optimizer state and yields a megabyte adapter instead of a full checkpoint.

Two invariants Lab 03 proves numerically. Zero-init start: one factor (B) is initialized to zero, so A · B = 0 and W' = W exactly at step 0 — training begins at the base model's behavior, not a random perturbation of it. The merge identity: because W'x = Wx + (alpha/r)(A·B)x is just the sum of two linear maps, folding (alpha/r)·A·B into W (merge_and_unload) produces a weight whose forward pass is mathematically identical to the unmerged adapter's — same outputs, zero added latency, at the cost that the adapter is no longer separable. The unmerged form keeps A·B as a second matmul per forward, buying the ability to swap adapters over one shared base. Lab 03 asserts merged_forward(x) == unmerged_forward(x) within floating-point tolerance, which is the concrete proof that LoRA is a constrained parameterization of the update, not a lossy approximation of a full fine-tune. That distinction is the mechanism-level answer to the most common LoRA misconception.