Phase 08 — Sampling, Decoding & Constrained Generation

The phase where you stop saying model.generate() and start building the function that turns logits into the next token. Every property people attribute to "the model" — creativity, determinism, repetition loops, valid JSON, function-call arguments — is actually a property of the decoding loop wrapped around it. The same weights are boring or wild, reliable or degenerate, depending on a handful of knobs you are about to implement from scratch. This is the layer that decides whether your product works.

Why this phase exists

A trained model does exactly one thing: it emits a vector of logits — one real number per vocabulary token — for the next position. That's it. It does not "choose a word." The choosing is a separate algorithm you control, and almost every production incident in generation lives there, not in the weights:

1. Softmax + temperature — logits become a probability distribution, and temperature is the single dial between "deterministic and repetitive" and "creative and incoherent."
2. The truncation filters (top-k, top-p, min-p) — the model assigns nonzero probability to every token, including garbage; these chop off the unreliable tail. Each one fixes a different failure mode of the others.
3. Repetition / presence / frequency penalties — the reason your model stops saying the same sentence forever.
4. The order of operations — penalty → temperature → top-k → top-p → min-p → softmax → draw. Get the order wrong and the knobs interact in ways that make sampling impossible to reason about. This is a favorite interview probe.
5. Constrained / structured generation — the difference between asking for JSON and guaranteeing it. A grammar/FSM + logit masking makes invalid output literally unrepresentable. This is the machinery under tool calling, function calling, and every "structured output" API.

Get fluent here and you can debug "the model won't stop repeating," "the JSON is malformed 3% of the time," and "why is temperature 0 still random?" — in minutes, with a mechanism, not a vibe.

Concept map

                      ┌─────────────────────────────────────────┐
                      │  Model emits LOGITS (one per token)      │
                      │  — it does NOT pick a token. You do.     │
                      └─────────────────────────────────────────┘
                                       │
                 ┌─────────────────────┴──────────────────────┐
                 ▼                                             ▼
        repetition penalty                           constrained / structured
        (÷ positive, × negative seen)                generation (a GUARANTEE)
                 │                                             │
                 ▼                                    FSM / regex / JSON-schema
            temperature  (T→0 = greedy)                       │
                 │                                       logit MASK → -inf
       ┌─────────┼──────────┐                          on every illegal token
       ▼         ▼          ▼                                 │
     top-k     top-p      min-p                               ▼
   (fixed)  (nucleus,   (relative to            "please return JSON" = best-effort
            adaptive)    max prob)               grammar = invalid is impossible
       └─────────┼──────────┘                                 │
                 ▼                                             │
            SOFTMAX (max-subtracted, once)  ◄─────────────────┘
                 │
                 ▼
           draw with seeded rng  →  next token id
                 │
       greedy ≡ sample(T→0) ≡ sample(top_k=1)   ← THE SOUL

The lab

The lab is a runnable, test-verified miniature — see the lab standard. Run it red (pytest test_lab.py), make it green, then read solution.py. All randomness flows through a seeded random.Random, so every test is reproducible.

Integrated scenario ideas

• Debug a repetition loop: a summarizer keeps repeating a sentence. Show how greedy / low-temperature causes it, and how repetition penalty + top-p fix it — with the entropy numbers.
• "Temperature 0 but still random": explain why a naive softmax-then-sample at T≈0 can still diverge across runs (float ties, library greedy != your greedy), and why the SOUL invariant (greedy ≡ sample(T→0) ≡ top_k=1) is the thing to assert.
• Make JSON a guarantee, not a hope: take a flaky "return JSON" prompt and replace it with a grammar-constrained decode; show the failure rate goes from ~3% to 0% by construction.
• Pick a sampling config for a use case: code completion (low T, top-p), brainstorming (higher T, top-p), function-call arguments (grammar-constrained, T≈0) — and defend each with the mechanism.

Deliverables checklist

• lab.py passes pytest test_lab.py -v (and LAB_MODULE=solution does too).
• You can derive the max-subtraction trick and prove it leaves softmax unchanged.
• You can state the canonical filter order and explain why softmax runs once, at the end.
• You can explain top-k vs top-p vs min-p — what each is, and the failure mode it fixes.
• You can explain why greedy is sample at the T→0 / top_k=1 limit (the SOUL invariant).
• You can explain logit masking and why a grammar is a guarantee where a prompt is a hope.
• You can turn any "the generation is wrong/repetitive/malformed" report into a named knob.

Key takeaways

• The model emits logits; the decoder picks the token. Creativity, determinism, repetition, and validity are properties of the decoding loop, not the weights — which is why this is its own phase and why most "the model is bad" tickets are really "the sampling is misconfigured."
• Temperature is the master dial, and the filters are tail-control. Temperature reshapes the whole distribution; top-k/top-p/min-p truncate the unreliable tail before you draw. Top-p is adaptive (the senior default); min-p scales the cutoff to the model's confidence.
• Order matters and softmax runs once. Penalty → temperature → top-k → top-p → min-p → softmax. Normalizing in the middle would make every later filter operate on a stale distribution.
• A grammar is a guarantee; a prompt is a hope. Logit masking sets every illegal token to -inf, so the engine cannot emit invalid output — the foundation of reliable tool/function calling (P12) and every "structured output" feature. "please return JSON" is best-effort; constrained decoding is correct by construction.

Next: Phase 09 — Inference & Serving Internals.