Hitchhiker's Guide — Sampling, Decoding & Constrained Generation

The compressed practitioner tour. If WARMUP.md is the professor, this is the senior who leans over and says "here's what you actually need to remember about the part of generate() nobody shows you."

The 30-second mental model

The model emits logits (one number per token) and nothing else. You turn logits into a token. Softmax (always max-subtracted) makes a distribution; temperature divides logits first (T→0 = greedy, T>1 = wild). The filters chop the unreliable tail before you draw: top-k (fixed count), top-p/nucleus (adaptive mass — the default), min-p (cutoff relative to the max). A repetition penalty stops loops. Order is fixed: penalty → temperature → top-k → top-p → min-p → softmax once → draw. And the killer: a grammar + logit mask makes invalid output impossible — "please return JSON" is a hope, a grammar is a guarantee.

The numbers to tattoo on your arm

Config cheat-sheet (by use case)

extraction / classification / tool-call args   → T=0 (greedy)            deterministic, correct
code completion                                → T≈0.2, top_p=0.95       mostly-best token
chat / general                                 → T≈0.7, top_p=0.9        the default
brainstorming / creative                       → T≈0.9–1.1, top_p=0.95   variety
"stop the repetition"                          → repetition/frequency penalty 1.1–1.3 + top_p
"output MUST be valid JSON"                    → constrained decode (grammar mask), T≈0

Back-of-envelope one-liners

# "Why does T=0 still vary across runs?"
non-deterministic argmax tie-break / float / batching → pin lowest-index tie-break, assert greedy≡sample(T→0)

# "Model loops the same sentence"
greedy/low-T in a self-reinforcing rut → add top_p≈0.9 + repetition/frequency penalty ~1.2

# "JSON is malformed ~3% of the time"
prompting is best-effort → constrain decoding to the schema grammar → 0% by construction

# "Which is more random, top_k=10 or top_p=0.9?"
depends on the step: top_p widens when the model is unsure, top_k can't — top_p is adaptive

The framework one-liners (where these live in real tools)

# Hugging Face — the ordered LogitsProcessor pipeline, your sample() in production
out = model.generate(**ids, do_sample=True, temperature=0.7, top_p=0.9,
                     top_k=50, repetition_penalty=1.2)              # GenerationConfig knobs

# vLLM — same knobs as SamplingParams
from vllm import SamplingParams
SamplingParams(temperature=0.7, top_p=0.9, min_p=0.05, repetition_penalty=1.2)

# Constrained / structured — a GUARANTEE, not a request
import outlines
gen = outlines.generate.json(model, MySchema)          # schema-constrained
# llama.cpp: --grammar-file schema.gbnf
# OpenAI: response_format={"type":"json_schema", "json_schema": {...}}
# Anthropic: tools=[{... input_schema ...}]  (tool args are schema-constrained)

War stories

• "Temperature 0 but the eval isn't reproducible." Two services with "T=0" gave different outputs. Cause: different argmax tie-breaking plus batched matmul nondeterminism. Fix: pin a deterministic greedy (lowest-index tie-break) and treat greedy as the ground truth, not "T=0" in some library. The greedy ≡ sample(T→0) test exists precisely for this.
• The 3% JSON tax. A pipeline "asked" for JSON in the prompt; ~3% came back malformed at a million calls/day → 30k retries/repairs daily, plus a fragile regex repairer. Swapped in schema-constrained decoding (logit mask). Failure rate: 0%, by construction. The repairer got deleted.
• The over-penalized model. Someone cranked repetition_penalty to 1.8 to kill loops; the model then couldn't repeat necessary tokens — it produced broken JSON because it refused to emit a second ". Lesson: penalty is a scalpel (1.1–1.3) plus top-p, not a hammer.
• Beam search for a chatbot. A team used beam search "for quality"; the bot was bland and looped. Beam search optimizes for the most-probable sequence, which is exactly the boring, degenerate text Holtzman warned about. Switched to top-p sampling; instantly more natural.
• Top-k too tight in the long tail. A creative tool used top_k=10; in high-uncertainty spots it cut off perfectly good continuations. Switched to top_p=0.92 (adaptive) and the dead-end feeling vanished.

Vocabulary (rapid-fire)

• Logits — raw per-token scores from the model; the only thing it outputs.
• Softmax — logits → probabilities; subtract the max for stability.
• Temperature — pre-softmax logit divisor; the master randomness dial.
• Greedy — argmax; deterministic, repetitive; the T→0/top_k=1 limit of sampling.
• Top-k / Top-p / Min-p — fixed-count / adaptive-mass / relative-to-max truncation.
• Nucleus — the top-p set: smallest set with cumulative prob ≥ p.
• Repetition / presence / frequency penalty — down-weight already-seen tokens (multiplicative / flat-once / proportional-to-count).
• Logit masking — set forbidden tokens to -inf → prob 0 → unpickable.
• FSM / grammar / GBNF — the legal-sequence automaton a constrained decoder enforces.
• Beam search — keep B best partial sequences; good for translation, bad for chat.
• Speculative decoding — draft-then-verify; 2–3× faster, identical distribution.

Beginner mistakes

• Treating sampling and greedy as different algorithms (greedy is sampling at T→0/top_k=1).
• Forgetting the max-subtraction → overflow/NaN on large logits.
• Re-normalizing (softmax) between filters instead of once at the end.
• Truncating before applying temperature (the knobs then fight).
• Stacking every penalty at high values → the model can't emit needed tokens.
• Using top-k everywhere when top-p (adaptive) is the better default.
• Believing "please return JSON" is reliable — it's best-effort; constrain decoding instead.
• Beam search for open-ended chat (optimizes for boring).

The one thing to take away

Generation behavior is a property of the decoder, not the weights. Master the knob → effect → failure-mode map (and that a grammar is a guarantee), and you'll fix "it's repetitive / random / malformed" in minutes — with a mechanism, not a vibe.