Warmup Guide — Sampling, Decoding & Constrained Generation

Zero-to-senior primer for Phase 08. We start from "the model emits a vector of numbers, and that is all it does," and end with a full decoding engine: the numerically-stable temperature softmax, greedy vs sampling, the truncation filters (top-k, top-p/nucleus, min-p), the repetition family of penalties, the exact order the operations are applied and why, beam search and why it's wrong for open-ended text, a preview of contrastive/speculative decoding (Phase 09), and the one that wins design reviews — constrained/structured generation, where a grammar/FSM and logit masking turn "please return JSON" from a hope into a guarantee. Every term is built from first principles: what it is, why it exists, how it works under the hood, and how it bites in production.

Chapter 1: The Model Emits Logits — You Pick the Token
Chapter 2: Softmax & the Temperature Knob
Chapter 3: Greedy vs Sampling — and Why Greedy Repeats
Chapter 4: The Truncation Filters — Top-k, Top-p/Nucleus, Min-p
Chapter 5: Repetition, Presence & Frequency Penalties
Chapter 6: The Order of Operations — and Why It Matters
Chapter 7: Beam Search, Contrastive & Speculative Decoding
Chapter 8: Constrained & Structured Generation — A Grammar Is a Guarantee
Lab Walkthrough Guidance
Success Criteria
Interview Q&A
References

Chapter 1: The Model Emits Logits — You Pick the Token

From zero. A language model's final layer produces, for the next position, a vector of real numbers — one per vocabulary token. These are the logits: unnormalized scores, in (-∞, +∞), where a larger value means "this token is more likely next." A 32k-vocab model emits a 32,000-long vector at every step. That is the entire output of the network. The model does not "choose a word," "decide to stop," or "return JSON." It returns numbers.

Why this framing matters. Everything a user perceives as model behavior during generation — that it's creative, that it's deterministic, that it loops, that it produced valid JSON, that it called a tool with the right arguments — is produced by the decoding loop you wrap around the model: take logits → turn them into a token → append it → feed the new sequence back in → repeat until a stop token or a length cap. The model is a function sequence → logits; the decoder is the algorithm logits → token. This phase is that algorithm.

The loop, concretely:

prompt ──► [ MODEL ] ──► logits[vocab] ──► [ DECODER ] ──► token id ──► append
              ▲                                                            │
              └──────────────────── feed the new sequence back ───────────┘
                       (stop on the eos token or a max-length cap)

The senior's habit. When someone reports "the model keeps repeating itself" or "the model returns broken JSON sometimes," the senior does not reach for a different model or a bigger prompt. They reach for the decoder: what temperature? what top-p? is there a repetition penalty? is the output constrained by a grammar or just requested in the prompt? Nine times out of ten the fix is a decoding knob, and it costs nothing.

Common misconception. "The model decided to stop / the model chose that word." No. The model assigned a high logit to the <eos> token (or to that word); the decoder turned that logit into an action. Sloppy language here leads to debugging the wrong layer for hours.

Chapter 2: Softmax & the Temperature Knob

What softmax is. To draw a token we need a probability distribution, not raw scores. Softmax maps a logit vector to probabilities that are positive and sum to 1:

$$ p_i = \frac{e^{x_i}}{\sum_j e^{x_j}}. $$

Bigger logits get exponentially more probability; the exponential is what makes a logit gap of "2" turn into a large probability ratio.

The max-subtraction trick (the actual lesson). Computed naively, $e^{x_i}$ overflows the moment a logit is large — $e^{1000}$ is inf, and inf/inf is nan. The fix is to subtract the maximum logit $m = \max_j x_j$ before exponentiating:

$$ p_i = \frac{e^{x_i - m}}{\sum_j e^{x_j - m}}. $$

This is mathematically identical — multiply top and bottom of the naive form by $e^{-m}$ and nothing changes — but now the largest exponent is $e^{0}=1$ and every other is in $(0, 1]$, so nothing overflows. Every softmax you will ever write or trust subtracts the max. A token whose logit is $-\infty$ (a masked token — Chapter 8) maps to $e^{-\infty}=0$: probability exactly zero. That is the whole mechanism behind constrained decoding.

Temperature. Temperature $T$ divides the logits before softmax:

$$ p_i = \frac{e^{x_i / T}}{\sum_j e^{x_j / T}}. $$

$T = 1$: the model's native distribution.
$T < 1$: divides by a small number → widens logit gaps → distribution sharpens toward the argmax → more confident, less diverse.
$T > 1$: shrinks logit gaps → distribution flattens toward uniform → more random.
$T \to 0$: the argmax dominates completely → a one-hot at the top token → greedy. (Dividing by ~0 is numerically dangerous, so in code we special-case $T < \varepsilon$ to return greedy directly — documented in the lab as TEMP_EPSILON.)
$T \to \infty$: every gap vanishes → uniform random.

 prob
  ▲     T=0.5 (sharp)        T=1.0            T=2.0 (flat)
  │      █                    █                █ █
  │      █                    █ ▄              █ █ ▄ ▄
  │      █ ▄                  █ █ ▄            █ █ █ █
  └──────█─█──────tok    ─────█─█─█───tok  ───█─█─█─█──tok
        low entropy        medium            high entropy

The right measure of "how random" is entropy $H = -\sum_i p_i \ln p_i$: lowering T lowers entropy, raising T raises it. The lab asserts exactly this (test_temperature_raises_and_lowers_entropy).

Production significance. Temperature is the single most-tuned generation knob. Code completion and extraction want low T (0–0.3): you want the most-likely, reliable token. Brainstorming and creative writing want higher T (0.7–1.0): you want variety. The temperature parameter in the OpenAI/Anthropic APIs and HF generate() is exactly this division.

Common misconception. "Temperature 0 means deterministic, always." It means greedy, which is deterministic if your argmax tie-break is deterministic. With float ties or a library that breaks ties differently than you expect, "T=0" can still surprise you across runs/implementations — which is why the lab pins a lowest-index tie-break and asserts greedy ≡ sample(T→0).

Chapter 3: Greedy vs Sampling — and Why Greedy Repeats

Greedy decoding. At each step, take the argmax — the single highest-logit token — and ignore the rest. It is deterministic, dirt cheap, and the natural choice when there is one right answer (classification, extraction, "what is 2+2"). In the lab it is greedy(logits), with ties going to the lowest index so the result is reproducible.

Why greedy is repetitive on open-ended text. Greedy is locally optimal and globally myopic. Once the model lands in a high-probability rut — "The cat sat on the mat. The cat sat on the mat." — the most-likely next token continues the rut, because the model has now seen the pattern and predicts more of it. Greedy can never escape because it never explores a slightly lower-probability token that would break the loop. This neural-text-degeneration loop is the single most famous failure mode in generation (Holtzman et al., 2019), and it is not a model defect — it is what argmax does to a self-reinforcing distribution.

Sampling. Instead of always taking the top token, draw one according to its probability: roll $u \sim U[0,1)$, walk the cumulative distribution, and take the first token whose cumulative probability exceeds $u$ (inverse-CDF / roulette-wheel sampling). Now the model can pick the second- or third-likely token sometimes, which is what produces variety and breaks the repetition rut. The cost: it can also pick the 5000th-likely token (pure noise), which is exactly what the truncation filters of Chapter 4 prevent.

probs:   [0.50][0.30][0.15][0.05]      cumulative: 0.50  0.80  0.95  1.00
u=0.62 ──────────────┤                 0.62 falls in the 2nd bucket → token 1

The SOUL invariant. Sampling is a generalization of greedy, not a different algorithm. At $T \to 0$ the distribution becomes a one-hot at the argmax, so sampling can only draw the argmax. With top_k = 1 you keep only the argmax, so again sampling must draw it. Therefore:

$$ \texttt{greedy}(x) ;=; \texttt{sample}(x, T!\to!0) ;=; \texttt{sample}(x, \texttt{top_k}=1). $$

This is the lab's headline test. If your sample does not collapse to your greedy at those limits, your pipeline is wrong somewhere — it is the cleanest correctness check in all of decoding.

Production significance. Determinism (T=0/greedy) is what you want for tests, evals, caching, and tool-call arguments; sampling is what you want for anything a human reads for variety. Knowing that one is the limit of the other is what lets you reason about both with a single mental model.

Common misconception. "Sampling is just noise; greedy is the 'correct' decode." Greedy maximizes the per-step probability, but the highest-probability whole sequence is rarely the best text, and greedy reliably degenerates on long open-ended generations. Sampling with good truncation usually reads better than greedy.

Chapter 4: The Truncation Filters — Top-k, Top-p/Nucleus, Min-p

The problem they solve. Softmax assigns nonzero probability to every token, including thousands of irrelevant ones. With a 32k vocab, the long tail collectively holds enough probability mass that pure sampling will occasionally draw garbage — a random foreign word, a control token, nonsense. Truncation chops the tail off before you draw: keep a trusted set of candidates, set the rest to $-\infty$ (probability 0 after softmax), renormalize, then sample. All three filters in this chapter do that; they differ in how they choose the set.

Top-k. Keep the k highest-probability tokens; mask the rest.

Mechanism: rank tokens by logit, keep the top k, set the others to $-\infty$.
k = 1 is greedy. k = vocab is no truncation.
Failure mode it has: k is a fixed count, but the right number of plausible tokens varies wildly by context. After "The capital of France is" there is one good token; after "I felt" there are hundreds. A fixed k=40 is too loose in the first case and too tight in the second.

Top-p (nucleus sampling). Keep the smallest set of tokens whose cumulative probability is ≥ p; mask the rest. (Holtzman et al., 2019.)

Mechanism: sort tokens by probability descending, walk the cumulative sum, keep tokens until it first reaches p, always keeping at least the top token.
What it fixes: the set size is adaptive. When the model is confident, the nucleus is tiny (one or two tokens); when it is unsure, the nucleus is large. This is why top_p ≈ 0.9 is the senior default for open-ended generation — it tracks the model's own confidence.
Boundary case the lab pins: if p is just below the top token's probability, the nucleus is exactly that one token; reaching p exactly on a cumulative step still includes that token.

probs:  0.64  0.24  0.09  0.03      cumulative: 0.64  0.88  0.97  1.00
top_p=0.8 keeps {0.64, 0.24}        (0.64<0.8, +0.24=0.88≥0.8 → stop)  → 2 tokens
top_p=0.6 keeps {0.64}              (0.64≥0.6 immediately)              → 1 token

Min-p. Keep tokens whose probability is ≥ min_p × (max probability); mask the rest.

Mechanism: find $p_{\max} = \max_i p_i$, set the threshold to $\texttt{min\_p}\cdot p_{\max}$, keep every token at or above it. The argmax is always kept (its prob is $p_{\max}$).
What it fixes: top-p can still admit junk when the distribution is very flat (a large nucleus of similarly-bad tokens). Min-p scales the cutoff to the model's confidence on this step: a peaked distribution gets a high absolute threshold (small set), a flat one gets a low threshold (large set), but always relative to the best token — which empirically keeps quality higher at high temperatures.

Production significance. These are the top_k, top_p, and min_p parameters in HF generate(), llama.cpp, vLLM, and the inference APIs. The defaults you'll see in the wild: top-p 0.9–0.95 (the workhorse), top-k 40–50 (often combined with top-p), min-p 0.05–0.1 (newer, gaining ground). You usually combine top-p with a temperature; you rarely need all three.

Common misconception. "Top-k and top-p are the same idea with different parameters." No — top-k fixes the count, top-p fixes the mass. Their behavior diverges exactly where it matters: in high-uncertainty contexts top-p widens automatically and top-k cannot.

Chapter 5: Repetition, Presence & Frequency Penalties

The problem. Even with sampling and truncation, models drift into repeating phrases and over-using the same words. Penalties fight this by down-weighting tokens the model has already produced, before sampling.

Repetition penalty (the multiplicative / Hugging Face & CTRL convention). For every token id that has already appeared in the generated text, adjust its logit $x$:

$$ x ;\leftarrow; \begin{cases} x / \theta & \text{if } x > 0 \ x \cdot \theta & \text{if } x \le 0 \end{cases} \qquad (\theta \ge 1). $$

The reason for the two-branch rule: a positive logit should shrink toward 0 (divide), and a negative logit should grow more negative (multiply) — both directions lower the post-softmax probability. A naive "subtract a constant" would help positive logits but is awkward for negatives; the divide/multiply rule is sign-correct everywhere. $\theta = 1$ is a no-op; $\theta = 1.1\text{–}1.3$ is the usual range. The lab implements exactly this in repetition_penalty and tests both signs.

Presence penalty (the OpenAI variant). Subtract a flat amount from a token's logit the moment it has appeared at all, regardless of how many times. It encourages introducing new tokens (topic diversity) but doesn't escalate with repetition.

Frequency penalty (the OpenAI variant). Subtract an amount proportional to how many times a token has already appeared. It escalates: the more you've said a word, the harder it is to say again. This is the one that most directly kills runaway loops.

Penalty	Rule	Strength scales with	Where
repetition (HF/CTRL)	`÷θ` if `+`, `×θ` if `−`	once seen, fixed	HF `repetition_penalty`
presence (OpenAI)	subtract flat `α` once seen	once seen, fixed	OpenAI `presence_penalty`
frequency (OpenAI)	subtract `β · count`	number of occurrences	OpenAI `frequency_penalty`

Production significance. A small repetition or frequency penalty is the standard fix for "the model keeps saying the same thing." But over-penalize and you get the opposite failure: the model avoids necessary words (it can't say "the" or repeat a required key in JSON), producing stilted or broken output. The senior move is a light penalty plus good top-p, not a heavy hand.

Common misconception. "Repetition penalty and frequency penalty are the same." They share a goal but differ in mechanism (multiplicative vs additive) and escalation (fixed-once-seen vs proportional-to-count). Naming the right one — and not stacking all of them at high values — is the difference between fixing repetition and breaking fluency.

Chapter 6: The Order of Operations — and Why It Matters

The canonical order. The lab's sample() applies the stages in exactly this sequence, and so do real samplers (HF runs an ordered LogitsProcessorList):

raw logits
   │
   ▼ 1. repetition / presence / frequency penalty   (adjust seen tokens, in logit space)
   ▼ 2. temperature                                 (scale logits; T→0 short-circuits to greedy)
   ▼ 3. top-k                                        (keep k highest)
   ▼ 4. top-p (nucleus)                              (keep cumulative-p set)
   ▼ 5. min-p                                        (keep ≥ min_p·max)
   ▼ 6. softmax  ← runs ONCE, here, at the very end
   ▼ 7. draw with seeded rng                         → next token id

Why penalties come first. They operate on the raw logit scores (the model's unmodified preferences), before temperature warps the scale. Penalizing after temperature would make the penalty's effect depend on T, which nobody wants.

Why temperature comes before truncation. Temperature reshapes the distribution; truncation then selects from the reshaped one. If you truncated first and scaled after, a high temperature could not re-flatten a set you'd already collapsed — the knobs would fight each other.

Why softmax runs once, at the end. Each filter works in logit space (it sets rejected tokens to $-\infty$). If you softmaxed in the middle to apply the next filter and then softmaxed again, you'd be normalizing a distribution that the next filter is about to truncate — wasted work and, worse, an inconsistent distribution that makes the pipeline impossible to reason about. Keep everything as logits; normalize exactly once, right before you draw. This is why the lab's filters all take and return logits, and only sample()'s final step calls softmax.

The T→0 / top_k=1 short-circuit. Because greedy must equal sampling at those limits (Chapter 3), sample() checks for top_k == 1 and T < ε up front and returns greedy(...) — it never tries to divide by ~0 or normalize a one-hot. This is both a numerical safeguard and the thing that makes the SOUL invariant hold by construction.

Production significance. "In what order are sampling operations applied?" is a direct senior interview question. The deeper point — filters live in logit space, softmax runs once — is what separates someone who used the knobs from someone who understands the pipeline.

Common misconception. "Apply softmax, then top-p on the probabilities, then softmax again to re-sample." You can compute top-p's membership from probabilities (you need them to walk the cumulative sum), but you apply the truncation by masking logits and softmax once at the end. Re-normalizing repeatedly is a classic off-by-a-distribution bug.

Chapter 7: Beam Search, Contrastive & Speculative Decoding

Beam search. Instead of committing to one token per step, keep the B highest-probability partial sequences ("beams") alive at once, expand each by every possible next token, and keep the best B of the expansions, scored by total (log-)probability. At the end, return the highest-scoring complete sequence. Beam search approximates "find the most probable whole sequence," which greedy (most probable next token) does not.

Where it's good: tasks with a single correct, high-probability target — machine translation, summarization to a tight spec, speech transcription. The right answer really is (close to) the most probable sequence, so searching for it helps.
Why it's bad for open-ended generation: the most-probable sequence is boring and repetitive. Holtzman et al. showed beam search on open-ended text produces bland, degenerate, looping output — exactly the neural-text-degeneration problem from Chapter 3, made worse by optimizing harder for high probability. Human language is not the maximum-likelihood sequence; it has the surprising, lower-probability turns that beam search prunes away. For chat and creative text, sampling with top-p beats beam search, which is why almost no chat model uses beams.

Contrastive search (preview). Picks the next token by balancing model probability against a penalty for being too similar to the already-generated context (a "degeneration penalty"). It aims for fluent-but-diverse output without sampling randomness — a deterministic middle ground.

Speculative decoding (preview, deep in Phase 09). A latency optimization, not a quality one. A small, fast draft model proposes several tokens ahead; the big target model verifies them all in one parallel forward pass and accepts the longest correct prefix. Because verification of k tokens costs about one decode step (it's a single batched pass), you get multiple tokens per expensive step — a 2–3× speedup — and, crucially, the output distribution is provably identical to sampling from the target model alone. The mechanism (why decode is memory-bandwidth-bound, so verifying many tokens at once is nearly free) is the heart of Phase 09's serving math.

Common misconception. "Beam search is the 'best' decoding because it searches more." It searches for highest probability, which is the wrong objective for open-ended language. Best decoding is task-dependent: beams for translation, top-p sampling for chat, greedy/T=0 for extraction and tool arguments.

Chapter 8: Constrained & Structured Generation — A Grammar Is a Guarantee

The problem. You need the model's output to be a valid structure — JSON matching a schema, a function call with the right argument names, an SQL statement, a date \d{4}-\d{2}-\d{2}. The naive approach is to ask: "Please respond with valid JSON." This is best-effort. The model usually complies, but at scale it will, some fraction of the time, emit a trailing comma, a missing brace, prose before the JSON, a hallucinated field. At 3% failure and a million requests a day, that's 30,000 broken responses you have to detect, retry, or repair.

The mechanism: logit masking. Constrained decoding makes invalid output impossible instead of merely unlikely. At each step, you compute the set of tokens that would be legal next given the structure so far, and set the logit of every illegal token to $-\infty$. After softmax those tokens have probability exactly 0 — the sampler cannot pick them, no matter what the model preferred. You are not asking the model to behave; you are removing its ability to misbehave.

state: just emitted '{'  → legal next tokens (by grammar): only  '"ok"'
logits:  '{'  '"ok"'  ':'  'true'  'false'  '}'
mask:   -inf    keep  -inf  -inf    -inf    -inf      ← only the grammar-legal token survives
softmax → P('"ok"') = 1.0 ; everything else 0          ← invalid is unrepresentable

Grammars and finite-state machines. The set of legal sequences is a formal language, and we track "where we are" in that language with a finite-state machine (FSM): states, transitions labelled by tokens, and a set of accepting (valid-ending) states. At any state, the outgoing edges are exactly the legal next tokens — so building the mask is "look up the current state's edges." The lab's FSM does precisely this, and constrained_logits_mask(fsm, state, token_to_str) returns the set of legal token IDs. Regular expressions compile to FSMs (DFAs); a JSON schema compiles (roughly) to a grammar that compiles to an automaton over the tokenizer's vocabulary — this is what Outlines does, building, ahead of time, a map from (state) → allowed-token mask so masking is O(1) per step instead of O(vocab).

The lab's concrete grammar. A tiny fixed-format object {"ok": <true|false>}, emitted as the token sequence { "ok" : (true|false) }. The FSM forces every structural token and offers a real choice only at the boolean value — so the model's logits matter exactly where the schema permits freedom, and nowhere else. The CONSTRAINED SOUL TEST loads the logits to scream for an invalid token (a huge logit on }) and proves that, for every seed, the decoder still only ever emits one of the two valid strings, and fsm.accepts(output) is always True. That is the whole point: correctness by construction, not by prompt.

Why "please return JSON" is best-effort but a grammar is a guarantee. A prompt biases the distribution; a mask zeroes the forbidden region. No amount of prompt engineering can make invalid output literally impossible, because the model can always assign some probability to a wrong token. Masking removes that probability. This is the foundation of reliable tool/function calling (Phase 12): the arguments to a function must match a schema, so production tool-calling pipelines constrain decoding to the argument grammar rather than hoping the model formats it correctly.

The hard part (the limit of the miniature). Real tokenizers are subword: a single grammar terminal like true might be one token, or split as tr + ue, and a single token might straddle two terminals. A production constrained decoder (Outlines, llama.cpp GBNF, guidance) must track partial matches across the tokenizer's actual vocabulary — building, for each automaton state, the exact set of vocabulary tokens whose string is a valid continuation. That precomputation is the genuine engineering; the idea is exactly the lab's FSM-plus-mask.

The libraries (what you'll actually use):

Outlines — regex- and JSON-schema-constrained generation; compiles the schema to a finite-state index over the vocab and masks logits. The reference open-source implementation.
guidance — interleaves program control flow with constrained generation (grammars, regex, selects) so you script the structure and let the model fill the holes.
llama.cpp GBNF grammars — a BNF grammar file that the llama.cpp sampler enforces by masking.
OpenAI structured outputs / response_format json_schema, Anthropic tool use — the hosted-API form: you supply a schema, the provider constrains decoding to it and returns guaranteed-valid structured output.

Common misconception. "Constrained decoding hurts quality / makes the model dumb." It removes invalid options, not good ones — within the grammar the model's preferences still drive the choice (the boolean branch in the lab is decided by the logits). It changes what's representable, not how the model thinks, and for structured tasks it strictly improves reliability.

Lab Walkthrough Guidance

The lab (lab-01-decoding-engine) turns these chapters into code. Suggested order (matches the file top-to-bottom):

softmax / apply_temperature — Chapter 2. The only trick is the max-subtraction and the T→0 greedy special-case. Get these right and the entropy test follows for free.
greedy — Chapter 3. Argmax with a lowest-index tie-break. Tiny, but the SOUL test depends on its determinism.
top_k_filter / top_p_filter / min_p_filter — Chapter 4. All return logits with rejects set to NEG_INF. Top-p must "always keep at least the top token"; min-p's threshold is min_p · max(probs).
repetition_penalty — Chapter 5. The divide-positive / multiply-negative rule; test both signs.
_sample_from_probs / sample — Chapters 3 & 6. The inverse-CDF draw, then the full ordered pipeline. Mind the top_k==1 and T<ε short-circuits — they are what make the SOUL TEST pass. (Note the repetition_penalty parameter shadows the function; the solution calls it via globals()[...].)
FSM + constrained_logits_mask + constrained_decode + build_bool_json_fsm — Chapter 8. The mask is "which token ids have a string that's a legal edge from this state"; the decoder masks, samples, and steps the FSM. The CONSTRAINED SOUL TEST is the soul of this half: invalid output is impossible for every seed.

Run red → green: pytest test_lab.py -v, then LAB_MODULE=solution pytest test_lab.py -v, then python solution.py for the worked output.

Success Criteria

All tests pass against your lab.py and against LAB_MODULE=solution.
You can write a stable softmax from memory and explain why subtracting the max changes nothing.
You can state what each of T<1, T>1, T→0, T→∞ does to entropy, and prove greedy ≡ sample(T→0) ≡ sample(top_k=1) (the SOUL invariant).
You can explain top-k vs top-p vs min-p — the rule each uses and the failure mode each fixes — and name the senior default (top-p ≈ 0.9 with a temperature).
You can state the canonical filter order and explain why penalties go first, temperature before truncation, and softmax exactly once at the end.
You can explain why beam search is right for translation and wrong for chat.
You can explain logit masking and why a grammar guarantees validity where a prompt only requests it — and how the CONSTRAINED SOUL TEST proves it for every seed.

Interview Q&A

"Why must softmax subtract the max?" — to prevent overflow of $e^{x}$ for large logits; it is mathematically identical (multiply num/denom by $e^{-m}$), so it costs nothing and is always done. A $-\infty$ logit cleanly becomes probability 0, which is the basis of masking.
"What does temperature do, mechanically?" — divides logits before softmax; T<1 sharpens (lower entropy), T>1 flattens (higher entropy), T→0 is greedy, T→∞ is uniform.
"Greedy vs sampling — when each?" — greedy/T=0 for deterministic, single-answer tasks (extract, classify, tool args, tests); sampling for variety. Greedy degenerates into repetition on open-ended text because it can't escape a self-reinforcing rut.
"Top-k vs top-p vs min-p?" — top-k keeps a fixed count; top-p keeps the smallest set with cumulative mass ≥ p (adaptive — the workhorse); min-p keeps tokens ≥ min_p·max_prob (cutoff scales with confidence). Top-p adapts to uncertainty where top-k can't.
"In what order are sampling operations applied, and why?" — penalties → temperature → top-k → top-p → min-p → softmax → draw. Penalties on raw logits; temperature before truncation so the two don't fight; softmax once, at the end, because filters live in logit space.
"How do you guarantee valid JSON?" — not by prompting; by constrained decoding: compile the schema to a grammar/FSM and mask every illegal next token to $-\infty$, so invalid output is unrepresentable. Prompting is best-effort; masking is correct by construction. Use Outlines / guidance / llama.cpp grammars / the provider's structured-output mode.
"Why is beam search bad for chat?" — it optimizes for the highest-probability sequence, which for open-ended language is bland and repetitive (Holtzman). It's right for translation, wrong for creative/chat text where sampling+top-p wins.
"Temperature is 0 but output still varies across runs — why?" — non-deterministic argmax tie-breaking (or float/library differences), batching/parallel reductions, or a different greedy path. Pin the tie-break and assert greedy ≡ sample(T→0).
"What's the difference between repetition, presence, and frequency penalty?" — repetition is multiplicative (÷ positive, × negative) and fixed once seen; presence subtracts a flat amount once seen; frequency subtracts proportional to count (the one that escalates against loops).
"What is speculative decoding and does it change the output?" — a small draft model proposes tokens, the big model verifies them in one parallel pass; a latency win (2–3×) with a provably identical output distribution. It's a serving optimization (Phase 09), not a quality knob.

References

Holtzman et al., The Curious Case of Neural Text Degeneration (2019) — nucleus (top-p) sampling and why beam search / greedy degenerate on open-ended text. The foundational paper of this phase.
Fan, Lewis, Dauphin, Hierarchical Neural Story Generation (2018) — top-k sampling.
Nguyen et al., Min-p Sampling (2024) — the relative-to-max cutoff and its high-temperature quality gains.
Keskar et al., CTRL: A Conditional Transformer Language Model (2019) — the multiplicative repetition penalty convention HF adopted.
Willard & Louf, Efficient Guided Generation for Large Language Models (2023) — the Outlines finite-state / regex-constrained decoding method.
Outlines, guidance, and llama.cpp GBNF docs — the production constrained-generation libraries.
OpenAI Structured Outputs and Anthropic tool-use docs — hosted schema-constrained generation.
Leviathan et al., Fast Inference from Transformers via Speculative Decoding (2023), and Chen et al., Accelerating LLM Decoding with Speculative Sampling (2023) — the speculative-decoding preview (built in Phase 09).
Hugging Face generation_strategies and LogitsProcessor docs — the real sampler whose ordered pipeline this lab mirrors.

Senior AI Engineer — LLMs, VLMs & Agentic Systems