Inference Parameters — Sampling and Generation Control

Phase 1 · Document 03 · LLM Vocabulary and Mental Models Prev: 02 — Parameters, Weights, and Checkpoints · Next: 04 — Model Capabilities

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Inference parameters are the knobs exposed on every LLM API and local server — the dials you turn daily. Getting them wrong produces the most common production failures: hallucinations (temperature too high for extraction), repetitive/boring output (temperature too low for creative work), truncated responses (max_tokens too low), runaway cost (max_tokens too high or unset), and broken automation (wrong settings for structured output). None of these are "the model is bad" — they're configuration. Mastering this section eliminates a whole class of incidents and lets you tune behavior without reaching for a bigger, more expensive model.

2. Core Concept

Plain-English primer (logits → probabilities → pick, from zero)

Every sampling parameter here is just a tweak to one tiny pipeline. Understand the pipeline and the parameters become obvious.

1. Logits — when the model finishes a step, it outputs one raw score (a "logit") for every token in its vocabulary. Bigger score = "more likely the next token." These are unbounded raw numbers, e.g. cat: 3.1, dog: 2.0, banana: -1.0.
2. Softmax → a probability distribution — softmax turns those raw scores into probabilities that sum to 1, so they're choosable. (Same softmax from 2.00.)
3. Sample → pick one token — choose a token according to those probabilities (or just take the most likely one).

Temperature is literally "divide the logits before softmax." That's the whole mechanism:

import math
def softmax(scores):
    m=max(scores); e=[math.exp(s-m) for s in scores]; t=sum(e); return [x/t for x in e]

logits = {"cat": 3.1, "dog": 2.0, "banana": -1.0}
names, vals = list(logits), list(logits.values())

for temp in (0.5, 1.0, 2.0):
    probs = softmax([v/temp for v in vals])     # ← temperature divides the logits
    print(temp, {n: round(p,2) for n,p in zip(names, probs)})
# 0.5 → {'cat': 0.89, 'dog': 0.11, 'banana': 0.0}   low temp = SHARP (picks the top, predictable)
# 1.0 → {'cat': 0.73, 'dog': 0.24, 'banana': 0.03}  default
# 2.0 → {'cat': 0.55, 'dog': 0.32, 'banana': 0.13}  high temp = FLAT (more random/creative)

So low temperature = sharper distribution = more deterministic; high temperature = flatter = more random. It changes how the dice are weighted, never the model's underlying knowledge — that's why "raise the temperature to make it smarter" is a myth.

The other dials just prune which tokens are even allowed before sampling: top-k keeps the K highest, top-p keeps the smallest set whose probabilities add up to p, min-p keeps tokens above a fraction of the top one. max_tokens caps how many tokens are generated; stop sequences end generation early; seed fixes the dice for reproducibility. Everything below is detail on these.

Sampling parameters all act on the logits (raw per-token scores) just before a token is chosen. Understanding where each acts makes them easy to reason about.

temperature

Divides the logits before softmax, sharpening or flattening the distribution.

• 0 (or near-zero): always pick the top token (greedy) — deterministic, factual, can repeat.
• 1.0: sample from the distribution as trained.
• >1.0: flatten it, raising the odds of unlikely tokens — more random/creative, eventually incoherent.

It changes diversity, not capability. A wrong answer at temperature=0 is still wrong.

0.0 deterministic · 0.2 mostly stable · 0.7 balanced · 1.0 full sampling · 1.5 chaotic

top_p (nucleus sampling)

Sort tokens by probability, keep the smallest set whose cumulative probability ≥ p, sample from that "nucleus." top_p=0.9 keeps the most probable mass and discards the long tail; 1.0 disables it. Use temperature or top_p as your primary dial, not both at extremes.

top_k

Hard cap: only the k highest-probability tokens are candidates. top_k=1 = greedy. Common in local inference (llama.cpp, Ollama); ~40 is a typical default.

min_p

Dynamic floor: exclude tokens below min_p × p(top_token). If the top token is very likely, the floor rises and the pool tightens; if the model is unsure, it widens. More adaptive than top_k; increasingly a local-inference default.

max_tokens / max_new_tokens

Hard cap on output tokens. Too low → truncated answers. Unset/too high → runaway generation and cost. Always set explicitly in production.

# risky: no cap — model generates until its EOS token or the context limit
client.chat.completions.create(model="...", messages=[...])
# safe:
client.chat.completions.create(model="...", messages=[...], max_tokens=500)

stop sequences

Strings that immediately halt generation (and are not included in output). Used for chat turn boundaries, structured-output delimiters, end-of-function markers: stop=["</answer>", "\n\nUser:", "###"].

seed

Sets the sampling RNG seed for reproducibility. Same seed + same input → same output in principle — but not all providers guarantee it, and batching can still introduce variance. For hard determinism prefer temperature=0.

logprobs

Returns log-probabilities of generated tokens (and optionally top alternates). Useful for calibration, uncertainty, reranking candidates, and flagging low-confidence answers. Not universally supported.

presence_penalty / frequency_penalty

Both subtract from a token's logit to reduce repetition. presence_penalty (0–2) penalizes any token that already appeared (topic diversity); frequency_penalty (0–2) penalizes proportional to how often it appeared (exact-repeat suppression). Above ~1.0 the model starts avoiding useful words; start at 0.1–0.3. Local stacks expose repetition_penalty as a multiplier (1.0 none, 1.1 slight, 1.3 strong).

stream

Returns tokens as Server-Sent-Event deltas as they're produced. Improves perceived latency (first token sooner) but does not reduce total generation time. Adds proxy complexity (partial chunks, disconnects, mid-stream errors).

response_format / JSON mode / structured output

Constrains output structure via grammar-constrained decoding (guaranteed valid JSON), schema-validated structured output, or soft post-processing+retry. Validate regardless — valid JSON syntax ≠ correct schema/semantics.

tool_choice

Controls tool use: "auto" (model decides), "none" (never), "required" (must call something), or a specific named function.

reasoning_effort / thinking budget

On reasoning models (OpenAI o-series, Claude extended thinking, etc.) controls how many hidden reasoning tokens the model spends. Higher → better on hard tasks, but slower and billed (often at output rates), with the thinking content frequently withheld. Not every task benefits — route simple tasks to cheaper, non-reasoning paths.

3. Mental Model

                         logits (raw scores)
                              │
        temperature ──────────┤  sharpen / flatten the whole distribution
        presence/frequency ───┤  subtract from repeated-token logits
                              ▼
              ┌─────────── candidate pool ───────────┐
   top_k ─────┤ keep K highest                       │
   top_p ─────┤ keep cumulative-prob nucleus         │  pick ONE token
   min_p ─────┤ keep ≥ min_p × p(top)                │
              └──────────────────────────────────────┘
                              │
        seed → reproducibility · max_tokens → length/cost ceiling
        stop → format boundary · stream → perceived latency
        response_format → structure · tool_choice → tool use · reasoning_effort → depth

Rule: temperature/penalties reshape the distribution; top_k/top_p/min_p prune the pool; the rest are control/output knobs.

4. Hitchhiker's Guide

What to set first, always: max_tokens (cost ceiling) and temperature (task-appropriate). These two prevent most incidents.

What to ignore at first: min_p, logprobs, exotic penalty tuning — reach for them only when a specific symptom appears.

What misleads beginners: thinking high temperature = smarter; thinking temperature=0 is bit-identical; thinking JSON mode guarantees a correct schema; tuning top_p and temperature simultaneously at extremes (they fight).

How experts reason: they pick one primary randomness dial (usually temperature), set it by task class (deterministic for extraction/code, higher for creative), always cap max_tokens, and reach for penalties/min_p only to fix an observed problem — then they measure the effect rather than guessing.

What matters in production: explicit max_tokens, deterministic settings for evals/tests (so results are comparable), validation after structured output, and budgeting reasoning tokens.

Debug/verify: reproduce with fixed seed + temperature=0; log the exact parameters with each request so you can correlate behavior changes to config changes.

Questions to ask providers: Is seed honored? Does temperature=0 mean greedy? Is structured output grammar-constrained or best-effort? Are reasoning tokens billed and at what rate? Which sampling params are ignored for this model?

What silently gets expensive/unreliable: unset max_tokens; reasoning_effort=high on simple tasks; high penalties degrading coherence; assuming determinism that the provider doesn't guarantee.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• temperature changes randomness, not capability — it cannot fix a wrong answer.
• Always set max_tokens in production; unset means "until EOS or the context limit," an open-ended cost.
• temperature=0 is near-deterministic; for guaranteed repeatability also pin seed and verify the provider honors it.
• JSON mode guarantees valid JSON syntax, not a correct schema — always validate.
• Output (and reasoning) tokens are typically 2–5× the price of input tokens, so max_tokens and reasoning_effort are direct cost levers.
• top_k/min_p are mostly local-inference controls; commercial APIs lean on temperature/top_p.
• Penalties above ~1.0 often make the model avoid useful words and degrade quality.
• Streaming improves perceived latency but not total generation time.

9. Observations from Real Systems

• OpenAI / Anthropic APIs expose temperature, top_p, max_tokens, stop, seed, and structured output; some models ignore or restrict temperature (notably reasoning models).
• llama.cpp / Ollama expose the full local kit — top_k, min_p, repeat_penalty, num_predict (their max_tokens) — via request fields or Modelfile defaults.
• vLLM accepts OpenAI-compatible sampling params and adds guided-decoding (grammar/JSON-schema constrained) for reliable structured output.
• OpenRouter passes sampling params through to the underlying provider; unsupported params are silently dropped — a common "why did my setting do nothing?" surprise.
• Cursor / coding assistants run near temperature=0 for edits (determinism, fewer diffs) and lift it only for brainstorming-style features.

10. Common Misconceptions

11. Engineering Decision Framework

Production defaults by use case:

Choosing settings:
  Deterministic task (extract/classify/code)? → temperature 0–0.2, structured output, validate.
  Creative task?                              → temperature 0.7–1.0, consider multiple candidates.
  Output looping/repeating?                   → add frequency_penalty 0.1–0.3 (then min_p locally).
  Hard reasoning task?                        → enable reasoning_effort; otherwise leave OFF (cost).
  Need repeatable eval?                       → temperature 0 + fixed seed; verify provider honors seed.
  Always:                                     → set max_tokens; stream for user-facing UIs.

12. Hands-On Lab

Goal

Build a parameter playground that demonstrates how each setting changes output, cost, and determinism.

Prerequisites

• Python 3.10+, an OpenAI-compatible API key (adaptable to any provider).

Setup

pip install openai rich
export OPENAI_API_KEY=sk-...

Steps

from openai import OpenAI
from rich.console import Console

client, console = OpenAI(), Console()
prompt = "Write the opening line of a mystery novel."

for temp in (0.0, 0.5, 1.0, 1.5):
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=temp, max_tokens=80,
    )
    console.print(f"\n[bold]temp={temp}[/bold]: {r.choices[0].message.content}")

# Determinism check
outs = [client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": "What is 2+2?"}],
            temperature=0, max_tokens=10).choices[0].message.content
        for _ in range(3)]
console.print(f"\nAll identical at temp=0: {len(set(outs)) == 1}")

Expected output

• temp=0 terse/repeatable; temp=1.5 varied, sometimes incoherent.
• Determinism check usually True (note any drift).

Debugging tips

• Setting changed nothing? The model/provider may ignore it (common for reasoning models / via OpenRouter).
• Truncated output? max_tokens too low — raise it or add a stop sequence.

Extension task

1. Structured output: extract {name, date, amount} from an invoice string; validate against a schema.
2. Stop sequences: generate a function and stop at \n}.
3. Streaming: measure TTFT vs total time.
4. Penalties: compare frequency_penalty 0 vs 1.5 on a 500-word generation.

Production extension

Wrap the call so every request logs its parameters + token usage + latency; build a tiny report correlating temperature to output-length variance and cost.

What to measure

Output variance by temperature; determinism at temp=0; cost delta from max_tokens; TTFT with streaming.

Deliverables

• 4-temperature output comparison.
• Determinism finding.
• A validated structured-extraction example.

13. Verification Questions

Basic

1. What temperature would you use for a classification task, and why?
2. What is the risk of leaving max_tokens unset?
3. Why might temperature=0 not be perfectly reproducible?

Applied 4. Output is repetitive. Which two parameters help, and how do you avoid over-penalizing? 5. Distinguish JSON mode from schema-constrained structured output — what must you still do in both?

Debugging 6. You set top_k=20 against a commercial API and nothing changes. Why might that be? 7. A reasoning model ignores your temperature. Is that a bug? Explain.

System design 8. Design the sampling config for an agent that must reliably call tools and never loop forever. Cover temperature, tool_choice, stop, max_tokens, and step limits.

Startup / product 9. Your costs spiked after enabling a "smarter" reasoning mode for all requests. Explain the likely cause and a routing strategy that preserves quality where it matters while controlling spend.

14. Takeaways

1. Temperature is the primary dial: low for facts/code, high for creativity — it changes diversity, not skill.
2. Always set max_tokens; it's both a correctness and a cost control.
3. top_k/min_p are mostly local; temperature/top_p dominate commercial APIs — use one primary randomness dial.
4. Structured output guarantees syntax, not semantics — validate every time.
5. Reasoning effort is a cost/quality trade-off — route, don't blanket-enable.
6. For evals/tests, pin temperature=0 + seed and confirm the provider honors them.

15. Artifact Checklist

• Code: parameter playground with a logging wrapper.
• Benchmark result: temperature-vs-variance and cost table.
• Notes: the "reshape vs prune vs control" mental model.
• Config reference: your production defaults-by-use-case table, adapted to your stack.
• Validated example: one structured-extraction with schema validation.
• Determinism report: seed/temperature behavior for your chosen provider.

Next: 04 — Model Capabilities