Cheatsheet 07 — Inference Parameters
The knobs
| Param | Range | Effect | Default to use |
|---|---|---|---|
temperature | 0–2 | Randomness. 0 = deterministic/greedy; ↑ = more diverse/creative | 0 for extraction/classification/code; 0.7 for creative |
top_p (nucleus) | 0–1 | Sample from smallest token set summing to p | 1.0 (or 0.9 with temp) |
top_k | int | Sample from k most likely tokens | off / large |
max_tokens | int | Cap output length | Set it! (cost + runaway protection) |
stop | strings | End generation at these strings | Task-specific delimiters |
frequency_penalty | -2–2 | Penalize repeated tokens | 0 (raise to reduce loops) |
presence_penalty | -2–2 | Encourage new topics | 0 |
seed | int | Reproducible sampling (where supported) | Set for evals/tests |
n | int | Number of completions | 1 (n>1 multiplies cost) |
logprobs | bool/int | Return token probabilities | For confidence/analysis |
Recipes by task
| Task | Settings |
|---|---|
| Extraction / classification / structured output | temperature=0, set max_tokens, schema/stop |
| Code generation | temperature=0–0.2 |
| Factual Q&A / RAG | temperature=0, low; rely on grounding |
| Brainstorming / creative writing | temperature=0.7–1.0, top_p=0.9 |
| Reproducible evals | temperature=0, seed set |
Gotchas
- Set
temperatureANDtop_pcarefully — adjusting both compounds; usually tune one. temperature=0≠ fully deterministic across runs/providers (hardware/batching nondeterminism); useseed+ same provider for tests.- Always set
max_tokens— uncapped output = runaway cost and latency. - Reasoning models ignore some sampling params and bill hidden reasoning tokens.
- n>1 / logprobs raise cost — use deliberately.
Interview line
"For anything that needs correctness — extraction, classification, code, RAG — I use temperature 0 and cap max_tokens. I only raise temperature for genuinely creative tasks, and I set a seed for reproducible evals."