Brother Talk — Sampling, Decoding & Constrained Generation

Off the record. The stuff I'd tell you over a beer, not in a design review.

This is the phase where you realize half of "the model is bad" is actually "the sampling is misconfigured." People file tickets blaming the model — "it repeats itself," "it's too random," "it ignored my JSON instruction" — and reach for a bigger model or a longer prompt. You'll be the one who quietly changes temperature, adds a top_p, or wraps the call in a grammar, and the problem disappears for free. Once you've done that a few times, you stop seeing a magic box and start seeing a logit vector with a loop around it. That shift is the whole point.

The "temperature 0 isn't deterministic" bug will find you. Somebody will swear their eval is reproducible because they set temperature=0, and it won't be. It'll be argmax tie-breaking, or batched-matmul nondeterminism, or two libraries disagreeing about what "greedy" means. The fix is boring and senior: pin your own deterministic greedy and assert greedy ≡ sample(T→0). The reason that test is the soul of the lab is that it's the soul of every real reproducibility incident.

Constrained decoding is the cheat code nobody outside this niche knows about. Most engineers genuinely believe "please return JSON" is how you get JSON, and they build retry loops and regex repairers around the 3% that comes back broken. When you say "just constrain the decoder to the schema — invalid output becomes impossible, not just unlikely," it lands like a magic trick. It isn't magic; it's a logit mask. But it deletes a whole category of bugs, and the people who don't know it will treat you like you have a superpower. Learn Outlines / GBNF / structured-outputs and reach for them by reflex.

Resist the urge to crank every knob. There's a temptation to stack high temperature + top-k + top-p + min-p + a fat repetition penalty and call it "tuned." That way lies a model that can't say "the" twice or close a JSON brace. The senior config is embarrassingly simple: a temperature, a top_p≈0.9, and maybe a light penalty if you actually see loops. Simple and explainable beats a pile of interacting dials you can't reason about at 2 a.m.

Beam search is a trap dressed as rigor. It sounds smart — "we search for the most probable sequence." For translation, sure. For a chatbot it gives you the blandest, loopiest text imaginable, because human language isn't the maximum-likelihood sequence. If someone proposes beams for open-ended generation, you've found a quick, citable win (Holtzman) and a chance to look like you read the literature. Because you did.

What's actually worth caring about: the knob → effect → failure-mode map (temperature, top-p, repetition penalty), the canonical order, the SOUL invariant, and "a grammar is a guarantee." That's it. That's the whole toolbox, and it covers ~90% of generation incidents. What's not worth losing sleep over: the exact default top-p some library ships, whether min-p beats top-p by two points on some benchmark, or memorizing every penalty variant's formula — know they exist and look them up.

The career framing. This phase is small in code and huge in leverage. It's the layer between "I call the API" and "I own how the product behaves," and almost nobody bothers to learn it properly because it hides behind one method call. Get it cold and you become the person who debugs generation in minutes and ships reliable structured output while everyone else writes JSON repairers. That reputation compounds straight into Phase 12 (tool/function calling) and Phase 09 (serving). Now go make pytest green.