Selecting Reasoning Models

Phase 5 · Document 04 · Model Selection Prev: 03 — Coding Models · Next: 05 — RAG Models

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Reasoning models (Phase 2.09) genuinely lift quality on hard math, code, and planning — but they're the easiest model class to misuse in selection: teams route everything through them and watch latency and cost explode for no gain on easy tasks. Selecting reasoning models well is really about deciding when reasoning is worth its cost and tuning the effort/budget — i.e., it's a routing problem more than a "pick one model" problem. This doc applies the framework to the reasoning class, building directly on the mechanism in Phase 2.09.

2. Core Concept

Plain-English primer

• Reasoning model — a model trained to emit hidden step-by-step "thinking" before answering; quality on hard problems scales with how many thinking tokens it spends (inference-time compute) (Phase 2.09).
• Reasoning effort / thinking budget — the dial that controls how much it thinks (low/med/high or a token budget). Higher → better on hard tasks, slower, costlier.
• Thinking tokens — generated (and usually billed at output rates) but often hidden from you (Phase 4.04).
• Difficulty routing — sending only hard requests to a reasoning model and easy ones to a cheap/fast model — the core production pattern.

What to optimize for (the reasoning-specific axes)

1. Quality gain on hard tasks — does reasoning actually beat a non-reasoning model on your eval, not on a benchmark? (Often big on multi-step math/code/planning, negligible on lookups/extraction.)
2. Cost per solved hard task — thinking tokens make each hard answer 5–50× a normal one; what matters is cost per correct hard answer, and how often you even need it.
3. Latency tolerance — reasoning is decode-heavy (Phase 2.07); responses can take many seconds to minutes. Unusable for real-time UIs unless budgeted low.
4. Effort controllability — can you cap/tune the budget per request? You need this to control cost.
5. Open vs closed — open reasoning models (DeepSeek-R1, Qwen QwQ) let you self-host and see the reasoning; closed ones (o-series, Claude/Gemini thinking) hide it (01).

The selection is mostly a routing decision

You rarely "replace your model with a reasoning model." You add a reasoning path and route to it by difficulty:

easy / lookup / extraction / chat  → cheap fast (non-reasoning) model
hard / multi-step math, code, planning, agentic decomposition → reasoning model (effort by hardness)

So "selecting a reasoning model" = (a) pick a reasoning model whose hard-task quality justifies its cost on your eval, and (b) design the router + a difficulty signal and the effort/budget caps.

When reasoning is and isn't worth it

• Worth it: competition-grade math, hard coding/refactors, complex planning, agentic task decomposition, deep analysis with a verifiable path.
• Not worth it: classification, extraction, formatting, short chat, retrieval-grounded Q&A — reasoning adds latency/cost with little/no gain and can overthink trivial tasks.

3. Mental Model

Reasoning = trade TOKENS (billed) + LATENCY for QUALITY on HARD tasks (Phase 2.09).

SELECTION ≈ ROUTING:
  difficulty signal → easy → cheap/fast model ; hard → reasoning model (effort by hardness, budget CAPPED)

Decide a reasoning model by:  Δquality_on_HARD_eval  vs  Δcost + Δlatency  (your eval, not benchmarks)
Open reasoning (R1/QwQ, self-host, visible CoT)  vs  closed (o-series/Claude/Gemini thinking, hidden CoT)

Default mistake: route EVERYTHING to reasoning → cost+latency blow up for zero gain on easy tasks.

4. Hitchhiker's Guide

What to decide first: what fraction of your traffic is genuinely hard, and whether reasoning beats a strong non-reasoning model on those hard cases in your eval. If hard traffic is rare, you need reasoning only on a small routed slice.

What to ignore at first: reasoning-benchmark leaderboards — verify the gain on your hard tasks.

What misleads beginners:

• Treating reasoning as a free quality upgrade for all requests.
• Forgetting thinking tokens are billed (and hidden).
• Using a reasoning model where latency must be real-time.
• Maxing effort everywhere instead of tuning per task.

How experts reason: measure Δquality on a hard eval slice vs Δcost/Δlatency; route by difficulty; cap the thinking budget; reserve high effort for the hardest. They consider open reasoning models when they need self-hosting or visible reasoning.

What matters in production: a difficulty router, capped effort/budgets, separate latency SLOs for the reasoning path, and cost tracking that counts reasoning tokens.

How to verify: run an easy set and a hard set through reasoning-off vs reasoning-on; adopt reasoning only where the hard-set quality gain justifies the cost/latency (Phase 2.09 lab).

Questions to ask: Are thinking tokens billed, at what rate, and visible? What effort/budget controls exist? Typical added latency? Open or closed? Does it beat my non-reasoning model on my hard tasks?

What silently gets expensive/unreliable: blanket reasoning; uncapped budgets; reasoning latency breaking UX; assuming benchmark gains transfer to your tasks.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Reasoning quality scales with thinking tokens (inference-time compute) — a different axis than size (Phase 2.09).
• Thinking tokens are billed (output rates) and often hidden — a hard answer can cost 5–50× a normal one.
• Reasoning is decode-heavy → high latency (Phase 2.07); often unusable for real-time without low budgets.
• Reasoning helps hard multi-step tasks; it's wasteful (and can overthink) on easy ones.
• Selection ≈ routing: add a reasoning path for the hard slice, not for everything.
• Cap the effort/budget; reserve high effort for the hardest tasks.
• Open reasoning models (R1, QwQ) allow self-hosting and visible reasoning (01).
• Verify on a hard eval slice — benchmark gains don't guarantee gains on your tasks.

9. Observations from Real Systems

• Cursor / coding agents route planning/hard reasoning to a reasoning model and edits/autocomplete to a fast model — textbook difficulty routing (Phase 11).
• Reasoning bill shock is a common incident when teams enable thinking globally (Phase 4.04).
• Open reasoning models (DeepSeek-R1, Qwen QwQ) are self-hosted when privacy or visible reasoning matters, and you can see the long reasoning traces locally (Phase 6).
• Latency dashboards show reasoning requests as long, decode-dominated outliers.
• models.dev flags reasoning support as a capability column (Phase 4.01).

10. Common Misconceptions

11. Engineering Decision Framework

1. ESTIMATE hard-traffic fraction. If tiny → reasoning is a small routed slice (don't over-invest).
2. EVAL on a HARD set: reasoning-off vs reasoning-on (and effort levels).
     adopt reasoning ONLY where Δquality justifies Δcost + Δlatency (Phase 2.09 lab).
3. ROUTE: build a difficulty signal (task type / cheap classifier) → hard → reasoning (effort by hardness),
     easy → cheap fast model.
4. CAP budgets; set a separate latency SLO for the reasoning path; track reasoning tokens.
5. OPEN vs CLOSED: need self-host/visible CoT/privacy → open reasoning (R1/QwQ); else closed thinking models.
6. Memo the decision + routing rule (3.07).

12. Hands-On Lab

Goal

Decide whether (and where) a reasoning model earns its cost by comparing reasoning-off vs reasoning-on on an easy and a hard eval set, then build a difficulty router.

Prerequisites

• Access to a reasoning-capable model + a fast non-reasoning model; a small hard set (multi-step problems) and easy set; the Phase 1.07 harness.

Steps

1. Build two sets: ~10 hard (multi-step math/code/planning) and ~10 easy (lookup/extract/classify) items with checkable answers.
2. Run 3 conditions: fast model (no reasoning); reasoning model low effort; reasoning model high effort. Record accuracy, latency, and (thinking) token cost per item.
3. Compute deltas: on the hard set and the easy set separately — Δaccuracy vs Δcost/Δlatency.
4. Decide adoption + effort: reasoning likely wins on the hard set, loses on the easy set — confirm and quantify.
5. Build a router: a difficulty signal (keyword/classifier) → route + effort; memo it.

def route(prompt, classify_difficulty):
    d = classify_difficulty(prompt)   # "hard" / "easy"
    if d == "hard":
        return ("reasoning-model", {"reasoning_effort": "high"})
    return ("fast-model", {"max_tokens": 256})

Expected output

• Tables showing reasoning's clear win on hard items and net loss (cost/latency) on easy items — justifying routing, not blanket use.

Debugging tips

• No gain even on hard items? Your "hard" set may be too easy, or the non-reasoning model already suffices — both are useful findings.
• Cost exploded? You maxed effort on easy items — that's the anti-pattern this lab proves.

Extension task

Add cost-per-correct-hard-answer as the headline metric and pick the effort level that optimizes it.

Production extension

Replace the keyword classifier with a cheap model that labels difficulty, log per-request model/effort/cost/latency, and feed the router into the Phase 8 gateway.

What to measure

Δaccuracy (hard vs easy), latency, thinking-token cost, cost per correct hard answer; routing mix.

Deliverables

• A reasoning-off/low/high comparison on hard + easy sets.
• A difficulty router + effort policy.
• A memo: when you route to reasoning and at what effort.

13. Verification Questions

Basic

1. What does reasoning_effort trade off, and how are thinking tokens billed?
2. Which phase (prefill/decode) makes reasoning slow, and why?
3. Why is selecting a reasoning model mostly a routing decision?

Applied 4. Give two task types where reasoning helps and two where it's wasteful. 5. How do you decide the effort level for the hard slice?

Debugging 6. After enabling reasoning everywhere, cost and latency spiked. Diagnose + fix. 7. Reasoning "overthinks" a classification task. What's the right response?

System design 8. Design difficulty routing + budget caps for a product mixing FAQ lookups and complex troubleshooting.

Startup / product 9. Justify offering "advanced reasoning" while protecting margins, using routing, budgets, and eval-gated adoption.

14. Takeaways

1. Reasoning trades billed thinking tokens + latency for quality on hard tasks — verify the gain on your hard eval.
2. Selection ≈ routing: add a reasoning path for the hard slice, not for everything.
3. Cap effort/budgets and set a separate latency SLO for the reasoning path.
4. Don't use reasoning for easy tasks (cost/latency waste, overthinking).
5. Open reasoning models enable self-hosting + visible CoT.
6. Decide on cost per correct hard answer, not benchmark rank.

15. Artifact Checklist

• Hard + easy eval sets with checkable answers.
• Reasoning-off/low/high comparison (accuracy, latency, cost).
• Difficulty router + effort policy.
• Cost-per-correct-hard-answer metric.
• Selection memo + routing rule (3.07).
• Notes: when reasoning is/ isn't worth it.

Next: 05 — RAG Models