Reasoning Models

Phase 2 · Document 09 · Transformer Foundations Prev: 08 — MoE and Dense Models · Up: Phase 2 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Reasoning models (OpenAI o-series, Claude extended thinking, Gemini thinking, DeepSeek-R1, Qwen QwQ) are the most impactful — and most misused — recent development. They genuinely lift quality on hard math, code, and planning, but they do it by generating large amounts of hidden, billed "thinking" tokens, which means higher cost and much higher latency. Treating them as a free quality upgrade and routing every request through them is the fastest way to blow your latency SLOs and your budget. Understanding what they are — still the same next-token transformer, just trained/prompted to think before answering — lets you route them correctly: hard tasks get reasoning, simple tasks don't.

2. Core Concept

What a reasoning model is (and isn't)

It is not a new architecture. A reasoning model is a standard decoder-only transformer (00) that has been post-trained (often with reinforcement learning on verifiable problems) to emit a long internal chain-of-thought before its final answer. The "thinking" is just more generated tokens — the same autoregressive loop (05), but the model spends tokens working through the problem first.

normal model:    prompt → answer
reasoning model: prompt → [long hidden reasoning tokens...] → answer
                          └── billed (usually at output rates), often NOT shown to you ──┘

This is inference-time compute scaling: quality improves with more thinking tokens, a different axis than scaling parameters or data. "Think longer" can beat "be bigger" on hard problems.

Controls: reasoning effort / thinking budget

Providers expose a knob — reasoning_effort (low/medium/high), a thinking token budget, etc. (Phase 1.03):

• Higher effort → more reasoning tokens → better on hard tasks, slower and costlier.
• Lower effort → faster/cheaper, less thorough.

Cost and latency reality

• Reasoning tokens are billed, usually at output rates, even though you often don't see them (chain-of-thought privacy).
• They are pure decode (07) → reasoning models are TPOT-dominated; latency can be many seconds to minutes.
• A single "hard" answer can cost 5–50× a normal answer in tokens. Budget and route accordingly.

When reasoning helps — and when it doesn't

• Helps: multi-step math, competitive coding, complex planning, agentic decomposition, hard analysis, anything with a verifiable/searchable solution path.
• Doesn't help (or hurts): simple lookups, classification, extraction, short chat, formatting, retrieval-grounded Q&A — here it adds latency/cost for no quality gain, and sometimes overthinks a trivial task.

The engineering rule: route by difficulty. Cheap/fast non-reasoning models for the bulk; reasoning models for the hard minority. (Many products use a cheap classifier or task type to decide.)

Chain-of-thought privacy and safety

Providers frequently hide the raw reasoning (to protect IP and prevent unsafe content leakage), returning only a summary plus the answer. You're still billed for the hidden tokens. Don't rely on the visible "reasoning" being the true internal process, and never feed sensitive data assuming the hidden chain is private to you.

3. Mental Model

REASONING MODEL = same transformer, trained to "show its work" (internally) before answering.
   quality ↑ with THINKING TOKENS  (inference-time compute scaling — a new axis)

   prompt ──► [════ hidden reasoning tokens ════] ──► answer
                     │ billed (output rates), latency-heavy (pure decode), often hidden │

   reasoning_effort:  low ── fast/cheap/shallow ──→ high ── slow/expensive/thorough

   ROUTE BY DIFFICULTY:
     hard (math/code/planning) → reasoning model (high effort)
     easy (lookup/extract/chat) → cheap fast model (no reasoning)

4. Hitchhiker's Guide

What to understand first: reasoning = extra billed/hidden tokens for quality on hard tasks; it's a routing decision, not a default.

What to ignore at first: the RL training recipe details. Know the behavior and cost, route accordingly.

What misleads beginners:

• "Reasoning models are just smarter, same cost." They cost (reasoning tokens) and add big latency.
• "Always use the best/reasoning model." Overkill for most requests; murders cost and latency.
• "The shown reasoning is the real reasoning." It's often a hidden or summarized process.
• "More effort always = better." Beyond the task's needs, you pay for nothing (and risk overthinking).

How experts reason: they route by difficulty, set effort per task class, cap reasoning/thinking budgets, and measure whether reasoning actually improves their eval before enabling it broadly.

What matters in production: difficulty-based routing, reasoning-token budgets/limits, separate latency SLOs (reasoning paths are slow), and cost tracking that counts reasoning tokens.

How to verify: run your eval (Phase 1.07) with reasoning on vs off on representative tasks; compare quality gain to the latency/cost increase; check whether usage reports reasoning tokens.

Questions to ask providers: Are reasoning tokens billed, and at what rate? Is the chain-of-thought returned, summarized, or hidden? What effort/budget controls exist? What's typical added latency?

What silently gets expensive/unreliable: blanket-enabling reasoning for all traffic; uncapped thinking budgets; reasoning-path latency breaking real-time UX; assuming hidden CoT is private.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• A reasoning model is the same transformer architecture, post-trained (often via RL) to think before answering.
• Reasoning tokens are billed (usually at output rates) and often hidden from you.
• Reasoning is inference-time compute scaling: quality rises with thinking tokens — a different axis than parameter/data scaling.
• Reasoning is pure decode → these models are TPOT-dominated and can be very slow.
• A hard answer can cost 5–50× a normal answer in tokens.
• Reasoning helps hard, multi-step tasks (math/code/planning) and wastes money on simple ones.
• The right pattern is difficulty-based routing, not blanket enablement.
• Don't assume the visible reasoning is the true/complete internal process, or that hidden CoT is private.

9. Observations from Real Systems

• OpenAI o-series expose reasoning_effort and bill reasoning tokens while hiding the raw chain; Anthropic extended thinking exposes a thinking budget.
• DeepSeek-R1, Qwen QwQ are open-weight reasoning models — you can run them locally and see the long reasoning traces (Phase 6).
• Cursor and coding agents route planning to a reasoning model and edits/autocomplete to a fast model — textbook difficulty routing (Phase 11).
• Provider usage objects report reasoning/thinking token counts so you can attribute their cost.
• Latency dashboards show reasoning requests taking seconds-to-minutes — pure decode/TPOT cost.
• models.dev flags reasoning support as a capability column (Phase 4).

10. Common Misconceptions

11. Engineering Decision Framework

Should THIS request use reasoning?
  Hard, multi-step, verifiable (math/code/planning/analysis)? → reasoning model, effort by hardness.
  Simple (lookup/extract/classify/format/chat)?              → cheap fast model, no reasoning.
  Unsure?  → cheap classifier or task type decides (difficulty routing).

Setting effort/budget:
  start LOW; raise only if your eval shows quality gains worth the latency/cost.
  always CAP the thinking budget to bound cost/latency.

Validate before rollout:
  run your eval with reasoning ON vs OFF on representative tasks;
  adopt only where Δquality justifies Δcost & Δlatency (Phase 1.07 / 12).

12. Hands-On Lab

Goal

Quantify the reasoning trade-off: measure quality, latency, and token cost with reasoning OFF vs ON across an easy and a hard task, then build a difficulty router.

Prerequisites

• Python 3.10+, access to a reasoning-capable model and a fast non-reasoning model (or an open reasoner locally).

Setup

pip install openai
export OPENAI_API_KEY=sk-...

Steps

import time
from openai import OpenAI
client = OpenAI()

tasks = {
  "easy": "What is the capital of Japan? One word.",
  "hard": "A train leaves A at 60mph, another leaves B (180mi away) toward A at 40mph 30min later. "
          "When and where do they meet? Show reasoning.",
}

def run(model, prompt, **kw):
    t0 = time.perf_counter()
    r = client.chat.completions.create(model=model, messages=[{"role":"user","content":prompt}], **kw)
    dt = time.perf_counter() - t0
    u = r.usage
    # reasoning tokens may appear in usage details depending on provider/model
    rt = getattr(getattr(u, "completion_tokens_details", None), "reasoning_tokens", 0) or 0
    return dt, u.completion_tokens, rt, r.choices[0].message.content[:80]

for label, p in tasks.items():
    dt1 = run("gpt-4o-mini", p, max_tokens=400)                 # fast, no reasoning
    dt2 = run("o4-mini", p, reasoning_effort="high")            # reasoning (model id illustrative)
    print(f"\n[{label}] fast: {dt1[0]:.1f}s out={dt1[1]} reason={dt1[2]}")
    print(f"[{label}] reasoning: {dt2[0]:.1f}s out={dt2[1]} reason={dt2[2]}")

# Simple difficulty router
def route(prompt):
    hard = any(k in prompt.lower() for k in ["show reasoning","prove","step","optimi","when and where","derive"])
    return ("o4-mini", {"reasoning_effort":"high"}) if hard else ("gpt-4o-mini", {"max_tokens":200})

Expected output

• Easy task: reasoning adds latency/cost for no quality gain.
• Hard task: reasoning improves correctness but with far higher latency and (reasoning) token count.
• A router that sends only hard prompts to the reasoning model.

Debugging tips

• reasoning_tokens is 0/None? Your provider may not expose it, or the model isn't a reasoner — note that and use total completion tokens.
• Use whatever reasoning-capable model IDs you actually have access to (the IDs above are illustrative).

Extension task

Add a cost column (reasoning tokens × output price) and compute the cost ratio reasoning/fast for easy vs hard tasks.

Production extension

Replace the keyword router with a cheap classifier model that labels difficulty, and log per-request model/cost/latency — the seed of difficulty routing in your Phase 8 gateway.

What to measure

Quality (correct?), latency, output + reasoning tokens, and cost ratio — for easy vs hard, reasoning on vs off.

Deliverables

• A table: easy/hard × reasoning-off/on → quality, latency, tokens, cost.
• A working difficulty router.
• A one-line policy: which tasks you'd route to reasoning and why.

13. Verification Questions

Basic

1. Is a reasoning model a new architecture? What actually makes it "reason"?
2. Are reasoning tokens billed and visible? Explain.
3. What does reasoning_effort trade off?

Applied 4. Which phase (prefill/decode) dominates reasoning-model latency, and why? 5. Give two task types where reasoning helps and two where it's wasteful.

Debugging 6. After enabling reasoning everywhere, latency and cost spiked. Diagnose and propose a fix. 7. A reasoning model "overthinks" a trivial classification. What's the right architectural response?

System design 8. Design difficulty-based routing for a product mixing FAQ lookups and complex troubleshooting. How do you decide per request?

Startup / product 9. Investors ask why your margins survive offering "advanced reasoning." Explain routing, budgets, and eval-gated enablement.

14. Takeaways

1. A reasoning model is the same transformer, post-trained to think (emit hidden CoT) before answering.
2. Quality scales with thinking tokens — inference-time compute scaling, a new axis.
3. Reasoning tokens are billed (output rates) and often hidden; latency is decode/TPOT-dominated.
4. Reasoning helps hard multi-step tasks and wastes money on simple ones.
5. The right pattern is difficulty-based routing with capped effort/budgets.
6. Validate with your eval before enabling reasoning broadly; don't trust visible CoT as the true process.

15. Artifact Checklist

• Code: reasoning on/off measurement + difficulty router.
• Table: easy/hard × reasoning → quality/latency/tokens/cost.
• Policy note: which task classes use reasoning.
• Cost analysis: reasoning/fast cost ratio.
• Notes: inference-time compute scaling and CoT-privacy caveats.
• Diagram: the reasoning flow + difficulty-routing decision.

Up: Phase 2 Index · Next: Phase 3 — Model Cards and System Cards