Cheatsheet 04 — Choosing a Model
The selection decision in one card. Full: Phase 5.
The framework (in order)
- Define the task + quality bar — what does "good" mean, and what accuracy is acceptable?
- Hard constraints first — context size, modality, tool calling, data residency / self-host (kills options fast), latency ceiling, budget.
- Shortlist 3–5 candidates from the catalog (cheatsheet 03).
- Evaluate on YOUR task — a golden set, not benchmarks (Phase 12.01).
- Score the axes: quality, cost per resolved task, latency (p95), reliability, safety (a gate).
- Decide with a weighted score + a safety gate; document in a selection memo.
The decision axes
| Axis | How to measure | Tool |
|---|---|---|
| Quality | Eval on golden set (task metric) | Phase 12 |
| Cost | $ per resolved task (not per request) | Phase 15.05 |
| Latency | p95 under load (TTFT + TPOT) | Phase 12.06 |
| Reliability | Error rate, uptime, provider stability | Phase 7.07 |
| Safety | Violation rate (a hard gate, not a weighted axis) | Phase 12.07 |
Quick heuristics
- Don't default to the biggest/best model — use the cheapest that meets the quality bar (route, cheatsheet 13).
- Reasoning model only for genuinely hard reasoning — they cost more (reasoning tokens) and are slower.
- Open-weight/self-host when residency/privacy/cost-at-scale demand it (cheatsheet 05).
- Two-model pattern: small/cheap for easy requests, premium for hard — route by difficulty.
- Re-evaluate quarterly — a newer/cheaper model may now beat your choice.
Anti-patterns 🚩
- Choosing on benchmark leaderboards.
- Choosing one model for everything (margin killer).
- Ignoring p95 (avg latency lies; tails hurt UX).
- No safety gate.
- Never re-evaluating after launch.
Interview line
"Selection = hard constraints → shortlist → eval on my task → weighted score across quality/cost/latency/reliability with safety as a hard gate. I pick the cheapest model that clears the bar and route harder requests up."
Next: 05 — Open-Source vs Commercial · Full: Phase 5