Phase 5 — Model Selection
How to choose the right model(s) for a job — a repeatable process, the open-vs-commercial and local-vs-cloud calls, per-use-case selection (coding, reasoning, RAG, agents, structured output, multimodal), and the cost-quality-latency framework that unifies them all.
Why this phase matters
Phases 3–4 taught you to read models and the market. Phase 5 is where you decide — defensibly, with eval and spike data, usually landing on a small set of models + a routing rule rather than one pick. This is the bridge from "understanding models" to "building systems" (Phases 6–11).
Documents
| # | Document | What you'll be able to do |
|---|---|---|
| 00 | Model Selection Framework | Run the requirements→filter→score→spike→memo process |
| 01 | Open-Source vs Commercial | Decide API vs open-weight (licenses, break-even, hybrid) |
| 02 | Local vs Cloud | Place the model: local / self-host cloud / managed |
| 03 | Coding Models | Select for apply-rate + test-pass on your repo |
| 04 | Reasoning Models | Decide when reasoning is worth it; route by difficulty |
| 05 | RAG Models | Select embedder + reranker + generator (retrieval dominates) |
| 06 | Agent Models | Select on per-step reliability (it compounds) |
| 07 | Structured-Output Models | Get guaranteed schemas (constrained ≠ correct values) |
| 08 | Multimodal Models | Handle in≠out modalities; price vision tokens; compose |
| 09 | Cost-Quality-Latency Framework | Quantify the trade-off; escape it with routing + caching |
| 10 | Provider Variance & Serving Fidelity | "Same model ID ≠ same model": detect quantization/context-cap/prompt-injection across providers |
How to work through it
Read 00 (the process) and 09 (the unifying trade-off) as the bookends; 01–02 are the deployment branch; 03–08 are per-use-case applications you can read as needed for your work. Every doc opens with a from-zero plain-English primer and ends with a lab that builds a real eval/decision artifact.
Phase 5 artifacts
- A requirements doc + scored candidate table + spike + selection memo (00)
- An open-vs-commercial break-even analysis (01) and a local/cloud cost-vs-utilization model (02)
- Per-use-case evals: coding apply/test-pass (03), reasoning hard/easy routing (04), RAG retrieval-vs-generator (05), agent valid-call/compounding (06), structured conformance/value-accuracy (07), multimodal accuracy/grounding/cost (08)
- A cost-quality-latency frontier plot + routing/caching simulation beating the best single model (09)