Model Selection Framework

Phase 5 · Document 00 · Model Selection Prev: Phase 5 Index · Next: 01 — Open-Source vs Commercial

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Choosing the wrong model is one of the most expensive mistakes in LLM engineering: too powerful wastes money, too weak forces costly workarounds, the wrong provider triggers a privacy violation, a missing feature flag breaks your agent, and the wrong pricing model blows the budget. Yet most teams pick by leaderboard or vibes. This doc gives you a repeatable selection process — define requirements → filter on hard constraints → score → spike → document — that makes model choices defensible and reversible. It's the hub of Phase 5: the per-use-case docs (03–08) and the cost-quality-latency framework all plug into this process.

2. Core Concept

Plain-English primer (the words in a selection decision)

• Requirement vs hard constraint vs preference — a requirement is anything you need; a hard constraint is a pass/fail gate (context too small, no tool calling, data can't leave region → eliminate); a preference is a soft trade-off you score (a bit cheaper, a bit faster).
• Spike — a small, time-boxed experiment: run ~50–100 real requests through a candidate to measure actual quality/latency/cost before committing. (Borrowed from agile engineering.)
• Pareto frontier — the set of "best possible trade-offs": models where you can't improve one of cost/quality/latency without worsening another. You pick a point on it for your use case (09).
• Routing — using different models for different requests (cheap for easy, premium for hard, private for sensitive) instead of one model for everything. The dominant production pattern.
• Selection memo — the written record of the decision (candidates, eval, rationale, fallback, cost) — Phase 3.07 template.

The five-step process

1. Define requirements — before looking at any model. Use case, quality bar (with examples), latency target (TTFT/total or "batch OK"), cost budget, data policy (regions/retention), required features, context and output needs, availability/SLA, and scale (now + 12 months).
2. Filter by hard constraints — eliminate any candidate that fails a gate: context < required, missing a required feature, data policy prohibits the provider, license forbids your use, clearly out of budget. This usually cuts the field from dozens to a handful (use the catalog).
3. Score survivors — on weighted criteria (below), with quality measured by your eval (Phase 1.07), not benchmarks.
4. Validate with a spike — 50–100 real requests; measure actual quality, latency (p50/p95), cost; test edge cases and failure behavior.
5. Document — write a selection memo; pin the versioned ID; define a fallback.

The scoring rubric (a starting point — reweight per use case)

final = Σ(weight × normalized_score). The weights are your priorities — set them deliberately.

Routing beats "the one best model"

There is rarely a single best model. Real systems route: by task complexity (simple→cheap, hard→premium/reasoning), by data sensitivity (private→self-hosted, standard→API), and by latency (real-time→fast/small, batch→quality). Selection, then, often means choosing a small set of models + a routing rule — the foundation of the Phase 8 gateway.

3. Mental Model

REQUIREMENTS ──► HARD-CONSTRAINT FILTER ──► SCORE survivors ──► SPIKE (real data) ──► MEMO + pin + fallback
(define first)    (pass/fail gates cut       (weighted: quality      (50–100 real         (document the
                   dozens → a few)            via YOUR eval,           requests; p50/p95,    decision)
                                              cost, latency,           edge + failure)
                                              reliability, fit)

OUTPUT is usually NOT one model but a SMALL SET + a ROUTING RULE:
   simple→cheap · hard→premium/reasoning · sensitive→self-hosted · real-time→fast · batch→quality

You pick a POINT on the cost/quality/latency Pareto frontier — you can't max all three.

4. Hitchhiker's Guide

What to do first: write requirements before browsing models. The hard constraints do most of the elimination for free.

What to ignore at first: leaderboard rank. It's a shortlist input; your spike decides.

What misleads beginners:

• Picking the "best" model for everything (overpays; ignores routing).
• Scoring quality by benchmarks instead of your eval.
• Skipping the spike (demo ≠ production behavior at p95 and on edge cases).
• Forgetting output limits, data policy, or fallback until they bite.

How experts reason: requirements → filter → score (quality = your eval) → spike → memo. They default to routing (a few models) over a single pick, pin versioned IDs, and design a fallback from day one.

What matters in production: a passing spike on real data, a documented decision, a fallback model, and an eval that tells you when to switch as the market churns.

How to verify a choice: the spike — actual quality/latency/cost on 50–100 real requests, including edge and failure cases — must meet your requirements, not just the happy path.

Questions to ask: What's my quality bar (with examples)? Which constraints are hard vs soft? What's the fallback? What would make me re-select (price, deprecation, a better model)?

What silently gets expensive/unreliable: one premium model for trivial tasks; no fallback; choosing on benchmarks; ignoring scale (a choice that's fine at 1k/day breaks at 1M/day).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Define requirements before looking at models — hard constraints eliminate most candidates instantly.
• Quality is scored by your eval, not benchmarks (Phase 1.07).
• You can't maximize cost, quality, and latency together — pick a point on the frontier (09).
• Always run a spike on 50–100 real requests (incl. edge + failure) before committing.
• The output is usually a small model set + a routing rule, not one model.
• Pin versioned IDs and define a fallback from day one.
• Document with a selection memo — decisions must be reproducible and reviewable.
• Re-select on triggers (deprecation, price change, a better model) — keep the eval ready.

9. Observations from Real Systems

• Production LLM apps route: cheap model for the bulk, premium/reasoning for the hard minority, self-hosted for sensitive data — implemented in gateways (Phase 8).
• Cursor and coding tools route planning→reasoning model, edits→fast code model (Phase 11) — selection as routing.
• OpenRouter/LiteLLM make "primary + fallbacks" a config, turning a selection memo into executable routing.
• Teams with an eval harness switch models calmly when a better/cheaper one appears; teams without it either churn or stagnate (Phase 4.05).
• Spikes catch what demos hide: p95 latency spikes, edge-case failures, and real cost — the reasons "it worked in the demo" features fail in production.

10. Common Misconceptions

11. Engineering Decision Framework

THE PROCESS (use for every non-trivial choice):
  1. Requirements: use case · quality bar (examples) · latency · budget · data policy ·
     features · context & OUTPUT · availability · scale (now + 12mo).
  2. Filter (hard constraints): context/output too small · missing feature · data/region · license · budget → ELIMINATE.
  3. Score survivors: weighted rubric; QUALITY = your eval (Phase 1.07).
  4. Spike: 50–100 real requests; p50/p95 latency, real cost, edge + failure cases.
  5. Decide: usually a SMALL SET + routing rule; pin versioned IDs; define fallback; write a memo (3.07).

WHICH WAY TO BRANCH (see dedicated docs):
  open vs commercial → 01 · local vs cloud → 02 · cost/quality/latency trade-off → 09
  coding → 03 · reasoning → 04 · RAG → 05 · agents → 06 · structured output → 07 · multimodal → 08

12. Hands-On Lab

Goal

Run the full selection process end-to-end for one real use case: requirements → filter → score → spike → memo.

Prerequisites

• A real use case; the catalog importer, a golden set (Phase 1.07), and an API key for 2–3 candidates.

Steps

1. Requirements: fill the template (use case, quality bar with examples, latency, budget, data policy, features, context+output, scale).
2. Filter: query your catalog for candidates passing every hard constraint; keep 2–3.
3. Score: set rubric weights to your priorities; score cost (model it) and feature fit now; leave quality/latency for the spike.
4. Spike: run your golden set + 50 real-ish requests through each survivor; record quality (eval), p50/p95 TTFT, total latency, and cost/request; test 3 edge cases and the "model unavailable" path.
5. Decide + memo: compute final scores, pick (or define a routing rule), pin versioned IDs, set a fallback, write the memo.

Expected output

A completed requirements doc, a scored candidate table backed by spike data, and a selection memo with a fallback.

Debugging tips

• All candidates "pass"? Your quality bar is too vague — add concrete examples and a failing case.
• Spike contradicts benchmarks? Trust the spike — that's its purpose.

Extension task

Turn the decision into a routing rule (e.g. cheap model default, premium on a complexity signal) and estimate the blended cost vs single-model cost (reuse Phase 1.08 calculator).

Production extension

Wire the spike into a repeatable eval so re-selection (on a deprecation/price change) is a button, not a project (Phase 4.05).

What to measure

Candidates eliminated by hard constraints; final weighted scores; spike quality/latency/cost; blended vs single-model cost (extension).

Deliverables

• Requirements doc + scored candidate table.
• Spike results (quality, p50/p95 latency, cost, edge/failure).
• A selection memo with pinned IDs + fallback.

13. Verification Questions

Basic

1. What are the five steps of the selection process?
2. What's the difference between a hard constraint and a scored preference?
3. Why score quality with your eval rather than benchmarks?

Applied 4. A model passes the quality bar but costs 10× budget. Give three options. 5. Your use case processes employee HR data. What hard constraints apply, and how do they steer selection?

Debugging 6. The chosen model demos great but users complain in production. What step did you under-do? 7. A new model tops the benchmarks. What do you run before switching?

System design 8. Design a routing-based selection (not one model) for a product mixing simple chat and hard troubleshooting.

Startup / product 9. Explain to investors how your selection process (not a specific model) is a durable advantage as models churn.

14. Takeaways

1. Requirements first; hard constraints eliminate fast.
2. Score quality with your eval, cost/latency with measurement — and spike on real data.
3. You can't max cost+quality+latency — pick a point on the frontier.
4. The output is usually a small model set + routing rule, not one model.
5. Pin versioned IDs, define a fallback, write a memo.
6. Re-select on triggers — keep the eval ready for a churning market.

15. Artifact Checklist

• Requirements doc for one real use case.
• Scored candidate table (weighted rubric).
• Spike results (quality, p50/p95 latency, cost, edge/failure).
• Selection memo with pinned IDs + fallback.
• (Extension) routing rule + blended-cost estimate.
• Notes: the five-step process and your rubric weights.

Next: 01 — Open-Source vs Commercial