Exercises 03 — Model Catalog & Selection (10)

Concepts: Phase 4, Phase 5. Cheatsheets: 03, 04.

Scenario: "Just use GPT-best for everything."
Task: Propose a selection process instead.
Constraints: Mixed workload, cost matters.
Expected answer: Hard constraints → shortlist from catalog → eval on your task → weighted score (quality/cost/latency/reliability) with safety as a gate → pick the cheapest that clears the bar, route harder requests up.
Rubric: Senior = the full framework + routing + safety gate.
Common mistakes: One model for all; selecting on benchmarks.
Extension: Add a CI regression gate to the process (Phase 12.08).

Scenario: Healthcare app, EU data, needs tool calling.
Task: Which constraints eliminate models fastest?
Constraints: Regulated.
Expected answer: Data residency/privacy (may force self-host or a region/ZDR provider) eliminates many options immediately; then tool calling, context, latency, budget. Filter hard constraints before comparing quality.
Rubric: Senior = leads with residency/privacy as the killer constraint.
Common mistakes: Comparing quality before filtering on constraints.
Extension: When does this push you to open-weight self-hosting? (cheatsheet 05)

Scenario: Model A is cheaper per token but fails more, triggering retries.
Task: Which metric decides?
Constraints: Agent workflow.
Expected answer: Cost per resolved task (= cost per task ÷ success rate), not per token. A "cheap" model that retries can cost more per successful outcome (Phase 15.05).
Rubric: Senior = uses resolved-task cost and accounts for retries/loops.
Common mistakes: Comparing per-token price only.
Extension: Compute resolved-task cost for two models given success rates.

Scenario: 90% of requests are easy, 10% hard; using a premium model for all is too costly.
Task: Design routing.
Constraints: Protect margin + quality.
Expected answer: Route easy requests to a small/cheap (or local) model, hard ones to premium; classify difficulty (heuristic or a cheap classifier); cache repeats (Phase 8.05).
Rubric: Senior = difficulty-based routing + caching + fallback.
Common mistakes: Single model; no fallback.
Extension: How to measure routing accuracy and its cost/quality impact?

Scenario: Autocomplete needs sub-second; chat can take seconds.
Task: Model strategy.
Constraints: IDE product.
Expected answer: Latency tiers: tiny/fast FIM model for autocomplete (<<1s), mid for inline edit, frontier for chat/agent (Phase 11.06). Latency budgets force multi-model.
Rubric: Senior = matches model to latency tier.
Common mistakes: One model across all latency tiers.
Extension: What's the p95 budget per tier?

Scenario: You picked a model 8 months ago; it's still the default.
Task: What's the risk and the practice?
Constraints: Fast-moving field.
Expected answer: A newer/cheaper model may now beat it for free. Re-evaluate on your golden set periodically (quarterly); keep selection a living decision, not one-time (Phase 12.08).
Rubric: Senior = scheduled re-eval against the golden set.
Common mistakes: "Set and forget."
Extension: Automate model bake-offs in CI.

Scenario: You copy a price from a catalog row into your cost model.
Task: What could go wrong?
Constraints: Production cost model.
Expected answer: Prices/limits drift; "Updated" date matters; price is per provider; context ≠ max output; reasoning tokens billed extra. Recheck before production (cheatsheet 03).
Rubric: Senior = treats catalog as a starting point, verifies live.
Common mistakes: Trusting a stale row.
Extension: Build a price-refresh check into the registry (Phase 8.04).

Scenario: B2B SaaS, growing volume, some enterprise prospects need on-prem.
Task: Recommend a model strategy.
Constraints: Cost + residency.
Expected answer: Hybrid: commercial APIs now for speed/quality; self-host open-weight for on-prem/residency tenants and the cheap bulk as volume passes break-even; route (cheatsheet 05).
Rubric: Senior = hybrid + break-even + residency reasoning.
Common mistakes: All-commercial or all-self-host dogma.
Extension: Estimate the self-host break-even volume.

Scenario: Model X scores highest on quality/cost but fails safety red-team.
Task: Can it win the bake-off?
Constraints: Consumer-facing.
Expected answer: No — safety is a gate, not a weighted axis. A model that fails safety is disqualified regardless of quality/cost (Phase 12.07).
Rubric: Senior = safety as hard gate.
Common mistakes: Trading safety for quality in the weighted score.
Extension: Design the safety eval that gates selection.

Scenario: You must choose among 3 models for a summarization feature.
Task: Design the evaluation.
Constraints: Decision in a week.
Expected answer: Golden set (representative + edge + unanswerable); task metric (faithfulness + conciseness, judge calibrated); measure p95 latency + cost-per-resolved-task; safety gate; weighted score → decision memo (template).
Rubric: Senior = full harness + documented decision.
Common mistakes: Eyeballing a few outputs.
Extension: Turn it into a reusable harness (labs/lab-05-eval-harness).

Next: 04 — Local Inference · Index: exercises/

LLM Inside-Out Mastery