Cost-Quality-Latency Framework

Phase 5 · Document 09 · Model Selection Prev: 08 — Multimodal Models · Up: Phase 5 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

This is the capstone of Phase 5: every model decision is, underneath, a trade-off between cost, quality, and latency — you cannot maximize all three. The per-use-case docs (03–08) each pick a point on this triangle; this doc gives you the explicit framework to quantify the trade-off, turn it into a single weighted score, and — crucially — escape the triangle's limits through routing and caching. Teams that internalize this stop arguing about "the best model" and start optimizing a measurable objective. It ties together the selection framework, pricing, serving, and eval.

2. Core Concept

Plain-English primer (the three axes)

• Quality — how good the output is on your task, measured by your eval (Phase 1.07), not benchmarks.
• Cost — $ per request (and per resolved task): input + output (+ reasoning) tokens × price, minus caching (Phase 4.04).
• Latency — how fast: TTFT (time to first token) + TPOT × output tokens (Phase 1.05, 2.07).

The trade-off (and the Pareto frontier)

These three pull against each other: bigger/reasoning models raise quality but cost more and run slower; small models are fast and cheap but lower quality. Plotting models on these axes, the Pareto frontier is the set of "best possible" models — ones where you can't improve one axis without sacrificing another. Models off the frontier are strictly worse (dominated) and should be discarded. Your job is to pick a point on the frontier that matches your use case's priorities, not to find a mythical model that wins all three.

Turning the trade-off into a decision: weighted scoring

Make the priorities explicit with weights that sum to 1, normalize each axis to 0–1 (higher = better), and combine (Phase 5.00):

score = w_quality·Q + w_cost·C + w_latency·L + w_reliability·R + w_fit·F

The weights are your product decision: a real-time autocomplete weights latency/cost; a legal-analysis tool weights quality. Same candidates, different winner, depending on weights. Set them deliberately before scoring to avoid rationalizing a favorite.

Escaping the triangle: routing + caching (the real superpower)

You're not stuck at one point. Two techniques let a system beat any single model:

• Routing — send each request to the cheapest model that meets its quality bar: easy → small/fast/cheap, hard → premium/reasoning. The blended cost/quality/latency lands better than any single choice because most traffic is easy (Phase 5.04, Phase 8).
• Caching — prompt/prefix caching (75–90% off repeats) and response caching cut cost and latency for repeated work (Phase 2.06, Phase 4.04).

So the framework's punchline: pick a frontier point per request via routing, and use caching to shift the whole frontier down-and-left.

Seasoned caveat: the provider also runs this exact trade-off — for its margin. A cheaper provider may quantize an open-weight model or cap its context to raise throughput, so the same model ID can sit at a worse frontier point than its canonical serving. Always measure quality per endpoint, not just per model, and pin the provider. Deep dive: 10 — Provider Variance and Serving Fidelity.

Cost per resolved task, not per token

The honest cost metric isn't $/token or even $/request — it's cost per successfully completed task. A "cheap" model that needs retries, more tokens, or human fixups can cost more per resolved task than a pricier model that nails it first try. Always evaluate at the task level.

3. Mental Model

        QUALITY
          ▲
          │   ● frontier/reasoning (high Q, high $, slow)
          │  ● mid-tier
          │ ● small/fast (low Q, low $, fast)
          └───────────────► SPEED
         ╱
     COST (per resolved task)

You CANNOT max all three → pick a POINT on the PARETO FRONTIER (discard dominated models).
WEIGHTS encode your priorities:  score = Σ wᵢ·(normalized axisᵢ).

ESCAPE the triangle:
  ROUTING  → cheapest model meeting each request's quality bar (easy→small, hard→premium)
  CACHING  → repeats cost ~0 and return instantly → shifts the frontier down-and-left

Honest metric: COST PER RESOLVED TASK (retries/fixups included), not $/token.

4. Hitchhiker's Guide

What to do first: set your weights (priorities) for the use case before looking at scores — and measure all three axes (quality via eval, cost via model, latency via spike).

What to ignore at first: dominated models (worse on every axis) — discard them immediately.

What misleads beginners:

• Hunting for one model that's best on all three (doesn't exist — it's a frontier).
• Optimizing $/token instead of cost per resolved task.
• Picking by quality alone (overpays + too slow) or cost alone (quality fails).
• Forgetting routing + caching can beat any single point.

How experts reason: they discard dominated models, set weights from the product's real priorities, score the frontier candidates, and then design routing + caching to do better than any single pick — measuring at the resolved-task level.

What matters in production: a weighted objective tied to product goals, a blended (routed) cost/quality/latency, caching where repeats exist, and continuous measurement so you can re-balance as priorities or models change.

How to verify: run candidates through your eval + a latency spike + a cost model; plot them on the triangle, discard dominated ones, score the rest, then simulate a routing rule's blended outcome.

Questions to ask: What are my axis priorities (weights)? Which models are dominated? What's cost per resolved task? Can routing + caching beat the best single model here?

What silently gets expensive/unreliable: one premium model for everything (cost/latency); $/token thinking that ignores retries; no caching on repeated prompts; weights that don't match the product (optimizing the wrong axis).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• You cannot maximize cost, quality, and latency together — choose a point on the Pareto frontier.
• Discard dominated models (worse on every axis) immediately.
• Weights encode your priorities — set them deliberately before scoring.
• Routing beats any single model by sending each request to the cheapest model that meets its bar.
• Caching shifts the whole frontier down-and-left (cheaper and faster on repeats).
• The honest metric is cost per resolved task, not $/token.
• Quality must be measured (your eval), latency spiked, cost modeled — don't guess any axis.
• Re-balance as priorities or models change (Phase 4.05).

9. Observations from Real Systems

• Artificial Analysis literally plots models on quality vs price vs speed — a live Pareto frontier you can read (Phase 4.03).
• Production routers (OpenRouter/LiteLLM, in-house gateways) send easy traffic to cheap models and hard traffic to premium — beating any single model's blended economics (Phase 8).
• Prompt caching routinely yields 20–40% cost cuts and lower TTFT for chatbots with fixed prompts (Phase 4.04).
• "$/token" optimizations that backfire: a cheaper model that retries/fixes up more can cost more per resolved task — only task-level measurement reveals it.
• Coding/agent tools route (fast for autocomplete, strong for hard edits) — the frontier framework applied per sub-task (Phase 11).

10. Common Misconceptions

11. Engineering Decision Framework

1. MEASURE all three axes for candidates: quality (your eval) · cost (model it) · latency (spike, p50/p95).
2. PLOT + PRUNE: drop dominated models (worse on every axis).
3. WEIGHT: set w_quality/cost/latency/reliability/fit from the PRODUCT's priorities (sum=1) — before scoring.
4. SCORE: score = Σ wᵢ·normalizedᵢ → rank frontier candidates.
5. ESCAPE THE TRIANGLE:
     ROUTING → cheapest model meeting each request's quality bar (easy→small, hard→premium/reasoning).
     CACHING → enable prompt/prefix + response caching on repeats.
   compute the BLENDED cost/quality/latency and compare to the best single model.
6. Decide on COST PER RESOLVED TASK; memo (3.07); re-balance on triggers.

12. Hands-On Lab

Goal

Plot candidates on the cost-quality-latency triangle, score them with your weights, then show that routing + caching beats the best single model on blended metrics.

Prerequisites

• 2–3 candidate models; a golden set (Phase 1.07); the cost calculator (Phase 4.04) and latency harness (Phase 1.05).

Steps

1. Measure axes: for each model, get quality (eval score), cost/request (model it), and latency (p50/p95 spike).
2. Prune + score: drop dominated models; normalize axes to 0–1; score with your weights.

def score(m, w):  # m: {"q":0-1,"c":0-1,"l":0-1,"r":0-1,"f":0-1} (higher=better), w sums to 1
    return sum(m[k]*w[k] for k in w)
weights = {"q":0.4,"c":0.3,"l":0.2,"r":0.05,"f":0.05}   # set to YOUR priorities

3. Simulate routing: label your eval items easy/hard; route easy→cheap model, hard→premium; compute blended quality, cost, latency.
4. Add caching: assume a fraction of requests share a prefix; apply the caching discount to cost (and TTFT).
5. Compare: blended (routed + cached) vs the best single model on all three axes; confirm the system wins.
6. Memo the chosen point/route + weights (Phase 3.07).

Expected output

• A scored frontier table, and a blended (routing + caching) result that dominates the best single model on cost/latency at equal quality — quantifying the "escape the triangle" lesson.

Debugging tips

• Routing doesn't help? Either your hard fraction is large (most traffic needs the premium model) or your "easy" model fails the bar — re-check the difficulty split and quality bar.
• A model scores high but feels wrong? Your weights don't match the product — adjust them, not the data.

Extension task

Sweep the hard-traffic fraction (5% → 50%) and plot blended cost — showing routing's value shrinks as more traffic is genuinely hard.

Production extension

Encode the routing rule + caching in the Phase 8 gateway and track blended cost/quality/latency on live traffic.

What to measure

Per-model quality/cost/latency; weighted scores; blended (routed+cached) cost/quality/latency vs best single model; sensitivity to hard fraction.

Deliverables

• A cost-quality-latency table + frontier plot (dominated models dropped).
• A weighted scoring of frontier candidates.
• A routing+caching simulation beating the best single model + memo.

13. Verification Questions

Basic

1. Why can't you maximize cost, quality, and latency simultaneously?
2. What does it mean for a model to be "dominated," and what do you do with it?
3. Why is cost per resolved task the honest metric?

Applied 4. Set reasonable weights for (a) real-time autocomplete and (b) legal analysis. Justify. 5. Explain how routing produces a better blended outcome than any single model.

Debugging 6. Costs are high despite a "cheap" model. Two likely causes (think retries + caching)? 7. Your weighted score picks a model that feels wrong for the product. What's misaligned?

System design 8. Design a routing + caching architecture that targets a quality bar at minimum blended cost/latency.

Startup / product 9. Show investors how routing + caching improve gross margin while holding quality — with the blended-cost logic.

14. Takeaways

1. Every model choice is a cost-quality-latency trade-off — pick a point on the Pareto frontier.
2. Discard dominated models; set weights from product priorities before scoring.
3. Routing beats any single model; caching shifts the whole frontier down-and-left.
4. Optimize cost per resolved task, not $/token.
5. Measure quality (eval), latency (spike), cost (model) — don't guess.
6. Re-balance as priorities and models change.

15. Artifact Checklist

• Cost-quality-latency table + frontier plot (dominated dropped).
• Weighted scoring with product-justified weights.
• Routing + caching simulation beating the best single model.
• Cost-per-resolved-task comparison.
• (Extension) hard-fraction sensitivity sweep.
• Memo of the chosen point/route + weights.

Up: Phase 5 Index · Next: Phase 6 — Local Inference