Latency and Cost Evals
Phase 12 · Document 06 · Evaluation Prev: 05 — Code Evals · Up: Phase 12 Index
Table of Contents
- Why This Matters
- Core Concept
- Mental Model
- Hitchhiker's Guide
- Warmup Readings
- Deep Readings and External References
- Key Terms
- Important Facts
- Observations from Real Systems
- Common Misconceptions
- Engineering Decision Framework
- Hands-On Lab
- Verification Questions
- Takeaways
- Artifact Checklist
1. Why This Matters
Quality is only one axis — a model that's accurate but too slow or too expensive is unshippable. Latency and cost are evaluation dimensions, not afterthoughts, and they belong in the same harness as quality so model/prompt/routing decisions optimize the whole trade-off (Phase 5.09, 08). These are the objective, measurable axes (unlike subjective quality, 02) — but they have their own traps: latency must be measured as percentiles under realistic load (not a single happy-path call), and cost must be per resolved task (not per token). Getting these evals right is what lets you make defensible cost-quality-latency decisions instead of being surprised by a slow p99 or a runaway bill in production.
2. Core Concept
Plain-English primer: measure the non-quality axes properly
Two objective axes to evaluate alongside quality:
- Latency — how fast (the user-experience axis).
- Cost — how much (the unit-economics axis, your COGS — Phase 7.09).
They're measurable exactly (no judge needed), but only meaningful when measured correctly.
Latency: two phases, and percentiles under load
LLM latency is two-phase (Phase 1.05, Phase 2.07):
- TTFT (time to first token) — felt responsiveness; what streaming optimizes (Phase 7.06). The metric for autocomplete/chat snappiness.
- TPOT (time per output token) — streaming smoothness; with output length sets total latency (
TTFT + TPOT × output_tokens).
Two rules for measuring it:
- Percentiles, not averages — track p50/p95/p99, not the mean. The tail (p99) is the user experience and the SLO; averages hide it (Phase 7.08).
- Under realistic load — a single idle-server call is the floor. Real latency depends on concurrency (batching, queueing — Phase 7.03), so load-test at expected traffic. Latency that's fine at batch-1 can blow the SLO at batch-50.
Cost: per resolved task, not per token
Cost per request is arithmetic (Phase 4.04, Phase 7.09):
cost = in_tokens×in_price + (out + reasoning)×out_price − cache_savings
But the honest metric is cost per resolved task, not per token or per request (Phase 5.09): a "cheap" model that needs retries, more steps, or human fixups can cost more per finished job than a pricier model that nails it first try. So in eval, divide total cost by successful tasks (using the task's success metric, 03–05). Include reasoning tokens (Phase 2.09) and caching (Phase 7.05) — both materially shift cost.
Throughput (the serving-side axis)
For serving/capacity decisions (Phase 7), evaluate throughput (aggregate tokens/sec or requests/sec under concurrency) and the throughput-vs-latency curve (the knee, Phase 7.03) — single-stream tok/s ≠ production capacity. This drives "how many users per GPU" and self-hosting economics (Phase 5.02).
These belong in the eval harness (alongside quality)
The point of measuring latency/cost in eval (not just in prod monitoring) is to decide: when comparing models/prompts/routes (08), capture quality + latency (p50/p95) + cost/resolved-task for each candidate, so you optimize the weighted trade-off (Phase 5.09) rather than quality alone. The harness should emit all three per candidate.
Offline + online (and the gate)
- Offline (in eval): measure latency/cost per candidate on the golden set; gate changes that blow a latency SLO or cost budget even if quality holds.
- Online: monitor p95/p99 latency and cost/request + cost/resolved-task trends in production (Phase 7.08), alerting on drift (a creeping cost/request signals bigger prompts, lost cache, or a route to a pricier model — Phase 7.09).
The trap: optimizing the wrong cost
The classic mistake is optimizing $/token and shipping a cheaper model that lowers task-resolution, raising cost per resolved task (more retries) and hurting quality. Always evaluate cost at the task level, tied to the success metric (Phase 5.09).
3. Mental Model
three axes: QUALITY (subjective, judge [02]) + LATENCY (objective) + COST (objective) — eval ALL THREE [5.09]
LATENCY two-phase: TTFT (felt, streaming [7.06]) + TPOT (smoothness) → total = TTFT + TPOT×out_tokens [2.07]
★ measure PERCENTILES (p50/p95/p99, not avg) UNDER REALISTIC LOAD (concurrency/batching [7.03]) — batch-1 = floor only
COST: in×price + (out+reasoning)×price − cache_savings → ★ honest metric = COST PER RESOLVED TASK (not $/token) [5.09]
include reasoning tokens [2.09] + caching [7.05]; cheap-but-retries can cost MORE per finished job
THROUGHPUT (serving): aggregate tok/s under concurrency + throughput-vs-latency KNEE [7.03] → users/GPU
put latency+cost in the HARNESS [08] alongside quality; OFFLINE gate (SLO/budget) + ONLINE p95/cost trend [7.08]
Mnemonic: latency and cost are objective eval axes — measure latency as p50/p95/p99 under load (TTFT + TPOT), and cost per resolved task (not $/token). Put both in the harness with quality; gate on SLO/budget; monitor trends.
4. Hitchhiker's Guide
What to look for first: are you measuring latency percentiles under realistic concurrency and cost per resolved task (not $/token)? Those two corrections fix most latency/cost eval mistakes.
What to ignore at first: micro-optimizing single-call latency. Establish p95 under load and cost/resolved-task first; tune later.
What misleads beginners:
- Average latency. The tail (p99) is the UX and SLO; averages hide it (Phase 7.08).
- Batch-1 latency. It's the floor — real latency needs load testing under concurrency (Phase 7.03).
- $/token optimization. The honest metric is cost per resolved task; cheap-but-retries can cost more (Phase 5.09).
- Forgetting reasoning/cache. Reasoning tokens inflate cost (Phase 2.09); caching cuts it (Phase 7.05) — include both.
- Quality-only model choice. Capture latency + cost in the harness too, or you'll ship something unshippable (08).
How experts reason: they treat latency and cost as first-class eval axes, measure p50/p95/p99 under realistic load (TTFT + TPOT), compute cost per resolved task (with reasoning + cache), evaluate throughput for serving, put all three in the harness (08), gate on SLO/budget, and monitor trends online (Phase 7.08).
What matters in production: p95/p99 TTFT/TPOT (SLO adherence), cost/request and cost/resolved-task trends, throughput/capacity (users per GPU), and alerts on latency/cost drift (Phase 7.09).
How to debug/verify: if "fast in testing, slow in prod" → you measured batch-1, not under load (load-test). If cost surprises → check per-token vs per-resolved-task, reasoning-token share, cache hit rate, and whether a route moved to a pricier model (Phase 7.09).
Questions to ask: latency as p95/p99 under load (not avg/batch-1)? TTFT and TPOT separately? cost per resolved task (incl. reasoning + cache)? throughput for capacity? latency/cost in the selection harness + gated?
What silently gets expensive/unreliable: average/batch-1 latency (hidden tail/load failures), $/token thinking (retries cost more), ignored reasoning/cache (wrong cost), and quality-only selection (unshippable latency/cost).
5. Warmup Readings
| Title | Why to read it | What to extract | Difficulty | Time |
|---|---|---|---|---|
| Phase 1.05 — Serving Terms | TTFT/TPOT/percentiles | the latency axes | Beginner | 20 min |
| Phase 5.09 — Cost-Quality-Latency | Cost per resolved task | the honest metric | Beginner | 20 min |
| Phase 7.08 — Observability | Percentiles + cost trends | p95/p99, cost attribution | Beginner | 20 min |
| Phase 7.03 — Continuous Batching | Latency under load | the throughput knee | Beginner | 20 min |
6. Deep Readings and External References
| Title | URL | Why it matters | Read first | Lab connection |
|---|---|---|---|---|
| Artificial Analysis | https://artificialanalysis.ai/ | Latency/throughput/price plotted | TTFT, tok/s, price | Benchmarking |
| vLLM benchmarking | https://docs.vllm.ai/en/latest/contributing/benchmarks.html | Throughput/latency under load | concurrency sweep | Load test |
| Phase 4.04 — Pricing Pages | (curriculum) | Cost arithmetic | in/out/cache pricing | Cost model |
| Phase 7.09 — Cost Controls | (curriculum) | Cost per resolved task | budgets, $/task | Cost eval |
| Google SRE — SLOs | https://sre.google/books/ | Percentile SLOs | latency SLO | Gate design |
7. Key Terms
| Term | Simple meaning | Technical meaning | Why it matters | Where it appears | How to use it |
|---|---|---|---|---|---|
| TTFT | Time to first token | Latency to first output | Felt responsiveness | latency | Streaming [7.06] |
| TPOT | Per-token time | Decode latency/token | Streaming smoothness | latency | × output → total |
| p50/p95/p99 | Percentile latency | Tail latency | UX + SLO | metrics | Not averages |
| Under load | Realistic concurrency | Batched/queued latency | Real-world latency | load test | Not batch-1 |
| Cost/request | Per-call $ | Σ token×price − cache | Per-call cost | cost | Include reasoning |
| Cost/resolved-task | Honest cost | $ per successful task | True economics | cost [5.09] | Optimize this |
| Throughput | Aggregate speed | tok/s or req/s under load | Capacity | serving [7.03] | Users per GPU |
| SLO | Latency promise | p95/p99 target | The gate | ops | Gate on it |
8. Important Facts
- Latency and cost are objective eval axes — measure them in the harness alongside quality, not as afterthoughts (08, Phase 5.09).
- Latency is two-phase: TTFT (felt) + TPOT (smoothness); total ≈
TTFT + TPOT × output_tokens(Phase 2.07). - Measure latency as p50/p95/p99, not averages — the tail is the UX and the SLO (Phase 7.08).
- Measure latency under realistic load (concurrency/batching) — batch-1 is the floor, not production (Phase 7.03).
- The honest cost metric is cost per resolved task, not $/token — cheap-but-retries can cost more (Phase 5.09).
- Include reasoning tokens and caching in cost (Phase 2.09/Phase 7.05).
- Throughput (under concurrency) ≠ single-stream tok/s — it sets capacity/users-per-GPU (Phase 7.03).
- Gate offline on SLO/budget; monitor p95/p99 + cost trends online (Phase 7.08/7.09).
9. Observations from Real Systems
- Artificial Analysis plots models on latency (TTFT/tok-s) × price × quality — the cost-quality-latency frontier as a live dashboard (Phase 5.09).
- The classic "fast in dev, slow in prod" incident is batch-1 measurement vs production concurrency (Phase 7.03).
- The classic cost surprise is $/token thinking — a cheaper model that retries more raised cost/resolved-task (Phase 5.09).
- Reasoning-model bills shock teams when reasoning tokens aren't counted (Phase 2.09); caching routinely cuts cost 20–40% (Phase 7.05).
- Production teams alert on cost/request drift — a creeping value flags bigger prompts, lost cache, or a pricier route (Phase 7.09).
10. Common Misconceptions
| Misconception | Reality |
|---|---|
| "Average latency is fine" | Track p95/p99; the tail is the UX/SLO |
| "Measure latency once (batch-1)" | Load-test under concurrency — batch-1 is the floor |
| "Optimize $/token" | Optimize cost per resolved task |
| "Reasoning/cache don't matter for cost" | They materially shift cost — include both |
| "Quality is the only thing to eval" | Latency + cost are first-class axes |
| "Single-stream tok/s = capacity" | Throughput under concurrency sets capacity |
11. Engineering Decision Framework
EVALUATE LATENCY + COST (with quality, in the harness [08]):
1. LATENCY: measure TTFT + TPOT → total; report p50/p95/p99 UNDER REALISTIC LOAD (concurrency [7.03]) — not avg/batch-1.
2. COST: in×price + (out+reasoning [2.09])×price − cache [7.05]; divide by SUCCESSFUL tasks → COST PER RESOLVED TASK [5.09].
3. THROUGHPUT (serving): aggregate tok/s under concurrency + the latency knee → capacity/users-per-GPU [7.03].
4. PUT all three (quality + p95 latency + cost/resolved-task) per candidate in the selection harness [08, 5.09].
5. GATE offline: block changes that breach a latency SLO or cost budget (even if quality holds).
6. MONITOR online: p95/p99 + cost/request + cost/resolved-task trends; alert on drift [7.08/7.09].
| Use case | Latency emphasis | Cost emphasis |
|---|---|---|
| Autocomplete | TTFT/p95 (extreme) [11.05] | per-call (fires constantly) |
| Chat | TTFT + total p95 | per-request |
| Agent | total p95 + steps | cost per resolved task |
| Batch | (relaxed) | cost/task (batch discount [7.09]) |
12. Hands-On Lab
Goal
Measure latency (p50/p95 under load) and cost per resolved task for 2 models, and show why batch-1 latency and $/token mislead.
Prerequisites
Steps
- TTFT + TPOT: for each model, on a decode-heavy prompt, measure TTFT and TPOT; compute total latency.
- Percentiles under load: fire 1, 8, 32 concurrent requests; record p50/p95 TTFT. Show p95 at concurrency ≫ batch-1 (the load lesson, Phase 7.03).
- Cost/request: compute per-request cost (incl. reasoning tokens if any, minus cache savings) (Phase 7.09).
- Cost per resolved task: run the task set, compute task-resolution (the success metric); divide total cost by successful tasks → cost/resolved-task. If a cheaper model retries/fails more, show its cost/resolved-task is higher despite lower $/token (Phase 5.09).
- Combine: tabulate quality + p95 latency + cost/resolved-task per model — the inputs to the selection harness (08).
Expected output
A per-model table: quality, p50/p95 TTFT (batch-1 vs under load), cost/request, cost/resolved-task — demonstrating that batch-1 latency and $/token can pick the wrong model.
Debugging tips
- p95 looks fine but prod is slow → you didn't test under concurrency.
- Cheaper model "wins" on $/token but loses on cost/resolved-task → it retries/fails more; trust cost/resolved-task.
Extension task
Sweep concurrency to plot the throughput-vs-latency knee (Phase 7.03); add cache and show the cost/latency drop (Phase 7.05).
Production extension
Wire p95/p99 + cost/request + cost/resolved-task into dashboards with SLO/budget alerts (Phase 7.08/7.09); add latency/cost gates to the selection harness (08).
What to measure
TTFT/TPOT, p50/p95 (batch-1 vs load), cost/request, cost/resolved-task, throughput knee; per-model comparison.
Deliverables
- A per-model latency (p50/p95 under load) + cost-per-resolved-task table.
- A batch-1-vs-load latency comparison.
- A $/token-vs-cost/resolved-task demonstration.
13. Verification Questions
Basic
- What are TTFT and TPOT, and how do they combine into total latency?
- Why measure p95/p99 instead of average latency?
- Why is cost per resolved task the honest metric?
Applied 4. Why must latency be measured under realistic load, not batch-1? 5. What two cost components do people forget (and which direction does each push)?
Debugging 6. "Fast in testing, slow in prod." Cause and fix. 7. A cheaper model raised your bill. How is that possible?
System design 8. Design latency + cost evals that feed the model-selection harness with gates on SLO/budget.
Startup / product 9. Why are cost per resolved task and p95 latency the numbers that decide unit economics and UX?
14. Takeaways
- Latency and cost are objective eval axes — measure them in the harness with quality (08).
- Latency = TTFT + TPOT; report p50/p95/p99 under realistic load, not averages/batch-1.
- Cost per resolved task (incl. reasoning + cache) is the honest metric — not $/token (Phase 5.09).
- Throughput under concurrency sets capacity (users/GPU), not single-stream tok/s (Phase 7.03).
- Gate offline on SLO/budget; monitor p95/p99 + cost trends online (Phase 7.08/7.09).
15. Artifact Checklist
- A TTFT + TPOT measurement and total-latency calc.
- p50/p95 latency under load (vs batch-1).
- Cost per resolved task (incl. reasoning + cache).
- A $/token-vs-cost/resolved-task comparison.
- Latency/cost feeding the selection harness + SLO/budget gates.
Up: Phase 12 Index · Next: 07 — Safety and Policy Evals