Cost Controls

Phase 7 · Document 09 · Production Serving Prev: 08 — Observability · Up: Phase 7 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

LLM cost is usage-based and unbounded by default — every token costs money, traffic is spiky, and a single bug (a runaway generation loop, a leaked API key, an abandoned stream, a prompt that ballooned) can turn a $500/month bill into a $50,000 surprise overnight. For a product, token cost is your cost of goods sold (COGS): it sets your gross margin and whether the business works at all. Cost controls are the guardrails — max_tokens, budgets, caching, routing, batch discounts — that keep spend bounded, predictable, and aligned to value. This is where the engineering of Phases 5–8 meets the economics of the startup playbook: the same levers that control cost (routing, caching) are the ones that make the unit economics work.

2. Core Concept

Plain-English primer: where the money goes

Cost per request is simple arithmetic (Phase 4.04, what-happens §10):

cost = input_tokens × input_price
     + output_tokens × output_price        ← output is usually 3–5× the input price
     + reasoning_tokens × output_price      ← hidden thinking tokens are billed [Phase 2.09]
     − cache_read_savings                   ← prefix caching discount [05]

Two facts dominate: output tokens are the expensive ones (and the ones you most control via max_tokens), and the honest metric is cost per resolved task, not per token (Phase 5.09) — a "cheap" model that retries or needs human fixups can cost more per finished job than a pricier one that nails it.

The cost-control toolkit (cheapest guardrails first)

1. max_tokens on every request. The single most important guard: caps runaway generation. Without it, a model can loop to its full output limit and bill you for thousands of unwanted tokens. Set it to the real need per endpoint.
2. Per-user / per-project budgets + spend limits. Track cumulative spend (from observability usage) and reject/throttle when over budget — protects against abuse, bugs, and runaway tenants.
3. Prompt/prefix caching. Reuse the stable prefix → cached input is ~75–90% cheaper (05). The biggest lever for repeated system prompts/RAG/chat.
4. Difficulty-based routing. Easy → cheap/small model, hard → premium — blended cost far below always-premium (07, Phase 5.09).
5. Batch endpoints (~50% discount). Most providers offer ~50% off for async batch processing — use for non-interactive work (offline classification, eval runs, doc processing).
6. Right-size context and reasoning. Trim/retrieve instead of stuffing (what-happens §3); reserve reasoning/thinking budget for genuinely hard steps (Phase 2.09) — thinking tokens are billed and add up fast.
7. Self-hosting above break-even. At high steady volume, owned GPUs at high utilization beat per-token pricing — but only past the break-even point (Phase 5.02, and utilization economics from 03/04).

Budgets and admission control (code shape)

# Per-request cap
resp = client.chat.completions.create(model=m, messages=msgs, max_tokens=1000)

# Budget enforcement at the gateway (pre-flight)
spend = usage_store.month_to_date(user.id)          # from observability usage [08]
if spend + estimate_cost(req) > user.budget:
    raise HTTPException(429, "Budget exceeded")      # reject or downgrade to a cheaper model

# Post-flight: record actual usage for billing/attribution
usage_store.record(user.id, resp.usage, model=m, cost=price(resp.usage, m))

Cost as COGS: the unit-economics view

For a product, roll per-request cost up to the units that matter: cost per user, per workspace, per document, per resolved task (Phase 15.05). Gross margin = (price − COGS) / price; if a seat sells for $20/mo and costs $14 in tokens, you have a margin problem that routing + caching must fix. Instrument cost from day one so you can answer "what does a customer cost?" — and so the levers above translate directly into margin.

3. Mental Model

   cost = in×price_in + out×price_out(3–5× higher) + reasoning×price_out − cache_savings
          └ output + reasoning are the expensive, controllable parts ┘
   honest metric: COST PER RESOLVED TASK (retries/fixups in), not $/token  [5.09]

   GUARDRAILS (cheap → strategic):
     max_tokens (always)  →  budgets/spend limits  →  prefix caching [05]  →
     difficulty routing [07]  →  batch endpoints (~50% off)  →  trim/retrieve + reasoning budget  →
     self-host above break-even [5.02]

   COGS view: $/request → $/user, $/workspace, $/resolved-task → GROSS MARGIN  [Phase 15]

Mnemonic: cap every request (max_tokens), budget every tenant, cache the prefix, route by difficulty, batch the offline work — and measure cost per resolved task as your COGS.

4. Hitchhiker's Guide

What to look for first: is max_tokens set on every path, and do you have per-tenant budgets + per-request cost attribution (08)? Those three prevent the catastrophic surprises.

What to ignore at first: squeezing the last 5% via exotic quant or self-hosting — first set the guardrails and turn on caching/routing, which capture most of the savings.

What misleads beginners:

• Optimizing $/token. The real metric is $/resolved task — a cheap model with retries can cost more (Phase 5.09).
• Forgetting output/reasoning dominate. Input is cheap; output and thinking tokens are where the bill lives — cap and reserve them (Phase 2.09).
• No max_tokens. The classic runaway-cost bug — a loop or verbose model bills to the cap.
• Ignoring abandoned streams. No cancel-on-disconnect = paying for tokens nobody reads (06).
• Self-hosting too early. Below break-even, idle GPUs cost more than API calls (Phase 5.02).

How experts reason: they treat cost as COGS with a target gross margin, set max_tokens + budgets everywhere, lean on caching + difficulty routing for the bulk of savings, move offline work to batch endpoints, right-size context/reasoning, and only self-host past a measured break-even. They attribute cost per user/project and alert on cost/request drift (08).

What matters in production: cost/request and cost/resolved-task trends, per-tenant spend vs budget, cache hit rate (cost lever), reasoning-token share, and a hard ceiling (global + per-tenant) so no single bug/abuser can run away.

How to debug a cost spike: check max_tokens (runaway?), prompt length (context ballooned? cache busted? 05), reasoning-token share, traffic volume (legit vs abuse), and whether a route accidentally moved to a pricier model (07).

Questions to ask providers: input/output/cache/reasoning pricing? batch discount + SLA? spend caps / hard limits on the account? usage webhooks for real-time attribution? (Phase 4.04)

What silently gets expensive: missing max_tokens, lost cache hits (volatile-first prompts), creeping context/reasoning, abandoned streams, idle self-hosted GPUs, and no per-tenant ceiling.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Cost is usage-based and unbounded by default — guardrails are mandatory, not optional.
• Output (and reasoning) tokens dominate cost (output ≈ 3–5× input price); max_tokens is the #1 guard.
• The honest metric is cost per resolved task, not $/token (Phase 5.09).
• Prefix caching (~75–90% off cached input) and difficulty routing are the biggest levers (05, 07).
• Batch endpoints give ~50% off for non-interactive workloads.
• Per-tenant budgets + cost attribution (from observability) prevent runaway spend (08).
• Abandoned streams cost money without cancel-on-disconnect (06).
• Self-host only past a measured break-even; idle GPUs are pure loss (Phase 5.02).
• Token cost is your COGS — it sets gross margin (Phase 15).

9. Observations from Real Systems

• Gateways (OpenRouter/LiteLLM) implement budgets, spend limits, and per-key/project cost tracking — cost control as a gateway feature (Phase 8).
• The classic cost incident is a missing max_tokens + a verbose/looping model, or a prompt that ballooned and busted the cache (05).
• High-volume bots cut 20–40% by enabling prefix caching alone, and more via difficulty routing (Phase 5.09).
• Offline pipelines (eval runs, bulk classification, doc processing) routinely use batch endpoints for ~50% savings.
• Reasoning-model bills surprise teams because hidden thinking tokens are billed — routing reasoning only to hard requests controls it (Phase 2.09, Phase 5.04).

10. Common Misconceptions

11. Engineering Decision Framework

CONTROL COST (apply in order):
 1. max_tokens on EVERY request (per-endpoint real need).
 2. Per-user/project BUDGETS + spend limits; reject/downgrade over budget (needs attribution [08]).
 3. PREFIX CACHING on (stable-prefix-first) — biggest repeat-workload lever.            [05]
 4. DIFFICULTY ROUTING: easy→cheap, hard→premium; reasoning only where needed.          [07, 2.09]
 5. BATCH ENDPOINTS (~50% off) for non-interactive work.
 6. RIGHT-SIZE context (trim/retrieve) and reasoning budget.                            [§3 what-happens]
 7. SELF-HOST only past a measured break-even at high utilization.                       [5.02]
 8. MEASURE cost/request and cost/resolved-task; alert on drift; report margin.          [08, Phase 15]

12. Hands-On Lab

Goal

Build a cost model + budget guardrail, then show how caching and difficulty routing cut blended cost and how max_tokens caps runaway spend.

Prerequisites

• Usage data from a real or simulated endpoint (08); provider price table; pip install openai.

Setup

PRICES = {  # $/1M tokens (illustrative)
  "premium": {"in": 3.0, "out": 15.0, "cache_read": 0.30},
  "cheap":   {"in": 0.15,"out": 0.60, "cache_read": 0.015},
}
def cost(u, m):
    p = PRICES[m]
    return (u["in"]*p["in"] + u["out"]*p["out"] + u.get("cache_read",0)*p["cache_read"]) / 1e6

Steps

1. Per-request cost + attribution: record input/output/cache tokens per request and compute cost; aggregate by user/project.
2. Budget guardrail: enforce a per-user monthly budget — reject (or downgrade to cheap) when projected spend exceeds it; test with a heavy user.
3. max_tokens cap: run a prompt that tends to over-generate with max_tokens unset vs set; show the cost difference (and finish_reason="length" when capped).
4. Caching savings: simulate a workload with a fixed 2k-token system prompt; compute cost with cache-read pricing on the prefix vs full input price — quantify the ~75–90% prefix savings (05).
5. Difficulty routing: classify a mixed workload easy/hard; route easy→cheap, hard→premium; compute blended cost vs always-premium. Then compute cost per resolved task if the cheap model has a higher retry rate — show when routing actually wins (Phase 5.09).
6. COGS roll-up: convert to cost/user and compute gross margin at a hypothetical seat price.

Expected output

A cost dashboard/table showing: per-tenant spend + budget enforcement; max_tokens savings; caching savings; blended cost from routing (and cost/resolved-task); and a gross-margin figure.

Debugging tips

• Routing "saves" but margin worsens → the cheap model's retries raise cost/resolved-task; re-check the quality bar.
• Cache savings look tiny → prefix isn't actually shared (volatile-first prompt) [05].

Extension task

Add a batch-endpoint path for offline jobs and show the ~50% reduction vs the interactive endpoint.

Production extension

Wire budgets + cost attribution + drift alerts into the gateway and observability; add a global + per-tenant hard ceiling.

What to measure

Cost/request, cost/resolved-task, per-tenant spend vs budget, max_tokens savings, cache savings, blended routing cost, gross margin.

Deliverables

• A cost model + per-tenant budget guardrail.
• A max_tokens / caching / routing savings comparison.
• A cost-per-resolved-task and gross-margin figure (COGS memo).

13. Verification Questions

Basic

1. Why is max_tokens the most important cost guard?
2. Why is cost per resolved task more honest than $/token?
3. Which tokens dominate cost, and which are easy to forget?

Applied 4. Rank the cost levers by typical impact for a high-volume fixed-prompt chatbot. 5. When does difficulty routing not save money (think retries)?

Debugging 6. The bill tripled with flat traffic. List five things to check. 7. Caching is enabled but savings are negligible. Why?

System design 8. Design per-tenant budgets + ceilings + cost attribution for a multi-tenant SaaS.

Startup / product 9. A seat sells for $20/mo and costs $14 in tokens. Which levers fix the margin, and how would you prove the new margin?

14. Takeaways

1. LLM cost is unbounded by default — guardrails are mandatory; max_tokens first.
2. Output + reasoning tokens dominate; the honest metric is cost per resolved task.
3. Prefix caching + difficulty routing are the biggest levers; batch endpoints give ~50% off offline work.
4. Per-tenant budgets + cost attribution prevent runaway spend; cancel abandoned streams.
5. Token cost is your COGS — instrument from day one and manage it to a target gross margin.

15. Artifact Checklist

• A per-request cost model with per-user/project attribution.
• A budget guardrail (reject/downgrade over limit) + global/per-tenant ceiling.
• A savings comparison (max_tokens, caching, routing).
• A cost-per-resolved-task + gross-margin figure.
• Cost drift alerts wired into observability.

Up: Phase 7 Index · Next: 10 — Production Runbook