Business and Pricing Terms — Providers, Routing, and Unit Economics

Phase 1 · Document 08 · LLM Vocabulary and Mental Models Prev: 07 — Evaluation Terms · Next: 09 — Complete Glossary

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

LLM features live or die on unit economics. The difference between a profitable AI product and one that bleeds money is whether the team understands input vs output pricing, cached-token discounts, who the provider actually is (vs the lab that made the model), what a rate limit or quota will do under load, and how routing/fallbacks keep you reliable and cheap. This is also the vocabulary of every pricing page, status page, and procurement conversation. Misreading it produces the two classic disasters: a margin-negative feature and a 3 a.m. outage when your sole provider rate-limits you. This document connects Phase 1 to Phase 5 selection, Phase 8 gateways, and the Phase 16 startup playbook.

2. Core Concept

Who's who in the supply chain

• Lab / creator: the org that trained the model (OpenAI, Anthropic, Google DeepMind, Meta, Mistral, Qwen).
• Provider / host: the service that serves it over an API. May be the lab itself, a cloud (Bedrock, Vertex, Azure), or a third party. The same model can have many providers at different prices/latencies.
• Aggregator / router: a service exposing many providers behind one API (OpenRouter-style).
• Gateway / proxy: an abstraction layer (often self-hosted, LiteLLM-style) for routing, keys, budgets, logging across providers.

Separating lab from provider is the first move in cost/latency/policy decisions.

Pricing model

Almost all commercial APIs price per token, input and output separately, quoted per 1M tokens:

request_cost = input_tokens/1e6 × input_price
             + output_tokens/1e6 × output_price
             + reasoning_tokens/1e6 × output_price        (if a reasoning model)
             − cached_input_tokens × (input_price − cached_input_price)/1e6   (discount)

• Output is typically 2–5× input price (decode is the expensive phase — see 05).
• Cached input price (prompt caching) can be 75–90% cheaper for repeated prefixes.
• Reasoning tokens are billed (usually at output rates) even though hidden.
• Image/audio/video tokens may be priced on a different schedule.

Limits, reliability, and routing

• Rate limit: requests or tokens per minute you may send (RPM/TPM). Exceed it → HTTP 429.
• Quota: a longer-horizon allotment (per day/month, often by tier).
• Spend limit / budget: a cap on cost (per key/user/project) that you enforce — essential for abuse and runaway-cost control.
• SLA (contractual uptime/latency guarantee) vs SLO (your internal target).
• Fallback: automatically retry on a different provider/model on error, rate limit, or timeout.
• Load balancing: spread traffic across providers/keys for throughput and resilience.
• Provider / region / data-policy routing: choose the endpoint by cost, latency, region (data residency), or compliance.

Lifecycle and identity

• Versioned model ID (gpt-4o-2024-08-06) is pinned and reproducible; an alias (gpt-4o, ...-latest) silently moves to newer versions — convenient but a reproducibility/quality risk.
• Deprecation: providers retire models on a schedule; pin versions and watch deprecation notices.
• License & weights availability: governs self-hosting, fine-tuning, and commercial use (open weights ≠ open source — see 02).
• Data retention / training-on-your-data: whether the provider stores or trains on your inputs — a hard gate for sensitive data (Phase 14).

Unit economics

cost_per_request → cost_per_user → cost_per_resolved_task
gross_margin = (price_to_customer − cost_to_serve) / price_to_customer

Levers: prompt caching, model routing (cheap model for easy tasks), retrieval (fewer input tokens), output caps, and self-hosting break-even at high volume.

3. Mental Model

LAB (makes it) ─┬─► PROVIDER A  ($, latency, region)  ┐
                ├─► PROVIDER B  ($, latency, region)  ├─► ROUTER/GATEWAY ─► your app
                └─► PROVIDER C  (self-hosted)         ┘     (routing, fallback,
                                                             budgets, logging)

COST = input_tokens×in$ + output_tokens×out$ (+reasoning) − cache_discount
       └ output is 2–5× input · caching saves 75–90% on repeats · route easy tasks to cheap models

RELIABILITY = rate limits + quotas + SLA  →  needs fallback + load balancing
IDENTITY    = pin versioned IDs (not aliases) · watch deprecation · check retention/license
MARGIN      = (price − cost_to_serve)/price   ← the number investors and your CFO care about

4. Hitchhiker's Guide

What to look for first: input price, output price, cached-input price, rate/quota limits, data-retention policy, and whether the ID is pinned or an alias.

What to ignore at first: marginal price differences between near-equivalent providers — reliability, latency, and policy usually matter more than a few cents/1M.

What misleads beginners:

• Quoting "the price" as one number — input and output differ, and output dominates many workloads.
• Assuming the lab is the only provider (you may get the same model cheaper/faster elsewhere).
• Using an alias in production and being surprised when behavior shifts under you.
• Ignoring rate limits until traffic spikes and everything 429s.

How experts reason: they model cost per resolved task, not per token; pin versioned IDs; design fallback + budgets from day one; and pick providers on the bundle of price, latency, limits, region, and retention — not price alone.

What matters in production: enforced per-key budgets, fallback chains across providers, rate-limit-aware retries with backoff, version pinning, and a data-retention posture that matches your data sensitivity.

Debug/verify: reconcile your token logs against the provider invoice; load-test against rate limits; confirm a fallback actually triggers by simulating a 429/timeout.

Questions to ask providers: Input/output/cached prices? RPM/TPM and how to raise them? Data retention and training-on-inputs policy? Region/residency options? Deprecation schedule and notice period? SLA terms?

What silently gets expensive/unreliable: unbounded reasoning tokens; aliases drifting to pricier/slower versions; no budget caps (abuse/runaway loops); single-provider dependence; non-English/JSON token inflation hitting margin.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Pricing is per token, input and output separately; output is typically 2–5× input.
• The same model often has multiple providers at different prices, latencies, and regions.
• Cached input can be 75–90% cheaper — caching stable prefixes is a top margin lever.
• Reasoning tokens are billed (usually at output rates) even though hidden — budget them.
• Aliases drift; pin versioned model IDs in production for reproducibility and stable cost.
• Rate limits cause 429s; production needs backoff, fallback, and budget caps.
• Data retention/training-on-inputs policies vary and gate sensitive-data use.
• Unit economics scale with caching, routing, and retrieval far more than with raw model price.

9. Observations from Real Systems

• models.dev explicitly separates Lab, Providers, Price, Weights, Release, and Updated — the supply-chain split in catalog form (detailed in Phase 4).
• OpenRouter lists multiple providers per model with per-provider price/latency and supports provider routing + fallbacks — the aggregator pattern.
• LiteLLM (self-hosted proxy) centralizes virtual keys, per-key budgets, rate limits, and fallbacks — the gateway pattern you build in Phase 8/9.
• OpenAI/Anthropic publish tiered rate limits and prompt caching discounts; Anthropic and others offer zero-retention / no-training options for sensitive data.
• VS Code BYOK / Cursor let you bring your own provider key so spend and data path stay on your account.
• Cloud resellers (Bedrock, Vertex, Azure OpenAI) serve the same models with different regions, SLAs, and compliance — a provider choice, not a model choice.

10. Common Misconceptions

11. Engineering Decision Framework

Choosing a provider for a chosen model:
  hard gates first → data retention/residency/compliance acceptable? license ok?
  then optimize    → price (in+out+cache) × expected mix, latency (p95), rate limits/quota, SLA.
  never single-home a critical path → define a FALLBACK provider.

Designing for cost (margin):
  1. Cache stable prefixes (system prompts, few-shot) → 75–90% input savings on repeats.
  2. Route easy tasks to a cheaper model; reserve premium/reasoning for hard ones.
  3. Retrieve less; cap max_tokens; avoid needless reasoning.
  4. At high steady volume → evaluate self-hosting break-even (Phase 5/6).

Designing for reliability:
  pin versioned IDs · per-key budgets · rate-limit backoff · fallback chain · watch deprecations.

12. Hands-On Lab

Goal

Build a cost-and-margin calculator and a tiny routing rule that sends easy requests to a cheap model and hard ones to a premium model, then compute the blended cost.

Prerequisites

• Python 3.10+; pricing numbers for 2–3 models (from a pricing page or models.dev).

Setup

pip install openai

Steps

# 1. Cost model with caching + reasoning
def request_cost(in_tok, out_tok, in_p, out_p, cached_tok=0, cached_p=None, reason_tok=0):
    cached_p = in_p if cached_p is None else cached_p
    billable_in = in_tok - cached_tok
    return (billable_in/1e6*in_p + cached_tok/1e6*cached_p
            + out_tok/1e6*out_p + reason_tok/1e6*out_p)

# Example: gpt-4o-mini-ish vs gpt-4o-ish prices ($/1M)
CHEAP   = dict(in_p=0.15, out_p=0.60)
PREMIUM = dict(in_p=2.50, out_p=10.00)

# 2. Naive routing: long/complex → premium, else cheap
def route(prompt):
    hard = len(prompt) > 400 or any(k in prompt.lower() for k in ("prove","optimize","debug","architecture"))
    return ("premium", PREMIUM) if hard else ("cheap", CHEAP)

# 3. Blended cost over a workload
workload = [("Summarize this note", 300, 150)]*800 + [("Debug and optimize this service architecture "*20, 3000, 1200)]*200
total = 0.0
for prompt, in_tok, out_tok in workload:
    _, price = route(prompt)
    total += request_cost(in_tok, out_tok, **price)
print(f"Blended daily cost: ${total:.2f}")

# 4. Caching impact: re-run with a cached 200-token system prefix at 10% price
cached_total = sum(request_cost(in_tok, out_tok, cached_tok=200, cached_p=route(p)[1]['in_p']*0.1, **route(p)[1])
                   for p, in_tok, out_tok in workload)
print(f"With prefix caching: ${cached_total:.2f}")

Expected output

• A blended daily cost; a visibly lower number once prefix caching is applied — quantifying two core margin levers.

Debugging tips

• Numbers wildly off? Recheck $/1M units and that you're billing input/output separately.
• Routing always premium? Tune the heuristic; in production, route on a cheap classifier or task type, not string length alone.

Extension task

Add a third "local/self-hosted" tier at $0/token but with a fixed daily GPU cost; compute the break-even volume where self-hosting beats the API.

Production extension

Pull live prices from models.dev (see Phase 4 lab) and parameterize the calculator; emit a per-model and blended margin report.

What to measure

Blended cost per day; caching savings %; routing mix (cheap vs premium); self-host break-even volume.

Deliverables

• A cost/margin calculator.
• A before/after caching + routing savings table.
• A break-even note for self-hosting.

13. Verification Questions

Basic

1. What's the difference between a lab and a provider?
2. Why is output usually more expensive than input?
3. What's the difference between a versioned model ID and an alias, and which belongs in production?

Applied 4. A request has 2,000 input (200 cached at 10%) and 800 output tokens at $2.50/$10.00 per 1M. Compute the cost. 5. Your feature charges $0.05/task and costs $0.04/task to serve. What's the gross margin, and name two levers to improve it.

Debugging 6. Traffic spikes cause intermittent 429s. Describe a resilient handling strategy. 7. A model's outputs changed overnight with no deploy on your side. What's the most likely cause?

System design 8. Design provider selection + fallback for a healthcare app with data-residency requirements. What gates come first?

Startup / product 9. Pitch how your unit economics improve as you scale, citing caching, routing, retrieval, and self-host break-even.

14. Takeaways

1. Separate lab from provider — the same model has many providers, prices, and policies.
2. Pricing is per token, input/output separate; output is 2–5×, and caching saves 75–90% on repeats.
3. Pin versioned IDs, watch deprecations, and check data retention before sending sensitive data.
4. Rate limits and quotas demand fallback, backoff, and budgets — design them from day one.
5. Model cost per resolved task, not per token; optimize margin with caching, routing, retrieval, output caps.
6. At high steady volume, self-hosting break-even can flip the economics.

15. Artifact Checklist

• Code: cost + margin calculator (with caching/reasoning).
• Code: simple routing rule + blended-cost report.
• Analysis: caching + routing savings table.
• Break-even note: self-host vs API crossover volume.
• Decision record: provider selection + fallback for one workload (with data-policy gates).
• Cheat card: the cost formula and margin levers.

Next: 09 — Complete Glossary