Cost Model and Unit Economics

Phase 15 · Document 05 · Startup Playbook Prev: 04 — MVP Design · Up: Phase 15 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

This is the doc that decides whether your AI startup is a business or a money-incinerator. Traditional SaaS has near-zero marginal cost per user — once built, serving one more customer is almost free, so gross margins run 80–90%. LLM products are different: every single request costs real money (tokens, GPU time), so a naive AI product can have negative gross margins — you lose money on every active user, and growth makes it worse. Founders who don't model unit economics discover this at scale, when it's nearly fatal. Worse, the easy mistake is pricing to compete (matching a rival's price) instead of pricing to margin (covering your real per-unit cost with room to spare). This doc teaches the cost metrics that matter for LLM products, how the levers from the engineering phases (caching, routing, right-sizing, self-hosting) move them, and how to build a unit-economics model that proves you have a viable business.

2. Core Concept

Plain-English primer: every request costs money — know how much

Unit economics = the revenue and cost associated with one unit of your business (a request, a user, a task). The core question: does each unit make or lose money? For LLM products the cost side is dominated by inference — tokens through an API (input + output priced per million tokens) or GPU time if self-hosted (Phase 6/Phase 7). Because that cost is per-request and non-trivial, you must measure it and design around it from day one (03/04).

TRADITIONAL SaaS: marginal cost per user ≈ $0 → 80–90% gross margin "for free"
LLM PRODUCT: every request costs tokens/GPU → margin must be ENGINEERED, or it goes NEGATIVE (lose money per user; growth = bigger losses)

The cost ladder: from token to resolved task

Model cost at increasing levels of aggregation — each answers a different question:

1. Cost per request — tokens in + tokens out × price; the atom. (Output tokens are usually pricier; long contexts/RAG inflate input tokens, Phase 9.07.)
2. Cost per task — a task may take several requests (agent loops, retries, tool calls, Phase 10); this is often much higher than per-request and the one people underestimate.
3. Cost per resolved task — cost per task ÷ success rate; if 1 in 3 tasks fails and retries, your effective cost triples. This is the true unit for outcome-based products (Phase 12.06).
4. Cost per user — cost per task × tasks/user/month; reveals whether a user is profitable at your price.
5. Cost per workspace / per document — for B2B/seat or doc-processing products, aggregate to the billing unit (a workspace's whole team, or per-document for automation).

The discipline: measure at the level you bill at, and at the level that hides surprises (per-resolved-task and per-user catch the agent-loop and retry costs that per-request hides).

Gross margin: the number that defines the business

Gross margin = (revenue − cost of goods sold) ÷ revenue. For LLM products, COGS is dominated by inference (plus retrieval, vector DB, infra). Healthy software margins are 70–80%+; venture investors expect a credible path there (09). A product at 20% margin (or negative) is a fundraising and survival problem. You engineer your way to good margins with the levers below — they're not optional optimizations, they're how the business exists.

The cost levers (this is where Phases 5–8 pay off)

• Token budget / prompt discipline — fewer tokens = lower cost. Trim system prompts, retrieve less but better (Phase 9.07), cap output length, avoid stuffing the whole context.
• Caching — repeated/overlapping requests served from cache cost ~nothing; prompt/prefix caching cuts input cost on shared context (Phase 7.05). High cache-hit workloads transform margins.
• Model routing / right-sizing — use the cheapest model that meets quality per request; route easy cases to small/cheap/local models, hard cases to premium (Phase 5/Phase 8.05/Phase 11.06). The single biggest margin lever for most products.
• Distillation / fine-tuning a small model — replace an expensive model with a cheap fine-tuned/distilled one for a narrow task (Phase 13.04).
• Self-hosting — at high volume, running open-weight models on your own GPUs can beat API pricing — but only past a break-even point (below).

Self-hosting break-even

APIs are pay-per-token (cheap at low volume, no ops); self-hosting is fixed GPU cost (expensive baseline, cheap marginal token) plus engineering/ops (Phase 6/Phase 7). There's a break-even volume: below it, APIs win (you're not utilizing a GPU); above it, self-hosting's lower marginal cost wins — if you can keep the GPUs well-utilized and absorb the ops burden.

APIs:        cost ≈ (tokens × price)         → low fixed cost, scales linearly, no ops    → win at LOW/medium volume
SELF-HOST:   cost ≈ (GPU $/hr + ops)         → high fixed cost, near-flat marginal token  → win at HIGH, steady volume (if GPUs stay utilized)
BREAK-EVEN = volume where self-host total < API total. Below it, self-hosting wastes idle GPU. Include ENGINEERING/ops cost, not just GPU $.

Most startups start on APIs (speed, no ops) and consider self-hosting only once volume is high and steady (Phase 6).

Pricing to margin, not to compete

Set price from cost + target margin + value delivered, not by matching competitors. Three implications:

• Cover your true unit cost (per resolved task, including retries/agent loops) with margin to spare — or each customer loses you money.
• Value-based pricing fits AI — if you save a lawyer 10 hours, you can charge a fraction of that value, decoupling price from token cost (00).
• Beware unlimited/flat plans on usage-driven costs — a few power users can blow your margin; add usage caps, budgets, or usage-based components (Phase 7.09/Phase 8.06).

The unit-economics model (the deliverable)

A simple spreadsheet that, per customer/user, computes: tasks/month × cost per resolved task = monthly cost; vs price = gross margin per customer; plus CAC (customer acquisition cost) and LTV (lifetime value) for the full picture (LTV:CAC ≥ 3 is the SaaS rule of thumb, 09). This model tells you if you have a business — and which lever (price up, cost down via routing/caching, or churn down) to pull.

3. Mental Model

   UNIT ECONOMICS = does ONE unit (request/task/user) make or lose money? LLM cost is dominated by INFERENCE (tokens/GPU) — PER REQUEST, non-trivial.
   ★ unlike SaaS (≈$0 marginal → 80–90% margin free), LLM margin must be ENGINEERED or it goes NEGATIVE (lose $/user; growth = bigger losses).

   COST LADDER (measure at billing level AND where surprises hide):
     per REQUEST (tokens in+out × price; output pricier; RAG/long-context inflate input [9.07])
     → per TASK (multiple requests: agent loops/retries/tools [10] — underestimated)
     → per RESOLVED TASK (÷ success rate; 1/3 fail → 3× cost) = true outcome unit [12.06]
     → per USER (× tasks/user) → per WORKSPACE / per DOCUMENT (billing unit)
   GROSS MARGIN = (rev − COGS)/rev; software-healthy = 70–80%+; investors expect a path there [09]. you ENGINEER it.

   LEVERS (Phases 5–8 pay off): TOKEN BUDGET/prompt discipline [9.07] · CACHING (prefix/prompt) [7.05] · ROUTING/right-sizing (cheapest model that works) [5/8.05/11.06] ·
                                DISTILL/FT a small model [13.04] · SELF-HOST at high volume
   SELF-HOST BREAK-EVEN: APIs (low fixed, linear, no ops) win LOW/med; self-host (high fixed GPU+ops, flat marginal) wins HIGH steady volume IF GPUs utilized. include OPS cost. start on APIs.
   ★ PRICE TO MARGIN not to compete: cover true unit cost (per resolved task) + margin; VALUE-BASED fits AI (charge fraction of value saved); beware UNLIMITED plans (cap/meter [7.09/8.06]).
   MODEL: tasks/mo × cost/resolved-task vs price = margin/customer; + CAC/LTV (LTV:CAC ≥ 3 [09]). tells you which lever to pull.

Mnemonic: every request costs money, so model cost up the ladder (request → task → resolved task → user) — the agent-loop and retry costs hide above per-request. Engineer margin with token budgets, caching, routing, distillation, and (at high volume) self-hosting; then price to margin and value, not to your competitor.

4. Hitchhiker's Guide

What to look for first: what is your cost per resolved task, and what's your gross margin at your price? If you don't know, you don't know if you have a business. Per-resolved-task (not per-request) is the number that catches agent loops and retries.

What to ignore at first: premature self-hosting and micro-optimizing token counts before you have volume. Start on APIs, measure, then pull the routing/caching levers that matter most.

What misleads beginners:

• Thinking like SaaS (zero marginal cost). LLM cost is per-request — margin must be engineered or goes negative.
• Measuring per-request only. Agent loops/retries make per-task and per-resolved-task far higher — measure those (Phase 10).
• Pricing to compete. Matching a rival's price below your cost = losing money per customer — price to margin/value.
• Unlimited plans on usage costs. Power users blow the margin — cap/meter (Phase 7.09).
• Premature self-hosting. Idle GPUs are pure loss below break-even — start on APIs (Phase 6).
• Ignoring the levers. One premium model for everything wrecks margins — route/cache/right-size (Phase 8.05).

How experts reason: they instrument cost from day one (03/04), model the full ladder (especially per-resolved-task and per-user), target 70–80%+ gross margin, engineer it with routing/caching/right-sizing/distillation (Phase 5/Phase 7.05/Phase 13.04), price to margin and value (not competitors), cap/meter usage, and consider self-hosting only past break-even. They track LTV:CAC for the full business picture.

What matters in production: a known cost per resolved task, healthy gross margin at the actual price, levers (routing/caching) actively managing cost, usage caps preventing margin blowouts, and a unit-economics model that proves viability.

How to debug/verify: compute cost per resolved task (incl. retries/loops) and margin at your price; if margin is thin/negative, pull a lever (route to cheaper model, add caching, raise price). Check whether a power user on an unlimited plan is unprofitable. Verify self-hosting actually beats APIs at your volume including ops.

Questions to ask: what's cost per request/task/resolved-task/user? gross margin at my price? which lever moves it most (routing? caching? right-sizing?)? am I pricing to margin/value or to compete? are usage caps in place? have I passed self-hosting break-even (with ops)?

What silently bankrupts you: SaaS-thinking, per-request-only measurement, pricing to compete, unlimited plans, premature self-hosting, and one premium model for everything.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Every LLM request costs real money (tokens/GPU) — unlike near-zero-marginal-cost SaaS, margin must be engineered or it goes negative.
• Model the cost ladder: per request → per task → per resolved task → per user → per workspace/document — measure at your billing level and where surprises hide (loops/retries).
• Cost per resolved task is the true outcome unit — divide by success rate; failures/retries multiply cost (Phase 12.06).
• Target 70–80%+ gross margin — investors expect a credible path there (09); you engineer it.
• The levers: token budget, caching, routing/right-sizing, distillation, self-hosting — routing/caching are usually the biggest (Phase 8.05/Phase 7.05).
• Self-hosting has a break-even volume — APIs win at low/medium volume (no ops); self-hosting wins at high steady volume if GPUs stay utilized and you include ops cost (Phase 6).
• Price to margin and value, not to compete — cover the true unit cost with margin; value-based pricing fits AI; cap/meter usage to prevent power-user margin blowouts (Phase 7.09).
• The unit-economics model (tasks × cost/resolved-task vs price, plus LTV:CAC ≥ 3) tells you if you have a business and which lever to pull.

9. Observations from Real Systems

• Negative-margin AI products are a recurring story — startups that priced to compete (or offered unlimited plans) discovered per-resolved-task cost exceeded revenue at scale (Phase 12.06).
• Routing + caching routinely cut costs by large factors — sending easy requests to small/cheap models and caching shared context is the standard margin fix (Phase 8.05/Phase 7.05).
• Agent products are cost-traps if measured per-request — multi-step loops and retries make per-resolved-task many times higher; teams that missed this mispriced (Phase 10).
• Value-based pricing is the AI advantage — products that charge a fraction of the expensive human work they replace enjoy fat margins decoupled from token cost (00).
• Self-hosting saves at scale but burns startups early — idle/underutilized GPUs below break-even are pure loss; most start on APIs and migrate selectively (Phase 6).

10. Common Misconceptions

11. Engineering Decision Framework

UNIT ECONOMICS (prove it's a business, then engineer margin):
 1. MEASURE the ladder: per request (tokens×price) → per TASK (loops/retries [10]) → per RESOLVED TASK (÷ success [12.06]) → per USER → per workspace/doc.
 2. COMPUTE gross margin at your actual price. Target 70–80%+ [09]. Negative/thin → pull levers.
 3. LEVERS (biggest first): ROUTING/right-sizing (cheapest model that works [5/8.05]) · CACHING (prefix/prompt [7.05]) · TOKEN budget [9.07] · DISTILL/FT small model [13.04].
 4. PRICE to MARGIN + VALUE (not to compete): cover true unit cost + margin; value-based where you replace expensive work; CAP/METER usage [7.09/8.06] — avoid unlimited.
 5. SELF-HOST only past BREAK-EVEN (high steady volume, utilized GPUs, incl. ops) [6]; otherwise APIs.
 6. MODEL the business: tasks/mo × cost/resolved-task vs price = margin/customer; track LTV:CAC ≥ 3 [09]. Decide: price up / cost down / churn down.

12. Hands-On Lab

Goal

Build a unit-economics model for your MVP (04): compute cost per resolved task and gross margin at your price, then apply levers to reach a healthy margin.

Prerequisites

• Your MVP design (04); provider token prices; a rough success rate and tasks/user estimate.

Steps

1. Cost per request: estimate input+output tokens for the core workflow × provider price (Phase 9.07 for RAG token inflation).
2. Cost per task → resolved task: multiply by requests/task (agent loops/retries, Phase 10); divide by success rate for cost per resolved task (Phase 12.06).
3. Per user + margin: × tasks/user/month = monthly cost/user; vs your price = gross margin per user.
4. Apply levers: model the margin improvement from routing easy cases to a cheap model (Phase 8.05) and caching shared context (Phase 7.05); recompute.
5. Pricing check: is the price to-margin/value, or to-compete? Add a usage cap or usage-based component to protect against power users (Phase 7.09).
6. Self-host break-even (stretch): estimate the volume at which self-hosting (GPU + ops) beats APIs (Phase 6).

Expected output

A unit-economics spreadsheet: cost per resolved task, gross margin at your price before/after levers, a pricing decision (to-margin/value), usage protections, and a self-hosting break-even estimate — proving (or fixing) the viability of the business.

Debugging tips

• If per-request looks fine but the business doesn't work, you ignored loops/retries — use per-resolved-task.
• If margin is negative, route to a cheaper model and/or raise price before anything else.

Extension task

Add CAC and LTV and compute LTV:CAC; model how churn and expansion change it (09).

Production extension

Wire real cost instrumentation (per request/task/user) and routing/caching into the product (Phase 8.05/Phase 7.05); enforce usage caps at the gateway (Phase 8.06).

What to measure

Cost per request/task/resolved-task/user, gross margin before/after levers, price-to-margin vs to-compete, break-even volume, LTV:CAC.

Deliverables

• A unit-economics model (cost ladder → margin per user).
• A before/after-levers margin comparison (routing + caching).
• A pricing decision + usage protections + a self-hosting break-even estimate.

13. Verification Questions

Basic

1. Why do LLM products differ from traditional SaaS on margins?
2. What's the difference between cost per request, per task, and per resolved task?
3. What gross margin should you target and why?

Applied 4. What are the main cost levers, and which usually moves margin most? 5. When does self-hosting beat APIs?

Debugging 6. Your per-request cost looks fine but you're losing money. What did you miss? 7. A few power users are destroying your margin. What do you change?

System design 8. Build a unit-economics model for an agent product (loops, retries, routing, caching, pricing).

Startup / product 9. Why price to margin and value rather than to your competitor, and how does value-based pricing help AI products?

14. Takeaways

1. Every LLM request costs money — margin must be engineered, or it goes negative (unlike near-zero-marginal-cost SaaS).
2. Model the cost ladder; cost per resolved task is the true unit — it catches agent loops and retries that per-request hides (Phase 12.06).
3. Engineer 70–80%+ gross margin with routing, caching, right-sizing, and distillation (Phase 8.05/Phase 7.05/Phase 13.04).
4. Self-host only past break-even (high steady volume, utilized GPUs, ops included); otherwise APIs (Phase 6).
5. Price to margin and value, not to compete; cap/meter usage — and use the unit-economics model (with LTV:CAC) to decide which lever to pull (09).

15. Artifact Checklist

• A cost ladder (per request → resolved task → user → workspace/doc).
• A gross-margin calc at your actual price.
• A levers analysis (routing + caching before/after).
• A pricing decision (to-margin/value) + usage caps.
• A self-hosting break-even estimate + LTV:CAC.

Up: Phase 15 Index · Next: 06 — Moat and Defensibility