Usage Metering

Phase 8 · Document 06 · LLM Gateways Prev: 05 — Routing Engine · Up: Phase 8 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Usage metering is how the gateway turns every request into a billable, attributable, governable event — the foundation of cost visibility, per-tenant budgets and rate limits, chargeback, and the unit economics that decide whether your product is profitable. It's the feature that makes a gateway more than a proxy: without metering you can't answer "what does this customer cost?", can't stop a runaway tenant, and can't bill. Phase 7.09 covered cost control as a serving discipline; this doc is the gateway component that records usage, computes cost from the registry, enforces budgets/quotas, and exposes the data the dashboard and your finance team need. Get it wrong and you either lose money silently or bill customers incorrectly — both fatal.

2. Core Concept

Plain-English primer: record every request, price it, enforce limits

Metering has three jobs:

1. Record a structured usage event per request (who, what model/provider, tokens, latency, cost, status).
2. Compute cost from the tokens and the registry's prices (Phase 4.04).
3. Enforce budgets, quotas, and rate limits — before the call (pre-flight check) and after (record actuals).

The token counts are ground truth from the provider's usage (mapped by the adapter); in streaming they arrive in the final chunk, so metering must capture them there (07, Phase 7.06).

The usage event (the atom of metering)

CREATE TABLE usage_events (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  request_id TEXT, ts TIMESTAMPTZ DEFAULT NOW(),
  user_id TEXT, project_id TEXT, api_key_id TEXT,     -- attribution dimensions
  model_id TEXT, provider TEXT, route TEXT,            -- what served it (route from [05])
  input_tokens INT, output_tokens INT,
  cached_input_tokens INT DEFAULT 0,                   -- prefix-cache savings [7.05]
  reasoning_tokens INT DEFAULT 0,                      -- billed thinking [2.09]
  input_cost NUMERIC(12,8), output_cost NUMERIC(12,8), total_cost NUMERIC(12,8),
  ttft_ms INT, duration_ms INT,
  finish_reason TEXT, fallback_used BOOL, error_type TEXT
);

Cost is the Phase 7.09/4.04 arithmetic, priced from the registry:

def cost(u, m):   # m = registry row [04]
    return ((u.input_tokens - u.cached_input_tokens) * m.input_price
            + u.cached_input_tokens * m.cached_input_price
            + (u.output_tokens + u.reasoning_tokens) * m.output_price) / 1e6

Budgets, quotas, rate limits (the enforcement side)

• Budget = a spend ceiling (per user/project/key, per period). Pre-flight: estimate the request's cost and reject/downgrade if it would exceed the budget; post-flight: add the actual cost to the running total (Phase 7.09).
• Quota = a count ceiling (requests or tokens per period).
• Rate limit = a velocity ceiling (RPM/TPM) to protect backends and prevent abuse — typically a token-bucket in Redis, per key.

# pre-flight (in the gateway, before routing/calling)
if spend_mtd(key) + estimate_cost(req) > budget(key):
    raise HTTP(429, "Budget exceeded")        # or downgrade to a cheaper model [05]
if not rate_limiter.allow(key):               # token bucket (Redis)
    raise HTTP(429, "Rate limit")
# ... route [05] + call ...
# post-flight
record(usage_event); increment_spend(key, usage_event.total_cost)

These tie to the routing engine's budget filter — over-budget candidates are filtered out, or the request is downgraded/rejected.

Attribution and chargeback

Every event carries dimensions — user, project, API key, model, provider, route — so you can aggregate cost per user / per workspace / per feature and do chargeback (bill internal teams or external customers for their usage). This is the data behind unit economics: cost per user, per workspace, per resolved task → gross margin (Phase 7.09, Phase 15.05).

Accuracy: reconcile against invoices

Your metered cost is an estimate from registry prices; the provider's invoice is truth. Reconcile them periodically — drift means a stale registry price (04), an unmetered path (e.g., uncaptured streaming usage, 07), or missed reasoning/cache tokens. Unreconciled metering quietly loses money or over-bills customers.

Reliability of the metering pipeline

Metering must be durable and non-blocking: emit events to a queue/async sink so a metering hiccup doesn't fail user requests, but ensure events aren't lost (at-least-once) or double-counted. Aggregate to per-period spend tables for fast budget checks. (Spend counters in Redis for hot-path budget checks; durable events in Postgres for billing.)

3. Mental Model

   every request → USAGE EVENT {who · model/provider/route · tokens(in/out/cached/reasoning)
                                · cost · ttft/duration · finish/fallback/error}
        cost = tokens × REGISTRY prices [04]   (streaming: usage in FINAL chunk [07])
   ENFORCE:  pre-flight budget/quota/rate-limit (reject/downgrade) → ... → post-flight record + increment
        budget filter feeds the ROUTER [05];  rate limit = token bucket (Redis)
   ATTRIBUTE by user/project/key/feature → chargeback → $/user, $/workspace, $/resolved-task → MARGIN [15]
   ACCURACY: reconcile metered cost vs provider INVOICE; pipeline durable + non-blocking

Mnemonic: record every request as a priced, attributed event; enforce budgets/quotas/rate-limits pre- and post-flight; reconcile against invoices. Metering = the gateway's billing + governance + unit-economics brain.

4. Hitchhiker's Guide

What to look for first: a complete usage event (with attribution dimensions + token breakdown) and pre-flight budget + rate-limit enforcement. Those give cost visibility and abuse protection immediately.

What to ignore at first: elaborate chargeback billing models and fancy dashboards — first get accurate, attributed events and budget enforcement; aggregation/UI follow.

What misleads beginners:

• Missing streaming usage. Token counts arrive in the final chunk — if the streaming proxy doesn't capture it, metering is blind (Phase 7.06).
• Forgetting cached + reasoning tokens. Cached input is cheaper (Phase 7.05); reasoning tokens are billed (Phase 2.09) — both must be in the event and the cost formula.
• Only post-flight enforcement. Without a pre-flight check, a runaway request/tenant blows the budget before you record it.
• No reconciliation. Metered cost drifts from invoices (stale prices, missed paths) and you lose money or over-bill.
• Blocking the request on metering. A metering outage shouldn't fail user traffic — make it async + durable.

How experts reason: they emit a rich, attributed usage event per request (durably, async), price from the registry, enforce pre- and post-flight budgets + token-bucket rate limits, aggregate to fast spend tables, and reconcile against invoices on a schedule. They treat metering as financial-grade: no lost or double-counted events.

What matters in production: event completeness/accuracy (incl. streaming, cached, reasoning tokens), reconciliation delta vs invoices, budget/rate-limit correctness, pipeline durability (no loss/double-count), and attribution granularity for chargeback.

How to debug/verify: reconcile a day's metered cost against the provider invoice; check streaming requests recorded usage; verify a budget rejects over-limit and a rate limit throttles; confirm events aren't lost under load.

Questions to ask: does it capture streaming + cached + reasoning tokens? pre-flight budget/rate-limit? per-user/project/key attribution? invoice reconciliation? durable, non-blocking pipeline?

What silently gets expensive/unreliable: uncaptured streaming usage (under-billing), missing cached/reasoning tokens (wrong cost), post-flight-only budgets (overruns), lost/double-counted events, and unreconciled drift.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Metering = record (event) + compute (cost from registry) + enforce (budgets/quotas/rate-limits).
• Token counts are ground truth from the provider usage; in streaming they're in the final chunk — capture them (07).
• Include cached + reasoning tokens in the event and cost formula (Phase 7.05, Phase 2.09).
• Enforce budgets pre-flight (reject/downgrade) and post-flight (record) — feeds the router's budget filter (05).
• Rate limits are velocity (RPM/TPM) — typically a Redis token bucket per key.
• Attribution dimensions (user/project/key/route) enable chargeback and unit economics (Phase 15).
• Reconcile metered cost vs provider invoices to catch drift (stale prices, missed paths).
• The pipeline must be durable + non-blocking — no lost/double-counted events, no failing user traffic on a metering hiccup.

9. Observations from Real Systems

• LiteLLM records spend per key/user/team, enforces budgets + RPM/TPM, and exposes usage — production metering you can reuse (02).
• OpenRouter returns usage/cost per request and tracks per-key spend — the aggregator's metering surface (01).
• Billing incidents usually trace to uncaptured streaming usage or stale registry prices — the two classic metering bugs (07, 04).
• Reasoning-model bills surprise teams when reasoning tokens aren't metered (Phase 2.09).
• Enterprises rely on per-team chargeback from gateway metering to allocate AI spend across departments (09).

10. Common Misconceptions

11. Engineering Decision Framework

METERING:
 1. EVENT: emit a rich usage event per request (attribution dims + token breakdown + latency + status),
    durably + async (no lost/double-count, no blocking traffic).
 2. COST: price from the REGISTRY (input/output/cached/reasoning); capture streaming usage from final chunk. [04,07]
 3. ENFORCE: pre-flight budget (reject/downgrade) + token-bucket rate-limit (Redis); post-flight record + increment. [05,7.09]
 4. ATTRIBUTE: aggregate by user/project/key/route → chargeback → $/user, $/workspace, $/resolved-task. [15]
 5. RECONCILE: schedule metered-vs-invoice checks; alert on drift (stale price / missed path).
 6. EXPOSE: feed the admin dashboard + cost alerts. [08,7.08]

12. Hands-On Lab

Goal

Build a metering layer: record priced, attributed usage events (incl. streaming + cached tokens), enforce a pre-flight budget and a token-bucket rate limit, and produce a chargeback report + an invoice reconciliation.

Prerequisites

• The registry from 04 for prices; SQLite/Postgres; (optional) Redis; a real or mock endpoint.

Steps

1. Event + cost: for each request, capture tokens (incl. cached_input_tokens; for streaming, parse the final chunk) and compute cost from registry prices; insert a usage_events row with attribution dims.
2. Pre-flight budget: maintain per-key month-to-date spend; before a call, estimate cost and reject (429) or downgrade if over budget; after, increment actual spend (05/Phase 7.09).
3. Rate limit: implement a per-key token bucket (RPM); fire a burst and confirm throttling.
4. Chargeback report: aggregate total_cost by project_id and by user_id for a period — the data finance needs.
5. Reconciliation: compare summed metered cost against a (mock or real) provider invoice figure; investigate any delta (e.g., flip a registry price to simulate drift and show the reconciliation catches it).
6. Streaming check: run a streaming request and confirm its usage was recorded (proving final-chunk capture, 07).

Expected output

A metering layer producing accurate, attributed, priced events; working pre-flight budget + rate limit; a per-project chargeback report; and a reconciliation that flags injected price drift.

Debugging tips

• Streaming requests show 0 tokens → not capturing the final-chunk usage [07].
• Reconciliation off → stale registry price, or missing cached/reasoning tokens in the formula.

Extension task

Add reasoning-token metering for a reasoning model and show its effect on cost (Phase 2.09); compute cost per resolved task if you have a success signal (Phase 5.09).

Production extension

Move spend counters to Redis for hot-path budget checks, events to a durable async sink (Postgres/queue), and wire chargeback + drift alerts into the dashboard and observability.

What to measure

Event completeness/accuracy (incl. streaming/cached/reasoning), budget + rate-limit enforcement, chargeback totals, reconciliation delta.

Deliverables

• A usage-event schema + recorder (priced from the registry).
• Pre-flight budget + token-bucket rate limit enforcement.
• A chargeback report + an invoice reconciliation (with injected drift caught).

13. Verification Questions

Basic

1. What three jobs does usage metering do?
2. Where do token counts come from in a streaming response?
3. Why include cached and reasoning tokens?

Applied 4. Write the pre-flight budget check and explain reject vs downgrade. 5. How do attribution dimensions enable chargeback and unit economics?

Debugging 6. Streaming requests record zero usage. Cause and fix. 7. Metered cost is 15% below the provider invoice. Three likely causes.

System design 8. Design a durable, non-blocking metering pipeline with pre-flight budgets, rate limits, and chargeback.

Startup / product 9. Which metering outputs prove your gross margin and let you bill customers accurately?

14. Takeaways

1. Metering records a priced, attributed usage event per request, computes cost from the registry, and enforces budgets/quotas/rate-limits.
2. Capture streaming (final-chunk), cached, and reasoning tokens or your cost is wrong.
3. Enforce budgets pre- and post-flight (feeds the router's budget filter); rate-limit by velocity (token bucket).
4. Attribution → chargeback → unit economics ($/user, $/workspace, $/resolved-task).
5. Reconcile against invoices and keep the pipeline durable + non-blocking.

15. Artifact Checklist

• A usage-event schema + recorder priced from the registry.
• Streaming/cached/reasoning token capture in the event.
• Pre-flight budget + token-bucket rate limit enforcement.
• A chargeback report (per project/user).
• An invoice reconciliation that catches injected price drift.

Up: Phase 8 Index · Next: 07 — Streaming Proxy