Observability

Phase 7 · Document 08 · Production Serving Prev: 07 — Routing and Fallbacks · Up: Phase 7 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

You cannot operate, debug, or cost-control what you cannot see. Every other doc in this phase produces a signal — TTFT/TPOT (01), KV-cache utilization and queue depth (04), prefix-cache hit rate (05), fallback rate (07), per-request cost (09) — and observability is what turns those signals into dashboards, SLOs, and alerts so you catch a latency regression, a capacity ceiling, a cost spike, or a quality drop before customers do. LLM serving adds wrinkles ordinary web observability doesn't have: latency that's two-phase (TTFT vs TPOT), cost that's per-token and variable, and "errors" that include quality failures a 200 OK won't reveal. This doc makes you fluent in the LLM-specific observability stack.

2. Core Concept

Plain-English primer: the three pillars + the LLM extras

Classic observability has three pillars, and LLM serving adds two LLM-specific concerns on top:

• Metrics — numeric time series (counters, gauges, histograms): TTFT, TPOT, RPS, error rate, KV utilization, cost/request. Cheap to store, great for dashboards/alerts. Tools: Prometheus + Grafana.
• Logs — structured per-request records (JSON): request ID, model, provider, tokens in/out, duration, error, user/project. Great for forensics. Tools: Loki/Elasticsearch/your warehouse.
• Traces — the path of one request across spans (gateway → router → provider → stream): shows where time went. Tools: OpenTelemetry → Jaeger/Tempo.
• (LLM extra) Token/cost accounting — usage per request → billing/cost dashboards (09).
• (LLM extra) Quality/eval observability — online quality signals (judge scores, JSON validity, refusal rate, user thumbs) — because a request can return 200 OK and be wrong (Phase 12).

The metrics that matter (and why percentiles, not averages)

Latency is two-phase (Phase 1.05, Phase 2.07): TTFT (felt responsiveness) and TPOT (streaming smoothness). Track both, and track them as percentiles — p50/p95/p99 — never just the mean. Averages hide the tail: a p50 of 300ms with a p99 of 8s means 1% of users have a terrible experience, and averages are dominated by easy requests while your SLO lives in the tail. Use histograms (Prometheus histogram/_bucket) so percentiles are computable.

Core serving metrics:

LATENCY:     ttft_seconds{p50,p95,p99} · tpot_seconds{...} · request_duration{...}
THROUGHPUT:  generation_tokens_per_second · requests_per_second · concurrent_requests
CAPACITY:    gpu_cache_usage_perc (KV ceiling [04]) · num_requests_waiting (backpressure)
             · gpu_utilization · preemptions
QUALITY/ERR: error_rate{type} · fallback_rate [07] · timeout_rate
             · prefix_cache_hit_rate [05] · (online) judge_score / json_valid_rate
COST:        input/output/cache tokens · cost_per_request · cost_per_resolved_task [09]

Self-hosted engines expose much of this for free: vLLM's /metrics gives gpu_cache_usage_perc, num_requests_running/waiting, TTFT/TPOT histograms, token counters (01).

Logs and traces for LLM requests

A good structured log per request (sampled, with prompts redacted/PII-scrubbed — Phase 14):

{"request_id":"...","ts":"...","user":"u_123","project":"p_9","model":"claude-...",
 "provider":"anthropic","route":"smart","fallback_used":false,
 "input_tokens":1203,"cache_read_tokens":11500,"output_tokens":420,
 "ttft_ms":310,"duration_ms":4200,"cost_usd":0.0134,"status":"ok","finish_reason":"stop"}

Traces tie the spans together: gateway.auth → router.select → provider.call (TTFT here) → stream → usage. When p95 spikes, the trace shows whether it was routing, the provider, or the stream.

SLOs and alerting

Turn metrics into promises: an SLO (e.g., "p95 TTFT < 1.5s and availability ≥ 99.9% over 30 days") with an error budget (the allowed 0.1% failure). Alert on SLO burn and leading indicators — rising num_requests_waiting/gpu_cache_usage_perc (capacity), rising fallback_rate (degrading primary, 07), cost/request creep (09) — not on every blip. Alert on symptoms users feel (p99 latency, error rate) plus causes you can act on (KV ceiling, queue depth).

3. Mental Model

   THREE PILLARS                LLM EXTRAS
   METRICS  (Prometheus/Grafana)   TOKEN/COST accounting → $/request, $/resolved-task [09]
   LOGS     (structured JSON)      QUALITY/EVAL signals → judge score, JSON valid, refusal [Phase 12]
   TRACES   (OpenTelemetry)
        │
   LATENCY is TWO-PHASE: TTFT (felt) + TPOT (smoothness) — track p50/p95/p99, NOT averages
   CAPACITY signals: gpu_cache_usage_perc (KV ceiling [04]) · num_requests_waiting (backpressure)
        │
   SLO (p95 TTFT < X, avail ≥ Y) + ERROR BUDGET → alert on BURN + LEADING INDICATORS
   key LLM truth: a 200 OK can still be WRONG → you must observe QUALITY, not just status codes

Mnemonic: metrics+logs+traces, plus tokens/cost and quality. Percentiles not averages; alert on user-felt symptoms + actionable causes (KV, queue, fallback, cost).

4. Hitchhiker's Guide

What to look for first: p95/p99 TTFT + TPOT, error rate, KV utilization + queue depth, and cost/request. Those five tell you if you're fast, healthy, at capacity, and affordable.

What to ignore at first: vanity averages and raw token totals without percentiles/attribution. A mean latency number is nearly useless operationally.

What misleads beginners:

• Averaging latency. The tail (p99) is the user experience and the SLO; means hide it.
• Treating 200 OK as success. LLMs fail by being wrong/irrelevant/hallucinated — you need quality signals, not just status codes (Phase 12).
• No cost attribution. Without per-user/project token+cost, you can't bill, budget, or find the expensive endpoint (09).
• Logging raw prompts. PII/secret leakage — redact and sample (Phase 14).
• Alert fatigue. Alerting on every spike trains people to ignore alerts; alert on SLO burn + leading indicators.

How experts reason: they instrument TTFT/TPOT as histograms, attribute tokens+cost per request/user/project, scrape engine capacity signals (gpu_cache_usage_perc, num_requests_waiting), trace the request path, add online quality signals, define SLOs with error budgets, and alert on burn + leading indicators. They redact/sample logs and keep cardinality sane.

What matters in production: p95/p99 SLO adherence, capacity headroom (KV/queue), fallback rate trend, cost/request and cost/resolved-task trend, and quality signals — all on dashboards owned by on-call with runbook links (10).

How to debug an incident: start at the SLO dashboard (which symptom?) → traces (which span?) → engine metrics (KV ceiling? queue? provider?) → logs (which requests/users?). The capacity signature is gpu_cache_usage_perc≈1 + rising num_requests_waiting (04).

Questions to ask vendors/gateways: do you expose token usage, TTFT/TPOT, cache hit rate, and error/fallback metrics? OpenTelemetry support? per-key/project attribution? webhook/export for billing?

What silently gets expensive/unreliable: no p99 visibility (tail pain unseen), no cost attribution (budget surprises), no quality observability (silent regressions), and high-cardinality labels blowing up your metrics bill.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Observability = metrics + logs + traces, plus token/cost and quality for LLMs.
• Latency is two-phase (TTFT, TPOT) and must be tracked as p50/p95/p99 — averages hide the tail.
• A 200 OK can be wrong — observe quality (judge scores, JSON validity, refusals), not just status (Phase 12).
• gpu_cache_usage_perc (KV) + num_requests_waiting are the capacity/backpressure signals (04).
• Attribute tokens + cost per request/user/project — required for billing, budgets, and finding spend (09).
• Fallback rate is a primary-health signal — a rising trend means the primary is degrading (07).
• Define SLOs with error budgets; alert on burn + leading indicators, not every blip.
• Redact/sample logs (PII/secrets) and control label cardinality (Phase 14).

9. Observations from Real Systems

• vLLM/TGI expose Prometheus metrics out of the box — gpu_cache_usage_perc, queue depth, TTFT/TPOT histograms — the basis of any self-hosted dashboard (01).
• Gateways (OpenRouter/LiteLLM) emit per-request usage/cost and latency, and LiteLLM integrates with Prometheus/OTel — the observability seam for managed-API stacks (Phase 8).
• LLM-specific observability tools (Langfuse, Helicone, Phoenix, OpenLLMetry) add prompt/trace/eval views and cost dashboards tuned for LLM apps.
• OpenTelemetry GenAI semantic conventions are standardizing how token usage, model, and latency are recorded in traces.
• The classic capacity incident is read straight off the dashboard: gpu_cache_usage_perc≈1 + rising num_requests_waiting → scale or shed load (04, 10).

10. Common Misconceptions

11. Engineering Decision Framework

INSTRUMENT a serving stack:
 1. METRICS (Prometheus, histograms): TTFT, TPOT, RPS, errors, gpu_cache_usage_perc,
    num_requests_waiting, prefix_cache_hit_rate, cost/request — all with p50/p95/p99.
 2. LOGS (structured, redacted, sampled): request_id, model, provider, route, tokens(in/out/cache),
    ttft, duration, cost, status, finish_reason, user/project.
 3. TRACES (OTel): gateway→router→provider→stream spans; attach token/cost attributes.
 4. QUALITY (online): judge score / JSON validity / refusal rate / user feedback.   [Phase 12]
 5. SLOs + error budgets (p95 TTFT, availability); alert on BURN + leading indicators
    (KV ceiling, queue depth, fallback rate, cost creep).
 6. Dashboards owned by on-call, linked from the runbook.                            [10]

12. Hands-On Lab

Goal

Stand up a minimal Prometheus + Grafana dashboard for an LLM endpoint with p95/p99 TTFT, KV utilization, cost/request, and an SLO alert.

Prerequisites

• A vLLM endpoint exposing /metrics (01); Docker for Prometheus/Grafana; a load script (03 lab).

Setup

# Prometheus scrape config (prometheus.yml): target vLLM's /metrics
# scrape_configs: [{job_name: vllm, static_configs: [{targets: ['host:8000']}]}]
docker run -p 9090:9090 -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
docker run -p 3000:3000 grafana/grafana

Steps

1. Scrape: confirm Prometheus is collecting vllm:gpu_cache_usage_perc, vllm:num_requests_waiting, and the TTFT/TPOT histograms.
2. Percentile panels: in Grafana, build p50/p95/p99 TTFT from the histogram buckets (histogram_quantile(0.95, ...)); add KV utilization and queue-depth gauges.
3. Generate load: run the concurrency sweep from 03; watch p95 TTFT and gpu_cache_usage_perc rise together past the knee.
4. Cost panel: instrument your client/proxy to emit per-request input/output/cache tokens and computed cost_usd; chart cost/request (and a rough cost/resolved-task if you have a success signal) (09).
5. SLO alert: define "p95 TTFT < 1.5s"; add a Grafana/Prometheus alert that fires when the 5-min p95 exceeds it, and trigger it by overloading the endpoint.
6. Quality signal (stretch): log JSON-validity (or a tiny judge score) per request and chart the rate to show a "200 OK but wrong" failure (Phase 12).

Expected output

A live dashboard with p50/p95/p99 TTFT, KV utilization, queue depth, cost/request, and a firing SLO alert under load — plus (stretch) a quality-rate panel.

Debugging tips

• Percentiles look wrong → you averaged instead of using histogram_quantile on buckets.
• No engine metrics → wrong scrape target/port; curl /metrics directly.

Extension task

Add OpenTelemetry traces through a small proxy so a single slow request shows which span (routing vs provider vs stream) dominated.

Production extension

Wire alerts to on-call, add per-user/project cost attribution and budget-burn alerts (09), and link each alert to the runbook.

What to measure

p50/p95/p99 TTFT/TPOT; KV utilization + queue depth; cost/request; SLO burn; (stretch) quality rate.

Deliverables

• A Grafana dashboard (latency percentiles, KV/queue, cost).
• A defined SLO + alert that fires under load.
• A structured log schema (redacted) + (stretch) a quality-signal panel.

13. Verification Questions

Basic

1. Name the three observability pillars and the two LLM-specific extras.
2. Why track p95/p99 instead of average latency?
3. Which two engine metrics signal you're at capacity?

Applied 4. A 200-OK response can still be a failure — how do you observe that? 5. Design the structured log fields you'd record per request (and what to redact).

Debugging 6. p99 TTFT spikes under load but p50 is fine. Where do you look, in order? 7. Cost/request crept up 30% with stable traffic. Three things to check.

System design 8. Define an SLO + error budget + alerting strategy for an interactive LLM endpoint.

Startup / product 9. Which observability signals prove your unit economics and reliability to a customer or investor?

14. Takeaways

1. Observe metrics + logs + traces, plus tokens/cost and quality — LLMs need the extras.
2. Latency is two-phase (TTFT/TPOT); track p50/p95/p99, never just averages.
3. gpu_cache_usage_perc + num_requests_waiting are your capacity signals; fallback rate flags a degrading primary.
4. A 200 OK can be wrong — instrument quality, and attribute cost per user/project.
5. Define SLOs with error budgets; alert on burn + leading indicators, and redact/sample logs.

15. Artifact Checklist

• A dashboard with p50/p95/p99 TTFT/TPOT, KV utilization, queue depth, cost/request.
• A defined SLO + error budget + alert that fires under load.
• A structured, redacted log schema with token/cost attribution.
• (Optional) a trace showing the slow span and an online quality signal.
• Alerts linked to the runbook.

Up: Phase 7 Index · Next: 09 — Cost Controls