Cheatsheet 17 — Production Readiness
The pre-launch checklist for an LLM system. Full: Phase 7.10, Phase 14.
Reliability
- Fallbacks across providers/models; retries with backoff; circuit breakers (7.07).
- Timeouts on every call; graceful degradation.
- Rate limits / quotas per user/tenant (noisy-neighbor) (14.04).
- Load-tested; p95/p99 under load within target (12.06).
- SLA defined; status page; DR/BCP.
Cost
- Cost instrumented per request/task/user (15.05).
- Spend caps + alerts; per-tenant budgets (7.09).
- Caching (prompt/prefix/response) where applicable (7.05).
- Routing (cheapest model that meets quality) (8.05).
- Gross margin known and positive.
Quality
- Golden eval set + harness; CI regression gate (12.08).
- Metric per task; retrieval vs generation split for RAG.
- Output validation (schema/grounding) (10.02, 9.08).
Observability
- Traces with correlation IDs across retrieval→model→tools (10.08).
- Latency (p50/p95/p99), cost, error rate dashboards (7.08).
- Audit logs (PII-safe, tamper-evident) (14.06).
- Alerting on anomalies, spend spikes, error spikes.
Security & privacy (Phase 14)
- Trust boundary — model proposes, app executes (14.01, 10.05).
- No secrets in prompts/weights/client/logs; vaulted + scoped + capped keys (14.03).
- Tenant isolation incl. vector index/cache/fine-tunes; leak tests (14.04).
- Guardrails in+out, fail closed; moderation; PII scrubbing (14.05, 14.02).
- Prompt-injection red-team done (direct/indirect/exfil) (14.01).
- Data: retention TTLs, erasure flow, provider DPA/no-train/ZDR (14.02).
Ops
- Runbook (incidents, rollback, on-call) (7.10).
- Versioned prompts/models; canary/rollback for changes.
- Incident-response plan; postmortems.
The 8 numbers to know before launch
p95 latency · cost per resolved task · gross margin · golden-set eval score · error/fallback rate · violation rate (safety) · cache hit rate · throughput (req/s).
Next: 18 — Startup Unit Economics · Full: Phase 7.10