Production Runbook

Phase 7 · Document 10 · Production Serving Prev: 09 — Cost Controls · Up: Phase 7 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

This is the capstone of Phase 7: the operational discipline that turns the components you've built (engine, batching, caching, streaming, routing, observability, cost controls) into a service you can launch, operate, and recover without heroics. A runbook is what lets a tired on-call engineer at 3am resolve an incident by following steps instead of guessing — and what lets you launch safely (checklist), change safely (canary/rollback), and respond predictably (incident playbooks). LLM systems add failure modes ordinary services don't have — KV-cache OOM, model/prompt regressions, cost spikes, silent quality drops, prompt-injection — so the runbook must cover them explicitly. Shipping without one is how a provider blip becomes a multi-hour outage and a quality regression ships unnoticed.

2. Core Concept

Plain-English primer: launch safely, change safely, recover predictably

A production runbook has three jobs, each a checklist or playbook:

1. Launch safely — the pre-launch checklist. A gate you pass before taking traffic: is everything from Phases 5–7 actually configured and tested?
2. Change safely — rollout & rollback. Models, prompts, and routes change constantly; you ship changes via canary / staged rollout with a fast rollback, and you pin versions so nothing changes under you (Phase 5.10).
3. Recover predictably — incident playbooks. For each common failure (latency, errors, cost, quality, capacity), a step-by-step diagnosis→mitigation, anchored to the observability dashboard.

The pre-launch checklist (LLM-specific)

QUALITY
☐ Eval harness passes on a golden set (incl. the FALLBACK models)        [Phase 12, 07]
☐ Tested on 100+ real production-like samples
☐ Online quality signal wired (judge/JSON-valid/refusal)                 [08]
CONFIG
☐ max_tokens set on every path; context cap (--max-model-len) sane       [09, 01]
☐ Model + provider + version PINNED (no aliases drifting)                [5.10]
☐ Prompt/prefix caching enabled where applicable                        [05]
RELIABILITY
☐ Fallback chain configured AND chaos-tested (kill primary)             [07]
☐ Circuit breaker + retries (typed by failure) configured               [07]
☐ Streaming tested e2e: forward, disconnect-cancel, mid-stream error    [06]
☐ Load-tested at expected concurrency; KV ceiling known                 [03, 04]
COST
☐ Per-user/project budgets + global ceiling set                         [09]
☐ Cost/request + attribution dashboards live                            [08, 09]
OBSERVABILITY
☐ p95/p99 TTFT/TPOT, error rate, KV usage, fallback rate dashboards live [08]
☐ SLOs defined + alerts wired (burn + leading indicators) → on-call     [08]
SECURITY/PRIVACY
☐ Secrets/API keys in a secrets manager (not plaintext env)             [Phase 14]
☐ PII redaction + log sampling; data-retention policy confirmed         [Phase 14]
☐ Prompt-injection / input validation considered                       [Phase 14]

Rollout & rollback

• Canary / staged rollout: send a small % of traffic (or shadow traffic) to the new model/prompt/route; compare quality, latency, cost against control before ramping. LLM changes are behavioral — a new model can pass smoke tests yet regress your task, so gate the ramp on eval + online metrics, not just "it returns 200."
• Fast rollback: one switch back to the previous model/prompt/route. Keep the prior version pinned and warm.
• Version pinning: pin model versions (not aliases) and prompt versions so behavior is reproducible and changes are deliberate (Phase 5.10).
• A/B for quality: beyond canary, run A/B with an eval/online metric to choose between models, not just to de-risk a deploy.

Incident playbooks (symptom → steps)

Each gets a written playbook with the exact dashboard panels, commands, and the rollback switch.

3. Mental Model

   LAUNCH SAFELY        →   CHANGE SAFELY        →   RECOVER PREDICTABLY
   pre-launch checklist     canary + rollback        incident playbooks
   (quality·config·         (gate ramp on EVAL,      (symptom → dashboard →
    reliability·cost·         not 200s; pin versions;   mitigation → rollback)
    observability·security)   fast rollback)
        │                         │                        │
        └──────── all anchored to the OBSERVABILITY dashboard + SLOs [08] ───────┘
   LLM-specific failures to plan for: KV OOM · model/prompt regression · cost spike ·
                                      silent quality drop · provider outage · prompt injection

Mnemonic: checklist before launch, canary+rollback for change, playbooks for incidents — all read off one dashboard, with LLM-specific failures planned in.

4. Hitchhiker's Guide

What to look for first: is there a written pre-launch checklist that's actually been run, a tested rollback, and incident playbooks linked from alerts? If any is missing, you're one bad deploy from an outage.

What to ignore at first: elaborate change-management process for a tiny service — but never skip max_tokens, budgets, a fallback, version pinning, and a rollback. Those are non-negotiable even at small scale.

What misleads beginners:

• "It passed smoke tests, ship it." LLM changes are behavioral — gate the ramp on eval + online quality, not status codes (Phase 12).
• No rollback plan. The fastest incident mitigation is reverting the last change — if you can't, every incident is long.
• Aliases, not versions. Floating aliases mean the provider can change your model under you (Phase 5.10).
• Untested fallback. A fallback you never chaos-tested won't work when you need it (07).
• No quality observability. Silent regressions ship and persist until users complain (08).

How experts reason: they make the system boring to operate: pin versions, gate changes behind canary + eval, keep a one-switch rollback, write playbooks tied to dashboard panels, run game days (chaos tests) so the team has muscle memory, and conduct blameless post-mortems that add the new failure mode to the runbook.

What matters in production: SLO adherence + error budget, MTTR (mean time to recover — driven by playbooks + rollback), change-failure rate (driven by canary/eval), and that every alert links to a playbook. The runbook is a living doc updated after every incident.

How to run an incident: declare it → check the SLO dashboard (which symptom?) → open the matching playbook → mitigate (often roll back or fail over first, diagnose later) → communicate → post-mortem → update the runbook.

Questions to ask before launch: Can I roll back in <5 min? Is the fallback chaos-tested? Are versions pinned? Are budgets + ceilings set? Does every alert link to a playbook? Has the eval gate been run on the exact deployed config?

What silently gets expensive/unreliable: no rollback (long MTTR), unpinned versions (silent drift), untested fallback, no quality gate (regressions ship), and stale playbooks no one updated after the last incident.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• A runbook does three jobs: launch safely (checklist), change safely (canary+rollback), recover predictably (playbooks).
• LLM changes are behavioral — gate rollouts on eval + online quality, not status codes (Phase 12).
• The fastest mitigation is usually rollback or failover — diagnose after you've stopped the bleeding.
• Pin model + prompt versions (not aliases) to prevent silent drift (Phase 5.10).
• Chaos-test the fallback — an untested fallback fails when needed (07).
• Every alert links to a playbook, anchored to dashboard panels (08).
• LLM-specific failures: KV OOM (04), cost spikes (09), silent quality regression, prompt injection (Phase 14).
• The runbook is a living doc — every post-mortem adds the new failure mode.

9. Observations from Real Systems

• Mature LLM platforms ship behind canary + eval gates and keep a one-click rollback — because a "better" model often regresses a specific task silently (Phase 12).
• The most common production incidents map exactly to earlier docs: KV-OOM/capacity (04), latency knee under load (03), cost spikes (09), provider outages (07), and prompt/model regressions (5.10).
• Gateways centralize the operational controls (budgets, fallback, version pinning, observability) that the runbook depends on (Phase 8).
• Provider status pages + health checks are the first stop in latency/error incidents — much LLM downtime is downstream.
• Teams that run game days recover faster — the playbook has been rehearsed, so MTTR is low when it's real.

10. Common Misconceptions

11. Engineering Decision Framework

OPERATE the service:
 LAUNCH:  run the pre-launch checklist (quality·config·reliability·cost·observability·security).
          Block launch on any unchecked critical item (max_tokens, budgets, fallback, rollback, SLO).
 CHANGE:  canary/shadow the new model/prompt/route → compare quality+latency+cost vs control
          → ramp only if eval + online metrics hold → keep prior version pinned & warm for rollback.
 RESPOND: alert fires → open linked playbook → MITIGATE FIRST (rollback / failover / shed load)
          → diagnose via dashboard+traces → comms → blameless post-mortem → UPDATE the runbook.
 PIN:     model+prompt versions; no floating aliases.                                   [5.10]
 REHEARSE: game-day the top incidents (provider outage, KV-OOM, cost spike) quarterly.

12. Hands-On Lab

Goal

Produce a real runbook artifact for your serving stack: a pre-launch checklist you've executed, a tested rollback, and one rehearsed incident playbook.

Prerequisites

• A deployed endpoint with observability (08) and routing/fallback (07).

Steps

1. Pre-launch checklist: copy the §2 checklist; for each item, mark configured/tested with evidence (dashboard link, test output). Fix any gaps; note any "accepted risk."
2. Canary + rollback drill: deploy a new model/prompt to 10% of traffic; compare quality (eval/online), p95 latency, and cost/request vs the 90% control for a window; then roll back and time how long it takes (target < 5 min).
3. Eval gate: show a case where the canary passes smoke tests (200s) but fails the eval gate (quality regression) — and that your process blocks the ramp. (Construct it with a deliberately worse prompt/model.)
4. Incident game day: pick one — provider outage (kill the primary, confirm circuit breaker + fallback, 07) or KV-OOM (overload until gpu_cache_usage_perc≈1, confirm admission control/shedding, 04). Follow your playbook; record detection time, mitigation, and MTTR.
5. Post-mortem: write a short blameless post-mortem (timeline, root cause, action items) and add the failure mode + fix to the runbook.

Expected output

A runbook document containing: an executed pre-launch checklist, a measured rollback time, an eval-gate-blocks-bad-canary demonstration, one rehearsed incident playbook with MTTR, and a post-mortem.

Debugging tips

• Rollback is slow → prior version not pinned/warm; fix that first.
• Canary "looks fine" but users complain → you gated on 200s, not quality; add the eval/online gate.

Extension task

Add shadow traffic (mirror prod requests to the candidate without serving its output) to compare quality/latency/cost with zero user risk before canary.

Production extension

Wire each alert to its playbook URL; schedule quarterly game days; track MTTR and change-failure rate as ops KPIs.

What to measure

Checklist completeness; rollback time; whether the eval gate blocks a bad canary; incident detection time + MTTR.

Deliverables

• An executed pre-launch checklist with evidence.
• A rollback drill result (time-to-revert).
• One incident playbook + a blameless post-mortem that updated the runbook.

13. Verification Questions

Basic

1. What three jobs does a production runbook do?
2. Why gate LLM rollouts on eval/online quality instead of status codes?
3. Why pin model versions instead of using aliases?

Applied 4. Walk through your first five moves when p95 latency breaches the SLO. 5. Design a canary process for swapping the primary model, including the gate and rollback.

Debugging 6. A new prompt shipped and quality dropped, but all responses are 200 OK. How do you detect and recover? 7. A cost spike with flat traffic — runbook steps?

System design 8. Write the incident playbook for a primary-provider outage (detection → mitigation → comms → post-mortem).

Startup / product 9. How do MTTR, change-failure rate, and SLO adherence translate into customer trust and enterprise-readiness?

14. Takeaways

1. A runbook = launch safely (checklist) + change safely (canary/rollback) + recover predictably (playbooks), all read off the observability dashboard.
2. Gate rollouts on eval + online quality — LLM changes are behavioral; 200 OK isn't success.
3. Mitigate first (rollback/failover), diagnose second; keep a <5-min rollback and pin versions.
4. Chaos-test fallbacks and run game days so MTTR is low when incidents are real.
5. The runbook is living — every post-mortem adds the new LLM-specific failure mode (04/09/Phase 14).

15. Artifact Checklist

• An executed pre-launch checklist with evidence + accepted risks.
• A tested rollback (time-to-revert measured) with pinned prior version.
• An eval/online quality gate that blocks a bad canary.
• At least one incident playbook tied to dashboard panels.
• A blameless post-mortem template + a living-runbook update process.

Up: Phase 7 Index · Next: Phase 8 — LLM Gateways