Template 04 — Incident Runbook
Copy and fill per system; keep it where on-call can find it. Full method: Phase 7.10.
Runbook: <system name>
Owner team: <team> · On-call: <rotation/link> · Last updated: <date>
System map
- Components:
<gateway, app, vLLM, vector DB, providers, cache, DB> - Dependencies:
<providers + regions, third-party APIs> - Dashboards:
<links: latency p95/p99, cost, error rate, GPU util> - Key correlation ID: propagated across retrieval→model→tools (Phase 10.08)
Alerts → first response
| Alert | Likely cause | First action |
|---|---|---|
| p95 latency ↑ | provider slowness / KV pressure / batch | check provider status; fail over; scale; reduce batch |
| Error rate ↑ | provider outage / rate limit / bad deploy | fail over to backup provider; rollback; check limits |
| Spend spike | loop/retry storm / power user / routing change | attribute by tenant/feature; cap; fix driver (Phase 7.09) |
| Safety violation | guardrail gap / jailbreak | tighten guardrail (fail closed); red-team; patch (Phase 14.05) |
| Cross-tenant leak | missing isolation filter | disable affected path; fix filter; audit blast radius (Phase 14.04) |
| Quality drop | model/prompt change / drift | rollback to last-good version; check eval gate |
Incident loop (Phase 14.06)
- Detect — alert fires / report received. Declare severity.
- Investigate — reconstruct via correlation ID; check dashboards + audit logs.
- Contain — fail over / disable feature / revoke key / rate-limit (Phase 14.03).
- Remediate — fix root cause; verify with eval/red-team.
- Communicate — status page; affected customers; (if data) regulators within deadlines (Phase 14.07).
- Postmortem — blameless; action items; add a regression test / eval case.
Rollback
- Models/prompts are versioned; rollback =
<command/process>. - Canary new changes; keep last-good for instant revert.
Key levers (quick reference)
- Fail over provider · enable/disable caching · raise/lower rate limits · scale pods · tighten guardrails (fail closed) · revoke a key.
Escalation
| Severity | Who | When |
|---|---|---|
| SEV1 (down/leak) | <lead + eng> | immediately |
| SEV2 (degraded) | <on-call> | within <X> |
Assume breach. The runbook + audit logs are how you detect, contain, and prove.