The ops runbook for the platform. Based on the incident runbook template and Phase 7.10. Fill <...> for your deployment.
- Components: gateway, product app, RAG, agent, vLLM/local, vector DB, cache, Postgres, providers.
- Dashboards:
<links: p95/p99 latency, cost per request/tenant, error/fallback rate, GPU util, cache hit>.
- Correlation ID: propagated request→retrieval→model→tools (reconstruct any request).
- Status page:
<link>.
| Alert | Likely cause | First action |
|---|
| p95 latency ↑ | provider slow / KV pressure / batch | fail over provider; scale serving; reduce batch/context |
| Error/fallback rate ↑ | provider outage / rate limit / bad deploy | confirm fallback firing; rollback; check limits |
| Spend spike | agent loops / power user / routing change | attribute by tenant/feature; cap; fix driver |
| Safety violation | guardrail gap / jailbreak | tighten guardrail (fail closed); red-team; patch |
| Cross-tenant leak | isolation filter gap | disable path; fix filter; audit blast radius; notify |
| Eval regression (CI) | prompt/model change | block deploy; roll back to last-good |
- Detect (alert/report) → declare severity.
- Investigate → reconstruct via correlation ID; dashboards + audit logs.
- Contain → fail over / disable feature / revoke key / rate-limit.
- Remediate → fix root cause; verify with eval + red-team.
- Communicate → status page; customers; (data incident) regulators within deadlines.
- Postmortem → blameless; add a regression test / eval case.
- Deploys: canary a fraction → watch p95/error/eval → promote or rollback. Models/prompts are versioned.
- Model bake-offs: scheduled re-eval vs new models on the golden set (12.08).
- Cost review: weekly cost-per-resolved-task + margin; tune routing/caching.
- Key rotation: scheduled + on suspicion; config change, no redeploy.
- Index freshness: RAG re-index job; monitor staleness.
fail over provider · toggle caching · adjust rate limits/quotas · scale app/serving pods · tighten guardrails (fail closed) · revoke a key · roll back model/prompt version.
| SLO | Target |
|---|
| p95 latency | <ms> |
| Availability | <99.9%> |
| Error rate | <%> |
| Eval score (golden set) | <≥ X, gate> |
| Severity | Who | When |
|---|
| SEV1 (down / data leak) | <lead + on-call> | immediately |
| SEV2 (degraded) | <on-call> | within <X> |