🛸 Hitchhiker's Guide — Phase 10: Observability & Production Ops
Read this if: you can build a GPU platform but couldn't yet see it failing or release it safely. Field notes; derivations in the WARMUP.
0. The 30-second mental model
You can't operate what you can't see, or release safely what you can't observe. Metrics (Prometheus/DCGM) say something's wrong, traces say where, logs say what. Alert on user-facing SLOs (goodput, p99 TTFT) + a few cause signals that predict death (Xid, ECC, throttling). Release through compatibility gate → canary → progressive rollout → auto-rollback, all governed by error budget.
1. GPU metrics card (with the traps)
| Metric | Type | Trap / signal |
|---|---|---|
| utilization.gpu | gauge | lies — "a kernel ran," not "SMs busy." Use DCGM SM-active. |
| SM-active / DRAM-active | gauge | the real saturation/bandwidth signals |
| memory reserved vs allocated | gauge | gap = cache (Phase 05); leak = allocated up |
| largest-free-block | gauge | fragmentation early warning — most don't export it |
| clocks + throttle-reasons | gauge | throttling = silent perf loss; temp lags |
| ECC correctable / uncorrectable | counter | uncorrectable = page (dying GPU) |
| Xid errors | counter | top alert — predicts node failure (Phase 04) |
| NVLink/PCIe throughput | counter | low NVLink during TP = topology misconfig (Phase 08) |
Gauge (up/down) vs counter (monotonic, alert on rate()). Get it right or every dashboard is wrong. Never label by request ID (cardinality explosion).
2. SLO discipline
SLI = measured user health (goodput, p99 TTFT). SLO = target ("p99 TTFT<300ms 99.9%"). Error budget = 1−SLO, spendable: ship while budget remains, freeze when exhausted. "Should we deploy?" becomes arithmetic.
3. Alerting that doesn't get ignored
- Page on symptoms (SLO burning → users hurt → always actionable).
- Ticket on causes (one GPU throttling → investigate in hours).
- Burn-rate, multi-window (fast burn = page, slow = ticket), not instant threshold.
- Every page: actionable + has a runbook. Exceptions paged anyway: Xid, uncorrectable ECC (reliably predict imminent impact).
4. Runbook library (one per earlier-phase failure)
| Failure | Detect | Respond |
|---|---|---|
| Xid / off the bus (P04) | Xid-rate alert | cordon → drain (ckpt-aware) → RMA → health-check |
| Fragmentation OOM (P05) | largest-free-block | find workload → restart / bucket / paged KV |
| Collective hang (P08) | collective timeout | culprit rank → drain → restart from ckpt |
| Driver skew (P04) | version mismatch | fleet-upgrade choreography (P04 Q6) |
Practice them (game-days). An untested runbook is a hypothesis.
5. Release pipeline (Lab 02)
compatibility gate (driver/CUDA/engine matrix, P03/P04 — HARD gate, pre-deploy)
-> canary (statistical SLI compare, not eyeballed) -> PROMOTE/HOLD/ROLLBACK
-> progressive % rollout (each stage gated on error budget)
-> auto-rollback on SLO burn (image-pinned = fast)
-> feature flags (decouple deploy from release)
The compatibility matrix is the highest-risk GPU gate — driver skew and JIT cold-start (P03) are subtle and fleet-wide.
6. Incident model
Detect → mitigate first (roll back/drain/failover — stop bleeding) → resolve → blameless post-mortem whose output is system changes (new alert, runbook, guardrail). No system change = theater. Air-gapped: all of this runs inside the customer's env (P07/P11).
7. War stories
- "90% util, terrible throughput" — memory-bound decode; utilization.gpu was meaningless. SM-active told the truth.
- "3am OOM at 60% memory" — fragmentation; nobody exported largest-free-block. Added it; now it's a daytime ticket.
- "Driver upgrade cut perf 8% on one SKU" — canary matrix didn't cover that SKU. Auto-rollback fired on budget burn; post-mortem added the SKU + a perf gate.
- "Alert storm, real page missed" — paging on causes (every throttle). Moved causes to tickets, paged only on SLO burn. Signal returned.
8. Exit bar
You can instrument a GPU platform with correct metric semantics, set SLOs and non-fatiguing alerts, write tested runbooks for the failure modes from Phases 04–08, and ship through a compatibility-gated, canary-analyzed, auto-rollback pipeline. You've built the exporter and the release tooling. Next: Phase 11 — the security/licensing primitives that make the platform a commercial, sovereign-ready product.