Phase 10 — Observability & Production Operations
Difficulty: ⭐⭐⭐⭐☆ | Estimated Time: 1.5 weeks Roles supported: Head of Engineering [GPU], SRE/Platform Engineer, Production ML-Systems Engineer Hardware needed: none — exporter and release tooling run on any machine
Why This Phase Exists
The JD requires "observability tooling" and "ship production software through a full release cycle, not just a prototype." A GPU platform you can't see is a platform you can't operate: you can't catch a thermal-throttling GPU dragging a training job (Phase 08), a fragmentation leak building toward a day-3 OOM (Phase 05), a KV-cache pressure spike blowing TTFT (Phase 07), or an Xid error about to take a node off the bus (Phase 04). And a platform you can't release safely — versioned, canaried, rollback-ready — is a prototype with customers attached.
You'll build a GPU metrics exporter (Prometheus format, DCGM-style metrics with the right semantics), define SLOs and alerts tied to the failure modes from earlier phases, write failure-drill runbooks, and build release-engineering tooling (version compatibility checks, canary analysis, rollback) — the operational layer that turns the platform into a service.
Concepts
- The three pillars: metrics (Prometheus/DCGM), logs, traces — and what each answers
- GPU metrics that matter and their semantics: utilization (and why "GPU util %" lies), SM activity, memory used/reserved (Phase 05's reserved-vs-allocated), HBM bandwidth, temperature/power/clocks (throttling), ECC errors, Xid errors (Phase 04), NVLink/PCIe traffic
- DCGM (Data Center GPU Manager) and the exporter pattern; Prometheus exposition format; labels and cardinality
- Serving SLIs/SLOs: TTFT/TPOT/goodput (Phase 07), queue depth, KV-cache utilization, error rate — and SLO/error-budget discipline
- Training SLIs: MFU, step time, straggler detection, collective health (Phase 08)
- Alerting: symptom-based vs cause-based, alert fatigue, the "page on SLO burn, ticket on cause" discipline
- eBPF for low-overhead tracing (overview); GPU-aware profiling in production
- Failure drills & runbooks: Xid handling, node drain, the fragmentation-restart, the collective-hang detection (Phase 08), driver-skew (Phase 04)
- Release engineering: semantic versioning, the driver/CUDA/engine compatibility matrix (Phase 03/04), canary deploys + automated analysis, progressive rollout, rollback, feature flags
- The on-call/incident model: severity, escalation, blameless post-mortems; air-gapped/sovereign ops (local-only telemetry, Phase 07/11)
Labs
Lab 01 — GPU Metrics Exporter (Python, Prometheus format)
| Field | Value |
|---|---|
| Goal | Build a Prometheus-format GPU metrics exporter with correct metric semantics (gauges/counters), DCGM-style metrics, and an HTTP /metrics endpoint — fed by a simulated GPU (or real nvidia-smi/DCGM if available). |
| Concepts | Prometheus exposition format, gauge vs counter, metric naming/labels/cardinality, the metrics that matter (Xid, reserved-vs-allocated, throttle, ECC). |
| Steps | 1) python solution.py — serves /metrics; curl localhost:9400/metrics shows GPU metrics. 2) Read the exporter against WARMUP Ch. 2–4. 3) Run the included scrape-and-alert check that fires on a throttling GPU and a fragmentation signal. 4) Extensions: real DCGM/nvidia-smi source, recording rules, a Grafana dashboard JSON. |
| Stack | Python (stdlib http.server) |
| Output | A working exporter + an alert-rules file + a scrape test that fires the right alerts. |
| How to Test | Built-in: /metrics parses as valid Prometheus exposition; the alert evaluator fires GPUThrottling and KVFragmentationRisk on the simulated bad states and stays quiet on healthy ones. |
| Talking Points | Why "GPU utilization %" is misleading (it's "any kernel ran," not "SMs busy"); reserved-vs-allocated as the leak signal (Phase 05); why Xid rate is a top alert; cardinality discipline. |
| Resume Bullet | "Built a Prometheus-format GPU metrics exporter (DCGM-style metrics, correct gauge/counter semantics) with alert rules for throttling, ECC/Xid, and KV-cache fragmentation; validated with an automated scrape-and-alert test." |
| Extensions | Source from real DCGM/nvidia-smi; Prometheus recording rules for goodput; a Grafana dashboard; per-MIG-instance metrics (Phase 06). |
Lab 02 — Release Engineering: Compatibility, Canary & Rollback (Python)
| Field | Value |
|---|---|
| Goal | Build the release-safety tooling a GPU platform needs: a driver/CUDA/engine compatibility checker (Phase 03/04), a canary analyzer (compare canary vs baseline metrics, decide promote/hold/rollback), and a rollout state machine with automatic rollback on SLO burn. |
| Concepts | Semantic versioning, compatibility matrices, canary analysis (statistical comparison), progressive rollout, automated rollback, error budgets. |
| Steps | 1) python solution.py — runs a compatibility check, a canary analysis on simulated metrics, and a rollout that auto-rolls-back on a bad canary. 2) Read against WARMUP Ch. 8–9. 3) Reproduce: an incompatible driver/CUDA pair is rejected before deploy; a canary with regressed p99 TTFT triggers rollback. 4) Extensions: progressive % rollout, error-budget gating, the driver-fleet-upgrade plan (Phase 04). |
| Stack | Python (stdlib) |
| Output | Compatibility checker + canary analyzer + rollout state machine + tests. |
| How to Test | Built-in: incompatible version pairs are flagged; a canary with a significant p99 regression yields ROLLBACK; a healthy canary yields PROMOTE; the rollout state machine reaches the right terminal state. |
| Talking Points | The driver/CUDA/engine compatibility matrix as a release gate (Phase 03/04); why canary analysis must be statistical not eyeballed; error-budget-based release decisions; the fleet driver-upgrade choreography (Phase 04 Q6). |
| Resume Bullet | "Built GPU-platform release tooling: a driver/CUDA compatibility gate, a statistical canary analyzer, and an auto-rollback rollout state machine driven by SLO burn — turning deploys from manual to policy-driven." |
| Extensions | Progressive percentage rollout with per-stage gates; error-budget-based freeze; integrate the Lab 01 exporter's metrics as the canary signal. |
Deliverables Checklist
-
Exporter serves valid Prometheus
/metrics; alert test fires correctly - You can name the GPU metrics that matter and their correct semantics
- Release tooling: compatibility gate + canary + rollback all pass tests
- One failure-drill runbook written (Xid, fragmentation, collective-hang, or driver-skew)
- One page: "our platform's observability + release strategy" (feeds Phase 12)
Interview Relevance
- "What GPU metrics do you alert on, and why is 'GPU utilization' misleading?"
- "nvidia-smi shows 90% util but throughput is low. What's your investigation?"
- "Design a safe release process for a GPU serving platform."
- "How do you catch a fragmentation leak before it OOMs at 3am?"
- "A driver upgrade regressed performance on 5% of the fleet. Walk me through detection and rollback."