Lab 01 — GPU Metrics Exporter (Prometheus format)

Phase: 10 — Observability | Difficulty: ⭐⭐⭐☆☆ | Time: 4–6 hours Language: Python (stdlib http.server) | Hardware: none (simulated GPU; real DCGM/nvidia-smi optional)

Concept primer: ../WARMUP.md Ch. 2–6.

Run

python solution.py                    # serves /metrics on :9400 + runs the alert test
# in another shell:  curl localhost:9400/metrics

0. The mission

Build a DCGM-style GPU metrics exporter with correct metric semantics (gauge vs counter), the metrics that actually matter (WARMUP Ch. 3), and a Prometheus /metrics endpoint — then prove the alert rules fire on the right bad states and stay quiet on healthy ones.

1. What the output shows

serving GPU metrics on http://localhost:9400/metrics

== scrape + alert evaluation ==
gpu 0: temp=71C clocks=1410MHz throttled=no  xid_rate=0  largest_free=18GB   OK
gpu 1: temp=88C clocks=1100MHz throttled=YES xid_rate=0  largest_free=16GB   -> GPUThrottling
gpu 2: temp=70C clocks=1410MHz throttled=no  xid_rate=2  largest_free=15GB   -> GPUXidErrors
gpu 3: temp=69C clocks=1410MHz throttled=no  xid_rate=0  largest_free=1GB    -> KVFragmentationRisk

alerts fired: GPUThrottling(gpu1), GPUXidErrors(gpu2), KVFragmentationRisk(gpu3)
healthy gpu0 fired nothing.

And /metrics emits valid Prometheus exposition:

# HELP gpu_temperature_celsius GPU die temperature
# TYPE gpu_temperature_celsius gauge
gpu_temperature_celsius{gpu="0"} 71
# TYPE gpu_xid_errors_total counter
gpu_xid_errors_total{gpu="2"} 2
gpu_memory_largest_free_bytes{gpu="3"} 1073741824
...
  • Correct semantics: temperature/clocks/largest-free are gauges; Xid and ECC totals are counters (you alert on their rate). Getting this wrong makes rate() and alerts misbehave (WARMUP Ch. 4).
  • The metrics that matter: throttling (silent perf loss), Xid (predicts node death), largest-free-block (fragmentation early warning — Phase 05) — not just the misleading utilization.gpu (WARMUP Ch. 3).
  • Alerts fire correctly: each bad state triggers its alert; the healthy GPU triggers nothing.

2. Reading order (solution.py)

  1. Metric (gauge/counter) and render_prometheus — the exposition format.
  2. simulate_gpus — the simulated fleet, including the three bad states.
  3. ALERT_RULES + evaluate_alerts — the rule logic (WARMUP Ch. 6).
  4. The HTTP handler serving /metrics.

3. Extensions

  1. Real source: replace the simulator with nvidia-smi --query-gpu=... --format=csv parsing, or DCGM (dcgmi dmon / the DCGM Python bindings) where a GPU exists.
  2. Recording rules: add a Prometheus recording rule that computes goodput from request metrics (Phase 07).
  3. Grafana dashboard: export a dashboard JSON with the key panels (temp, throttle, Xid rate, largest-free trend).
  4. Per-MIG metrics: emit metrics per MIG instance (Phase 06), with the right labels.
  5. Burn-rate alert: implement a multi-window burn-rate alert on a simulated p99 TTFT SLO (WARMUP Ch. 6).

4. Common pitfalls

  1. Counter as gauge (or vice versa): temperature must be a gauge; total Xid errors a counter. Emit the wrong # TYPE and rate() breaks.
  2. Cardinality explosion: never label by request ID or timestamp — unbounded label values blow up the TSDB (a real production cost). Labels are bounded dimensions (gpu, uuid, model).
  3. Alerting on utilization.gpu: it's the misleading metric (WARMUP Ch. 3); alert on SM-active / SLOs / cause-signals instead.
  4. Instantaneous-threshold alerts: real alerts need a for: duration / burn-rate to avoid flapping.

5. What this lab proves about you

You can instrument a GPU platform with correct metric semantics and alert on the signals that predict real failures (not the ones that look impressive). When an incident hits, your dashboards tell the truth — which is the difference between a 5-minute diagnosis and a 5-hour one.