Warmup Guide — Observability & Production Operations

Zero-to-expert primer for Phase 10. Builds on every prior phase's failure modes (you monitor what you understand). By the end you can instrument a GPU platform, set SLOs, and ship it through a safe release cycle.

Table of Contents

Chapter 1: Why Observability Is a Platform Feature, Not an Add-On

Every prior phase produced a failure mode that is invisible without instrumentation:

  • Phase 04: an Xid error takes a GPU off the bus — silent until jobs fail.
  • Phase 05: a fragmentation leak builds toward a day-3 OOM — silent until 3am.
  • Phase 07: KV-cache pressure spikes TTFT past SLO — silent until customers churn.
  • Phase 08: one thermal-throttling GPU drags a 512-GPU job to its speed — silent in aggregate metrics.

A GPU platform you can't see is a platform you operate by waiting for outages. Observability is what converts "the cluster feels slow" into "GPU 37 on node 12 is throttling at 84°C, pulling its TP group's MFU to 40%." It's not an SRE afterthought — for a compute platform it's a product feature (especially for sovereign customers who need local-only telemetry, Phase 07 Ch. 9), and the JD lists it explicitly.

The leadership framing: you can't manage what you can't measure, and you can't release safely what you can't observe. This phase is the difference between a prototype and a service.

Chapter 2: The Three Pillars — Metrics, Logs, Traces

The standard decomposition, with what each answers:

  • Metrics — numeric time series (GPU temp, TTFT p99, queue depth). Cheap, aggregatable, the backbone of dashboards and alerts. Answers "is something wrong, and how much?" — but not why for a specific request. Prometheus is the de facto standard (Ch. 4).
  • Logs — discrete timestamped events (an Xid error, a request failure, a rollback). Answer "what happened to this specific thing?" High cardinality, expensive at scale; structured logging + sampling are the disciplines.
  • Traces — the path of one request across services (gateway → scheduler → engine → GPU). Answer "where did this request's latency go?" Essential for the serving stack (Phase 07); OpenTelemetry is the standard.

The mental model: metrics tell you something's wrong, traces tell you where, logs tell you what. A platform needs all three; this phase's labs focus on metrics (the alerting backbone) and the operational glue, with logs/traces as extensions.

Chapter 3: GPU Metrics That Matter (and Their Semantics)

The metrics, and the semantics traps that catch people (Lab 01 implements these correctly):

  • "GPU utilization %" (nvidia-smi's utilization.gpu): the famous lie. It means "the fraction of time at least one kernel was running" — not "the SMs were busy." A kernel using 5% of the SMs at batch 1 (memory-bound decode, Phase 01) shows ~100% utilization while the GPU is mostly idle. For real work, use SM activity / occupancy (DCGM DCGM_FI_PROF_SM_ACTIVE) and achieved memory bandwidth (DCGM_FI_PROF_DRAM_ACTIVE).
  • Memory: used vs the framework's allocated — the reserved-vs-allocated gap (Phase 05 Ch. 4) is cache, not a leak; the leak signal is allocated trending up, and the fragmentation signal is largest-free-block shrinking.
  • Temperature / power / clocks: throttling shows as clocks dropping below base under load (Phase 03's "thermal throttle" note). clocks_throttle_reasons is the smoking gun; temp alone is a lagging indicator.
  • ECC errors: correctable (logged) vs uncorrectable (a failing GPU — page).
  • Xid errors (Phase 04 Ch. 10): the GPU's error codes (Xid 79 = fell off the bus, Xid 48 = double-bit ECC, Xid 13/31 = app/memory faults). Xid rate is a top-tier alert — it predicts node failures.
  • NVLink / PCIe throughput: low NVLink traffic when you expected TP activity = a topology misconfiguration (Phase 08).

Gauge vs counter (the format trap, Ch. 4): temperature is a gauge (goes up and down); total ECC errors is a counter (monotonic, you alert on its rate). Mixing them up makes every dashboard wrong.

Chapter 4: The Exporter Pattern and Prometheus Format

Prometheus pulls metrics: your exporter serves a /metrics HTTP endpoint in a text format; Prometheus scrapes it every N seconds. (DCGM-exporter is NVIDIA's; Lab 01 builds one.)

The exposition format (simple, and Lab 01 emits it correctly):

# HELP gpu_temperature_celsius GPU die temperature
# TYPE gpu_temperature_celsius gauge
gpu_temperature_celsius{gpu="0",uuid="GPU-abc"} 71
# HELP gpu_xid_errors_total Cumulative Xid errors
# TYPE gpu_xid_errors_total counter
gpu_xid_errors_total{gpu="0"} 3

The rules that matter:

  • Naming: unit-suffixed (_celsius, _bytes, _total for counters), namespaced (gpu_, dcgm_).
  • Labels: dimensions (gpu, uuid, model). Cardinality discipline — never label by something unbounded (request ID!) or you explode the time-series database. (A real production-cost lesson.)
  • Gauge vs counter (Ch. 3): get it right or rate() and alerts misbehave.

The exporter pattern is universal (node-exporter, dcgm-exporter, your app's /metrics); building one teaches the semantics that make dashboards and alerts correct.

Chapter 5: SLIs, SLOs, and Error Budgets

The discipline that makes "is the platform healthy?" answerable:

  • SLI (indicator): a measured signal of user-visible health — for serving, goodput / p99 TTFT / p99 TPOT / error rate (Phase 07); for training, MFU / step-time / straggler-free-fraction (Phase 08).
  • SLO (objective): the target — "p99 TTFT < 300ms, 99.9% of the time" or "goodput ≥ X at SLO." Set from user needs, not what's easy to hit.
  • Error budget: 100% − SLO. A 99.9% SLO allows 0.1% bad — that budget is spendable: you ship features/releases (Ch. 8) while budget remains, and you freeze risky changes when it's exhausted. This turns "should we deploy?" into a data question.

The leadership point: SLOs aligned to user experience (goodput, not raw throughput — Phase 07 Ch. 6) plus error budgets give you an objective release and reliability policy, replacing arguments with arithmetic. It's also how you balance the JD's "ship fast" against "production-grade."

Chapter 6: Alerting Without Fatigue

The failure mode of monitoring is too many alerts — on-call ignores them, and the real one slips through. The disciplines:

  • Page on symptoms, ticket on causes. Page when users are affected (SLO burning, goodput dropping) — that's always actionable. File a ticket (not a page) for causes that might lead there (one GPU throttling, ECC correctable rate up) — investigate in business hours unless they breach an SLO.
  • Alert on rate-of-burn, not instantaneous breach. "p99 > 300ms for 5 minutes" not "p99 > 300ms once." Multi-window burn-rate alerts (fast burn = page now, slow burn = ticket) are the SRE-book standard.
  • Every page is actionable + has a runbook (Ch. 7). A page you can't act on is noise; delete it or make it a ticket.
  • Xid/uncorrectable-ECC are exceptions — cause-based but page-worthy because they reliably predict imminent user impact (a node about to die).

Lab 01's alert rules embody this: GPUThrottling (ticket-class symptom), KVFragmentationRisk (early warning), tied to thresholds that mean something.

Chapter 7: Failure Drills and Runbooks

A runbook is a written, tested response to a known failure. You write them for the failure modes the earlier phases taught (this is the "production cycle" discipline the JD wants):

  • Xid error / GPU fell off the bus (Phase 04): detect via Xid-rate alert → cordon the node → drain jobs (checkpoint-aware, Phase 08) → RMA/reset → health-check → uncordon.
  • Fragmentation OOM risk (Phase 05): largest-free-block alert → identify the fragmenting workload → (mitigate) scheduled restart / (fix) size bucketing, paged KV (Phase 07).
  • Collective hang (Phase 08): collective-timeout alert names the culprit rank → check that GPU's health → drain → restart from checkpoint.
  • Driver/CUDA skew (Phase 04): version-mismatch detection → the fleet-upgrade choreography (Phase 04 Q6).

The drill discipline: practice them (game-days), because an untested runbook is a hypothesis. A platform leader's deliverable is the runbook library plus a culture that exercises it.

Chapter 8: Release Engineering — Compatibility, Canary, Rollback

The JD's "ship through a full release cycle, not just a prototype." The pipeline (Lab 02 builds the core of it):

  1. Compatibility gate: before deploy, verify the driver/CUDA/engine/model matrix (Phase 03/04). A build needing CUDA 12.4 on a node with a driver supporting only 12.2 must be rejected pre-deploy, not discovered at runtime ("driver insufficient for runtime", Phase 04 Ch. 6). This is a hard gate.
  2. Canary: deploy to a small slice, compare its SLIs to the baseline statistically (not eyeballed — a p99 regression hides in noise). Decide PROMOTE / HOLD / ROLLBACK from the comparison.
  3. Progressive rollout: expand by percentage with a gate at each stage, watching error budget (Ch. 5). Any stage that burns budget halts.
  4. Automated rollback: SLO burn → roll back automatically, fast — image- pinned so rollback is a re-deploy, not archaeology (Phase 04 Q6).
  5. Feature flags: decouple deploy from release; turn a risky path on for a cohort, off instantly if it misbehaves — no redeploy.

The semantic-versioning discipline underneath: major = breaking, minor = additive, patch = fixes; the ABI/compatibility rules from Phase 09 Ch. 6 apply to your platform's own interfaces. For GPU platforms the compatibility matrix is the highest-risk gate because the failure modes (driver skew, kernel rebuilds — Phase 03 cold-start) are subtle and fleet-wide.

Chapter 9: Incident Response and the Operating Model

When prevention fails (it will), the operating model:

  • Severity + escalation: SEV levels by user impact; clear escalation paths; an incident commander role for big ones.
  • Detect → mitigate → resolve → learn: mitigate first (roll back, drain, failover — stop the bleeding) before root-causing; resolve and verify; then learn.
  • Blameless post-mortems: the failure is the system's, not a person's; output is action items that change the system (a new alert, a runbook, a guardrail) — which feed back into Chapters 6–8. A post-mortem without system changes is theater.
  • Air-gapped/sovereign ops (Phase 07/11): no cloud monitoring SaaS, no phone-home — telemetry and incident tooling run inside the customer's environment, which changes your packaging and your on-call model.

The leadership reframe: incidents are inevitable; a learning operating model (every incident hardens the system) is what compounds reliability over time — the same continuous-improvement loop as Phase 08's "failures are the steady state."

Lab Walkthrough Guidance

Order: Lab 01 → Lab 02 (instrument first, then release safely on those signals).

Lab 01 (exporter):

  1. python solution.py; curl localhost:9400/metrics and read the format against Ch. 4. Verify gauge/counter types are correct.
  2. Run the scrape-and-alert check; confirm GPUThrottling and KVFragmentationRisk fire on the bad simulated states and stay quiet on healthy ones — tie each to Ch. 3/6.
  3. Extensions: real DCGM/nvidia-smi source; a recording rule for goodput; a Grafana dashboard JSON.

Lab 02 (release tooling):

  1. python solution.py; read the compatibility gate, canary analyzer, and rollout state machine against Ch. 8.
  2. Reproduce: an incompatible driver/CUDA pair rejected pre-deploy; a canary with regressed p99 → ROLLBACK; healthy canary → PROMOTE.
  3. Extensions: progressive % rollout, error-budget gating, wiring Lab 01's metrics as the canary signal.

Success Criteria

  • You can explain the three pillars and what each answers (Ch. 2)
  • You can name the GPU metrics that matter, their gauge/counter type, and why "GPU util %" is misleading (Ch. 3) — your exporter emits them correctly
  • You can write a valid Prometheus exposition and explain cardinality (Ch. 4)
  • You can define SLI/SLO/error-budget and tie them to goodput (Ch. 5)
  • You can apply "page on symptoms, ticket on causes" + burn-rate alerts (Ch. 6)
  • You can write a failure-drill runbook for an earlier-phase failure (Ch. 7)
  • You can describe the compatibility-gate → canary → rollback pipeline (Ch. 8) — Lab 02 implements it
  • You can describe a blameless, learning incident model (Ch. 9)

Interview Q&A

Q1: What GPU metrics do you alert on, and why is "GPU utilization" misleading? A: nvidia-smi's utilization.gpu means "fraction of time ≥1 kernel ran," not "SMs busy" — a memory-bound batch-1 decode (Phase 01) shows ~100% while using a few percent of the SMs, so it's useless as a saturation signal. I alert on user-facing SLOs first (goodput/p99 TTFT, Phase 07), then on cause-based signals that predict impact: Xid error rate and uncorrectable ECC (page — imminent node death, Phase 04), clock throttling under load (thermal/power — silent perf loss), largest-free-block shrinking (fragmentation leak, Phase 05), and allocated memory trending up (real leak vs cache). For utilization I use DCGM's SM-active and DRAM-active, not utilization.gpu. Each alert is gauge-or-counter-correct and has a runbook.

Q2: nvidia-smi shows 90% utilization but throughput is low. Investigate. A: 90% utilization.gpu just means kernels are running, not that they're doing useful work — classic memory-bound or badly-occupied kernels (Phase 01/02). I'd check DCGM SM-active (are the SMs actually busy or is one warp running?) and DRAM-active (is it bandwidth-bound — expected for decode?), then the roofline (Phase 01): if it's memory-bound decode, "low throughput at high util" is normal and the fix is batching/quantization (Phase 07), not kernel tuning. If SM-active is also low, it's latency-bound (occupancy/stalls — Phase 02 Ch. 11) or the GPU is throttling (check clocks). If clocks are fine and it's genuinely under-utilized, look upstream: data loading, sync points, or PCIe transfers in the hot path (Phase 02/05). The lesson: utilization.gpu is the start of the investigation, never the conclusion.

Q3: Design a safe release process for a GPU serving platform. A: Five gates. (1) Compatibility: a hard pre-deploy check of the driver/CUDA/ engine/model matrix (Phase 03/04) — reject incompatible pairs before they reach a node, because the failure mode (driver-insufficient, JIT cold-start) is fleet-wide and subtle. (2) Canary: deploy to a small slice, compare SLIs (goodput, p99 TTFT/TPOT) to baseline statistically, decide promote/hold/ rollback. (3) Progressive rollout by percentage, each stage gated on error budget (Phase 07/this phase Ch. 5). (4) Automated rollback on SLO burn — fast, image-pinned so it's a re-deploy not archaeology. (5) Feature flags to decouple deploy from release. Underneath: semantic versioning and the Phase 09 ABI discipline for our own interfaces. The whole thing is policy-driven by error budget, so "should we ship?" is arithmetic, not opinion.

Q4: How do you catch a fragmentation leak before it OOMs at 3am? A: Export the right signal. The OOM is preceded by largest-free-block shrinking while allocated stays flat (external fragmentation, Phase 05 Ch. 5) — so I alert on largest-free-block trending toward the size of typical large allocations, well before reserved hits capacity. That distinguishes it from a real leak (allocated trending up) and from healthy cache (reserved high, largest-block fine). The alert is a ticket (early warning, business hours) that escalates to a page if it crosses into SLO-risk territory. The runbook (Ch. 7): identify the fragmenting workload, mitigate with a scheduled restart, fix with size bucketing / paged KV (Phase 07). The point is the day-3 OOM is predictable from the right metric — most platforms just don't export largest-free-block.

Q5 (leadership): A driver upgrade regressed performance on 5% of the fleet. Walk me through detection and response. A: Detection should have happened in canary (Ch. 8) — a statistical p99/goodput comparison on the upgraded slice vs baseline would flag a regression before fleet rollout; if it reached 5%, either canary coverage missed the affected hardware SKU or the regression was load-dependent. Response: (1) the progressive rollout halts automatically on error-budget burn (Ch. 5); (2) auto-rollback the affected cohort — image-pinned so it's a fast re-deploy (Phase 04 Q6); (3) confirm recovery via the SLIs. Then learn (Ch. 9): blameless post-mortem → action items — add the affected GPU SKU to the canary matrix, add a perf-regression gate to the compatibility check, maybe a driver-version performance benchmark in CI. Driver upgrades shipping perf regressions is a known risk (Phase 04), so the systemic fix is a perf-conformance gate (Phase 09 Ch. 7) on driver changes, not just "be more careful."

References

  • Beyer et al., Site Reliability Engineering (Google) — SLOs, error budgets, alerting — https://sre.google/books/
  • The follow-up, The SRE Workbook — multi-window burn-rate alerts
  • NVIDIA DCGM documentation + dcgm-exporter — https://docs.nvidia.com/datacenter/dcgm/ , https://github.com/NVIDIA/dcgm-exporter
  • Prometheus exposition format + naming/cardinality best practices — https://prometheus.io/docs/practices/naming/
  • NVIDIA Xid errors reference — https://docs.nvidia.com/deploy/xid-errors/
  • OpenTelemetry (traces) — https://opentelemetry.io
  • Brendan Gregg on eBPF / production profiling — https://www.brendangregg.com/ebpf.html
  • "Canary Analysis" (Netflix Kayenta) — automated statistical canary — https://github.com/spinnaker/kayenta
  • Cross-track: Phase 04 WARMUP Ch. 6/10 (Xid, driver skew), Phase 05 Ch. 5 (fragmentation), Phase 07 Ch. 6 (goodput), Phase 08 Ch. 8 (failure at scale)