Warmup Guide — Telemetry & Observability

Zero-to-expert primer for Phase 07. You need no monitoring background: this guide starts at "what is a metric" and ends with you designing a unified rack telemetry pipeline, writing PromQL alerts, and correlating metrics/logs/traces. This is how you see the fleet — the prerequisite for operating and debugging it (Phase 10).

Table of Contents

Chapter 1: Observability vs Monitoring; the Three Pillars

Monitoring = watching known signals for known problems (is CPU > 90%?). Observability = being able to ask new questions about your system's behavior from its outputs, including ones you didn't anticipate ("why is this specific rack's tail latency high since the firmware update?"). You build monitoring; you design for observability.

The classic three pillars:

  • Metrics — numeric time-series sampled over time (temperature, power, request rate). Cheap to store, easy to aggregate and alert on; the backbone of dashboards. Weakness: they tell you that something is wrong, rarely why at fine grain.
  • Logs — timestamped, ideally structured (Phase 02) events. The detail behind a metric anomaly. Weakness: volume and cost; hard to aggregate.
  • Traces — the end-to-end path of one request/workflow across components, broken into spans with timings. The answer to "where did the time go / where did it fail." Weakness: needs instrumentation and (usually) sampling.

The power is in correlation: a metric alert → jump to the traces in that window → jump to the logs for the failing span, all tied by shared IDs. For a rack fleet, metrics catch the thermal/power/PCIe problems, logs explain the specific failure, traces show which step of a provisioning/firmware workflow stalled.

Chapter 2: Metrics and the Prometheus Model

Prometheus is the de-facto open metrics system (the JD names it). Its defining choice is the pull model: Prometheus periodically scrapes an HTTP /metrics endpoint on each target and stores the samples in its time-series database (TSDB).

Why pull (a guaranteed interview question):

  • Targets are dumb; the monitoring system controls scrape frequency and knows the target list (via service discovery) — so a missing target is itself a signal ("up == 0").
  • No push infrastructure for thousands of targets to overwhelm; backpressure is natural.
  • Easy local testing: curl /metrics shows exactly what Prometheus sees.

When you do push: short-lived batch jobs that may finish before any scrape (use the Pushgateway), and across network boundaries you can't scrape (remote-write to a central Prometheus/Mimir/Thanos). For a rack, the agents/exporters expose /metrics and Prometheus scrapes them; ephemeral provisioning jobs might push to a Pushgateway.

A time-series is identified by its name + a set of labels: node_temp_celsius{rack="r1", node="n0",sensor="accel0"}. Labels are the dimensions you slice and aggregate by — and the source of cardinality danger (Chapter 6).

Chapter 3: Metric Types and the Exposition Format

The exposition format is plain text an exporter serves:

# HELP node_temp_celsius Temperature in Celsius
# TYPE node_temp_celsius gauge
node_temp_celsius{rack="r1",node="n0",sensor="accel0"} 71
node_temp_celsius{rack="r1",node="n0",sensor="inlet"} 27
# TYPE pcie_correctable_errors_total counter
pcie_correctable_errors_total{node="n0",bdf="0000:13:00.0"} 250

The four metric types — choosing correctly is a senior signal:

  • Counter: monotonically increasing (only goes up, resets on restart). Errors, bytes, requests. You almost always query rate(counter[5m]), never the raw value. PCIe AER errors, packets, firmware-update attempts.
  • Gauge: a value that goes up and down. Temperature, power draw, fan RPM, free memory, CDU ΔT. Most rack hardware telemetry is gauges.
  • Histogram: samples bucketed into ranges + a sum + a count; lets you compute quantiles (p99) server-side via histogram_quantile. Scrape latency, provisioning step duration.
  • Summary: client-side quantiles (less flexible for aggregation than histograms; prefer histograms when you'll aggregate across instances).

The rule of thumb: did it only ever increase? → counter; can it decrease? → gauge; do you need a distribution/quantile? → histogram.

Chapter 4: Exporters — The Unification Problem

Prometheus scrapes HTTP; your hardware speaks Redfish/IPMI/SNMP/Modbus (Phases 03–04). An exporter bridges the gap: it pulls from the device's native protocol and re-exposes the data as Prometheus metrics on /metrics. The ecosystem has node_exporter (host metrics), snmp_exporter, ipmi_exporter, redfish_exporter — and for a rack with heterogeneous gear, you build a unified exporter that uses the Phase 03 driver abstraction (Redfish-first, IPMI/SNMP fallback) and normalizes everything into one consistent metric/label scheme.

This is the JD's "unified telemetry pipelines" in concrete form, and it's Lab 01. The design choices that matter:

  • Consistent naming/labels across protocols (a temperature is node_temp_celsius regardless of whether it came from Redfish Thermal or IPMI SDR) — so dashboards and alerts don't care about the source.
  • Scrape-time vs background-poll: BMCs are slow/fragile (Phase 03 Ch. 9), so the exporter usually polls devices on its own schedule (with caching, retries, timeouts — Phase 02) and serves the cached values on scrape, rather than hammering a BMC synchronously on every Prometheus scrape.
  • An up/health metric per device so you can alert on "exporter can't reach this BMC."

Chapter 5: PromQL, Recording Rules, and Alerting

PromQL queries the TSDB. The patterns you'll use constantly:

  • Aggregate across a label: avg by (rack) (node_temp_celsius), sum by (rack) (node_power_watts).
  • Rate of a counter: rate(pcie_correctable_errors_total[5m]) — errors per second.
  • Threshold for an alert: node_temp_celsius > 85, sum by (rack)(node_power_watts) > 17000 (the Phase 01 budget!).
  • up == 0 — a target Prometheus couldn't scrape (device/exporter down).

Recording rules precompute expensive/常用 queries into new series (e.g., per-rack power) so dashboards and alerts are fast.

Alerting rules are PromQL conditions with a for: duration (must hold N minutes to fire, avoiding flapping), labels (severity), and annotations (description/runbook link → Phase 12). They fire to Alertmanager, which dedupes, groups, silences, and routes (page/Slack/email). Example rack alerts:

- alert: AcceleratorHot
  expr: node_temp_celsius{sensor=~"accel.*"} > 85
  for: 5m
  labels: { severity: warning }
  annotations: { summary: "Accelerator hot on {{$labels.node}}", runbook: "..." }
- alert: RackPowerOverBudget
  expr: sum by (rack)(node_power_watts) > 17000
  for: 2m
  labels: { severity: critical }
- alert: NodeDown
  expr: up{job="rack-exporter"} == 0
  for: 1m
  labels: { severity: critical }

Chapter 6: Cardinality — The Footgun

The single most important Prometheus operational lesson. Cardinality = the number of distinct time-series, which is the product of every label's distinct values. Each unique label-set is a separate series consuming memory and disk.

The trap: putting a high-cardinality value in a label — a timestamp, a request ID, a serial number on a high-churn metric, a full error message — multiplies series explosively and can OOM Prometheus. http_requests_total{user_id="..."} with a million users = a million series per metric.

The discipline:

  • Labels are for bounded, low-cardinality dimensions you aggregate/filter by: rack, node, sensor, severity, job. Not unbounded identifiers.
  • Put high-cardinality detail in logs/traces (Chapter 8), not metric labels.
  • Watch series count; alert on it; review new metrics for label explosions in code review.

For a rack fleet (thousands of nodes × dozens of sensors), even legitimate labels add up — which is why naming and label design is a real engineering task, not an afterthought.

Chapter 7: Grafana and Dashboard Design

Grafana visualizes metrics (and logs/traces) from data sources like Prometheus. Dashboards are JSON (dashboards-as-code — version them, review them, Phase 11). Design matters more than prettiness:

  • Variables/templating: a $rack/$node dropdown so one dashboard serves the whole fleet.
  • Two proven methods: USE (Utilization, Saturation, Errors) for resources (per accelerator/PDU/CDU); RED (Rate, Errors, Duration) for services (the control-plane API). For a rack overview: power vs budget, temps with thresholds, PCIe error rates, node up/down, CDU ΔT/flow, firmware-version spread.
  • The hierarchy: a fleet view (which racks are unhealthy) → a rack view (which nodes) → a node view (which sensor) — so an operator drills from symptom to cause.
  • Avoid the "wall of graphs nobody reads"; every panel should answer a question or back an alert.

Chapter 8: Logs, Traces, and Correlation

Logs (Phase 02 structured JSON) get aggregated (Loki, Elasticsearch) so you can query across the fleet. The key for correlation: include a trace_id and stable labels (node, rack, request_id) in every line.

Traces (OpenTelemetry — the vendor-neutral standard) capture a workflow as a tree of spans, each with a name, start/end (duration), attributes, and a parent — sharing a trace_id and each having a span_id. Context propagation carries the trace_id/ span_id across function calls and across services (via headers), so a provisioning workflow that spans the orchestrator, a BMC call, and a firmware job is one trace. Traces answer "where did the time go / which step failed," which metrics and logs alone don't.

Correlation is the payoff: a metric alert → the traces in that window (via exemplars linking a metric sample to a trace) → the failing span → its logs (same trace_id). This is the debugging loop (Phase 10) that turns "the fleet is slow" into "step 3 of provisioning on node-237 times out calling the BMC, here's the log." The OTel collector receives, processes (sampling), and exports telemetry to backends (Jaeger/Tempo for traces, Prometheus for metrics) — one pipeline, vendor-neutral. Lab 02 builds the tracing + correlation.

Chapter 9: SLIs/SLOs and Alerting Philosophy

The operational maturity layer:

  • SLI (Service Level Indicator): a measured quantity reflecting user/operator experience (e.g., "fraction of provisioning runs that succeed," "p99 control-plane API latency," "fraction of nodes healthy").
  • SLO (Objective): a target for an SLI over a window ("99% of provisions succeed," "99.9% node availability/month").
  • Error budget: 1 − SLO. The allowed failure. Spend it on velocity (ship changes) until it's low, then slow down — a shared language between dev and ops.

Alert on symptoms, not causes. Page on the thing that matters (SLO breach: provisioning failing, a rack offline, accelerators throttling) — not on every cause (a single retry, a transient blip). Cause-based alerts produce alert fatigue: too many pages, people stop reading them, the real one gets missed. Every page should be actionable and link a runbook (Phase 12). For a rack: page on node-down, rack-over-budget, sustained accelerator overheat, CDU leak (Phase 04), and SLO breaches; don't page on a single correctable PCIe error (dashboard it, alert only on a rate).

Lab Walkthrough Guidance

Order: Lab 01 (exporter/metrics) → Lab 02 (tracing/logs). Metrics are the backbone; tracing is the deep-dive when metrics flag a problem.

Lab 01 (Redfish→Prometheus exporter):

  1. python3 solution.py, then curl localhost:9101/metrics — read the exposition format.
  2. Note the metric types (gauge for temp/power, counter for AER errors) and the bounded labels (rack/node/sensor) — and that there are no high-cardinality labels (Ch. 6).
  3. Read alerts.yml (the PromQL rack alerts) and dashboard.json (Grafana).
  4. Extensions: run real Prometheus scraping it + import the dashboard into Grafana; wire it to the Phase 04 PDU/CDU/PCIe sim.

Lab 02 (OTel tracing + logs):

  1. python3 solution.py — a traced provisioning-style workflow with an injected failure.
  2. Read the span tree (timings, parent/child) and the structured logs carrying the trace_id.
  3. See how the failed span + its correlated logs pinpoint the problem.
  4. Extensions: export OTLP to Jaeger/Tempo; add a metric exemplar linking a slow metric to its trace.

Phase capstone question: "Design the telemetry pipeline for a 10,000-node accelerator fleet with Redfish/IPMI/SNMP devices. What do you alert on, and how do you keep it scalable and debuggable?" (Answer in system-design/02: unified exporters near hardware → Prometheus (federated/remote-write to Mimir/Thanos for scale) → Grafana fleet→rack→node dashboards → Alertmanager on SLO symptoms with runbooks; logs to Loki and traces via OTel, correlated by IDs; ruthless cardinality discipline.)

Success Criteria

You're done with this phase when — without notes:

  • You can explain the three pillars and what each is good/bad at (Ch. 1)
  • You can explain Prometheus pull vs push and when to push (Ch. 2)
  • You can pick counter/gauge/histogram for a given rack signal and read exposition format (Ch. 3)
  • You can describe the exporter pattern and how you'd unify Redfish/IPMI/SNMP (Ch. 4)
  • You can write PromQL for an aggregation, a rate, and an alert threshold (Ch. 5)
  • You can explain cardinality and avoid the label-explosion trap (Ch. 6)
  • You can design a fleet→rack→node Grafana dashboard with USE/RED (Ch. 7)
  • You can explain traces/spans/context propagation and log↔trace correlation (Ch. 8)
  • You can define SLI/SLO/error budget and "alert on symptoms not causes" (Ch. 9)

Interview Q&A

Q1: Design a telemetry pipeline for a rack fleet with Redfish, IPMI, and SNMP devices. A: I'd build unified exporters near the hardware that use the Phase 03 driver abstraction (Redfish-first, IPMI/SNMP fallback), poll each device on their own schedule with caching/ retries (BMCs are fragile), and re-expose everything as Prometheus metrics with a consistent naming/label scheme so a temperature is node_temp_celsius regardless of source. Prometheus scrapes the exporters (pull); at fleet scale I federate or remote-write to a scalable backend (Mimir/Thanos). Grafana provides fleet→rack→node dashboards (USE for hardware, RED for the control plane). Alertmanager fires on SLO symptoms (node down, rack over power budget, sustained overheat, CDU leak) with runbook links, deduped and routed. Logs go to Loki and traces via OpenTelemetry, all correlated by node/rack/trace_id. The two disciplines that keep it healthy: strict cardinality control (bounded labels only) and alerting on symptoms not causes.

Q2: Why does Prometheus pull instead of push, and when would you push? A: Pull means the monitoring system owns the target list (via service discovery) and the scrape schedule, so a target simply being unreachable is itself a signal (up == 0), there's no push infrastructure to overwhelm, backpressure is natural, and I can curl /metrics to see exactly what Prometheus sees. I push in two cases: short-lived batch jobs that may finish before any scrape (via the Pushgateway), and crossing network boundaries I can't scrape (remote-write to a central store). For a rack, exporters expose /metrics and Prometheus pulls; an ephemeral provisioning job might push its result to a Pushgateway.

Q3: Counter vs gauge vs histogram — give a rack example of each. A: A counter only increases (resets on restart) and you query its rate — e.g., pcie_correctable_errors_total, queried as rate(...[5m]) to see error velocity. A gauge goes up and down — e.g., node_temp_celsius or cdu_delta_t_celsius or node_power_watts; most hardware telemetry is gauges. A histogram buckets samples so you can compute quantiles server-side — e.g., provisioning_step_duration_seconds so I can alert on p99 step latency across the fleet. The rule: only-ever-up → counter, up-and-down → gauge, need-a-distribution → histogram.

Q4: What is cardinality and how do you keep it from blowing up? A: Cardinality is the number of distinct time-series, which is the product of each label's distinct values — every unique label-set is a separate series costing memory/disk. It blows up when you put an unbounded value in a label (a request ID, timestamp, full error string, or a serial on a high-churn metric), which can OOM Prometheus. The discipline: labels are only for bounded, low-cardinality dimensions you aggregate/filter by (rack, node, sensor, severity); high-cardinality detail goes in logs/traces, not metric labels; and I watch the series count, alert on it, and review new metrics in code review for label explosions. At thousands of nodes × dozens of sensors, even legitimate labels add up, so label design is real engineering.

Q5: What would you alert on for an AI rack, and how do you avoid alert fatigue? A: Alert on symptoms that matter, with a for: duration to avoid flapping and a runbook on each: node/exporter down (up==0), rack over power budget (sum by(rack)(power) > derated), sustained accelerator overheat or thermal throttling, CDU leak or ΔT/flow anomaly (Phase 04), and SLO breaches (provisioning success rate, control-plane latency). I avoid fatigue by not paging on every cause — a single correctable PCIe error or one retry gets dashboarded, and I alert only on a rate or sustained condition; non-urgent issues go to a ticket, not a page. Every page must be actionable; if it isn't, it becomes a dashboard or a tuned threshold. Symptom-based, SLO-driven alerting keeps the signal-to-noise high so the real page gets read.

Q6: Metrics show provisioning latency spiked. Walk me from alert to root cause. A: The alert (an SLO symptom: p99 provisioning duration breached) is the entry point. I pivot to traces in that window — each provisioning run is a trace of spans (discover, firmware, OS install, configure, validate). The span tree shows which step got slow or errored; say the firmware span's duration ballooned. I follow the span's trace_id into the logs (structured, correlated) to see the specific error — e.g., the BMC returning 503s. I cross-check the metrics for that node/BMC (up, scrape latency, BMC error rate) to confirm it's the device, not my code. So: metric alert → trace to localize the step → logs to get the specific failure → metrics to confirm scope. That metrics→traces→logs correlation, tied by IDs, is exactly why you invest in all three pillars (and it's the Phase 10 RCA loop).

References

  • Prometheus docs — data model, exposition format, metric types, PromQL — https://prometheus.io/docs/
  • Prometheus — instrumentation & naming best practices; histograms vs summaries — https://prometheus.io/docs/practices/
  • Brian Brazil, Prometheus: Up & Running — the canonical book
  • Grafana docs — dashboards, variables, provisioning (as-code) — https://grafana.com/docs/
  • OpenTelemetry — traces/metrics/logs, context propagation, the collector — https://opentelemetry.io/docs/
  • Google SRE Book — SLIs/SLOs/error budgets; "alerting on symptoms" — https://sre.google/books/
  • The USE method (Brendan Gregg) & the RED method (Tom Wilkie) — dashboard frameworks
  • redfish_exporter / snmp_exporter / ipmi_exporter — reference exporter implementations
  • Cross-track: Head of SWE — GPU, Phase 10 (observability/production)