Lab 01 — Unified Rack Telemetry → Prometheus Exporter

Phase: 07 — Telemetry & Observability | Difficulty: ⭐⭐⭐☆☆ | Time: 4–6 hours Language: Python 3 (stdlib) + real Prometheus/Grafana config | Hardware: none

Concept primer: ../WARMUP.md Ch. 3–7, ../HITCHHIKERS-GUIDE.md §1–§6.

Run

python3 solution.py     # starts the exporter, prints /metrics, self-tests
# in another shell while it runs:  curl localhost:<port>/metrics

Files

  • solution.py — the exporter (serves real Prometheus exposition format on /metrics).
  • alerts.yml — Prometheus alerting rules (the rack symptoms you page on).
  • dashboard.json — a Grafana dashboard-as-code (fleet→rack hardware view).

0. The mission

Build the "unified telemetry pipeline" the JD asks for: one exporter that turns normalized Redfish/IPMI/SNMP rack telemetry (Phase 03) into Prometheus metrics, with the alert rules and Grafana dashboard that make it operable. The point is unification — one consistent metric/label scheme regardless of which protocol the data came from.

1. What to study in the output

  • Metric types: temp/power are gauges, AER errors a counter (queried as a rate), and a per-node up gauge — the right type for each signal (WARMUP Ch. 3).
  • Bounded labels only: rack, node, sensor, job — the self-test asserts no other labels exist, so cardinality can't explode (WARMUP Ch. 6). No serials, IDs, or timestamps in labels.
  • No stale data for down devices: node-02 is unreachable → up=0 and we don't emit a fake temperature for it.
  • The hot accelerator (92°C) is exactly what alerts.yml's AcceleratorHot (>85, for 5m) fires on; node-01's power pushes toward the RackPowerOverBudget rule (the Phase 01 budget).

2. The real config

alerts.yml shows symptom-based alerting (for: to avoid flapping, severity, runbook links, and rate for PCIe errors not raw count). dashboard.json shows a $rack variable and USE- style panels (nodes up/down, power vs budget threshold, accel temps with 85/95 thresholds, PCIe error rate).

3. Extensions (do them)

  1. Run the real stack: docker run prom/prometheus with a scrape config pointing at this exporter, load alerts.yml, run docker run grafana/grafana, add Prometheus as a data source, and import dashboard.json. Watch the alert fire on the 92°C accelerator.
  2. Wire it to Phase 04: pull live PDU/CDU/PCIe values from the Phase 04 Lab 02 simulator so the metrics reflect ΔT, leak state, and degraded links.
  3. Add a histogram: scrape_duration_seconds and provisioning_step_duration_seconds; query p99 with histogram_quantile.
  4. Background polling + cache: poll the (fake) BMCs on a timer with retries/timeouts (Phase 02) and serve cached values on scrape — the production pattern for fragile BMCs.
  5. Cardinality experiment: add a high-cardinality label (a timestamp) and observe how the series count would explode; then remove it and explain why.

4. What this lab proves

You can build the metrics backbone of a rack-management platform — correctly typed, cardinality- safe, unified across protocols, alertable, and dashboardable. "Design a telemetry pipeline for a rack fleet" becomes a demonstration, and this exporter is reused in the Phase 13 capstone and expanded in system-design/02 (the fleet-scale pipeline).