Phase 07 — Telemetry & Observability
Difficulty: ⭐⭐⭐☆☆ | Estimated Time: 2 weeks Roles supported: Rack Management SWE (Senior/Staff), SRE, Platform/Infra SWE Hardware needed: none — the exporter + Prometheus/Grafana run locally (Docker optional)
Why This Phase Exists
The JD is explicit and repeated: "Develop dashboards and unified telemetry pipelines using Prometheus and Grafana" and "Implement observability features such as logging, metrics, and tracing to support troubleshooting and operational reliability." You cannot operate — let alone debug in production (Phase 10) — what you cannot see. Observability is how a rack fleet stops being a black box.
The hard, role-specific problem is unification: a rack's signals come from Redfish nodes, IPMI BMCs, SNMP PDUs/switches, and CDUs (Phases 03–04), each with its own model. Your job is to normalize them into one telemetry pipeline — metrics (Prometheus), dashboards (Grafana), plus logs and traces — so an operator sees one coherent view and alerts fire on the right signals (the ΔT/PCIe/power thresholds from Phases 01/04).
You'll build a Redfish→Prometheus exporter that serves real exposition-format metrics, with PromQL alert rules and a Grafana dashboard, and an OpenTelemetry tracing + structured logging demo that makes a multi-step rack workflow debuggable.
Concepts
The three pillars (and the fourth)
- Metrics: numeric time-series (temperature, power, error counts); cheap, aggregatable, the basis of dashboards and alerts
- Logs: discrete structured events (Phase 02); the detail when a metric says "something's wrong"
- Traces: the path of a request/workflow across components (where time went, what failed)
- (Events/alerts): state changes and the notifications they trigger
Prometheus (the metrics standard)
- The pull model: Prometheus scrapes
/metricsHTTP endpoints on an interval (vs push) - Exposition format:
metric_name{label="value"} number— the text format an exporter serves - Metric types: counter, gauge, histogram, summary; when to use each
- Exporters: the adapter pattern —
node_exporter,redfish_exporter,snmp_exporter,ipmi_exporter; you write one for rack devices - PromQL: rate(), aggregation (sum/avg by label), threshold queries; recording rules; alerting rules → Alertmanager
- Cardinality (the #1 Prometheus footgun): labels explode series count; design labels carefully
- Service discovery (k8s/file-based); scrape configs; the TSDB
Grafana (dashboards)
- Data sources, panels, variables/templating; dashboards-as-code (JSON); per-rack/per-fleet views
- Designing a useful dashboard: the USE/RED methods; what to put on a rack overview
Logs & tracing
- Structured logging (Phase 02) → aggregation (Loki/ELK); correlation IDs
- OpenTelemetry: traces, spans, context propagation, trace_id/span_id; the OTel collector
- Tying logs↔traces↔metrics with shared IDs (exemplars); the debugging payoff
Operational practice (the senior layer)
- SLI/SLO/error budgets; alert on symptoms (SLOs) not causes; alert fatigue
- The unified pipeline architecture: collectors near hardware → central TSDB/log store → dashboards/alerts
- What to alert on for a rack: thermal (ΔT, throttle), power (over-budget, PSU), PCIe (AER, degraded link), node down, firmware drift
Labs
Lab 01 — Redfish → Prometheus Exporter + Grafana
| Field | Value |
|---|---|
| Goal | Build a Prometheus exporter that pulls rack telemetry (Redfish/IPMI/SNMP, normalized — Phase 03) and serves it in exposition format on /metrics, with PromQL alert rules and a Grafana dashboard. |
| Concepts | Pull model, exposition format, metric types, labels/cardinality, PromQL alerts, Grafana dashboards-as-code, the exporter pattern. |
| Steps | 1) python3 solution.py — starts the exporter; curl localhost:9101/metrics. 2) Read the exposition output and the metric/label design. 3) Read alerts.yml (PromQL) and dashboard.json (Grafana). 4) Extensions: scrape it with real Prometheus + Grafana. |
| Stack | Python 3 stdlib (http.server) + real Prometheus alert rules + Grafana JSON |
| Output | A working /metrics endpoint + alert rules + a Grafana dashboard for a rack. |
| How to Test | Asserts verify exposition-format correctness, metric types/labels, and that threshold values would fire the alert rules. |
| Talking Points | Pull vs push; counter vs gauge; the cardinality trap; alert on symptoms not causes; why a unified exporter across Redfish/IPMI/SNMP. |
| Resume Bullet | "Built a Prometheus exporter unifying Redfish/IPMI/SNMP rack telemetry into exposition-format metrics, with PromQL alerting rules and a Grafana dashboard for thermal/power/PCIe health." |
| Extensions | Run real Prometheus scraping it + Grafana importing the dashboard; add a histogram for scrape latency; add recording rules; wire it to the Phase 04 PDU/CDU/PCIe sim. |
Lab 02 — OpenTelemetry Tracing + Structured Logging
| Field | Value |
|---|---|
| Goal | Make a multi-step rack workflow (e.g., provisioning, Phase 05) debuggable: emit spans across steps with a propagated trace context, and structured logs carrying the trace_id, so you can answer "where did it fail and how long did each step take?" |
| Concepts | Traces/spans, context propagation, trace_id/span_id, structured logs, log↔trace correlation, the OTel data model. |
| Steps | 1) python3 solution.py — runs a traced workflow with an injected failure. 2) Read the span tree + the correlated logs. 3) See total/step timing and the failed span. 4) Extensions: export to a real OTel collector/Jaeger. |
| Stack | Python 3 stdlib (a minimal OTel-shaped tracer) |
| Output | A trace (span tree with timings) + structured logs correlated by trace_id, pinpointing a failure. |
| How to Test | Asserts verify span parent/child nesting, context propagation, error-span marking, and that logs carry the active trace_id. |
| Talking Points | Why traces beat logs for "where did time go"; context propagation across components; correlating logs/traces/metrics by ID; sampling. |
| Resume Bullet | "Instrumented a rack workflow with OpenTelemetry-style tracing and trace-correlated structured logging, pinpointing failures and per-step latency across a distributed flow." |
| Extensions | Export OTLP to Jaeger/Tempo; add metrics exemplars linking a slow metric to its trace; propagate context across a real HTTP call. |
Deliverables Checklist
-
Exporter serves valid Prometheus exposition format on
/metrics - You can choose counter vs gauge vs histogram correctly and explain why
- You can explain the cardinality trap and design labels to avoid it
- Alert rules fire on the right rack symptoms (thermal/power/PCIe/node-down)
- You can read/build a Grafana dashboard and explain USE/RED
- Tracing demo shows a span tree with timings and a marked failure
- Logs carry the trace_id so logs↔traces correlate
- You can articulate SLI/SLO and "alert on symptoms, not causes"
Interview Relevance
- "Design a telemetry pipeline for a rack fleet with Redfish, IPMI, and SNMP devices."
- "Prometheus pull vs push — why pull? When would you push?"
- "Counter vs gauge vs histogram — give an example of each for a rack."
- "What is metric cardinality and how do you keep it under control?"
- "What would you alert on for an AI rack, and how do you avoid alert fatigue?"
- "Metrics say latency spiked. How do traces and logs help you find why?"
- "What's an SLO and an error budget, and why alert on symptoms not causes?"