🛸 Hitchhiker's Guide — Phase 07: Telemetry & Observability
Read this if: "Prometheus, Grafana, tracing" is a stack you've seen but not built. Compressed map of metrics, the cardinality trap, and the metrics→traces→logs loop.
0. The 30-second mental model
Three pillars: metrics (cheap time-series — that something's wrong), logs (the detail — what), traces (the path — where the time went / what failed). Tie them by shared IDs and you can debug a fleet. The role-specific hard part is unification: one exporter normalizing Redfish/IPMI/SNMP/CDU into consistent metrics. Two laws keep you sane: bound your label cardinality and alert on symptoms, not causes.
1. Prometheus in one breath
Pull model: Prometheus scrapes /metrics on a schedule → TSDB. A series = name{labels}.
Pull (not push) because the system owns the target list (so up==0 is a signal), no push
infra to overwhelm, and curl /metrics shows what it sees. Push only for batch jobs
(Pushgateway) or across boundaries (remote-write).
2. Metric types (pick right)
| type | meaning | rack example | query |
|---|---|---|---|
| counter | only goes up | pcie_correctable_errors_total | rate(x[5m]) |
| gauge | up & down | node_temp_celsius, node_power_watts, cdu_delta_t_celsius | raw / avg by |
| histogram | distribution | provisioning_step_duration_seconds | histogram_quantile(0.99,...) |
Exposition format: name{label="v"} 123. Only-up→counter; up/down→gauge; need p99→histogram.
3. The exporter pattern (the unification job)
Hardware speaks Redfish/IPMI/SNMP; Prometheus speaks HTTP. An exporter bridges it — use the
Phase 03 driver abstraction, poll devices on your own schedule (cache, retry, timeout — BMCs
are fragile), and serve consistent metrics so a temp is node_temp_celsius regardless of
source. Add an up/health metric per device.
4. PromQL you'll actually write
avg by (rack) (node_temp_celsius) # aggregate
sum by (rack) (node_power_watts) > 17000 # over the Phase 01 budget
rate(pcie_correctable_errors_total[5m]) > 1 # error velocity
up{job="rack-exporter"} == 0 # device/exporter down
Recording rules = precompute; alerting rules = PromQL + for: + severity + runbook → Alertmanager.
5. The cardinality law
series = product of label values. Never put unbounded values (IDs, timestamps, messages) in labels → OOM. Labels = bounded dimensions (rack/node/sensor/severity). Detail → logs/traces.
6. Grafana dashboards
JSON (version it). $rack/$node variables. USE (Utilization/Saturation/Errors) for
hardware, RED (Rate/Errors/Duration) for services. Hierarchy: fleet → rack → node. Every
panel answers a question or backs an alert.
7. Traces + correlation (the debug loop)
Trace = tree of spans (name, duration, parent) sharing a trace_id; context propagation carries it across calls/services. Logs carry the trace_id. The loop: metric alert → traces in window (exemplars) → failing span → its logs. OTel = the vendor-neutral standard; the collector exports to Jaeger/Tempo/Prometheus.
8. SLO discipline
SLI (measured) → SLO (target) → error budget (1−SLO). Alert on symptoms (SLO breach), not causes — cause alerts → fatigue → missed real page. Every page is actionable + has a runbook (Phase 12).
9. The "done this before" tells
"Counter or gauge?" · "What's the cardinality of that label?" · "Are we alerting on a symptom or a cause?" · "Does the log carry the trace_id?" · "Pull or push for that?" · "What's the SLO?"
10. How this phase pays off later
The exporter is reused in the Phase 13 capstone; alerts/runbooks feed Phase 10 RCA and
Phase 12 ops; the fleet pipeline is system-design/02; thresholds come from Phases 01/04.