System Design 02 — Unified Telemetry Pipeline

"Design fleet-wide telemetry for heterogeneous rack hardware (Redfish/IPMI/SNMP/CDU)." Phases: 03 (drivers), 07 (observability), 01/04 (what to measure), 10 (debugging consumer).

1. Drive the requirements

  • Scale: 10,000 nodes × dozens of sensors each (temps, power, fans, PCIe AER, CDU ΔT/flow/leak).
  • Heterogeneity: Redfish nodes, IPMI BMCs, SNMP PDUs/switches, Modbus/SNMP CDUs.
  • Consumers: dashboards, alerting, and debugging (the metrics→traces→logs loop, P10).
  • Constraints: BMCs are slow/fragile (don't hammer them); cardinality must stay bounded.

2. Architecture

 [BMC/PDU/CDU/switch] --Redfish/IPMI/SNMP/Modbus--> [Unified exporter (per site/proxy)]  (P03 drivers)
        |  (poll on OWN schedule, cache, retry/timeout — not on every scrape)
        v
   /metrics (consistent names/labels)  --scrape-->  [Prometheus per shard]
                                                         | remote-write / federate
                                                         v
                                              [Mimir/Thanos (long-term, global)]  --> [Grafana]
   structured logs (P02) -> [Loki]                                         |
   traces (OTel, P07) ----> [Tempo/Jaeger]                                  +--> [Alertmanager -> page/Slack]
            (all correlated by node/rack/trace_id)

3. The hard decisions (and the tradeoffs)

  • Unified exporter via the driver abstraction (P03/P07): one consistent metric/label scheme (node_temp_celsius regardless of source). Tradeoff: a mapping layer to maintain, vs. dashboards/ alerts that don't care about protocol.
  • Poll-on-own-schedule + cache, serve cached on scrape (P07): BMCs are fragile/slow, so the exporter polls them on its cadence with retries/timeouts and serves cached values to Prometheus. Tradeoff: staleness bounded by the poll interval, vs. hammering BMCs synchronously per scrape.
  • Pull (Prometheus) + remote-write to a scalable backend: pull gives up==0 as a signal and easy testing; one Prometheus can't hold 10k×dozens of series, so shard per site and remote-write to Mimir/Thanos for the global, long-term view. Tradeoff: more infra, but it scales and survives a shard loss.
  • Events for state changes, polling for metrics (P07): Redfish EventService/SSE + SNMP traps for faults (low latency), scheduled polling for metrics and as the backstop (events get lost).
  • Cardinality discipline (P07): bounded labels only (rack/node/sensor/severity); high-cardinality detail (request/serial/message) goes to logs/traces, never metric labels. This is the #1 thing that kills a fleet telemetry system.

4. What to alert on (P07/P01/P04/P08)

Symptoms, with for: + severity + runbook: node/exporter down (up==0), rack over power budget (sum by(rack)(power) > derated), sustained accelerator overheat/throttle, CDU leak/ΔT anomaly, rising PCIe AER rate, SLO breaches. Not single correctable errors (dashboard those). Alert on rates and sustained conditions to avoid fatigue.

5. Failure modes & operations

  • A BMC stops responding → the exporter's per-device up metric fires NodeDown; cached values go stale and are marked.
  • A telemetry shard dies → Mimir/Thanos retains history; another shard's scrape continues; alert on the shard.
  • Alarm storm from a PDU loss (P01) → correlate by failure domain to one root cause, not 6 pages.
  • Debugging consumer (P10): an SLO alert → traces in the window → logs for the failing span → RAS/telemetry for the hardware — the unified pipeline is what makes that loop possible.

6. Honest limits / what I'd measure

The cardinality budget (series count per node × fleet), the poll interval vs. staleness tradeoff per metric class, and the cost of long-term storage. I'd start with a tight metric set and grow it under review (every new metric reviewed for label explosions).