Lab 02 — OpenTelemetry Tracing + Structured Logging

Phase: 07 — Telemetry & Observability | Difficulty: ⭐⭐⭐☆☆ | Time: 3–5 hours Language: Python 3 (stdlib) | Hardware: none

Concept primer: ../WARMUP.md Ch. 8, ../HITCHHIKERS-GUIDE.md §7.

Run

python3 solution.py     # a traced workflow (success + failure) with correlated logs

0. The mission

Make a multi-step rack workflow debuggable: emit a tree of spans with timings and a propagated trace context, and structured logs that carry the active trace_id — so "where did it fail and how long did each step take?" is answerable in seconds, and any log line links to its span. This is the deep-dive half of observability (metrics flag that; traces show where).

1. What to study

  • Context propagation: a contextvars current-span carries the trace across nested with span(...) blocks — no manual passing — exactly like OpenTelemetry's implicit context.
  • The span tree: provision is the root; discover/firmware/os_install/configure/validate are children sharing one trace_id. The printed tree shows per-step duration_ms.
  • Failure pinpointing: node-B's os_install span is marked ✗ ERROR, and the error propagates to the root span — you instantly see which step failed and how long the others took.
  • Log↔trace correlation: every log(...) line auto-includes the active trace_id/span_id, so from a log you jump to the trace and vice-versa.

2. Extensions (do them)

  1. Use real OpenTelemetry: pip install opentelemetry-sdk opentelemetry-exporter-otlp, replace the toy tracer, and export OTLP to Jaeger or Tempo (docker run jaegertracing/ all-in-one); view the trace tree in the UI.
  2. Cross-service propagation: split the workflow across a real HTTP call and propagate the traceparent header so the trace spans both processes.
  3. Metric exemplars: link a slow provisioning_step_duration_seconds sample (Lab 01) to its trace, so a dashboard spike jumps straight to the trace (the metrics→traces leap).
  4. Sampling: add head/tail sampling and discuss the cost/visibility tradeoff at fleet scale.

3. What this lab proves

You can instrument a workflow so production failures are diagnosable — the JD's "logging, metrics, and tracing... to support troubleshooting." "Latency spiked — find why" becomes the metrics→traces→logs loop (Phase 10 RCA), demonstrated end to end. Combined with Lab 01's metrics/alerts, you have the full observability surface a rack platform needs.