« Phase 14 · Warmup · Track Overview

Core Contributor — Working on the Engines Themselves

What it takes to contribute to OpenTelemetry, Prometheus, a tracing backend, or the SRE tooling your bank builds. Read this if you want to understand the systems rather than configure them.

Table of Contents

• 1. Why read the engines
• 2. OpenTelemetry: the architecture
• 3. Semantic conventions, and contributing one
• 4. The Collector
• 5. Tail sampling, mechanically
• 6. Prometheus: TSDB and the query engine
• 7. Recording rules and rule evaluation
• 8. Alertmanager
• 9. Building in-house SRE tooling
• 10. Testing observability code
• 11. Contributing

1. Why read the engines

Because the failure modes are internal. "Prometheus OOMed during the incident" is explained by the inverted index's memory model and by nothing in the configuration reference.

Because the semantic conventions are being written now. The GenAI conventions are young and actively evolving, which means a bank running agents in production has genuinely useful experience to contribute — this is one of the few areas in this track where you can affect the standard rather than just adopt it.

2. OpenTelemetry: the architecture

Three layers, and the separation is the point:

   API        ── what your code calls; a no-op by default
    │
   SDK        ── the implementation: sampling, batching, resource detection
    │
   Exporter   ── the wire format (OTLP) ──► Collector ──► backends

The API is a no-op unless an SDK is installed. Which is why a library can instrument itself with the OTel API and impose nothing on its consumers — they pay nothing unless they opt in. That is the design decision that made OTel adoptable, and it is why library authors should use the API and never the SDK.

The SDK's pipeline:

   Tracer ──► Span ──► SpanProcessor ──► SpanExporter
                          │
                    BatchSpanProcessor (queue, batch, retry, drop)

BatchSpanProcessor is where the production behaviour lives, and its parameters are the ones that matter:

The first row is the one that causes confusion: under load, spans are dropped, and the drop is counted in an internal metric nobody looks at. A trace missing its middle is usually this, not a propagation bug.

Worth reading in open-telemetry/opentelemetry-python: sdk/trace/__init__.py (span lifecycle), sdk/trace/export/ (the processors), sdk/trace/sampling.py.

3. Semantic conventions, and contributing one

Attribute names are a specification, not a convention in the loose sense (open-telemetry/semantic-conventions). Using the spec's names is what lets any backend chart your data without custom queries.

The GenAI set, which is what this phase depends on:

gen_ai.system:                  "azure.openai"
gen_ai.operation.name:          "chat"
gen_ai.request.model:           "gpt-frontier-2026-02-11"
gen_ai.request.temperature:     0.0
gen_ai.request.max_tokens:      2048
gen_ai.response.model:          "gpt-frontier-2026-02-11"   # may DIFFER from the request
gen_ai.response.finish_reasons: ["stop"]
gen_ai.usage.input_tokens:      4812
gen_ai.usage.output_tokens:     380

The gen_ai.response.model row is worth pausing on: it exists precisely because a provider can serve a different model than you asked for, and recording both is what makes the warmup's "did the provider move us?" question answerable.

Where the conventions are still thin, and where a bank running agents has something to say:

Contributing one is a real process and worth knowing: open an issue describing the use case, propose attributes in a PR against the YAML model, iterate with the SIG, and it lands as experimental before stabilizing. The bar is a genuine use case with more than one implementer — which is exactly what a production bank platform has.

And the discipline in the meantime: prefix your own attributes with your namespace (bank.agent.side_effect) so they never collide with a future standard name. Inventing gen_ai.cost is how you get a conflict when the spec lands.

4. The Collector

A separate process that receives, processes and exports telemetry — and it is the piece that makes OTel operationally viable.

receivers:  [otlp]
processors: [memory_limiter, tail_sampling, batch]   # ← ORDER MATTERS
exporters:  [prometheus, otlphttp/tempo]

Processor order is a real correctness concern:

• memory_limiter first, always. It refuses data when memory is high, and any processor before it can OOM the collector — which loses everything in flight.
• tail_sampling before batch. Sampling after batching means you have already paid to assemble batches you then discard.
• batch last, so everything downstream sees efficient batches.

Deployment topology, and the choice matters more than it looks:

Tail sampling requires the gateway topology and requires all spans of a trace to reach the same collector instance — which means a load-balancing exporter keyed on trace id in front of it. Getting that wrong produces partial traces sampled inconsistently, which is worse than not sampling: you keep half of the interesting traces.

Worth reading in open-telemetry/opentelemetry-collector-contrib: processor/tailsamplingprocessor/, exporter/loadbalancingexporter/.

5. Tail sampling, mechanically

   spans arrive ──► buffered by trace_id ──► decision_wait elapses
                                          ──► evaluate policies ──► keep or drop

The policies compose with OR — any matching policy keeps the trace:

tail_sampling:
  decision_wait: 30s          # ← must exceed your longest trace
  num_traces: 100000          # ← in-memory cap
  policies:
    - { name: errors,  type: status_code, status_code: { status_codes: [ERROR] } }
    - { name: slow,    type: latency,     latency: { threshold_ms: 5000 } }
    - { name: actions, type: string_attribute,
        string_attribute: { key: bank.tool.side_effect,
                            values: [write_non_idempotent, irreversible] } }
    - { name: sample,  type: probabilistic, probabilistic: { sampling_percentage: 5 } }

Three operational realities:

decision_wait must exceed your longest trace. A 40-second agent run under a 30-second wait is decided on partial data — and the late spans arrive after the decision and are dropped, so the kept trace is truncated. For an agent platform this is the parameter to get right, and 30 s is usually too short.

Memory is num_traces × spans × span size. At 100k traces × 12 spans × 2 KB that is ~2.4 GB, and exceeding num_traces evicts the oldest — silently.

The collector is now on the path. If it fails, telemetry stops. It must fail open — the application must never block on export — and it should have its own alerting, which is a slightly uncomfortable recursion worth planning for.

6. Prometheus: TSDB and the query engine

   /metrics ──scrape──► TSDB
                          │
                    head block (in memory, 2h) ──► WAL
                          │  compaction
                    persistent blocks (2h, 6h, 24h...)

The head block holds the last two hours in memory plus a write-ahead log. On restart the WAL is replayed, which for a large head is minutes — and if the head was large because of cardinality, the replay OOMs and you get a crash loop. That is the failure from §10 of the deep dive, and reading the head-block code is what makes it concrete.

Series storage:

   series = metric name + label set  →  a unique ID
   samples: delta-of-delta timestamps + XOR-compressed float64  (Gorilla)

The compression is excellent — ~1.3 bytes per sample — which is why people underestimate the cost. The samples are cheap; the index is not. Each active series costs 1–3 KB of memory for its entry in the inverted index, regardless of how few samples it has. Which is exactly why cardinality is a memory problem rather than a disk problem, and why a metric with a million label combinations and one sample each is catastrophic.

PromQL evaluation:

   rate(http_requests_total[5m])
     → for each series matching the selector
     → take samples in the window
     → (last − first) / seconds, with extrapolation to the window edges

Two behaviours that surprise people: rate() extrapolates to the window boundaries, so it can report a value slightly outside the observed range; and it requires at least two samples, so a series that appears mid-window produces nothing.

Worth reading in prometheus/prometheus: tsdb/head.go, tsdb/index/, promql/engine.go.

7. Recording rules and rule evaluation

groups:
- name: sli
  interval: 30s
  rules:
  - record: sli:availability:ratio_rate5m
    expr: |
      sum(rate(requests_total{outcome="success",valid="true"}[5m]))
        / sum(rate(requests_total{valid="true"}[5m]))

Why they exist: a 30-day window over raw data is a 1–10 second query. Precomputing at several window lengths turns a dashboard panel from unusable to instant.

Three properties worth knowing:

Rules within a group evaluate sequentially, so a rule can depend on one defined above it in the same group. Across groups, evaluation is concurrent and the ordering is not guaranteed — which makes a cross-group dependency a race that appears as an occasionally-empty panel.

A naming convention exists and it is worth following: level:metric:operations — sli:availability:ratio_rate5m reads as "aggregated at the SLI level, the availability metric, a 5-minute rate ratio". A directory of ad-hoc rule names becomes unnavigable at about fifty rules.

Rule evaluation is itself a load. prometheus_rule_evaluation_duration_seconds is a metric worth alerting on: rules that take longer than the interval fall behind, and the symptom is a dashboard that is quietly stale rather than an error.

8. Alertmanager

   Prometheus ──alerts──► Alertmanager
                             │
                   dedupe ──► group ──► inhibit ──► silence ──► route ──► notify

The pipeline order is the design, and each stage exists for a failure somebody had:

Dedupe — multiple Prometheus replicas send the same alert.

Group — group_by: [service, severity] with group_wait: 30s batches alerts from one incident into one notification. This is the single highest-value setting for reducing page volume, and it is usually left at the default.

Inhibit — a firing critical suppresses the related warnings (§6 of the deep dive).

Silence — a time-bounded mute, applied during maintenance. Worth having a policy that silences expire and require a reason, or the silence list becomes permanent.

Route — a tree matching on labels, with continue: true for multi-destination.

The clustering detail worth knowing: Alertmanager instances gossip (memberlist) so that N replicas send one notification rather than N. If the gossip is misconfigured, you get duplicate pages and it looks like an alerting bug rather than a clustering one.

9. Building in-house SRE tooling

The parts you build, and the properties that make them last.

The SLI definition is a specification, in code, reviewed. Not a Grafana panel:

@dataclass(frozen=True)
class SloSpec:
    name: str
    target: float
    window_days: int
    validity: ValidityPredicate      # ← in code, versioned, reviewed
    owner: str                       # ← a named human

Generate the recording rules, the alert rules and the dashboard from that spec. One definition, three artifacts, no drift — and the drift is the actual problem: a dashboard, an alert and a report that disagree about the same SLO is a weekly argument.

Cardinality checked in CI. A metric registry with estimated cardinalities, and a test that fails the build. This is the only enforcement that works, because the alternative notices during the crash loop.

Everything deterministic. Injected clock, no time.time() in a code path a test touches. The lab's 132 tests run in 0.12 s with no flakiness, and that is a direct consequence.

Property tests on the invariants:

# SLI ∈ [0, 1] for any event sequence
# an empty window is exactly 1.0
# remaining budget is never negative
# total self time across a tree == the root's duration
# adding a label never decreases the series count
# the ladder's active steps are always a prefix of the ladder
# a burn rate of 0 never fires any rule

The self-time one is the most valuable — Hypothesis finds the overlapping-children case immediately, and that is exactly the bug that makes teams abandon self time.

Generate the post-mortem timeline from the trace, the decision records and the alert history. Memory during an incident is unreliable, and the artifacts already exist.

10. Testing observability code

Two that are usually missing.

promtool test rules is the one nobody uses and it is built in: you write a synthetic series and assert which alerts fire at which times. An alert rule is production code with no tests otherwise, and rule changes are exactly the kind of thing that silently stops firing.

tests:
- interval: 1m
  input_series:
  - series: 'requests_total{outcome="success",valid="true"}'
    values: '0+100x60'
  - series: 'requests_total{outcome="error",valid="true"}'
    values: '0+10x60'
  alert_rule_test:
  - eval_time: 60m
    alertname: PlatformErrorBudgetFastBurn
    exp_alerts: [ { exp_labels: { severity: page } } ]

Alert drills. Fire a real page into the rota, deliberately, quarterly. It tests three things at once: that the routing works, that the runbook is current, and that the on-call engineer can actually resolve it. Every drill finds something, and the thing it finds is usually the runbook.

11. Contributing

OpenTelemetry (open-telemetry) — many repos, SIG-structured, welcoming. The highest-value contribution from this phase's perspective is to semantic-conventions: the GenAI set is experimental and there are real gaps (cost, agent runs, tool side-effect classes, guardrails). A bank running agents in production has exactly the evidence the SIG asks for.

Language SDKs and instrumentation libraries are the other approachable entry point — an instrumentation for a framework you use is self-contained and immediately useful.

Collector contrib (opentelemetry-collector-contrib) — Go, very active. Processors and exporters are modular; the tail-sampling processor in particular has open work around agent-shaped traces.

Prometheus (prometheus/prometheus) — Go, mature, high bar for TSDB changes. promtool, documentation and exporters are the accessible surface. Read the TSDB code regardless of whether you contribute; it is the clearest explanation of why cardinality behaves the way it does.

Alertmanager (prometheus/alertmanager) — Go, smaller, and the routing/inhibition logic is readable in an afternoon.

Grafana (grafana/grafana, and Tempo, Loki, Mimir) — Go + TypeScript, active, and Tempo's TraceQL is a good area for someone who has spent time querying agent traces and knows what is missing.

Sloth (slok/sloth) — generates Prometheus SLO rules from a spec. Small, focused, and directly the §9 idea; a good first contribution and a good thing to read before building your own.

For all of them the useful preparation is the same: implement the mechanism yourself first — the lab is a small version of exactly that — then read theirs and find every place they differ. The differences are where the real engineering is.