« Phase 14 · Warmup · Track Overview

Deep Dive — Mechanisms and Failure Modes

The warmup established what the pieces are. This takes them apart: how each mechanism actually works, what breaks, and what the fix costs.

Table of Contents

• 1. The event ratio, mechanically
• 2. Where the validity predicate lives
• 3. Windows: rolling, calendar, and alignment
• 4. Burn-rate arithmetic, in full
• 5. for: versus the short window
• 6. Alert composition and inhibition
• 7. Self time, precisely
• 8. Sampling
• 9. Context propagation
• 10. Cardinality, and how backends actually die
• 11. Exemplars
• 12. Ladder dynamics
• 13. Cost attribution
• 14. Forecasting, and what a linear fit hides
• 15. Performance
• 16. Failure modes

1. The event ratio, mechanically

    SLI(window) = |{e ∈ window : valid(e) ∧ good(e)}| / |{e ∈ window : valid(e)}|

Three properties that follow directly, and each is why the shape was chosen:

Composability across windows. Sum the numerators, sum the denominators. Two hours at 99% and 99.5% with equal traffic is 99.25% over the pair — and you can compute it from the stored counts without the raw events. A stored average of ratios would be wrong under unequal traffic, which is a real bug in dashboards that aggregate pre-computed SLIs.

Composability across dimensions. Regional SLIs sum to a global one the same way. Which is exactly what a percentile cannot do (§5 of the warmup).

A budget falls out. allowed_bad = valid × (1 − SLO). Countable, spendable, allocatable.

The implementation detail that matters: store two counters, not a ratio. good_total and valid_total as monotonic counters, with the ratio computed at query time over a rate(). Storing the ratio loses the ability to re-aggregate, and it is the decision that is expensive to reverse.

And the empty-window convention. No valid events means 0/0. Define it as 1.0:

• zero traffic is not an outage;
• a service with a nightly quiet period would otherwise burn its whole budget every night;
• and an alert that fires when nobody is using the system is an alert people disable.

In Prometheus this is the rate() of an empty vector, which produces no series — so the rule needs or vector(0) on the numerator, and that is a real gotcha rather than a detail.

2. Where the validity predicate lives

Three placements, and only one survives:

Emitting a valid label at the source is right for a reason that is not obvious: the predicate is a product decision, and it should be reviewable in code alongside the handler that classifies the outcome. A predicate that lives in a Grafana panel is a predicate nobody reviews.

The cost is that changing it does not apply retroactively — historical data carries the old classification. Which is arguably correct: an SLO whose denominator silently changed last month is an SLO you cannot reason about across the boundary. Version the predicate, and note the change on the dashboard.

And the second-order effect worth anticipating: valid is now a label, so it participates in cardinality (§10). It is a boolean, so it costs a factor of two — which is affordable and should be counted.

3. Windows: rolling, calendar, and alignment

Rolling — the last 30 days, continuously. The honest measure, and the one to use.

Calendar — this month. Resets on the 1st, so an outage on the 31st costs nothing and the same outage on the 1st costs everything. It exists because contracts are written monthly, which is a billing artifact rather than a reliability one.

The implementation problem with rolling windows is storage: a true 30-day rolling ratio needs 30 days of resolution. Prometheus does this with increase(...[30d]), which is expensive, so the standard approach is a recording rule that pre-computes the ratio at several window lengths:

- record: sli:availability:ratio_rate1h
  expr: sum(rate(good_total[1h])) / sum(rate(valid_total[1h]))
- record: sli:availability:ratio_rate30d
  expr: sum(rate(good_total[30d])) / sum(rate(valid_total[30d]))

Two subtleties:

rate() extrapolates. Over a short window with sparse data it can produce a value slightly outside the observed range, which for a ratio near 1.0 occasionally yields 1.0001. Clamp it, or use increase() over aligned windows.

Window alignment. A 30-day window evaluated every 15 seconds is a sliding window, so the budget is continuous rather than stepped. That is what you want, and it means "budget remaining" changes even when nothing is happening — old failures aging out. Somebody will report that as a bug.

4. Burn-rate arithmetic, in full

    error_rate    = bad / valid                       (in the window)
    budget_rate   = 1 − SLO                           (sustainable, per unit time)
    burn_rate     = error_rate / budget_rate

Worked, for a 99.5% SLO (budget 0.005):

Reading that table is what makes the ladder intuitive. 14.4× is not "a disaster" — it is a 7.2% error rate, which is a bad afternoon that would exhaust the month in two days. That is exactly the right thing to page on, and it is much lower than people guess.

The inverse, which is the useful form for a runbook:

    hours_to_exhaustion = 720 × budget_remaining_fraction / burn_rate

At 14.4× with a full budget, 50 hours. At 100× with 20% left, 1.4 hours — which is the number that tells an incident commander whether to degrade now or investigate first.

The floating-point trap. A mathematically-exact 14.4 computes as 14.399999999999986, so a naive >= never fires. And a budget consumed exactly to its limit leaves ~3.5e-14 remaining rather than 0. Both need an epsilon, and both are the kind of bug that produces "the alert did not fire and I cannot explain why".

5. for: versus the short window

They look like the same mechanism and are not:

for: requires the condition to hold continuously for a duration before firing. It is anti-flap on the leading edge.

The short window is part of the expression — it is anti-stale on the trailing edge. Without it:

   t=0    a 5-minute outage burns 2% of the budget
   t=5    fixed
   t=5..60  the 1-hour window still shows 2% consumed → the alert stays lit for 55 minutes

An alert that stays lit for 55 minutes after the fix is an alert people learn to close without reading, which is how a good alerting system becomes decorative.

Use both. And note for: interacts with the volume guard: with for: 2m and a 15-second evaluation interval, the condition is checked eight times, which on a low-traffic service means eight chances for a transient count to satisfy it.

6. Alert composition and inhibition

One incident should produce one page. Without inhibition, a total outage fires fast-burn, medium-burn and slow-burn, plus the latency SLO, plus every dependent service's SLO.

Alertmanager's mechanism:

inhibit_rules:
- source_matchers: [ severity="critical" ]
  target_matchers: [ severity="warning" ]
  equal: [ service ]                        # ← same service only

Three composition rules worth having:

Severity inhibition. A firing fast-burn suppresses medium-burn and slow-burn for the same service. They are the same information at different sensitivities.

Dependency inhibition. If the model gateway is down, the agent kernel's SLO alert is a symptom. Suppress the symptom, page on the cause — which requires a dependency graph (Phase 00) that somebody maintains.

Grouping. Alerts for the same incident arrive within seconds; group_wait batches them into one notification.

The failure mode of getting this wrong is not noise — it is that the page that mattered is item seven in a list of nine, and the responder starts at the top.

7. Self time, precisely

def self_time(span):
    intervals = sorted((c.start, c.end) for c in children(span))
    covered, cursor = 0, span.start
    for start, end in intervals:
        start = max(start, cursor)          # ← the union, not the sum
        if end > start:
            covered += end - start
            cursor = end
    return max(0, span.duration - covered)

The max(start, cursor) is the whole subtlety. Consider a parent 0–100 with two children both 10–60:

Concurrency is normal in an agent run — a vector search and a BM25 search run in parallel — so the naive sum produces zero or negative self time regularly. Teams clamp it at zero, the number becomes meaningless, and self time gets abandoned in favour of total duration, which blames the root span for everything.

Two related quantities:

Critical path — the longest root-to-leaf chain by summed duration. What to optimize: shortening anything off the critical path changes nothing.

Wall-clock vs work. Total self time across all spans equals the root's duration. Total duration across all spans exceeds it whenever there is concurrency, and the ratio is a measure of how parallel the run was — a useful number in its own right.

8. Sampling

The decision that determines whether tracing is affordable. An agent platform at 100 requests/second with 12 spans per run and ~2 KB per span is ~2.4 MB/s, ~200 GB/day — more than the logs.

Head-based decides at the root, before anything has happened. Cheap and stateless, and its weakness is fundamental: at 1%, you keep 1% of the errors too — and errors are the traces you actually want.

Tail-based buffers the whole trace and decides at the end, so it can keep every error, every slow run, and a sample of the rest. Much better, and it needs a collector holding traces in memory for the trace's duration plus a grace period — which for a 40-second agent run is a real memory footprint and a real availability dependency.

The pragmatic policy for an agent platform:

    keep 100% of:  errors, guardrail blocks, policy denials, runs over the p99
                   anything with a side-effecting tool call
    keep 100% of:  a fixed low-volume "always trace" tenant, for baseline comparison
    keep   ~5% of: everything else

And the rule that makes it defensible: anything that is evidence is never sampled away (Phase 15). A trace that supports an audit claim is not a debugging artifact you may discard — which means the sampling policy is a governance decision, not just a cost one.

9. Context propagation

A trace only reconstructs if the context crosses every boundary:

   traceparent: 00-<32-hex trace-id>-<16-hex span-id>-01
                └W3C Trace Context, the standard header┘

Where it breaks, in decreasing frequency:

The queue case is the common one and it has a standard answer: put the traceparent in the message headers and use span links rather than parent-child, because the consumer's work is causally related but not synchronously nested.

For an agent platform there is a specific decision worth making deliberately: is a multi-turn conversation one trace or many? One trace per turn, linked by a session.id attribute, is right — a single trace spanning an hour-long conversation is unbuildable in most backends and unreadable in all of them.

10. Cardinality, and how backends actually die

    series = ∏ cardinality(label)

The death is specific and worth knowing, because it does not look like a metrics problem:

Prometheus holds an inverted index in memory — roughly 1–3 KB per active series. At 10 million series that is 10–30 GB, and the process OOMs. On restart it replays the WAL, takes minutes, and OOMs again. The failure is a crash loop during an incident, and the symptom is that your monitoring disappears exactly when you need it.

Cortex/Mimir/Thanos enforce per-tenant limits and reject writes past them, which is better — the rejection is visible and bounded. But the rejection is of the whole scrape, so one bad metric loses the good ones alongside it.

The high-cardinality labels, in the order they get added:

That last row is the one that catches teams: tenant is a safe label right up to the day the cardinality assumption changes, and nothing re-checks it. Which is the argument for the check being an estimate that is reviewed, not a one-time approval.

The enforcement that works is at definition time — a metric spec with estimated cardinalities, checked in CI. A dashboard that notices afterwards notices during the crash loop.

11. Exemplars

The bridge between metrics and traces, and the reason the cardinality rule is not a loss:

   histogram bucket le="2.0"  count=4821  exemplar={trace_id="abc123", value=1.87}

A metric bucket carries a sample trace id. So "p99 latency is bad" becomes "here is a p99 trace" in one click, without trace_id ever being a label.

This is what makes the warmup's rule — identifiers on traces, not metrics — cost nothing: you keep the aggregate and the drill-down. It is supported by Prometheus (with --enable-feature=exemplar-storage), OTel and Grafana, and it is off by default nearly everywhere, which is why most teams do not have it.

For an agent platform, the exemplars worth attaching: the slowest bucket of the latency histogram, the most expensive bucket of the cost histogram, and every error bucket. Three lines of configuration and it removes most "find me a trace like this" work.

12. Ladder dynamics

The ladder is a feedback controller, and it has the failure modes of one.

Oscillation. Degrade → load drops → restore → load returns → degrade. The period is roughly twice the evaluation interval, and the user sees answer quality flapping — which is worse than staying degraded, because it is unpredictable.

The fixes, in order of how much they help:

Asymmetry is the important one. Descending must be immediate and can skip levels — during a real incident, stepping down one rung at a time is too slow to matter. Ascending must be one rung at a time with a hold, because each restoration re-adds load.

The measurement lag. The burn rate is computed over a window, so it reflects the past. A 1-hour window means the ladder is responding to an average that includes 55 minutes of health. For a control loop that is a long delay, and it argues for driving the ladder from the short window while alerting on both.

And the honest limit: the ladder sheds load and cannot fix a dependency. If core banking is down, every rung still fails — the ladder's value there is that it stops the platform also falling over, not that it keeps working.

13. Cost attribution

   span: gen_ai.usage.input_tokens=4812, output_tokens=380, model=gpt-frontier
     → cost = 4812 × price_in + 380 × price_out
     → attributed to (tenant, agent, tool, trace)

Four attribution problems that make a cloud bill useless for this:

Shared infrastructure. GPU nodes, the gateway, the vector store. Split by usage share, and be explicit that it is an allocation rather than a measurement.

Cached tokens. Prompt caching means the billed input tokens are lower than the sent tokens, at a different rate. A meter that counts sent tokens over-reports, sometimes by a lot — this is the usual source of a 20–30% gap between your accounting and the invoice.

Retries. A retried call costs twice and produces one result. Attributing both to the successful action is correct; attributing only the successful attempt understates real cost.

Batch versus interactive. Different pricing tiers, sometimes 50% apart, so the same tokens cost different amounts depending on a routing decision the user never sees.

Which produces a discipline worth adopting from Phase 12: reconcile your accounting against the provider invoice monthly. A persistent gap is a meter bug, and finding it in month two is much cheaper than finding it in the annual review.

And the trend that matters more than the total: cost per successful action over time. A platform whose total cost is flat while its per-action cost rises is getting less efficient as it grows, and that is invisible in the bill.

14. Forecasting, and what a linear fit hides

The lab fits a straight line. Real AI-platform demand is not linear, and the differences matter:

Three refinements, in increasing order of effort:

Forecast the peak, not the mean. Quota limits are enforced per minute, so the daily mean is not the constraint. Fit against the daily p95.

Add seasonality. Weekly and daily components (Holt-Winters, or Prophet) turn a projection that is off by weeks into one that is off by days.

Report an interval, not a point. "The limit is reached in 6.0 periods" implies a precision the model does not have. "Between 4 and 9 periods, 80% confidence" is honest and changes the conversation — procurement responds to a range.

And the framing that survives contact with a finance team: the forecast is an argument for starting a conversation, not a prediction. Its job is to fire early enough that the decision is unhurried, which is why the safety factor is deliberate over-caution rather than a modelling error.

15. Performance

Two numbers shape the design. The 30-day raw query is why recording rules exist and why nobody should put a 30-day window in a dashboard panel. And trace storage is why sampling is a decision rather than a default: at 100% an agent platform generates more trace data than log data, and the bill is visible.

What is not worth optimizing: the emission path. A counter increment is 50 ns against a 2-second model call — nine orders of magnitude. Somebody will propose sampling metrics to save CPU.

16. Failure modes