« Phase 14 · Warmup · Track Overview

Hitchhiker's Guide — SRE for Non-Deterministic AI

The fast orientation. What the pieces are, what they are called, and how they fit — before the deep dive takes them apart.

Table of Contents

• 1. Don't panic: the one-paragraph version
• 2. The map
• 3. The vocabulary
• 4. The signal table
• 5. The burn-rate ladder, memorized
• 6. The five things that will surprise you
• 7. Reading an alerting rule
• 8. Where the neighbouring phases connect
• 9. What to build first

1. Don't panic: the one-paragraph version

Correctness here is a distribution, not a predicate, so the discipline splits in two: availability and latency are hard SLOs that page, and quality is a gated objective that blocks deploys and never wakes anyone. Everything is measured as an event ratio — good over valid — because a ratio composes and has a budget. Alerts fire on burn rate with two windows (the short one is the reset) and a minimum-volume guard (or a low-traffic service pages at 3 a.m. and somebody disables it). Debugging is trace-first, because you cannot re-run to reproduce, and self time is what tells you where the latency went. And two signals nobody's golden-four includes — cost per successful action and safety-block rate — can move catastrophically while everything else is green.

2. The map

   requests ──► RequestEvent(outcome, latency, cost, tenant, trace_id)
                     │
        ┌────────────┼───────────────────────────────┐
        ▼            ▼                               ▼
   VALIDITY      SPAN TREE                      COST METER
   PREDICATE     (OTel, GenAI conventions)      per successful action
        │            │                               │
        ▼            ▼                               ▼
      SLI        self time,                    circuit breaker
   good/valid    critical path                  (per tenant)
        │
        ▼
   ERROR BUDGET  ──► BURN RATE ──► ALERTING (2 windows + min volume)
        │                              │
        │                              ▼
        │                        DEGRADATION LADDER
        │                        (descend fast, ascend slowly)
        ▼
   CAPACITY FORECAST ──► alert a procurement lead time early

3. The vocabulary

4. The signal table

The two rows to be able to defend: quality never pages, and cost has a breaker rather than an alert — because in both cases the useful response is an enforcement mechanism, not a human at 3 a.m.

5. The burn-rate ladder, memorized

    burn_rate = (budget consumed) / (period elapsed)

And the budget itself, as time, for a 30-day window:

6. The five things that will surprise you

1. An empty window is 1.0, not 0.0. Zero traffic is not an outage. Get this wrong and every quiet Sunday burns the whole budget.

2. A safety block is valid but not good. Excluding them entirely means a guardrail that refuses everything shows green.

3. Self time needs the union of child intervals. Sum them and two concurrent children give the parent negative self time, which people clamp to zero and then stop trusting.

4. Cardinality is a product. One new label multiplies every series. A metric goes from affordable to unaffordable in a single commit, and the backend falls over during an incident — because that is when a new label seemed useful.

5. Cost per request improves when you fail faster. Which is why the denominator is successful actions.

7. Reading an alerting rule

The Prometheus shape, with the parts that matter marked:

- alert: PlatformErrorBudgetFastBurn
  expr: |
    (
      sum(rate(requests_total{outcome!="success", valid="true"}[1h]))
        / sum(rate(requests_total{valid="true"}[1h]))
    ) > (14.4 * 0.005)                                    # ← threshold x budget
    and
    (
      sum(rate(requests_total{outcome!="success", valid="true"}[5m]))
        / sum(rate(requests_total{valid="true"}[5m]))
    ) > (14.4 * 0.005)                                    # ← THE SHORT WINDOW
    and
    sum(increase(requests_total{valid="true"}[1h])) > 10   # ← THE VOLUME GUARD
  for: 2m
  labels: { severity: page }
  annotations:
    runbook: https://.../fast-burn

Four things to notice:

• valid="true" is the validity predicate, baked into the metric at emission time. Doing it in the query instead means every dashboard reimplements it slightly differently.
• The and on a 5-minute window is the reset. Delete that clause and the alert stays lit for an hour after the fix.
• The volume guard is the third clause and the one most rules omit.
• for: 2m is a separate mechanism from the short window — it stops a single scrape spike firing, where the short window stops a resolved incident staying lit.

8. Where the neighbouring phases connect

9. What to build first

1. The event schema and the validity predicate. One field — valid — decided and emitted at the source. Retrofitting it means every dashboard has a different denominator.
2. Availability and latency SLIs as ratios. Before any dashboard, because the shape determines everything downstream.
3. Trace instrumentation with GenAI conventions. Early, because a trace you did not emit is gone — there is no re-run.
4. The cardinality budget, before the first custom metric. The alternative is an emergency migration.
5. Burn-rate alerting with both windows and the volume guard. All three at once; a rule missing any of them gets disabled within a quarter.
6. Cost attribution per tenant and agent. Then the breaker.
7. The degradation ladder, written down and agreed, before it is needed.
8. The capacity forecast, once you know the provider quotas — and sized to the procurement lead time, not to a utilization number.