« Phase 14 · Lab 01 · Track Overview

Warmup — SRE for Non-Deterministic AI, from Zero to Principal

Table of Contents

• 0. Where this sits
• 1. From first principles: what "correct" means here
• 2. SLIs, SLOs and the event-ratio model
• 3. The denominator is the argument
• 4. Why quality is not in the availability SLI
• 5. Latency as a ratio
• 6. Error budgets
• 7. Deriving the burn-rate ladder
• 8. Two windows, and the low-traffic guard
• 9. The golden signals, extended
• 10. Traces, not logs
• 11. Cardinality
• 12. The degradation ladder
• 13. Incident response for a probabilistic system
• 14. Post-mortems
• 15. Capacity planning
• 16. Cost governance
• 17. Numbers worth carrying
• 18. Interview questions, answered
• 19. References

0. Where this sits

Phase 00 introduced SLOs, error budgets and burn-rate alerting as architecture inputs — numbers you use to decide a design. This phase is the run-state discipline that operates them once the thing is live and you carry the pager.

It also consumes almost every earlier phase:

And it produces the inputs for Phase 15: the trace, the decision record and the eval history are the evidence pack.

1. From first principles: what "correct" means here

Start with what an ordinary SRE assumes without noticing:

For a given request, there is a fact of the matter about whether the system behaved correctly, it is observable at the time, and it is attributable to the system.

For a REST API that is true. A 500 is a 500.

For an AI platform, all three parts fail:

There is no fact of the matter. The same question can produce a good answer, an adequate one and a subtly wrong one. "Correct" is a distribution, and a single sample is not evidence.

It is not observable at the time. Whether the answer was right needs a judge — a human, or an evaluator model, or a downstream outcome that arrives days later. Nothing at request time knows.

It is not attributable. A wrong answer may be the model, the retrieval, the corpus, the prompt, the user's question, or the world having changed since the corpus was built.

So the discipline splits into two:

That split is the single most important idea in the phase, and §4 defends it.

2. SLIs, SLOs and the event-ratio model

The vocabulary, precisely, because it gets used loosely:

The measurement model that works, and the one to insist on:

    SLI = good events / valid events

Not an average of latencies, not a percentile, not a gauge. A ratio of counted events, because a ratio:

• composes across windows (you can add the numerators and denominators);
• composes across regions and tenants;
• has a budget you can spend, allocate and run out of;
• and means the same thing at any aggregation level.

A percentile has none of those properties, which is why §5 exists.

3. The denominator is the argument

Every argument about an SLO is really an argument about which events are valid. Six events, three predicates, three different numbers — and all three are defensible:

The three standard exclusions, each with the reason it is not obviously right:

Client errors. A malformed request is not an availability failure — unless your endpoint returns 400 for a valid request, in which case it very much is. The exclusion is only safe if 4xx is genuinely the client's fault, which is worth checking rather than assuming.

User aborts. They closed the tab. Counting these makes the SLI track user patience, which is a real signal and a different one.

Synthetic probes. They are for detection, not for the number. Including them means you can improve the SLO by probing more, which is Goodhart's law with a cron job.

And the row that is specific to this phase: a safety block is not good, but it is valid.

A guardrail refusing a request is the platform working correctly, so it is not a success. But if the guardrail starts refusing everything, availability should degrade — a total outage caused by a bad guardrail deploy must not show green. Excluding safety blocks entirely is the mistake, and it is a tempting one because it makes the number look better.

4. Why quality is not in the availability SLI

The proposal arrives in every design review: "shouldn't the SLO include whether the answer was good?" Four reasons it should not, and they are worth having in order.

One — it is not measurable in real time. Nothing at request time knows whether the answer was right. Any real-time proxy is a model judging a model, which is slow, expensive and itself wrong sometimes.

Two — it is not attributable. A wrong answer may be the model, the corpus, the question, or the world. An SLI that mixes platform failures with model quality cannot be acted on by the platform team.

Three — it makes the metric un-actionable during an incident. At 3 a.m. you need to know whether the platform is up. A blended number that dropped because quality dipped tells you nothing about what to do.

Four — it destroys the error budget's meaning. The budget is a decision-making tool: spend it on risky deploys, freeze when it is gone. A budget that drains because a model got worse is a budget that stops governing deploys.

So the treatment is:

"Gated objective" is the phrase to have. Quality is not unmonitored — it is monitored harder than availability, with an eval suite and a promotion gate (Phase 09). It just never wakes anyone up, because there is nothing to do at 3 a.m. about a distribution shift.

5. Latency as a ratio

The instinct is p99 < 2s. It is the wrong shape, for three reasons:

Percentiles do not average. The p99 of two regions is not the average of their p99s. There is no correct way to combine them, so a global latency SLO built from regional p99s is not a number.

There is no error budget for a percentile. "We used 40% of our p99" is meaningless. Budgets need countable events.

It hides the tail's size. p99 < 2s says nothing about whether the 1% took 3 seconds or 3 minutes.

The right shape is the same event ratio:

    99% of valid requests complete in under 2 seconds

which composes, has a budget, and is directly comparable across regions and tenants.

Report percentiles anyway — as a diagnostic, on the dashboard, next to the ratio. They are how you see the shape of the tail during an investigation. They are just not what you alert on.

And for an agent platform, one refinement: latency is multi-modal, not a single distribution. A cached answer is 200 ms, a single-step answer is 2 s, a five-tool investigation is 40 s. A single threshold across all of them is meaningless, so the SLO is per class of work, and the class is a span attribute you set deliberately.

6. Error budgets

    error budget = 1 − SLO
    99.5% over 30 days = 0.5% = 3h 36m of downtime, or 5,000 failures in 1,000,000

Three operational properties.

Rolling, not calendar. A calendar-month budget resets on the 1st, so an outage on the 31st costs nothing. A rolling 30-day window is the honest measure.

Clamped at zero, with overspend reported separately. A dashboard showing −340% tells you nothing you did not already know from −1%. But "how far past" is a real question, so it goes in a separate number rather than being lost to the clamp.

Allocated, not shared. A budget shared across the gateway, the kernel, retrieval and the action gateway is a budget nobody owns. Allocate it — proportionally to complexity or historical failure rate — so each team has a number that is theirs and learns from it rather than from an incident.

And the policy that gives it teeth, agreed in advance:

The policy is the point. Without one, a budget is a chart.

7. Deriving the burn-rate ladder

Burn rate is how fast the budget is being spent, as a multiple of the sustainable rate. 1.0 means exactly on budget for the period. 14.4 means the whole month's budget in an hour.

Everyone quotes 14.4. Derive it instead:

    burn_rate = (fraction of budget consumed) / (fraction of period elapsed)

    2% of a 30-day budget in 1 hour:
        0.02 / (1 / 720) = 14.4

That is the entire trick, and it generalizes:

Being able to derive it matters because the numbers change with your SLO period and your tolerance. Copying 14.4 into a 7-day-window SLO gives you an alert that means something else.

8. Two windows, and the low-traffic guard

Each rule uses two windows, and the second one is the whole trick.

The long window is the signal: has enough budget been burned to care?

The short window is the reset: is it still happening?

Without the short window, a five-minute blip that consumed 2% of the budget keeps the alert firing for the rest of the hour — long after the problem is gone. And an alert that stays lit after the fix is an alert people learn to close without reading.

   fires only when:  long_window_burn ≥ threshold  AND  short_window_burn ≥ threshold

The short window is conventionally 1/12 of the long one — five minutes for an hour, thirty minutes for six hours.

Then the guard that determines whether any of this survives contact with production:

A minimum event count.

On a service doing two requests an hour, one failure is a 50% error rate and a burn rate in the hundreds. Page on that a few times at 3 a.m. and somebody raises the threshold until nothing ever fires — and now the alerting is decorative, which is worse than not having it.

So each rule carries a minimum volume, and below it the rule simply does not evaluate. The correct behaviour for a genuinely low-traffic service is a different alert entirely: absolute error counts, or synthetic probes, or an availability SLO with a longer window.

9. The golden signals, extended

The four golden signals — latency, traffic, errors, saturation — are necessary and not sufficient here. Three more:

Cost per successful action. The unit economic. It can move 10× while every other signal is green: a retry loop, a prompt that grew, a fallback to an expensive model, a cache that stopped working. Note the denominator is successful actions — cost per request improves when you fail faster, which is exactly the wrong incentive.

Safety-block rate. From Phase 11. A sudden rise means either an attack or a broken guardrail, and both matter. A sudden fall means the guardrail stopped running, which is worse and is invisible unless you watch the rate.

Quality, sampled offline. Gated, never paged (§4).

And two more worth having on the dashboard:

Escalation rate — how often a task needs a human. It is the platform's actual usefulness, and it moves before quality metrics do.

Token efficiency — output tokens per successful action. A quiet regression here is a prompt that grew or a loop that got longer, and it shows up in cost before anyone notices behaviour.

The general form of the argument: the traditional signals measure whether the system responded. These measure whether it was worth responding.

10. Traces, not logs

In a deterministic system, logs plus a reproduction get you there. Here:

You cannot reproduce. Re-running the same input gives a different run — different tool calls, different retrieval, different answer. Whatever the trace did not capture is gone.

A run is a tree, not a sequence. An agent step calls a model, which decides on a tool, whose output feeds another model call. A flat log stream cannot reconstruct the parent-child structure.

The interesting question is "where did the time go?" — which is a tree-shaped question.

So: OpenTelemetry, with a span per agent step, model call, tool call, retrieval, policy decision and guardrail check. And use the GenAI semantic conventions for the attribute names (gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens) rather than inventing your own — it is what lets any OTel-aware backend chart token usage without custom queries, and inventing names is a migration you will do later under pressure.

The technique that makes a trace useful:

Self time — a span's duration minus the time covered by its children.

Total duration blames the root span for everything. Self time says the model call is 2.5 s of the 4.1 s and the reranker is 690 ms. And it must subtract the union of child intervals, not their sum, or two concurrent children produce negative self time — which people clamp to zero and then stop trusting the number.

And the property that pays for the whole thing: the debugging artifact and the audit artifact are the same object. The span tree that tells you where the latency went is the record that tells an examiner what the agent did (Phase 15).

11. Cardinality

A time-series database stores one series per unique label combination. Series are the product of label cardinalities, not the sum:

    tenant (12) × deployment (6) × outcome (7)              =     504 series
    ... × model (8) × region (3)                            =  12,096 series
    ... × user_id (40,000)                                  = 483M series   ← the outage

Which is why the failure is a cliff, not a slope: adding one label multiplies everything, and a metric goes from affordable to unaffordable in a single commit. The backend does not degrade gracefully — it falls over, and it falls over during an incident, because that is exactly when a new label seemed useful.

The rule:

High-cardinality identifiers live on traces and in the accounting store. Metrics carry tenant, deployment and outcome.

A trace backend is built for unbounded ids; a time-series backend is not. So trace_id, user_id, session_id, prompt and document_id are forbidden as metric labels, and the enforcement has to be at definition time — a check that fails a metric before it ships, not a dashboard that notices afterwards.

And the connection people miss: exemplars bridge the two. A metric bucket carries a sample trace id, so "p99 is bad" becomes "here is a p99 trace" without putting the trace id in the label set. It is the single highest-value observability feature most teams have not enabled.

12. The degradation ladder

Written in daylight, executed at 3 a.m.

The pre-agreed order in which capability is shed, so the decision is made by people who are awake and have time to think about the trade-offs:

The order is the design decision, and it is cheapest-loss-first: shed accuracy before capability, capability before availability.

Three operational properties:

Descend fast, ascend slowly. During a real incident, stepping down one rung at a time is too slow — jump straight to the level the burn rate implies. Coming back up quickly re-creates the load that caused it, so ascend one rung at a time with a hold period.

Hysteresis is required. Without it the ladder oscillates — degrade, load drops, restore, load returns — and the user sees answer quality flapping, which is worse than staying degraded.

Degradation must be visible to the user. A silent quality change is how a platform loses trust: the answers got worse and nobody said anything. "Operating in reduced mode; answers may be less detailed" is a sentence worth writing in advance.

13. Incident response for a probabilistic system

The distinctive question: what changed? — when nothing in your logs is an error.

The technique is not clever inference. It is having pinned everything:

Then the diagnosis is mechanical: compare the pinned versions against the baseline, and each one is confirmed or excluded.

The interesting case is when nothing you control changed and the eval still dropped. That leaves exactly two hypotheses: the provider changed behaviour behind a stable version string, or the input distribution moved. "It is one of these two" is a much better position than "something changed", and it is only available because somebody pinned the versions.

Which is the argument for pinning, stated operationally: if the model version is not pinned, "the provider changed the model" is a hypothesis you can never confirm or exclude — so every incident involving quality ends in a shrug.

Two more practices worth naming:

A canary with a fixed eval set, run continuously against production configuration. It is the tripwire that turns a silent provider change into an alert.

Shadow the previous model version for a period after a switch, so "is the new one worse?" is answerable by comparison rather than by memory.

14. Post-mortems

Standard SRE practice, with two AI-specific additions.

Blameless, with a timeline, contributing factors and action items. The action items' completion rate is itself a tracked metric — a post-mortem process whose actions are never done is a writing exercise.

The two additions:

"Was this deterministic?" Would the same input have produced the same failure? If not, the fix is different: you are not fixing a bug, you are narrowing a distribution, and "we fixed it" needs a measurement rather than a diff.

"What did the eval suite not catch?" Every quality incident should produce a new eval case. That is the ratchet that makes the suite grow toward the failures you actually have rather than the ones you imagined (Phase 09).

And a habit worth stealing from the rest of this track: the post-mortem's timeline should be generated from the trace, the decision records and the alert history, not reconstructed from memory. Memory during an incident is unreliable, and the artifacts are already there.

15. Capacity planning

Two things make this different from ordinary capacity work.

The operative limit is the provider's quota, not CPU. You will hit an Azure OpenAI tokens-per- minute ceiling, or a GPU quota, long before a machine is busy. A CPU-based headroom alert is silent all the way to a hard 429.

The lead time is weeks to months. A PTU commitment is a procurement conversation. A GPU quota increase takes weeks. Physical GPU capacity takes months. So the alert must fire a lead time early, not at 90% utilization — an alert that arrives after the decision point is a notification, not a control.

Which gives the shape:

    periods_to_limit = (limit − current) / growth_per_period
    alert when        periods_to_limit ≤ lead_time × safety_factor

The same growth curve therefore alerts or does not depending only on what you are buying: a pay-as-you-go quota bump alerts at 70% utilization, a GPU procurement alerts at 55%.

The safety factor (1.5 is reasonable) is deliberate over-caution, and the justification is asymmetric: starting a procurement conversation early costs a meeting; starting it late costs a quarter.

16. Cost governance

Cost is a reliability concern here, which is not true of most systems.

Cost per successful action is the unit economic — the number that tells you whether the platform is viable, and the one to put in front of a product owner. Successful, not total, per §9.

Attribution needs to reach tenant, agent, model and tool, which means it comes from the trace and the gateway's accounting ledger (Phase 04) rather than from a cloud bill. A cloud bill tells you the platform cost $40,000; it does not tell you that one agent's retry loop was $12,000 of it.

The cost circuit breaker is an availability control. A runaway agent loop can spend a month's budget in an hour, and every traditional signal reports health — the requests succeed. So the breaker trips per tenant, which is the property that matters: one tenant's loop must not exhaust another tenant's budget or the platform's.

Two thresholds, because "warn then stop" is what makes it usable: an alert at 80% gives somebody a chance to look before a tenant is cut off.

And the FinOps discipline around it: budgets per tenant, a forecast, an anomaly alert on the rate rather than the total, and a monthly review of the cost-per-action trend. A cost that is flat in total and rising per action is a platform getting less efficient while it grows, which is invisible in the bill.

17. Numbers worth carrying

18. Interview questions, answered

Q1. "What SLIs would you define for an AI platform?"

Availability and latency at the ingress, both as event ratios — good over valid — because a ratio composes across windows and regions and has a budget you can spend. Latency as "99% of requests under 2 seconds" rather than "p99 under 2 seconds", because percentiles do not average and there is no error budget for a percentile.

Then three that are specific to this: cost per successful action, safety-block rate, and quality from sampled offline evaluation.

And the important part is which of those are hard SLOs. Availability and latency page. Quality is a gated objective — it blocks deploys and is reviewed weekly, and it never pages. Cost has a circuit breaker rather than an alert, because the enforcement is what matters.

The thing I would spend the most time on in the design is the denominator. Almost every argument about an SLO is really an argument about which events are valid, and writing that predicate down — 4xx excluded, aborts excluded, synthetic excluded — settles it in advance.

Q2. "Why not include answer quality in the SLO?"

Four reasons, and they compound.

It is not measurable at request time — nothing knows whether the answer was right without a judge. It is not attributable, because a wrong answer may be the model, the corpus, the question or the world. It makes the metric un-actionable during an incident, when what you need to know is whether the platform is up. And it destroys the error budget's meaning as a deploy-governance tool, because a budget that drains when a model gets worse stops governing deploys.

So quality is a gated objective instead. It is monitored harder than availability — an eval suite, a promotion gate, a weekly review — it just never wakes anyone up, because there is nothing to do at 3 a.m. about a distribution shift.

One nuance I would add: safety blocks are excluded from "good" but counted in "valid". A guardrail refusing is the platform working, so it is not a success. But if a bad guardrail deploy starts refusing everything, availability should degrade — and excluding them entirely hides a total outage behind a green dashboard.

Q3. "Where does 14.4 come from?"

Burn rate is the fraction of budget consumed divided by the fraction of the period elapsed. Two percent of a thirty-day budget in one hour is 0.02 divided by 1/720, which is 14.4.

It generalizes: 5% in six hours is 6×, 10% in a day is 3×. I would derive it rather than copy it, because the numbers change with your SLO period — putting 14.4 into a seven-day-window SLO gives you an alert that means something else entirely.

And each rule uses two windows. The long one is the signal; the short one is the reset. Without the short window, a five-minute blip that burned 2% keeps the alert firing for the rest of the hour, long after the fix — and an alert that stays lit after the fix is one people learn to close without reading.

Q4. "How do you avoid paging on noise?"

Three mechanisms, and the third is the one that actually decides it.

Two windows, so an alert clears when the problem does. A severity ladder, so a slow burn is a ticket and only a fast burn pages. And a minimum event count per rule.

That last one is what keeps the alerting alive. On a service doing two requests an hour, one failure is a 50% error rate and a burn rate in the hundreds. Page on that a few times at 3 a.m. and somebody raises the threshold until nothing ever fires — and now the alerting is decorative, which is worse than not having it.

For a genuinely low-traffic service the right answer is a different alert entirely: absolute error counts, or synthetic probes, or a longer SLO window.

Q5. "How do you debug an agent that gave a wrong answer?"

Traces, not logs — because you cannot reproduce it. Re-running gives a different run, so whatever the trace did not capture is gone.

A run is a tree: an agent step calls a model, which picks a tool, whose output feeds another model call. I want a span per step, model call, tool call, retrieval, policy decision and guardrail check, with OTel's GenAI semantic conventions for the attributes so any backend can chart tokens without custom queries.

The technique that makes it useful is self time — duration minus the time covered by children. Total duration blames the root for everything; self time says the model call is 2.5 seconds of the 4.1 and the reranker is 690 milliseconds. And it has to subtract the union of child intervals, not the sum, or concurrent children give you negative self time.

The property that pays for it: the debugging artifact and the audit artifact are the same object. The tree that shows where the latency went is the record that shows an examiner what the agent did.

Q6. "A model regression in production. How do you find out what changed?"

The technique is not clever inference — it is having pinned everything: model version, prompt version, corpus version, policy version, and an eval baseline. Then the diagnosis is mechanical: each is confirmed or excluded by comparison.

The interesting case is when nothing we control changed and the eval still dropped. That leaves exactly two hypotheses — the provider changed behaviour behind a stable version string, or the input distribution moved. "It is one of these two" is a much better position than "something changed", and it is only available because somebody pinned the versions.

Which is the argument for pinning stated operationally: if the model version is not pinned, "the provider changed the model" is a hypothesis you can never confirm or exclude, so every quality incident ends in a shrug.

I would also run a canary with a fixed eval set against production configuration continuously — that is the tripwire that turns a silent provider change into an alert rather than a discovery.

Q7. "Your error budget is exhausted. What happens?"

The policy is agreed in advance, or the budget is just a chart. Above 50% remaining, ship freely. Between 20 and 50, ship with care. Below 20%, only reliability work and critical fixes. At zero, freeze — the next deploy is a reliability improvement.

Two things I would add for an AI platform. Allocate the budget per layer — gateway, kernel, retrieval, action gateway — because a shared budget is a budget nobody owns, and the retrieval team should learn that the gateway spent it from their own number rather than from an incident.

And the budget should drive the degradation ladder automatically: at a high burn rate, shed capability in a pre-agreed order rather than waiting for a human. Disable reranking, then route to a smaller model, then serve from cache, then read-only. Written in daylight, executed at 3 a.m., and the order is cheapest-loss-first.

Q8. "How do you plan capacity?"

Against the provider quota, not CPU. You hit a tokens-per-minute ceiling or a GPU quota long before a machine is busy, and a CPU-based headroom alert is silent all the way to a hard 429.

And the alert has to fire a lead time early. A PTU commitment is a procurement conversation, a GPU quota bump is weeks, physical capacity is months. So I project growth against the limit and alert when the periods remaining fall inside the lead time times a safety factor.

Which means the same growth curve alerts at different utilizations depending on what you are buying: a pay-as-you-go quota bump at 70%, a GPU procurement at 55%. A 90%-utilization alert arrives after the decision point, which makes it a notification rather than a control.

The safety factor is deliberate over-caution, and the justification is asymmetric: starting the conversation early costs a meeting, starting it late costs a quarter.

19. References

SRE foundations

• Google SRE Book — Service Level Objectives
• The SRE Workbook — Implementing SLOs · Alerting on SLOs — where the burn-rate ladder comes from
• Google SRE Book — Postmortem Culture
• Implementing Service Level Objectives (Alex Hidalgo)

Observability

• OpenTelemetry documentation · GenAI semantic conventions
• Prometheus — naming and labels · cardinality
• Exemplars · Grafana Tempo
• Azure Monitor — Application Insights

AI-specific operations

• Azure AI Foundry — monitoring generative AI
• LangSmith · Arize Phoenix · Langfuse
• Azure OpenAI quotas and limits

Cost

• FinOps Foundation · FinOps for AI
• Azure Cost Management