« Phase 00 · Warmup · Track Overview

Core Contributor Notes — How the Real Systems Do This

The maintainer's lens: how production tooling actually implements availability modelling, error budgets, burn-rate alerting and admission chains — the non-obvious decisions, the sharp edges, and what our stdlib miniature deliberately simplifies.

Table of Contents

• 1. SLOs in real backends: the event-ratio model
• 2. Burn-rate alerts as Prometheus rules
• 3. Where "availability" numbers actually come from
• 4. Admission chains in real infrastructure
• 5. Policy distribution: how fail-static is built
• 6. Token accounting in a real gateway
• 7. Sharp edges
• 8. What the miniature simplifies

1. SLOs in real backends: the event-ratio model

Our ErrorBudget takes an SLO and a window and returns minutes. Real SLO implementations almost never work in minutes — they work in event ratios, because time-based availability is ambiguous the moment traffic is uneven (is a minute with 1 request as bad as a minute with 10 000?).

The canonical model, used by Google's SLO tooling, Nobl9, Datadog, Grafana SLO and Azure Monitor workbooks alike:

good_events   = count of requests that met the SLI predicate
valid_events  = count of requests eligible to be judged
SLI           = good / valid
budget_spent  = (valid - good) / (valid * (1 - SLO))

Two consequences worth carrying into design:

• valid is a design decision, not a given. Requests rejected for being malformed, or from an unauthenticated caller, or over quota, are usually excluded — they are the client's fault. Getting this wrong in either direction either flatters your SLO or punishes you for enforcing your own limits. Write the eligibility predicate down in the SLO definition, not in code comments.
• Budget is consumed continuously, not by incidents. Our BudgetLedger.consume(layer, minutes) is a teaching device. Real systems compute the rolling ratio every evaluation cycle and never "attribute" it to a layer at all — attribution to layers happens in the incident review, and is at best an approximation. That is why §5.4 of the WARMUP recommends allocating by historical incident minutes: it is the honest reconstruction of something the metrics pipeline does not natively give you.

Two SLI shapes you will meet:

Mixing them silently is a classic reporting bug: window-based SLIs make short total outages look mild and long partial degradations look catastrophic, relative to request-based.

2. Burn-rate alerts as Prometheus rules

The lab's MultiWindowAlertPolicy calls bad_ratio_over(window_hours) twice per rule. A production implementation precomputes the ratios as recording rules so the alert expression is cheap and the same numbers appear on the dashboard:

# Recording rules: one per window, computed once.
- record: platform:sli_error_ratio_rate1h
  expr: |
    sum(rate(platform_requests_total{result="bad"}[1h]))
      / sum(rate(platform_requests_total[1h]))
- record: platform:sli_error_ratio_rate5m
  expr: |
    sum(rate(platform_requests_total{result="bad"}[5m]))
      / sum(rate(platform_requests_total[5m]))

# Alert: the two-window conjunction, with the burn rate inlined as (1 - SLO) * B.
- alert: PlatformErrorBudgetFastBurn
  expr: |
    platform:sli_error_ratio_rate1h > (14.4 * 0.001)
      and
    platform:sli_error_ratio_rate5m > (14.4 * 0.001)
  for: 2m
  labels: {severity: page}

Three implementation details the YAML hides:

• for: 2m is not the short window. It is a debounce on the conjunction, guarding against a single scrape glitch. People confuse the two constantly; the short window is about the incident ending, the for is about scrape noise.
• The threshold is written as burn × (1 − SLO) rather than as a burn rate, because Prometheus compares ratios. That multiplication is where the float boundary issue from DEEP-DIVE §9 hides in production — 14.4 * 0.001 is 0.014400000000000001, and a measured ratio of exactly 0.0144 does not exceed it.
• Low-traffic windows produce garbage. With 3 requests in 5 minutes, one failure is a 33% error ratio and a burn rate of 333. Real rules add a minimum-volume guard (and sum(rate(...[5m])) > 0.1) or switch to a longer short window for low-traffic services. Our lab has no such guard, and adding one is a good extension.

3. Where "availability" numbers actually come from

Our Component(availability=0.998) assumes the number is known. In practice, obtaining a defensible per-component availability is most of the work, and there are exactly three sources, in descending order of trustworthiness:

1. Your own measurement at the calling boundary. The gateway's success ratio for calls to that provider, over 90 days. This is the only number that reflects your traffic shape, your region, your timeouts, and your retry policy. Use it when you have it.
2. The dependency's published SLI, if it publishes one and you can reconcile it against (1). Reconciliation matters: a provider's "availability" often excludes throttling (429), which for an LLM gateway is your most common failure.
3. The contractual SLA. The weakest source, because it is a floor with financial remedies, not a forecast. Cloud SLAs are typically 99.9% for single-instance PaaS and higher with zone or region redundancy, but they exclude a long list of causes. Never model with an SLA number if you have a measured one, and never model with an SLA number without reading the exclusions.

The practice that separates seniors from principals: keeping a dependency register with, per dependency, the measured availability, the measurement window, the source, the exclusions, and the date. It takes an afternoon and it makes every subsequent availability conversation five minutes long.

4. Admission chains in real infrastructure

Our AdmissionPipeline is one function. In production the same chain is five components in three processes:

Two structural differences from the miniature that matter:

Real chains short-circuit, and leak less. The first denial ends evaluation, and the caller receives a generic error while the audit record receives the specific one. Returning SCOPE_MISSING to an attacker tells them the tool exists and they are close; returning 403 Forbidden tells them nothing. Our lab returns everything because it is teaching, and because the defence_depth metric is worth having in shadow mode.

Envoy/Istio external authorization (ext_authz) is the standard mechanism for putting a PDP in front of a service without changing the service. It has one famous sharp edge: the failure_mode_allow flag. Set to true, an unreachable PDP allows all traffic — the fail-open posture from PRINCIPAL-DEEP-DIVE §5. It defaults to false (fail shut) for good reason, and the correct answer for a bank platform is neither: run the PDP as a sidecar with a locally cached bundle, so "unreachable" almost never happens and the posture question becomes moot.

5. Policy distribution: how fail-static is built

The pattern every mature policy system converges on — OPA's bundle API is the clearest reference implementation:

1. The control plane builds a signed bundle (policies + data) and publishes it with a version and an ETag.
2. Each data-plane instance runs a local evaluator (an OPA sidecar, or an embedded library) that polls for a new bundle on an interval, using the ETag so unchanged polls are cheap.
3. On a successful download the bundle is verified (signature, expected keys present) and activated atomically. A malformed bundle is rejected and the previous one stays active — this is the mechanism that makes fail-static real, and it is why bundle verification is not optional.
4. Every decision records the bundle version it used. This is the artifact that answers the examiner's "which policy allowed this?"
5. Staleness is a metric with an alarm, and past a hard threshold the evaluator refuses to serve. OPA exposes bundle status (last_successful_activation, last_successful_download) precisely so you can alert on it.

The sharp edge: bundle size and evaluation cost grow with your data, not your policy. Teams put the entire agent registry inside the bundle, then discover a 200 MB bundle and 40 ms evaluations. The fix is to keep large, fast-changing facts out of the bundle and fetch them at decision time from a local cache — which reintroduces a (local) lookup, so the design converges on: policy in the bundle, entitlement facts in a local store, both versioned.

6. Token accounting in a real gateway

Our CostModel computes cost from token counts you supply. Real gateways get the counts from the provider's response usage block, and there are four traps:

• Streaming responses may omit usage unless you ask for it. If your gateway streams by default and forgets the option, your cost metrics quietly under-report for exactly the traffic that matters most.
• Cached-token accounting is provider-specific. Some report cached input as a separate field; some report it inside the input count with a discount applied at billing. Normalizing these into one schema is a substantial part of what a model abstraction layer is for (Phase 04).
• Failed and cancelled requests still cost money if the model produced tokens before the failure. A cost model that only counts successes under-reports, and it under-reports most during incidents.
• Reconcile against the billing export. Gateway-measured spend and the provider's invoice will disagree; the gap is where dropped usage blocks, retries you did not count, and unit misunderstandings live. Reconciling monthly is unglamorous and catches real bugs.

7. Sharp edges

Rolling windows are expensive and lie at the edges. A "30-day rolling" budget recomputed continuously is a heavy query. Many implementations use calendar windows instead — cheaper, but they reset abruptly, which means a team can burn 90% of the budget on the 29th and be "fine" on the 1st. Pick deliberately and say which you chose.

Percentiles do not aggregate. You cannot average the p95s of ten instances to get the fleet p95. If your metrics store keeps per-instance percentiles you have already lost the information; you need histograms (Prometheus histogram_quantile over a summed bucket set) to aggregate correctly. Teams discover this when a dashboard and an SLO disagree.

Error-budget policies die silently. They are signed, celebrated, and then the first time a freeze would bite, an exception is granted. After two exceptions the policy is decoration. The mechanism that keeps it alive is making the exception expensive and visible: a written exception, time-bounded, approved at the level above both owners, recorded in the same place as the ADRs.

Latency budgets rot. They are written once at design time and never updated as stages get slower. The fix is to make the budget a test: assert in CI (or in a synthetic canary) that the sum of measured stage p95s is under target, and fail the build when it is not. A budget nobody verifies is a document, not a control.

8. What the miniature simplifies

None of these simplifications changes the reasoning. That is the point of the miniature: the arithmetic and the ordering are identical, and everything the real systems add is plumbing you can now recognize rather than plumbing you must take on faith.

References

• Google, The Site Reliability Workbook, Ch. 2 and Ch. 5 — the event-ratio SLI model and the multi-window multi-burn-rate alert derivation, including the threshold table.
• Prometheus documentation — recording rules, histogram_quantile, and the reasons percentiles do not average.
• Open Policy Agent — bundle API and status/decision-log documentation; the reference design for fail-static policy distribution.
• Envoy — external authorization filter, including failure_mode_allow, and outlier detection (the breaker that lives in the mesh).
• AWS — Cedar policy language and Verified Permissions; a useful contrast with Rego's general-purpose evaluation.
• Nygard, Release It!, 2nd ed. — the stability patterns these components implement.