« Phase 14 · Warmup · Track Overview

Principal Deep Dive — The Trade-offs You Own

The deep dive covered how the mechanisms work. This covers the decisions where there is no correct answer, only a defended one.

Table of Contents

• 1. The central tension: sensitivity against trust
• 2. Choosing the SLO number
• 3. What to page on, and what not to
• 4. The quality argument, in full
• 5. Sampling as a governance decision
• 6. Buy or build the observability stack
• 7. Who owns the error budget
• 8. Cost as a product conversation
• 9. On-call for a platform nobody understands yet
• 10. Setting the numbers
• 11. Migration: instrumenting a live platform
• 12. What I would not build

1. The central tension: sensitivity against trust

Every alerting decision trades detection against the responder's willingness to act.

   SENSITIVE                                                        TRUSTED
      │                                                                  │
   alert on         + burn rate    + two windows    + volume    + inhibition
   error rate                                        guard
      │                  │               │              │            │
   pages hourly     pages daily     pages weekly   pages rarely  one page
                                                                 per incident
      │                                                                  │
   disabled in                                                    acted on
   a month                                                        immediately

The failure mode on the left is not noise. It is that the alerting is disabled, formally or informally — thresholds raised, the channel muted, the page acknowledged without reading. And a disabled alert is strictly worse than no alert, because the org believes it has coverage.

So the position I would defend, and it is uncomfortable at first:

Prefer under-alerting. A missed incident that a customer reports is recoverable. A team that has stopped believing the pager is not, and it takes a year to rebuild.

Which produces a per-class table:

2. Choosing the SLO number

The number is a product decision that engineering informs. Getting that direction right is most of the battle — an SLO chosen by engineering is a target; one chosen with the product owner is a commitment.

Do not start from a number. Start from four questions:

1. What does the user do when it fails? Retry in thirty seconds (cheap), or call the branch (expensive)? That ratio sets the order of magnitude.
2. What is the current, measured availability? Set the SLO slightly below it. An SLO above current performance is permanently breached, which means the error-budget policy never applies and the whole apparatus is decorative.
3. What would the next nine cost? Usually 3–10× in engineering and infrastructure.
4. What do the dependencies allow? You cannot promise 99.95% on top of a core banking system that offers 99.9% (Phase 00).

Question 4 is the one that ends most conversations, and it is worth doing the arithmetic in the room: a platform composing four dependencies at 99.9% each has a ceiling of 99.6% before it does anything of its own.

The practical starting points for an internal bank platform:

And the thing to resist: 99.95%+ for an internal platform. It is a 21-minute budget, which means a single bad deploy exhausts a month. The cost is real and the marginal user value is usually zero, and asking "what does the user do in those 21 minutes" makes that concrete.

3. What to page on, and what not to

The list is shorter than people expect, and defending its shortness is the job.

Page on:

Do not page on:

That last row is a policy worth adopting literally: an alert without a current runbook is deleted. It sounds harsh and it is the only thing I have seen that keeps runbooks current, because the alternative — an alert that fires at 3 a.m. into a runbook describing a system from two years ago — costs more than the alert is worth.

And the review discipline that keeps the list short: every page is reviewed weekly. Was it actionable? Was the action in the runbook? Would a ticket have been enough? Alerts that fail that review are downgraded or deleted, and the metric to watch is pages per week per engineer — above two, people start ignoring them.

4. The quality argument, in full

You will have this conversation repeatedly, usually with someone senior and usually reasonably. Their position: "if the answers are wrong, the platform is not working, so it should be in the SLO."

They are right about the first half. The disagreement is about what an SLO is for.

An SLO is a decision-making tool, not a scorecard. Its job is to answer "may we ship?" and "must we stop?" For that it has to be measurable now, attributable to a team, and actionable during an incident. Quality is none of the three.

Then the constructive half, which is what makes the argument land:

The sentence that usually settles it: "quality is monitored more strictly than availability — it can block a deploy, which availability cannot. It just never wakes anyone up, because there is nothing to do at 3 a.m. about a distribution shift."

And the honest concession worth offering: if a quality collapse is sudden — an eval score halving within an hour — that is not drift, it is an incident, and a canary eval running continuously against production should page. That is a different signal from the SLO, with a different threshold and a different runbook, and offering it makes the boundary principled rather than defensive.

5. Sampling as a governance decision

Usually presented as a cost decision. For a regulated platform it is not.

    100% sampling  →  ~200 GB/day at 100 rps  →  a visible line item
      1% sampling  →  affordable, and 99% of the evidence is gone

The governance constraint from Phase 15: anything that supports an audit claim cannot be sampled away. Which produces a policy rather than a rate:

Two consequences worth planning for.

The retention tiers differ by three orders of magnitude, so they are different stores. A trace backend tuned for 30-day debugging is not an evidence store, and trying to make one thing do both gives you an expensive debugging system or an unqueryable archive.

Tail sampling is required, not optional — you cannot decide at the root whether a run will make a payment. Which means a collector holding traces in memory, which is a real availability dependency that must not take the platform down when it fails. Fail-open on the collector, always.

6. Buy or build the observability stack

The line: buy the plumbing, build the definitions.

The row worth arguing about is LLM-specific tracing. Those tools are genuinely good at prompt-level debugging — seeing the exact rendered prompt, diffing runs, annotating outputs. What they are not is your SLO system, and the trap is ending with two observability stacks: OTel for the platform, a vendor for the AI parts, and no way to answer a question that spans them.

The position I would take: one OTel pipeline, exported to both a general backend and (if it earns its keep) an LLM tool. One instrumentation, two consumers. The moment there are two instrumentations, the trace of a slow agent run stops at the boundary.

7. Who owns the error budget

The question that decides whether error budgets work at all.

Not the SRE team. If SRE owns it, it becomes a stick SRE hits product with, and product responds by disputing the measurement.

The team that ships to the surface — and for this platform, the two-in-a-box pair: the engineering lead and the product owner (Phase 16). Together, because spending the budget is a product decision (ship the feature) with an engineering consequence (accept the risk).

Which makes the budget the pair's shared instrument, and the conversation it enables is the point:

"We have 30% of the budget left with eleven days to go. This feature is risky. Do we ship it now, or do we spend the next week on the retry path and ship it after the reset?"

That is a decision two people can make together in ten minutes. Without a budget it is an argument about feelings.

Two mechanisms that make ownership real:

Allocate per layer — gateway, kernel, retrieval, action gateway — so a team learns it spent the budget from its own number rather than from an incident review.

Publish the policy in advance. Below 20%, only reliability work. At zero, freeze. Agreed when nobody is under pressure, because the whole value is that the decision is not made in the moment.

And the failure mode to watch: a budget that is never exhausted is set too loosely. If twelve months pass without a freeze, the SLO is below what the platform actually delivers and it is governing nothing. That is a signal to tighten it, not a success.

8. Cost as a product conversation

Cost per successful action is not an engineering metric. It is the platform's unit economics, and it belongs in the same conversation as adoption.

The framing that works with a product owner:

Three decisions that are genuinely the product owner's, and should be presented as such:

The cost/quality trade. A larger model gives a measurably better answer at 5× the cost. Whether that is worth it depends on what the answer is for, and engineering does not know.

The degradation ladder's order. Rung 2 is "route to the small model", which is a visible quality reduction. Somebody who owns the user experience should have agreed to that in daylight.

The per-tenant budget. What happens when a business unit exceeds it? Cut them off, bill them, degrade them? That is a commercial decision with an engineering mechanism.

And the discipline that keeps it honest: reconcile against the provider invoice monthly (Phase 12). A persistent 25% gap — usually cached tokens counted as sent — makes every number above untrustworthy, and it is much cheaper to find in month two.

9. On-call for a platform nobody understands yet

A new AI platform has a specific on-call problem: the responder often cannot tell whether the system is broken. "The agent gave a bad answer" arrives as a page, and it might be an outage, a model change, a prompt regression, or the user asking something the platform was never good at.

Four things that help, in order of value:

A triage decision tree, not a runbook per alert. The first question is always "is this availability or quality?", and the answer routes to completely different work. Most new platforms have runbooks for the second and no way to distinguish it from the first.

A "known limitations" document, maintained, that the responder can check in thirty seconds. Half of the quality pages are the platform working as designed on a task it was never good at, and the document is what makes that a two-minute close rather than a two-hour investigation.

Pinned versions and a canary eval (§13 of the warmup). Without them, "what changed?" is unanswerable and every quality incident ends in a shrug.

A named escalation to someone who understands the models. For the first year that is a small number of people, and pretending otherwise produces a rota where most shifts cannot resolve the most common page.

And the staffing reality worth saying out loud: do not put an AI platform on a general on-call rota in its first year. The rota will not have the context, the pages will not be actionable, and the result is a rota that has learned to escalate everything — which is the same as not having one.

10. Setting the numbers

SLO. §2 — from what the user does when it fails, floored by the dependency ceiling, set slightly below measured performance.

Burn-rate thresholds. Derived from your budget-consumption tolerance, not copied. 2%/1h, 5%/6h, 10%/24h is a reasonable starting tolerance for a 30-day window.

Minimum volume. From the traffic distribution: the count at which one failure is not a meaningful error rate. At 100 rps, 10 events in an hour is trivially exceeded; on a service doing 2 requests an hour, no burn-rate rule should evaluate at all and the right alert is an absolute error count.

Sampling. §5 — a policy per category, not a rate.

Cardinality budget. 10k series per metric, ~200k total, as a starting point. The real constraint is the backend's memory (1–3 KB per active series), so derive it from what you are willing to spend on the metrics tier.

Retention. 30 days for debugging traces, 90 for errors, 7 years for anything that is evidence. Three tiers, three stores.

Cost budgets. Per tenant, from the business case rather than from current spend — a budget set to current spend has no headroom and trips on the first busy week.

Capacity lead time. From procurement reality: 1 week for a PAYG quota bump, 4 for PTU, 12 for GPU hardware. Times a 1.5 safety factor.

Pages per engineer per week. Target under two. Above that, the review in §3 has stopped happening.

11. Migration: instrumenting a live platform

Starting state: a platform in production, logs only, no SLOs, alerts on CPU.

Phase 1 — instrument, do not alert. OTel spans and the request event schema, with the valid label. No SLOs, no alerts. Two weeks of data before anything is decided.

Phase 2 — measure, and pick the SLO from the measurement. Now you know what the platform actually delivers, so the SLO can be set slightly below it (§2) rather than aspirationally.

Phase 3 — dashboards, still no pages. SLI, budget, burn rate, cost per action, visible. People learn what normal looks like, which is what makes the first page credible.

Phase 4 — the fast-burn page only. One alert. Review it weekly for a month. Adding a second alert before the first is trusted is how you get to nine alerts nobody reads.

Phase 5 — the ladder down the severity list. Medium-burn page, slow-burn ticket, with inhibition from the start.

Phase 6 — cost attribution and the breaker. Attribution first, in observe mode, because the first month of attribution always surprises somebody.

Phase 7 — the error-budget policy. Last, because it is a social contract and it needs the numbers to be trusted before it can bind anyone.

Phase 8 — the degradation ladder, written and agreed, then wired to the burn rate.

The mistake is starting at Phase 4. Alerting on a platform nobody has watched yet produces pages that nobody can act on, and the pager loses credibility in the first month — after which every subsequent alert inherits that.

12. What I would not build

A metrics or tracing backend. Not close.

A custom trace format. OTel exists and every backend speaks it. A proprietary format is a migration you will do later, under pressure, having lost the history.

An LLM-judge quality SLI on the request path. A model judging a model, on every request: slow, expensive, itself wrong sometimes, and it makes the availability number depend on a second model's availability.

Anomaly detection on everything. It generates alerts nobody can act on, and it trains people to ignore the channel. Anomaly detection is useful on one or two signals — cost rate is the good one — where the response is well-defined.

A "single pane of glass" that re-implements Grafana. Every platform team proposes this. It ends as a worse Grafana with one maintainer.

Alerting on every dependency. Alert on your SLO. A dependency being slow only matters if it burns your budget, and alerting on both means two pages for one problem.

Predictive alerting — "we forecast an SLO breach in four hours". The forecast is wrong often enough that people stop believing it, and the actionable version of this already exists: it is the burn rate.

A second observability stack for the AI parts. One OTel pipeline, multiple consumers. Two instrumentations means a trace that stops at the boundary, which is exactly the boundary you need to see across.