« Phase 14 · Warmup · Track Overview

Staff Notes — Judgment, Review Signal & Seniority

Table of Contents

• 1. Build vs buy
• 2. A decision framework for a proposed alert
• 3. Review red flags
• 4. Production war stories
• 5. The interview signal
• 6. Mentoring notes

1. Build vs buy

The line: buy the plumbing, build the definitions.

And the specific trap: ending with two observability stacks — OTel for the platform and a vendor tool for the AI parts. Then a trace of a slow agent run stops at the boundary, which is exactly the boundary you needed to see across. One OTel pipeline, multiple exporters.

2. A decision framework for a proposed alert

Somebody wants a new alert. Eight questions, in order. Most proposals die at question 2 or 5:

1. What user-visible symptom does this represent? If none, it is a dashboard, not an alert.
2. What would the responder DO at 3 a.m.? If the answer is "look at it", it is a ticket.
3. Is the action in a current runbook? If not, write the runbook first.
4. Is it a cause or a symptom? Alert on symptoms; causes go in the runbook as things to check.
5. What is the false-positive rate on the last 30 days of data? Replay it. If nobody has, that is the first task.
6. Does an existing alert already cover it? Usually yes, and the proposal is really about improving an existing runbook.
7. What is the minimum volume below which it is noise?
8. Who is on call for it, and have they agreed?

Question 2 is the sharpest. "Look at it" means a ticket, and reframing the request that way is usually welcome rather than resisted — nobody actually wants to be woken up to look at something.

3. Review red flags

In a design document

• Answer quality in the availability SLO.
• Latency as a p99 target rather than a ratio.
• No stated validity predicate.
• Safety blocks excluded from the denominator entirely.
• Synthetic probes counted in the SLI.
• A burn-rate threshold copied without derivation.
• Single-window alerting.
• No minimum-volume guard.
• An SLO above measured current performance.
• An SLO that ignores the dependency ceiling.
• No error-budget policy — just a chart.
• A shared budget with no per-team allocation.
• Logs as the primary debugging artifact.
• No sampling strategy, or a flat percentage.
• Sampling that could discard evidence.
• user_id, trace_id or prompt as a metric label.
• No cardinality budget.
• Cost absent from the signal list.
• Cost per request rather than per successful action.
• No degradation ladder, or one invented during an incident.
• Model, prompt and corpus versions not pinned.
• On-call staffed by a general rota in the platform's first year.

In code / config

# Red flag: quality in the availability SLI
good = r.status == 200 and r.answer_quality > 0.8     # un-actionable at 3am

# Red flag: empty window scores zero
ratio = good / valid if valid else 0.0                # every quiet night burns the budget

# Red flag: exact float comparison
if burn_rate >= 14.4: page()                          # 14.399999999999986

# Red flag: self time as a sum
self = span.duration - sum(c.duration for c in children)   # negative on concurrency

# Red flag: the dedup/validity logic in the dashboard
# (every panel reimplements it slightly differently)

# Red flag: cost per request
cost_per = total_cost / len(requests)                 # improves when you fail faster

# Red flag: single window
expr: error_rate_1h > 0.072                           # stays lit for an hour after the fix

# Red flag: no volume guard
# (1 failure in 2 requests is a 100x burn rate)

# Red flag: an unbounded label
labels: [tenant, user_id]                             # 483M series

# Red flag: head sampling only
sampler: parentbased_traceidratio                     # keeps 1% of errors too

# Red flag: decision_wait shorter than the trace
tail_sampling: { decision_wait: 10s }                 # a 40s agent run is truncated

# Red flag: an alert with no runbook annotation

In an incident review

• "The alert was firing but we'd muted it" → too sensitive, disabled informally.
• "The pager didn't go off" → volume guard too high, or a float comparison.
• "We couldn't tell what changed" → versions not pinned.
• "We couldn't find the trace" → head sampling, or context not propagated.
• "Prometheus was down too" → cardinality.
• "Nine alerts, we started at the top" → no inhibition.
• "It cost how much?" → cost is not a golden signal.

4. Production war stories

The SLO that included quality. A blended metric: available AND the answer scored above 0.8 by a judge model. It looked rigorous. Then the judge model was updated by its provider, scores shifted down 0.06, and the platform "breached its SLO" for a week. Engineering spent four days investigating an availability incident that never happened, and the error budget — the actual deploy-governance tool — was gone for the month.

Every quiet night. The empty-window case returned 0.0. The platform had almost no traffic between 02:00 and 06:00, so every night contributed four hours of "0% availability". The budget was exhausted by the 4th of every month and the freeze policy was quietly abandoned in the second month.

The alert that stayed lit. Single-window burn-rate alerting. A four-minute outage burned 2% of the budget; the one-hour window kept the alert firing for 56 minutes after the fix. Within a quarter, the on-call habit was to acknowledge it and wait — including the time it was still happening.

Three in the morning, one request. A low-traffic internal tool, one failure out of two requests, 100× burn rate, page. Three times in two weeks. The fix applied was raising the threshold to 500×, which meant it never fired again — including during a real four-hour outage nobody noticed until Monday.

Negative self time. Self time computed as duration minus the sum of children. Agent runs do retrieval concurrently, so the parent regularly came out negative. It was clamped to zero, the number became meaningless, and the team went back to total duration — which blamed the root span for everything and sent two engineers optimizing the wrong component for a week.

483 million series. During an incident, an engineer added user_id to a latency metric to find which users were affected. Entirely reasonable. Prometheus OOMed within ten minutes, crash-looped on WAL replay, and the monitoring was gone for the remaining three hours of the incident.

One percent of the errors. Head-based sampling at 1%. Six months in, a customer reported a recurring wrong answer. There were no traces for any of the reported cases, because 1% sampling keeps 1% of errors too. The migration to tail sampling took a quarter and could have been a config change on day one.

The runaway loop. An agent hit a tool that returned a malformed response, retried, re-planned, retried. Eleven hours, $47,000 of tokens. Availability 100%, latency normal, error rate zero — every request succeeded. Found by the monthly cloud bill.

Twenty-five percent off. Token accounting counted tokens sent; the provider billed cached input at a quarter rate. The internal cost dashboard and the invoice disagreed by 25% for eight months, and every cost-per-action figure quoted to the business in that period was wrong.

The flapping ladder. A degradation ladder with symmetric transitions. Burn rate crossed the threshold, it shed the reranker, load dropped, it restored, load returned, it shed again — a four-minute cycle for ninety minutes. Users saw answer quality change every few minutes, which generated more complaints than the original degradation would have.

"Something changed." A quality regression, no pinned versions. Six people, three days, and the conclusion was "the model seems worse". Nobody could confirm or exclude a provider change, a prompt edit from two weeks earlier, or an index rebuild. The action item was to pin the versions, which took an afternoon.

Nine alerts. A model-gateway outage fired fast-burn, medium-burn, latency, and each of the four dependent services' SLOs. Nine notifications in ninety seconds. The responder worked down the list in order and reached the actual cause seventh, eleven minutes in.

The general rota. The AI platform was added to the org's existing on-call rota in month three. Most engineers on it had never seen the system. The most common page was a quality complaint, which none of them could triage. Within two months the rota's documented procedure for anything AI-related was "escalate", which meant the same three people were on call regardless.

5. The interview signal

Signal 1 — correctness is a distribution, not a predicate. Said early, and used to structure everything else.

Signal 2 — quality is a gated objective, not an SLO. With all four reasons, and the constructive half: it blocks deploys, which availability cannot.

Signal 3 — the denominator is the argument. Almost every SLO disagreement is about which events are valid, and writing the predicate down settles it in advance.

Signal 4 — a safety block is valid but not good. Very few people get this, and it is the difference between an SLO that hides a total outage and one that does not.

Signal 5 — an empty window is 1.0. With the consequence: otherwise every quiet night burns the budget.

Signal 6 — latency as a ratio, because percentiles do not average. And the corollary: there is no error budget for a p99.

Signal 7 — 14.4 derived, not recited. 0.02 / (1/720). And ideally the inverse: at 14.4× the month goes in two days, which is a 7.2% error rate — much lower than people guess.

Signal 8 — two windows, and the short one is the reset. With the failure mode: an alert lit for 55 minutes after the fix is one people learn to close.

Signal 9 — the minimum-volume guard, and why its absence ends with the threshold raised until nothing fires.

Signal 10 — cost and safety-block rate as golden signals. With a concrete case: a runaway loop where every traditional signal is green.

Signal 11 — cost per successful action, because cost per request improves when you fail faster.

Signal 12 — self time with the union of child intervals. The concurrency case, unprompted.

Signal 13 — sampling is a governance decision. Evidence cannot be sampled away, which forces tail sampling and a policy per category rather than a rate.

Signal 14 — cardinality is a product, and the backend dies during the incident. Because that is when a new label seemed useful.

Signal 15 — the ladder is asymmetric. Descend fast, ascend slowly, with hysteresis — and degradation must be visible to the user.

Signal 16 — pinning is the diagnostic technique. "Nothing we control changed" is only sayable if you pinned it, and it narrows an unbounded question to two hypotheses.

Signal 17 — you prefer under-alerting. With the reason: a missed incident is recoverable, a team that has stopped believing the pager is not.

Anti-signals:

• Quality in the SLO.
• p99 as an SLO target.
• 14.4 recited.
• Single-window alerting.
• No minimum volume.
• Logs as the primary debugging artifact.
• Cost absent entirely.
• "We'll alert on everything and tune later."

The question to ask them: "Availability is 99.97% against a 99.5% target, latency is normal, error rate is flat — and the platform is broken. Give me three ways that is true." A strong answer reaches a runaway cost loop, a guardrail refusing everything, and a silent model change, and then explains what signal would have caught each.

6. Mentoring notes

Three exercises, in order of how much they change behaviour:

1. Compute the error budget as time. "99.9% is forty-three minutes a month. How long was our last deploy freeze?" Thirty seconds, and it converts an abstract percentage into something people argue about correctly.
2. Replay last month's data against a proposed alert. How many times would it have fired? How many were actionable? Nobody proposes an untested alert twice, and it usually kills the proposal in a way the author agrees with.
3. Show them a negative self time. Build a trace with two concurrent children, compute self time as a sum, and watch it go negative. Then fix it with the union. Five minutes, and it is the bug that otherwise makes teams abandon the most useful number in tracing.

And the framing for the platform team: this is the phase where the discipline is subtraction. Every other phase adds a control. This one is mostly about not alerting — not on quality, not on distribution shifts, not on causes, not on low-traffic noise — because the scarce resource is the responder's trust, and it is spent by every page that did not need them.

The argument that gets it funded is not observability in the abstract. It is: "last month an agent spent forty-seven thousand dollars in eleven hours and every dashboard was green, because cost is not one of the four golden signals. Two of our seven signals do not exist yet, and both of them are the ones that would have caught it."