« Phase 16 · Warmup · Track Overview

Staff Notes — Judgment, Review Signal & Seniority

Table of Contents

• 1. Build vs buy
• 2. A decision framework for a new mechanism
• 3. Review red flags
• 4. War stories
• 5. The interview signal
• 6. Mentoring notes

1. Build vs buy

The line: buy the tooling, write the policy. A paging system is a product. A policy that says what happens when the budget is exhausted is a two-page document that two named people signed, and no vendor can supply it.

And the trap: a platform-engineering vendor will offer to be your "golden path" with a built-in readiness checklist. Take the plumbing; write the criteria yourself. An ORR checklist you did not derive from your own incidents is a form, and everyone will treat it as one.

2. A decision framework for a new mechanism

Somebody proposes a new process. Eight questions, and most proposals die at 2 or 4:

1. What specific failure has happened, more than once? A mechanism for a hypothetical is overhead.
2. Have we made this decision the same way twice already? If not, a rule will be wrong.
3. Where in the existing path can it be enforced? A new gate is expensive; a rule inside an existing function is nearly free.
4. What is the friction, per use, in minutes? Multiply by frequency. This kills most proposals, correctly.
5. Who does it constrain? If it constrains only other people, expect resistance and deserve it.
6. What is the paved road? A control that blocks with no compliant alternative is an obstacle.
7. How will we know it is working? A mechanism with no signal cannot be tuned or retired.
8. When do we review it? Every mechanism needs a date at which it is re-justified.

Question 5 is the one that predicts adoption better than any other. The first mechanism you introduce should visibly constrain you.

3. Review red flags

In how a team describes its operating model

• Two-in-a-box described but the pager is not shared.
• "We'll decide when it happens" about budget exhaustion.
• No written error-budget policy.
• Exceptions with no expiry, or granted by one person.
• No stated exception rate, or one nobody has looked at.
• Disagreements resolved by seniority, and everyone knows it.
• No escalation path — or one nobody has used.
• Architecture that lives in two people's heads.
• ADRs edited in place.
• ADRs with one option, or no negative consequences.
• An ORR that has never failed anything.
• ORR criteria with no evidence definition.
• Alerts never tested by injecting failure.
• Runbooks rehearsed only by their authors.
• Post-mortems written, action completion untracked.
• Actions owned by "the team".
• The same deck taken to every governance forum.
• Standards in a wiki that the pipeline does not enforce.

In an ORR submission

   "SLOs defined"            with no dashboard link
   "alerts configured"       ← configured is not tested
   "runbook written"         ← written is not rehearsed
   "rollback supported"      ← supported is not tested
   "evals passing"           with no case count
   "red-team passed"         reporting a DETECTION rate
   evidence from staging     for a production readiness review
   evidence dated 8 months ago

In an ADR

   ## Options considered
   1. The thing we did            ← one option is a description

   ## Consequences
   ### Positive
   - it is better                 ← and no Negative section at all

In an incident review

• "We couldn't agree so we did neither."
• "Both of us were up all night" for a sev3.
• The commander was also the person debugging.
• The incident was closed at mitigation.
• Six actions, all owned by the same person, all "improve monitoring".

4. War stories

The policy that was never signed. Error budgets were measured and a dashboard existed. The policy was "we'll discuss it if we breach". The first breach was on the 18th, with a feature committed to a customer for the 22nd. The discussion took four days, both owners left it with less trust than they started, and the feature shipped anyway. The policy was written the following week, in an afternoon, and it would have taken the same afternoon eleven months earlier.

Exception creep. Month four, one exception. Month five, another. By month nine the budget state was not mentioned in release planning. Nobody made a bad decision; each exception was locally reasonable. The tell, visible in retrospect, was that nobody had ever looked at the rate.

Both owners, every incident. A well-intentioned reading of shared accountability: both awake for everything. After six weeks both were exhausted, the product owner asked to be removed from the rota, and the model reverted to a partition. The fix was escalation criteria — severity, duration, or a decision needing both — and it should have been there from the first week.

The commander who was debugging. A sev1 where the incident commander was also the only person who understood the failing component. Nobody decided to degrade for fifty minutes, because the person who would have decided was in a stack trace. The degradation ladder existed and was never executed.

The averaged architecture. A genuine values disagreement about autonomy: one owner wanted autonomous release under a threshold, the other wanted human approval always. They compromised on autonomous release with a post-hoc human review. It had the latency of neither and the assurance of neither — a human reviewing a completed payment cannot prevent it — and it was replaced six months later after an incident that the post-hoc review did not catch.

The ADR that was edited. A decision from March, superseded in practice by a different approach in June. Rather than writing a second ADR, somebody updated the first "to keep it current". In September a new joiner read it and implemented the June approach in a component that still needed the March one, because the ADR gave no indication that anything had changed.

The ORR that never failed. Eighteen services, eighteen first-time passes. It was cited in a governance forum as evidence of engineering maturity. An audit sampled three submissions: one evidence link pointed at a staging rollback test, one at a document that did not mention the criterion, and one at a dashboard that had never had data.

"Alerts configured". An ORR row satisfied by a screenshot of the alert rule. Four months later a real outage produced no page: the alert queried a metric label that had been renamed in a refactor. The rule was syntactically perfect and had never fired, in testing or otherwise.

The runbook rehearsed by its author. Rehearsal was recorded. The author could follow it in four minutes. During an incident the on-call engineer — a different person — could not, because step three said "restart the affected service" and there were nine, and the author had known which one.

Forty actions, one completed. A year of post-mortems, diligently written, with action items in a spreadsheet nobody owned. The completion rate was never computed. When it finally was, it was 8%, and three of the incidents that year were recurrences of earlier ones whose actions were still open.

Dropping the hard ones. After completion tracking was introduced, the rate rose to 85% within a quarter. It looked like a success. What had happened was that the difficult actions were being dropped and the trivial ones completed — and the drop reasons, when somebody finally read them, were mostly "deprioritized".

The same deck. A platform review deck taken unchanged to Enterprise Architecture, Cyber, Model Risk, Internal Audit and the CTTO over five weeks. EA wanted a convergence story. Cyber wanted a threat model and heard a claim that prompt injection was prevented, which cost the rest of the session. Model Risk asked what the model was and got an architecture diagram. Audit asked to see an artifact. The CTTO asked about cost per action. Five meetings, five reschedules.

"We prevent prompt injection." Said in a Cyber forum, in good faith, meaning "we have guardrails". The room contained two people who had read the literature. The remaining forty minutes were spent re-establishing credibility rather than discussing the containment design, which was actually good.

The standard in the wiki. A documented requirement that every tool declare a side-effect class. Adoption was about 60%, and the 40% were the tools written under time pressure — which correlated exactly with the ones that most needed it. Moving the check into publish() took twenty minutes and adoption became 100% the same afternoon.

5. The interview signal

Signal 1 — you define two-in-a-box as undivided accountability, not a split. And immediately name the shared pager as the mechanic that makes it real.

Signal 2 — you explain why an AI platform specifically. The autonomy band is a product decision and a risk decision; a partition puts it on a boundary where neither party has the full picture.

Signal 3 — shared accountability needs shared instruments. Without them it is two people blaming each other after an incident.

Signal 4 — the policy is signed before the first breach. A policy agreed while the budget is healthy is a rule; one negotiated during a breach is an argument somebody wins on seniority.

Signal 5 — a freeze still permits reliability work. Otherwise the freeze extends itself.

Signal 6 — exceptions expire, need both owners, and are counted. And the reframe: a high exception rate means the SLO is wrong, so renegotiate it rather than keep granting or start refusing.

Signal 7 — "what would change your mind?" as the factual/values test, with the observation that most disagreements that feel like values turn out to be factual once somebody asks.

Signal 8 — never average a values disagreement. The midpoint has the costs of both and the benefits of neither.

Signal 9 — escalate both written positions. Not a summary by one of them, which is a recommendation with extra steps.

Signal 10 — an ADR needs at least two options and its negative consequences. One option is a description; no negatives means it has not been thought about.

Signal 11 — superseded, never edited. With the reason: an ADR records what was decided at the time, and editing turns it back into a state document.

Signal 12 — any mandatory ORR failure fails at any score. Otherwise the review is a negotiation, and the thing negotiated away is always the runbook rehearsal.

Signal 13 — alerts tested by injecting failure, and the page checked on the device. An untested alert is a belief.

Signal 14 — the runbook rehearsed by somebody outside the team. The author has context the 3 a.m. responder does not.

Signal 15 — mitigation is not resolution. Two timestamps, and conflating them is how incidents recur.

Signal 16 — action-item completion is the only honest measure of a post-mortem culture, with dropped items leaving the denominator and their reasons reviewed.

Signal 17 — five forums, five answers. And specifically: do not tell Cyber you prevent prompt injection, do not tell Model Risk the model is the weights, and do not describe a control to Internal Audit when they asked for the artifact.

Signal 18 — a standard in code is a control. With a concrete example from your own work.

Signal 19 — you can say when two-in-a-box is the wrong model. Advocating for it universally is a weaker signal than knowing its preconditions.

Anti-signals:

• Two-in-a-box described as "we work closely together".
• No shared pager.
• "We'd discuss it" about budget exhaustion.
• Disagreements resolved by escalating the person.
• ADRs as documentation rather than as decisions.
• An ORR that is a form.
• "We prevent prompt injection."
• Post-mortems described without action tracking.
• Every mechanism constraining somebody other than the speaker.

The question to ask them: "It is the 18th, the error budget is exhausted, and a feature your product owner committed to a customer is ready. Walk me through the next hour." A weak answer negotiates. A strong one cites a policy that was signed in advance, explains the exception path and why it is expensive, and — the best answers — notes that if this keeps happening the SLO is the thing that is wrong.

6. Mentoring notes

Three exercises, in order of how much they change behaviour:

1. Have them write an ADR for a decision already made. Then ask for the negative consequences. Most people cannot list any on the first attempt, and that gap — not the template — is the lesson. Fifteen minutes.
2. Give them a real ORR submission and ask what they would reject. Include one "alerts configured" and one staging-environment evidence link. Watching somebody find the difference between configured and tested is the fastest way to teach why evidence definitions exist.
3. Run the disagreement protocol on a live disagreement. Ask both people "what would change your mind?" before anything else. About half the time the disagreement dissolves in the next two minutes, and everyone present remembers it.

And the framing for the platform team: this is the phase where the mechanisms are cheap and the timing is everything. The error-budget policy takes an afternoon and is worth ten times as much signed in month one as in month eleven. The ORR takes a day and cannot be retrofitted onto something already live without a negotiation you will lose. The disagreement protocol is one page and is useless once the first disagreement has been resolved badly.

Almost nothing here is difficult. All of it is easy to defer, and deferring it is what turns a working pair into two people who cannot agree.

The argument that gets it taken seriously is not process rigour. It is: "we are going to disagree about something that matters, probably about autonomy, probably in the next six months. Right now that is an abstract conversation we can have in an afternoon. Then, it will be a decision one of us loses."