« Phase 00 · Warmup · Track Overview

Staff Notes — Judgment, Review Signal & Seniority

The staff-engineer lens: what to build vs buy, how to decide, what to catch in review, what actually goes wrong, and precisely what an interviewer is listening for.

Table of Contents

• 1. Build vs buy for the Phase 00 concerns
• 2. A decision framework you can use live
• 3. Code-review and design-review red flags
• 4. Production war stories
• 5. The interview signal
• 6. How to disagree well in two-in-a-box

1. Build vs buy for the Phase 00 concerns

The pattern: buy anything with a specification, build anything with a policy. SLO math has a specification. What counts as a valid event on your platform does not.

2. A decision framework you can use live

When someone asks "should this be a separate layer / a separate service / a separate check," run these five in order. It takes ninety seconds and it is visibly senior.

1. What does it deny that nothing else denies? If nothing, it is ceremony. Delete it on paper and enumerate what now gets through.
2. What is its blast radius when it fails? A new serial dependency must earn its unavailability. Quantify: at 99.9% it costs you 43 minutes a month of the platform's budget.
3. Can it be degradable? If a failure can produce a worse but valid answer instead of an error, it stops being a serial dependency and the availability arithmetic changes by an order of magnitude.
4. What artifact does it emit? If a control emits nothing, audit will conclude the control does not exist, and they will be right.
5. Who operates it at 3 a.m.? A component with no runbook, no alert, and no owner is a future incident with a known cause.

For the cost version of the same question, substitute: what does it cost per successful action, what does it cost in latency headroom, and what does it cost in cognitive load for the twenty teams who now have to understand it.

3. Code-review and design-review red flags

In a design document

• An SLO stated without the composition that produces it. Ask: "show me the product."
• A dependency described as "highly available" with no number and no source.
• A fallback with no timeout, or a timeout larger than the remaining latency budget.
• A retry policy that does not mention idempotency or the side-effect class.
• "We'll add observability later." The whole point of tracing an agent platform is that non-determinism is undebuggable without it; retrofitting traces means re-running incidents you already had.
• A diagram with five layers and no statement of what each denies.
• Any cache without a tenant in the key.
• A control-plane call on the synchronous request path with no caching and no stated posture.

In code

# Red flag: availability composed with the wrong identity
def compose(components):
    total = 0.0                       # should be 1.0 for a product
    for c in components: total *= c.availability
    return total                      # always 0.0 — and it "passes" on empty input

# Red flag: budget that can go negative and cancel out
remaining = allocated - consumed      # one layer's overspend hides another's headroom

# Red flag: exact float threshold on a measured ratio
if error_ratio / (1 - slo) >= 14.4:   # never fires at exactly 14.4

# Red flag: tenant from the request
tenant = request.json["tenant_id"]    # should come from the verified token, always

# Red flag: parallel stages summed
committed = sum(s.p95_ms for s in stages)   # a parallel group contributes its max

In an incident review

• A timeline with no "how we detected it" line — that is the action item.
• Action items with no owner and no date.
• A contributing factor phrased as a person's name.
• No answer to "what would have caught this one layer earlier?"

4. Production war stories

The SLO that could not be met by construction. A platform published 99.9% for a request path with six serial components, four of which were third-party. The composed ceiling was 99.4%. Three months of "reliability work" moved nothing, because the constraint was topological. The fix was architectural — degradable retrieval and a second provider — and it took a quarter. The lesson is not "compose your SLO"; it is that the composition is an early-design artifact, because the remedies are architectural and slow.

The alert that never fired. A burn-rate rule written as ratio > 14.4 * (1 - 0.999). The threshold evaluated to 0.014400000000000001; sustained incidents at exactly the design error rate never tripped it. Discovered during a game day, four months in. Lesson: test your alerts by injecting the failure, not by reading the expression.

The fallback that doubled the payments. A gateway retried on timeout. Some of those requests had already executed a tool call downstream. No idempotency key, because "the gateway only calls models." It also called tools, on the agent's behalf, through a path nobody had classified. Lesson: retry policy must be derived from a declared side-effect class, and the class must be a required field, not an optional one.

The 40-tool agent. Success rate 41%. The team's hypothesis was model quality; they upgraded the model and got to 47%. Consolidating eleven primitive tools into two composite ones took it to 88% and cut cost 60% because the plans got shorter and the scratchpad stopped growing quadratically. Lesson: p^n first, model second.

The metric that fell over. agent_requests_total{agent, tool, tenant, model, status, region}. Two hundred agents later, the metrics backend was refusing writes and the on-call dashboard was blank during an incident. Lesson: decide the label budget in Phase 00; move high-cardinality identifiers to traces.

The evidence that did not exist. Internal Audit asked for the authorization trail behind a set of agent actions. Logs existed; the links did not — no policy version, no model version, no approval-to-execution join key. Six weeks of remediation, and a finding. Lesson: every control emits an artifact, and the artifacts must share join keys. Design that in Phase 00; retrofitting it is the most expensive work in this track.

5. The interview signal

For this JD, the Phase 00 material is probed in the first twenty minutes, and the interviewer is listening for four specific things.

Signal 1 — you compute instead of asserting. Weak: "we'd target three nines." Strong: "the composed number at measured availabilities is 99.10%; here are the two changes that get it to 99.66%, and here is why the action path is capped at 99.36% by core banking." The tell is whether numbers appear unprompted.

Signal 2 — you separate what you control from what you inherit. Publishing two SLOs, naming the dependency that caps the action path, and refusing to promise past it. Candidates who promise one optimistic number are signalling that they have never had to hold one.

Signal 3 — you can name the layer that denies. Given an attack, enumerate the independent denials in order. Then the follow-up that separates senior from staff: "and if I could only keep two of those five, I'd keep the action gateway's scope check and the control plane's tool-set check, because they're the two that don't depend on the model behaving."

Signal 4 — you treat evidence as a design input. Mentioning, unprompted, what artifact a control emits and who will ask for it. In a regulated JD this is often the single highest-value sentence you say all interview.

Anti-signals, in rough order of how badly they land:

• Quoting an SLO with no composition.
• Proposing a fallback with no latency budget.
• "Prompt engineering" as the answer to a reliability problem.
• Describing two-in-a-box as a reporting structure.
• Talking about cost per token rather than per successful action.
• Being unable to say what happens when the control plane is unreachable.

The question you should ask them (interviews are two-way, and this one is diagnostic): "Does the platform publish an error budget today, and has a freeze ever actually triggered?" The answer tells you whether the operating model is real or aspirational, and it signals that you know the difference.

6. How to disagree well in two-in-a-box

The mechanic that makes shared accountability survive contact with a real disagreement:

1. Classify the decision first. Reversible or not? Externally visible or not? Reversible and internal → whoever is closest decides now, ADR after. Irreversible or externally visible → both owners, or escalate.
2. Separate the disagreement into facts and values. Most architectural disagreements are secretly factual ("will the fallback fit the budget?"), and factual disagreements are measurable. Agree the measurement, run it, and the disagreement usually dissolves.
3. If it is genuinely a values disagreement (risk appetite, speed vs safety), do not average. An averaged architecture is typically worse than either option. Write both positions down, take them to the architecture board, and commit to the outcome publicly.
4. Disagree and commit, visibly. The failure mode is a decision that is nominally made and quietly relitigated in implementation. If you lost, say so in the ADR's consequences section and then build the thing properly.
5. Review the disagreement protocol after incidents, not during them.

Say this in an interview and it will land, because it is the part of the JD that most candidates treat as boilerplate: "Two-in-a-box only works if we agreed, in advance and in writing, how we decide when we disagree — and if the error-budget policy is something we signed before the first breach rather than negotiated during it."