« Phase 02 · Warmup · Track Overview

Staff Notes — Judgment, Review Signal & Seniority

Table of Contents

• 1. Build vs buy
• 2. A decision framework for publishing a tool
• 3. Review red flags
• 4. Production war stories
• 5. The interview signal
• 6. Mentoring notes

1. Build vs buy

The line is the same as everywhere in this track: buy anything with a specification, build anything with a policy. MCP has a specification. "Which agent may see which tool" does not.

One temptation to resist: building a nicer tool-authoring framework on top. That is how a registry becomes a framework, and then you own an SDK. Publish a schema, a description, and metadata; let teams write handlers however they like.

2. A decision framework for publishing a tool

Six questions, in order. It takes five minutes in a design review and catches most of what goes wrong.

1. Is this one tool or a workflow? If a human could write the orchestration as deterministic code, publish the composite, not the primitives. p^n and quadratic token growth both punish the alternative.
2. What is its side-effect class? Required, no default. If the answer is write_non_idempotent or irreversible, the conversation is now about idempotency keys and approvals, not about schemas.
3. What is the tightest schema that accepts every legitimate call? Closed objects, patterns on identifiers, enums on currencies, bounds on amounts. Strictness is free once a repair pass exists.
4. Who should be able to see it? Scopes, tenants, classification. If the answer is "everyone," ask again — a tool visible to every agent is 200 tokens on every turn of every run in the bank.
5. Does the description say when not to use it? The negative case is what prevents the wrong-tool selection that dominates per-step failure.
6. Who owns it at 3 a.m.? A tool is a service. If nobody is on-call for it, it is not ready.

A seventh, for anything third-party: what does its description do to my model's context, and who reviewed it?

3. Review red flags

In a design document

• "We'll expose all our APIs as MCP tools." That is an M×N problem with a new protocol.
• A tool count in the hundreds with no discovery filtering.
• No version constraint on any caller — a fleet-wide change on every publish.
• A tool with no owner, or an owner who is a team that no longer exists.
• Unentitled and unknown tools returning different errors.
• A third-party server with no registry pin on its descriptions.
• sampling granted to a server "because the SDK asked for it."
• Descriptions that read like API reference documentation rather than selection guidance.

In code

# Red flag: replying to a notification
def handle(msg): return {"jsonrpc": "2.0", "id": msg.get("id"), "result": ...}
#                                                ^ None for a notification, and it is now a reply

# Red flag: bool passes as an amount
if isinstance(args["amount"], int): ...          # True is an int

# Red flag: leaking the control model into the prompt
{"name": ..., "description": ..., "requiredScopes": ["payments.release"]}

# Red flag: enumerable estate
if not authorized: raise Forbidden(f"you lack {tool.required_scopes}")

# Red flag: filtering after the model has seen the list
tools = server.list_tools()
visible = [t for t in tools if allowed(t)]       # the full list already crossed a boundary

# Red flag: mutable version
registry[name][version] = new_spec                # every pin just changed meaning

# Red flag: side-effect class with a default
side_effect: SideEffect = SideEffect.READ         # a payment tool is now retryable

# Red flag: first-error validation
raise ValidationError(errors[0])                  # three round-trips instead of one

In an incident review

• "Four teams broke when we shipped X" → was the change classified? Were there pins? Was listChanged emitted?
• "The agent called the wrong tool" → how many tools were visible, and does the description say when not to use it?
• "We couldn't tell who was affected" → the registry does not record callers. That is the action item.

4. Production war stories

The minor bump that wasn't. Covered in the phase README, and worth repeating because it is the single most common estate-wide outage: a required field added in a minor release. Three independent controls each would have caught it, and a mature estate has all three.

The estate that was enumerable. Unentitled tools returned 403 with the missing scope in the message; nonexistent ones returned 404. A red team recovered the full tool catalogue — every capability the bank's agents had — without a single successful call. The fix was two lines and the finding was a page long.

The 12 000-token preamble. Every agent saw all sixty tools. Cost per turn was dominated by schemas, and selection accuracy was poor because the model had sixty choices. Filtering discovery by scope and task cut context 90% and improved task success more than the model upgrade the team had been lobbying for.

The tool nobody owned. A customer.lookup tool published during a hackathon, used by four production agents eighteen months later, owned by a team that had been reorganized twice. When its backing service was decommissioned, four agents broke and it took two days to find anyone who knew what it did.

The helpful description. A third-party server's tool description contained a paragraph addressed to the model — instructions about how to behave. It went into every agent's context on every turn. It was "just documentation," so it was never reviewed as prompt content.

The retryable payment. side_effect had a default of read. The gateway's retry-on-timeout did exactly what it was configured to do, twice.

5. The interview signal

Signal 1 — you name what MCP does not do, unprompted. The candidate who says "MCP gives me one protocol and one inventory; authorization, tenancy, versioning, tool identity, idempotency and audit are still mine" has thought about the platform. The one who says "we standardized on MCP" has read a blog post.

Signal 2 — the who-breaks rule, stated as a rule. Not a list memorized, but the underlying question: does the new contract accept everything the old one accepted? Then the asymmetry falls out and you can classify any change live.

Signal 3 — you connect discovery to p^n. Filtering the tool list is usually pitched as security. A staff-level answer gives cost, accuracy and security, in that order of increasing persuasiveness, and notices they all point the same way.

Signal 4 — the probing corollary. "An unentitled tool must return the same error as a nonexistent one." Very few candidates volunteer this, and it demonstrates the security habit of thinking about information leakage, not just access control.

Signal 5 — you treat descriptions as prompt surface. Including the third-party injection implication. This is the observation that lands hardest in a regulated interview, because it reframes a documentation field as an attack surface.

Anti-signals:

• "MCP handles auth." (It handles connection authentication for HTTP transports. Different thing.)
• No version story.
• Proposing to expose every internal API as a tool.
• Treating a tool failure and a schema violation as the same thing.
• Being unable to say who owns a tool.

The question to ask them: "When a tool's schema changes, how do you find out who breaks?" If the answer is "we announce it in a channel," you have learned the maturity of the estate and signalled that you know the right question.

6. Mentoring notes

Three exercises:

1. Hand them six schema diffs and ask for patch/minor/major. Include the two counter-intuitive ones (removing a required property; widening an enum). Engineers who have not internalized "who breaks" get those wrong every time, and getting them wrong once in a review is memorable.
2. Have them write a tool description, then evaluate it. Give them a set of ten user questions and see how often the right tool is selected. Then have them add a "do not use this when…" sentence and re-run. The effect size surprises people and permanently changes how they write descriptions.
3. Red-team their own estate. "Without calling anything successfully, tell me what tools exist." Ten minutes of diffing error messages teaches the probing lesson better than any explanation.

And one framing for the platform team: the registry is where a policy becomes a control. Every rule that lives in a wiki page will be broken by a well-meaning team on a Tuesday. The same rule in publish() fails a build instead of a bank. That argument is how this work gets prioritized against the feature backlog.