« Phase 02 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor

Phase 02 — Staff Engineer Notes: Tool Calling, Structured Output & the Repair Loop

This is the judgment layer. The other docs cover how the mechanism works and how it fits a platform. This one is about the seniority gap: what separates the engineer who passes tools=[…] to an SDK from the one who owns the tool layer — and what an interviewer or an architecture review is actually listening for.

The gap: SDK caller vs tool-layer owner

Anyone can wire a tools array into an SDK call and get a demo working. The person who owns the layer is distinguished by what they do about the things the SDK hides: that arguments comes back double-encoded, that a schema-valid call can still be semantically wrong, that a repair loop without a bound is an outage waiting to happen, that 40 tools is a reliability regression not a feature. The SDK makes the happy path trivial. Ownership is the unhappy paths — and every one of them is a decision, not a line of code.

The decisions a staff engineer owns

• Schema strictness. How tight is additionalProperties, how narrow are the enums and ranges, do you go strict/Structured Outputs or leave slack for the model. Tighter schemas reject more bad calls but reject more borderline-good ones too; the calibration is yours.
• Tool surface design. How many tools, how scoped, what the descriptions say. This is the single biggest lever on reliability (see the Principal Deep Dive), and it is a design decision no SDK makes for you.
• Repair budget. max_repairs, and what happens when it is hit — human handoff, fallback tool, hard fail. A budget with no defined terminal behavior is a hang.
• Where validation ends and business rules begin. The schema checks shape (amount_cents is a positive integer). Business rules check policy (this order is refund-eligible, this amount does not exceed the original charge). Deciding which invariants live in the schema versus a rules layer versus the tool body is an architecture call — and putting a business rule in the schema, or a shape check in the tool body, is a smell in both directions.

A decision framework: which structured-output mode

Candidates who conflate the three modes are a tell. The framework:

• Tool / function mode — when the model should choose whether and which action to take from a set. The model is deciding, and the decision is the point. Use for agents that act.
• JSON / response mode — when you always want exactly one structured answer and there is no choice of tool. Extraction, classification, a typed record. You are not asking the model to decide whether, only to fill a shape.
• Constrained / grammar decoding (or strict Structured Outputs) — when you need a hard guarantee the output parses and matches the shape, because a parse failure downstream is unacceptable or expensive. It buys syntactic validity by construction; it does not touch semantics.

The staff-level addition to all three: each still needs semantic validation. None of them makes the values correct. If a candidate reaches for constrained decoding and stops there, they have missed the entire point of the phase.

"Constrained ≠ correct" — the signal phrase

This is the phrase, and it is the fastest way to separate levels. Constrained (or strict) decoding guarantees syntactic validity — the JSON parses, the shape matches. It says nothing about semantic validity — whether the values make sense. A grammar can force {"amount_cents": <integer>}; it cannot force the right amount. Most candidates conflate the two: they say "we use Structured Outputs so the output is guaranteed correct." The correct framing is "guaranteed well-formed, then validated and repaired for correctness." Saying "constrained ≠ correct" out loud, and being able to name the syntactic/semantic distinction behind it, is the single clearest seniority signal in this domain.

Code-review red flags

These are the things that stop a review cold:

• Running the tool before validating. The side effect fires on bad input. This is a correctness and a security bug, not a style nit.
• isinstance(x, int) as the integer check. It passes True, because bool subclasses int. A boolean sails into a field that should be an integer. Reject bools explicitly.
• Assuming arguments is a dict. It is a JSON string on OpenAI, every time. The code works in the demo and breaks on the first real call.
• 40 overlapping tools. Every ambiguous pair is a coin flip in the trajectory, and coin flips compound. A large fuzzy tool surface is a reliability regression.
• An unbounded repair loop. A model missing data it does not have will loop forever. No max_repairs, no terminal behavior — that is a hang in production.
• "Validation is optional, the prompt will handle it." The prompt is a suggestion to a stochastic generator. Validation is the control. Anyone who says the prompt makes validation unnecessary has not run one of these in production.

Production war stories

• The double-encoded-arguments bug that ate an afternoon. A tool call "randomly" fails validation with "expected object." The arguments look right in the logs. Hours later: arguments is a JSON string, the code passed the string to the validator, and the validator was correctly reporting that a string is not an object. One json.loads fixes it. Everyone hits this once; you only hit it once if you remember it.
• The valid-but-wrong amount that reached the payment API. The call was schema-valid — a positive integer, right shape, passed every type check. It was also wrong: the model computed the refund from the wrong line item. Schema validation is necessary and not sufficient; a business-rule check ("amount ≤ original charge") is what would have caught it. The schema is the shape gate, not the policy gate.
• The repair loop that looped on an impossible call. The model was asked to fill a required field it had no data for. With no bound, it re-emitted a guess, failed validation, re-emitted, forever — burning tokens and latency on a call that could never succeed. The fix is not a better prompt; it is a bound plus a terminal handoff. Some calls are impossible, and the loop's job is to detect that and stop, not to try harder.

The exact interview signal

An interviewer probing this area is listening for a specific chain of reasoning, in this order: (1) the model only proposes — your code validates and executes, so all safety is on your side; (2) you validate before executing so the side effect never fires on a bad call and the failure becomes a model-actionable observation; (3) constrained decoding gives syntax not semantics — "constrained ≠ correct" — so you validate values and run a bounded repair loop; (4) tool design (few, well-scoped, well-described, model-actionable errors, IDs over free text) is the biggest reliability lever, and reliability compounds. A candidate who walks that chain unprompted is signaling they have owned the layer. A candidate who says "we pass tools to the SDK and it handles it" is signaling they have only called it.

Closing takeaways

• The model proposes; your code disposes. Every safety, correctness, and authorization decision is on the trusted side of the boundary. Validate before you execute, and let dispatch return a structured error rather than raise.
• Constrained ≠ correct. Syntactic validity is free with strict/grammar decoding; semantic validity you earn with validation plus a bounded repair loop. Never conflate the two.
• Tool design is reliability engineering. Few, sharp, well-described tools with model-actionable errors move per-step \(p\) more than any prompt — and \(p\) compounds over the trajectory.
• Bound the repair loop and define its terminal behavior. Some calls are impossible; the loop's job is to detect that and hand off, not to retry forever.
• The schema is the shape gate, not the policy gate. Types and ranges live in the schema; refund-eligibility and amount-limits live in a business-rule layer. Keep them separate and know which is which.
• Own the unhappy paths. Double-encoded arguments, bool-as-int, valid-but-wrong values, unbounded loops. The SDK hides them; owning the layer means handling every one deliberately.