« Phase 02 README · Warmup · Hitchhiker's · Principal Deep Dive · Core Contributor · Staff Notes

Phase 02 — Deep Dive: Tool Calling, Structured Output & the Repair Loop

This is the mechanism-level pass. We ignore product framing and look at the bytes on the wire, the data structures that check them, the exact ordering of operations, and the specific Python semantics that make a naive implementation wrong. The load-bearing idea: a tool call is an untrusted message that must be validated against a machine-checkable contract before any side effect fires. Everything below is that sentence made mechanical.

The wire protocol, both directions

Two payloads cross the boundary. Inbound to the model is a list of tool definitions. Each definition is a name (an identifier the model will echo back), a description (natural language — the model's only guidance on when to call it), and a parameters object that is a JSON Schema. In OpenAI's envelope this nests as {"type": "function", "function": {"name", "description", "parameters"}}; Anthropic calls the schema field input_schema. Cosmetic differences; identical semantics.

Outbound from the model is a tool call: {"name": "issue_refund", "arguments": {...}}. The model selected the tool by name and populated arguments. Your code owns everything after this point. The model has finished; it did not run anything.

The invariant to hold in your head: the schema you send in is the exact contract you enforce on what comes out. The parameters schema is not documentation — it is the predicate your validator evaluates against arguments.

JSON Schema as the machine-checkable contract

The subset that carries real weight:

• type — "object", "array", "string", "integer", "number", "boolean", "null". The root dispatch of the validator.
• properties — a map from field name to sub-schema; the recursion points.
• required — a list of keys that must be present. Checked independently of properties.
• enum — the closed set of allowed values.
• minimum / maximum — inclusive numeric bounds.
• items — the sub-schema every array element must satisfy.
• additionalProperties: false — reject any key not named in properties. This is what turns a permissive contract into a tight one; without it, the model can smuggle extra fields past you.

A tool's parameters and a structured-output schema are the same object checked by the same code. There is no second validator for "output" — coercing a returned record and validating a tool call are one mechanism.

The validator: a recursive walk that collects all pathed errors

The core is validate(instance, schema, path) -> list[str]. It recurses in lockstep with the schema's structure. At an object node it checks required, then descends into each present property under properties, then (if additionalProperties is false) sweeps for unexpected keys. At an array node it validates every element against items, carrying an index into the path ($.tags[2]). At a scalar node it checks type, then enum, then range bounds.

Two design decisions are load-bearing at the mechanism level.

It collects every error, it does not fail fast. A fail-fast validator returns the first violation and stops. That is fine for a human reading a stack trace, but catastrophic for a repair loop: if a call has three problems and you report one, the model fixes that one, re-emits, and you report the second — three round trips for what should be one. Collecting the full pathed list ($.amount_cents: expected integer, got string and $.reason: required in a single response) lets the model fix everything in one re-emit. Repair success rate is a direct function of error completeness. This is why the walk accumulates into a list rather than raising.

Every error carries a path. $.line_items[0].sku: required tells the model precisely where to act. A bare "validation failed" tells it nothing it can use. The path is built by string-threading the current location through each recursive call.

The bool-is-a-subclass-of-int gotcha

This is the single sharpest mechanism-level trap in the phase. In Python, bool is a subclass of int: isinstance(True, int) is True, and True == 1. So the "obvious" integer check —

if not isinstance(instance, int):
    errors.append(f"{path}: expected integer")

— accepts True as a valid integer. A model that emits {"amount_cents": true} sails through, and downstream true behaves as 1: a one-cent refund, or a boolean flag flowing into arithmetic. The correct check excludes bools explicitly:

if isinstance(instance, bool) or not isinstance(instance, int):
    errors.append(f"{path}: expected integer")

Order matters — test bool first, because a bool is an int. This is not pedantry; it is a genuine schema bypass that a correct validator closes.

Validate before execute — the ordering invariant

The dispatch sequence is fixed and non-negotiable:

1. Resolve the tool by name. Unknown name → structured error, return.
2. validate(arguments, tool.parameters). Non-empty error list → structured error, return.
3. Only now call the real function.
4. Wrap its return (or a caught exception) in a structured result.

The reason validation precedes execution is that step 3 has side effects — it charges a card, deletes a row, sends mail. If you run first and catch exceptions after (the "run-then-catch" anti-pattern), a malformed call has already half-fired the side effect before your except block runs. You get a partial charge, a cryptic traceback, and no clean, model-actionable message. Validate-first guarantees the side effect never fires on a call that fails the contract, and turns rejection into a recoverable observation rather than a crash. dispatch therefore never raises — it returns ToolResult(ok=False, ...).

The double-encoded arguments gotcha

parse_tool_call must survive a wire quirk: arguments frequently arrives as a JSON string, not a JSON object:

{"name": "search", "arguments": "{\"q\": \"tokyo\"}"}

Here arguments is the string "{\"q\": \"tokyo\"}". You must json.loads it a second time to recover the dict. Code that assumes arguments is already a dict passes a string into the validator, which reports "expected object" — a confusing error that points at the wrong layer. The robust parser branches: if arguments is a str, parse it again; if it is already a dict, use it directly; anything else is malformed and raises ValueError. This is not an edge case to defend against occasionally — for some providers the string form is the normal path.

Constrained decoding: syntax, never semantics

Constrained (grammar-guided) decoding operates one level below all of this, at generation time. At each decoding step the model produces a logit vector over the vocabulary; a constraint engine masks to \(-\infty\) every token that would make the partial output un-parseable against a grammar or schema-derived automaton, then samples only from the survivors. Conceptually (Willard & Louf) the JSON grammar compiles to a finite-state machine, and the current FSM state indexes a precomputed set of allowed tokens — the mask is a table lookup, not a re-parse, which is what makes it cheap enough to run per step.

The result: the output is syntactically valid by construction — it will parse, it will match the shape. What it does not guarantee is that the values are correct. A grammar can force {"amount_cents": <integer>}; it cannot force the right integer. "Constrained ≠ correct" is precisely this gap between syntactic and semantic validity. It is why you still validate (ranges, enums, business rules) and still repair after generation.

The bounded repair loop

run_with_repair(fixer_policy, registry, call, max_repairs) closes the semantic gap:

1. dispatch the call.
2. If ok, return (result, repairs_used).
3. Otherwise feed the specific error list back to fixer_policy(errors, call), which returns a corrected call.
4. Re-validate. Repeat, incrementing a counter.
5. Cap at max_repairs — a model missing data it does not have will loop forever; the bound converts an infinite loop into a surfaced failure for a human or fallback.

The fixer_policy is injected as a pure function so the loop is testable without a live model. The error list is the entire lever: the model fixes only what you name.

Worked trace

Schema: issue_refund(order_id: string, amount_cents: integer ≥ 1, reason: string), all required. Model emits:

{"name": "issue_refund", "arguments": {"order_id": "A-1", "amount_cents": "5000"}}

• Parse. arguments is a dict — no second json.loads.
• Validate. $.amount_cents: expected integer, got string; $.reason: required. Two pathed errors, collected together. dispatch returns ok=False; the payment API is never touched.
• Repair round 1. fixer_policy receives both errors, re-emits {"order_id": "A-1", "amount_cents": 5000, "reason": "duplicate charge"}.
• Re-validate. Empty error list. dispatch now calls the real refund function. repairs_used == 1.

Had the validator been fail-fast, it would have reported only the type error; the reason omission would surface on a second round trip. Had it used isinstance(x, int) naively and the model sent true, the bad value would have reached the payment API. Had the parser assumed a dict on a double-encoded payload, validation would have failed against the wrong layer with an unactionable message. Each naive shortcut breaks the mechanism at a specific, identifiable point — which is exactly why the lab makes you build all four correctly.