« Phase 02 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Staff Notes

Phase 02 — Core Contributor Notes: Tool Calling, Structured Output & the Repair Loop

This is the maintainer's view: what the real systems your lab miniature stands in for actually do, where they are sharp, and what our stdlib validator deliberately leaves out. If you were committing to jsonschema, to an inference server's constrained-decoding path, or to instructor, these are the things you would already know.

jsonschema (the Python package) vs our subset

Our validate implements a hand-picked subset: type, properties, required, enum, minimum/maximum, items, additionalProperties: false. The real jsonschema package targets the full Draft 2020-12 specification (and older drafts via versioned validator classes), and the gap is where the interesting engineering lives:

  • $ref and $defs — schemas reference other schemas, including recursively (a comment thread whose replies are comment threads). Supporting $ref means building a reference resolver with a resolution scope stack and cycle detection — a whole subsystem our miniature omits entirely.
  • anyOf / oneOf / allOf / not — boolean combinators. oneOf must match exactly one subschema (so you validate against all branches and count successes), anyOf at least one. This inverts our "collect all errors" model: a failing branch is not necessarily an error, so error reporting for combinators is genuinely hard — which branch's errors do you surface when none matched?
  • pattern (regex) and format (email, date-time, uri) — string constraints. format is famously annotation-only by default in the spec: many validators do not enforce it unless you opt in, a compatibility footgun maintainers field constantly.
  • dependentRequired / dependentSchemas — conditional requirements ("if card is present, cvv is required").
  • Real jsonschema raises ValidationError objects with a rich structure — absolute_path, schema_path, validator, message — and an iter_errors() generator that yields all errors, plus best_match() heuristics to pick the most relevant one from a oneOf explosion. Our flat list of pathed strings is the same idea, deliberately stripped to its essence.

The mechanism our lab teaches — walk the schema, recurse, collect pathed errors — is how the real validator's core works. We omit the reference resolver, the combinators, and the format registry, not the shape of the algorithm.

OpenAI: function calling and Structured Outputs

The evolution here is the part a committer keeps straight, because each stage changed the guarantee:

  1. JSON mode (response_format: {"type": "json_object"}) — guarantees the output is syntactically valid JSON. It does not guarantee the JSON matches any schema. You still get valid JSON of the wrong shape; you still validate and repair.
  2. Function / tool calling — you pass function schemas; the model returns a tool_calls array. Early on the model could still emit arguments that violated your schema, so validation on your side was mandatory.
  3. Structured Outputs (response_format: {"type": "json_schema", "json_schema": {..., "strict": true}}, and strict: true on function tools) — the strong guarantee. With strict on, the platform compiles your JSON Schema into a context-free grammar / FSM and constrains decoding so the emitted JSON is valid against the schema by construction. This is the same constrained-decoding idea you would build with Outlines, run inside the provider.

The maintainer caveat on strict: it does not accept the whole Draft. It requires additionalProperties: false, requires every property to be listed in required (optionality is expressed with a nullable type union, not by omission), and rejects or ignores various keywords (minimum/maximum and format support has been limited). The compilation step is why the schema must be constrained — you cannot compile an arbitrary schema to a finite grammar cheaply.

The crucial real-world fact: even under function calling, OpenAI returns arguments as a JSON string, every time — choices[0].message.tool_calls[0].function.arguments is a str you must json.loads. The double-encode is the normal path, not an edge case. Code that assumes a dict works in exactly zero real OpenAI integrations. This is precisely why parse_tool_call in the lab handles the string form first.

Anthropic: tool use

Anthropic's shape is close but not identical, and the differences bite in porting. Tools are defined with name, description, and input_schema (not parameters). Tool calls come back as content blocks of type: "tool_use", each with id, name, and — notably — input as an already-parsed object, not a JSON string. So the double-encoding hazard is an OpenAI-ism; a multi-provider client must branch on provider, which is exactly the robustness parse_tool_call models. You return results as tool_result blocks keyed by the tool_use id, which is the correlation mechanism for parallel calls.

Constrained-decoding libraries

The self-hosted side of the "valid by construction" guarantee:

  • Outlines compiles a regex or JSON Schema into a finite-state machine, precomputes for each FSM state the set of vocabulary tokens that keep the output valid, and at each decode step masks the logits to that set (Willard & Louf). Because the allowed-token sets are precomputed and indexed by FSM state, the per-step overhead is a lookup, not a re-parse — the insight that made schema-constrained generation practical.
  • guidance interleaves program control flow with generation, constraining spans with regex/grammar and letting you pin literal structure so the model only fills the holes.
  • llama.cpp GBNF grammars — a BNF-style grammar file the sampler enforces token-by-token; the same masking idea at the C++ inference layer.

All three guarantee syntax, not semantics. A grammar forces {"amount": <integer>}; nothing forces the right integer. This is the "constrained ≠ correct" boundary implemented in three codebases.

instructor and Pydantic AI: schema-from-signature plus auto-repair

These are the libraries whose entire value proposition is the machinery your lab builds:

  • You declare a Pydantic model (a typed signature). The library derives a JSON Schema from it via Pydantic's model_json_schema().
  • It ships that schema to the provider (as a tool, as response_format, or as a constrained-decoding grammar).
  • It parses the response and validates it back through the Pydantic model. On a ValidationError, it runs an auto-repair / retry loop: it feeds the validation errors back to the model as a new message and asks for a corrected object, bounded by a max_retries count.

That loop is run_with_repair. instructor's retry-on-validation-error and Pydantic AI's output retries are the production instance of the injected fixer_policy. Pydantic AI wraps tools with @agent.tool and derives each tool's schema from the function signature — the same derivation as to_openai_schema, done by introspecting type hints.

LangGraph ToolNode and the OpenAI Agents SDK executor

On the orchestration side, dispatch corresponds to the framework's tool executor. LangGraph's ToolNode reads the tool calls off the last message, looks each up in a registry, invokes it, and appends a ToolMessage per call — including error handling that returns a tool message on failure rather than raising, so the graph can continue (validate-before-run / never-crash, at the graph level). The OpenAI Agents SDK tool runner does the analogous thing around the SDK's tool abstraction. Both are the "resolve by name, run, wrap the result as an observation" loop, with framework-specific message plumbing on top.

Sharp edges a committer carries

  • Optionality under strict/Structured Outputs is expressed as a nullable union with the field in required, not by leaving it out of required. Porting a permissive schema to strict silently changes its meaning if you miss this.
  • format is annotation-only by default in JSON Schema — a format: "email" that "does nothing" is spec-correct behavior, not a bug.
  • additionalProperties defaults to permissive. If you forget additionalProperties: false, extra fields pass. Strict mode forces you to be explicit; hand-rolled schemas often forget.
  • Grammar compilation has a cost and coverage limit. Not every schema compiles to an efficient FSM; deeply recursive or highly branching schemas blow up, which is why providers restrict the accepted subset.

What our stdlib validator deliberately simplifies

We omit $ref/$defs (no resolver, no recursion-through-reference), the anyOf/oneOf/allOf/not combinators (and their hard error-attribution problem), pattern/format, dependentRequired, and structured ValidationError objects. We keep the essence on purpose: a recursive walk that collects pathed errors, strict bool-not-int typing, additionalProperties: false, and validate-before-execute. That essence is exactly the core the real systems build on — the omitted parts are volume and edge-handling, not a different algorithm. When you pip install jsonschema or reach for instructor in production, you are adding the combinators and the resolver to a mechanism you have already built by hand and understand from the inside.