Lab 01 — Tool-Calling Engine & Repair Loop

Phase 02 · Lab 01 · Phase README · Warmup

The problem

Phase 01 parsed the model's free text — Action: / Action Input: — across the trust boundary. Production stacks moved on: modern models emit a native tool call, structured JSON in a dedicated field ({"name": ..., "arguments": {...}}), and you validate it against a JSON Schema contract before you run anything. The wire format changed; the discipline did not. The model still only proposes; your code still executes, after validating.

Build that engine from scratch — stdlib only, no jsonschema package — because when a tool call is silently rejected at 2 a.m., the person who wrote the validator is the one who fixes it:

• A JSON-Schema (subset) validator — the contract the model's output must satisfy.
• Typed dispatch — validate arguments, then run; a schema failure is a structured result, never a crash.
• A schema-guided repair loop — feed the validation errors back to the model (here, an injected fixer) and let it correct its own output, bounded so it terminates.
• Structured-output coercion — parse and validate what the model returns, tolerating the ```json fenced-block habit.

The lesson the tests nail down: constrained ≠ correct. Validation guarantees the call is schema-valid, not that it is semantically right — a valid-but-wrong call still dispatches.

What you build

File map

Run it

pytest test_lab.py -v                       # your lab.py (red until you implement)
LAB_MODULE=solution pytest test_lab.py -v   # the reference (green)
python solution.py                          # the four-part worked example

Success criteria

• Your validate returns [] for a conforming instance and a pathed error string ($.days: …) for each violation, recursing into nested objects and arrays.
• Your validator rejects True for a type: integer field — bool is a subclass of int in Python, and letting it through is a real schema-bypass.
• parse_tool_call decodes a double-encoded arguments string (a second json.loads) and raises ValueError on malformed input.
• dispatch validates before running and turns every failure — unknown tool, schema violation, tool exception — into a structured ToolResult, never an exception.
• run_with_repair fixes a wrong-type argument in one round (repairs == 1) and gives up at max_repairs with a structured failure.
• coerce_output parses fenced JSON and raises on invalid JSON or a schema violation.
• All 30 tests pass under both lab and solution.

How this maps to the real stack

• validate is the jsonschema package (Draft 2020-12) that sits behind every serious tool layer — and the strict schema OpenAI/Anthropic enforce for tool arguments and structured outputs. Real validators add $ref, oneOf, pattern, formats; the mechanism — walk the schema, collect pathed errors, recurse — is exactly this.
• Tool + to_openai_schema is what @openai_function / Anthropic's tools=[…] / Pydantic-AI's @agent.tool / the instructor library generate for you: a JSON Schema derived from a typed function signature, shipped to the model in the request.
• parse_tool_call is the client-side decode of the model's tool_calls array. The double-encoded arguments is not a contrived edge case — OpenAI returns arguments as a JSON string every time, so the second parse is the normal path, not the exception.
• dispatch is the framework's tool executor (OpenAI Agents SDK, LangGraph ToolNode). Validate-before-run is the trust boundary from Phase 00 made mechanical.
• run_with_repair is the auto-repair loop in instructor, Pydantic-AI's output retries, and Outlines/Guidance's constrained decoding — the reliability mechanism that turns a flaky generator into a typed API. Constrained decoding enforces syntax at generation time; a repair loop enforces the schema after the fact. Both leave semantics to you.
• coerce_output is "JSON mode" / "structured outputs" — response_format={"type": "json_schema", …}. The fenced-block tolerance is the workaround for models (and JSON mode's weaker cousins) that wrap the payload in Markdown.

Limits. A real validator implements the full Draft spec ($ref, allOf/anyOf, pattern, format, dependentRequired); constrained decoding can guarantee schema-valid JSON at generation time (a GBNF grammar, Outlines' FSM) so the repair loop rarely fires; and the model is a live, sampling oracle. The control flow — schema in, call out, validate, repair — is exactly this.

Extensions (your own machine)

• Add pattern (regex) and format (email, date-time) keywords, and a oneOf/anyOf combinator, and watch the error-message design get hard — that is why good validators are worth their weight.
• Wire a real model behind the fixer_policy interface (one function swap): on a schema error, re-prompt with the error strings and take the model's corrected JSON. Measure how often it fixes in one round vs two.
• Add parallel tool calls: parse_tool_calls(raw) -> list[ToolCall] and a dispatch_all that validates and runs a batch, collecting per-call results (the shape modern APIs return).
• Add an idempotency key to ToolCall and make dispatch a no-op replay for a repeated key — the bridge to durable execution (Phase 08).

Interview / resume signal

"Built a native tool-calling engine from scratch — a JSON-Schema validator, typed validate-before-dispatch, and a bounded schema-guided repair loop — that turns an LLM's stochastic JSON into a typed, self-correcting tool API. Handles the double-encoded-arguments wire gotcha and structured-output coercion, and proves the 'constrained ≠ correct' boundary: validity is enforced in code, semantics are verified separately."