Lab 10 — Structured Output & Validation

Schema/validation parts run offline; generation needs an API key. Concepts: Phase 10.02, cheatsheet 15.

Goal

Get reliable structured output and prove that constrained ≠ correct — add a semantic validation + repair loop.

Run (offline, zero deps)

python3 validate.py    # schema + SEMANTIC validation + a repair loop; shows a schema-valid-but-WRONG case

validate.py runs dependency-free (stdlib json) and demonstrates the core lesson with mock model outputs. In production, use Pydantic (snippet below) + your model's native structured output.

Suggested files

lab-10-structured-output/
  README.md
  schema.py        # Pydantic models / JSON Schema (offline-testable)
  extract.py       # call model with structured output, validate, repair
  validate.py      # SEMANTIC checks beyond schema (ranges, cross-field, grounding)

Key snippet

from pydantic import BaseModel, ValidationError
import json

class Product(BaseModel):
    name: str
    price_usd: float
    in_stock: bool

def parse_and_repair(raw: str, retry_fn) -> Product | None:
    for attempt in range(2):
        try:
            obj = Product(**json.loads(raw))
        except (json.JSONDecodeError, ValidationError) as e:
            raw = retry_fn(f"Fix to match schema. Error: {e}\nPrevious: {raw}")
            continue
        # SEMANTIC validation — schema-valid is not enough:
        if obj.price_usd < 0:           # constrained != correct
            raw = retry_fn(f"price_usd must be >= 0; got {obj.price_usd}")
            continue
        return obj
    return None

Steps

  1. Define a schema (Pydantic) for an extraction task.
  2. Call a model with native structured output / tool calling; parse against the schema.
  3. Add semantic validation (ranges, cross-field consistency, grounding against the source).
  4. Add a repair loop (re-prompt on schema or semantic failure; cap retries).
  5. Eval on a golden set: schema-valid rate vs semantically-correct rate (use lab-05 scorers.py).

Deliverables

  • A schema + extraction that returns valid JSON.
  • A case where output is schema-valid but wrong, caught by semantic validation.
  • schema-valid % vs correct % on ≥10 examples.

Why it matters (interview)

"Constrained ≠ correct" is a senior agent/app signal — interviewers love it (interview-prep 01, 06).