Selecting Structured-Output Models

Phase 5 · Document 07 · Model Selection Prev: 06 — Agent Models · Next: 08 — Multimodal Models

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

The moment an LLM feeds another system — an API, a database, a workflow, an agent's tool call — its output must be machine-readable and reliable: valid JSON matching an exact schema, every time. "Mostly valid JSON" is a production outage waiting to happen (one malformed response breaks the pipeline). Selecting for structured output is about guaranteed schema conformance + correct values, and knowing the difference between valid syntax and correct data. This doc applies the framework to extraction/automation/pipeline use cases, and underpins reliable agents (06) and data extraction.

2. Core Concept

Plain-English primer

• Structured output — constraining the model to return data in a fixed shape, almost always JSON matching a schema (which fields, which types). Use case: extraction, automation, APIs, agent tool calls.
• JSON Schema — a standard way to describe the required shape ({"type":"object","properties":{...},"required":[...]}). The contract the output must satisfy.
• Three enforcement levels (this is the core distinction):
  1. Prompt-only ("please return JSON") — best-effort; the model usually complies but can emit prose, markdown fences, or invalid JSON. Lowest reliability.
  2. JSON mode — the provider guarantees syntactically valid JSON, but not that it matches your schema (fields/types/values).
  3. Schema-constrained / "structured outputs" (grammar-constrained decoding) — the decoder is physically prevented from emitting tokens that violate the schema, so the output is guaranteed to match the schema's structure. Highest reliability.
• Validation — checking the parsed output against the schema (and business rules) in your code — required regardless of enforcement level.

The distinction that trips everyone: valid ≠ correct

Even grammar-constrained output guarantees only structure (right fields, right types). It does not guarantee the values are correct (e.g. it can put the wrong date in a correctly-typed date field, or hallucinate a value). So selection has two axes:

1. Conformance — does it reliably produce schema-valid output? (Solved by schema-constrained decoding.)
2. Value accuracy — are the extracted/structured values right? (A model-quality + eval problem — measure it.)

Always validate, always eval the values. Schema enforcement removes parse failures; it does not remove wrong answers.

What to optimize for

1. Schema conformance rate — % of outputs that parse and match the schema (target ~100% with constrained decoding).
2. Value accuracy — correctness of the extracted/generated values on your eval (Phase 1.07).
3. Enforcement support — does the provider/model offer true schema-constrained decoding, or only JSON mode / prompt-only? (Big reliability difference.)
4. Schema complexity handling — nested objects, enums, arrays, optional fields, unions — some models/providers degrade on complex schemas.
5. Latency/cost — usually a smaller model suffices for extraction; reserve big models for hard value accuracy.

How enforcement is implemented (so you can reason about it)

Grammar/schema-constrained decoding works at the sampling step (Phase 1.03): at each token, the engine masks out any token that would make the output violate the schema's grammar, so only schema-legal tokens can be sampled. This is why it guarantees structure. Self-hosted engines (vLLM, llama.cpp via grammars/outlines/xgrammar) expose this; commercial providers expose it as "structured outputs" / "response_format with a schema."

3. Mental Model

THREE enforcement levels (reliability ↑):
  "please return JSON" (prompt-only) → JSON mode (valid syntax) → SCHEMA-CONSTRAINED (guaranteed structure)

  schema-constrained = decoder MASKS illegal tokens each step → can't emit invalid structure (Phase 1.03)

TWO axes to select on:
  CONFORMANCE (structure valid?)  → solved by schema-constrained decoding (~100%)
  VALUE ACCURACY (values right?)  → model quality + YOUR eval (constrained ≠ correct!)

ALWAYS validate in your code; ALWAYS eval the values. Extraction usually needs only a small model.

4. Hitchhiker's Guide

What to check first: does the model/provider support true schema-constrained decoding (not just JSON mode)? That single capability mostly solves conformance.

What to ignore at first: raw "smartness" rankings — extraction is usually well within a small model's reach; conformance + value accuracy matter more.

What misleads beginners:

• Believing JSON mode guarantees your schema (it guarantees valid JSON, not your fields/types).
• Believing schema-constrained output guarantees correct values (it guarantees structure only).
• Skipping validation because "the model supports structured output."
• Picking a big expensive model for simple extraction.

How experts reason: prefer schema-constrained decoding for conformance, then eval value accuracy on their data, pick the cheapest model that's accurate, and always validate the parsed output in code (plus retry/repair on failure).

What matters in production: ~100% conformance (constrained decoding), measured value accuracy, robust validation + retry, and graceful handling of the occasional bad value.

How to verify: run your extraction eval — measure conformance rate (parse+schema) and field-level value accuracy across candidates and enforcement modes (Phase 1.07).

Questions to ask: True schema-constrained decoding or just JSON mode? How does it handle nested/enum/union schemas? What's the value accuracy on my data? Does the smaller tier suffice?

What silently gets expensive/unreliable: prompt-only "JSON" breaking parsers; trusting JSON mode for schema; unvalidated wrong values flowing downstream; over-paying for a big model on trivial extraction.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Three enforcement levels: prompt-only < JSON mode (valid syntax) < schema-constrained (guaranteed structure).
• JSON mode guarantees valid JSON, not your schema; schema-constrained guarantees structure, not correct values.
• Always validate the parsed output in code, regardless of enforcement — and eval value accuracy separately.
• Schema-constrained decoding masks illegal tokens at sampling time (Phase 1.03) → ~100% conformance.
• Complex schemas (deep nesting, unions) can degrade conformance/accuracy on weaker models — test them.
• Extraction usually needs only a small model — reserve big models for hard value accuracy.
• Self-hosted engines (vLLM, llama.cpp + grammars/outlines) provide constrained decoding too.
• Tool calling is structured output under the hood — these capabilities overlap (06).

9. Observations from Real Systems

• OpenAI "Structured Outputs" / Anthropic tool schemas / Gemini response schemas provide schema-constrained decoding — the production way to get ~100% conformance.
• vLLM guided decoding and Outlines/xgrammar bring the same guarantee to self-hosted/open models (Phase 7).
• JSON-mode-only setups still hit schema-mismatch bugs (right syntax, wrong/missing fields) — caught by validation.
• Extraction pipelines routinely run on small/cheap models with constrained decoding + validation — cheap and reliable.
• Value-accuracy failures (correctly-typed but wrong data) are the residual risk after conformance is solved — only an eval reveals them.

10. Common Misconceptions

11. Engineering Decision Framework

1. ENFORCEMENT: require true SCHEMA-CONSTRAINED decoding if available → ~100% conformance.
     only JSON mode? → add strict validation + retry/repair.  prompt-only? → avoid for production.
2. VALUE ACCURACY: eval field-level correctness on YOUR data (constrained ≠ correct) → pick cheapest accurate model.
3. SCHEMA COMPLEXITY: test nested/enum/union schemas on candidates; weaker models degrade.
4. ALWAYS validate parsed output in code; retry/repair on failure; log conformance + accuracy.
5. Prefer the SMALL tier for extraction; escalate only for hard value accuracy. Memo (3.07).

12. Hands-On Lab

Goal

Compare enforcement levels and value accuracy: extract a fixed schema from messy text using prompt-only vs JSON mode vs schema-constrained, across a small and a large model.

Prerequisites

• A target JSON schema (e.g. {invoice_id, date, total, line_items[]}); ~20 messy source texts with known-correct values; access to a model supporting structured outputs (+ a small + large tier).

Steps

1. Define the schema (JSON Schema) and the 20 gold examples (text → correct values).
2. Run 3 enforcement modes (prompt-only, JSON mode, schema-constrained) on the small model; for each, measure conformance rate (parses + matches schema) and value accuracy (field-level vs gold).
3. Vary model size under schema-constrained mode; compare value accuracy and cost.
4. Validation + repair: add code-side validation; on failure, retry once with the error; measure improvement.
5. Decide: smallest model + enforcement that hits your conformance (~100%) and accuracy targets; memo it.

import json, jsonschema
def evaluate(model, mode, examples, schema):
    conform = acc = 0
    for ex in examples:
        out = generate(model, ex["text"], mode=mode, schema=schema)  # mode: prompt|json_mode|constrained
        try:
            obj = json.loads(out); jsonschema.validate(obj, schema); conform += 1
            acc += field_accuracy(obj, ex["gold"])                   # fraction of correct fields
        except Exception:
            pass
    return conform/len(examples), acc/len(examples)

Expected output

• Conformance jumps to ~100% with schema-constrained decoding; value accuracy varies by model (proving constrained ≠ correct); a small model often suffices.

Debugging tips

• Constrained conformance < 100%? The provider may only offer JSON mode — confirm capability.
• Conformance 100% but bad values? That's the lesson — eval and validate values; consider a stronger model.

Extension task

Stress-test a complex nested schema (arrays of objects, enums, optional unions) and see which models/enforcement degrade.

Production extension

Wrap extraction with constrained decoding + validation + one repair retry; log conformance and accuracy as a regression gate (Phase 13).

What to measure

Conformance rate and value accuracy per (model × enforcement); cost/latency; repair-retry lift; complex-schema degradation.

Deliverables

• A schema + gold extraction eval set.
• A (model × enforcement) conformance/accuracy table.
• A selection (smallest model + enforcement meeting targets) + memo.

13. Verification Questions

Basic

1. Name the three enforcement levels and what each guarantees.
2. Why does schema-constrained output not guarantee correct values?
3. Why validate even with structured outputs?

Applied 4. You need 100% schema conformance for a pipeline. What enforcement, and what else must you measure? 5. Why is a small model often enough for extraction?

Debugging 6. Your pipeline crashes on ~3% of responses. What enforcement gap caused it, and the fix? 7. Output is always schema-valid but values are wrong. What do you change?

System design 8. Design a robust extraction service: enforcement, validation, retry/repair, and value-accuracy monitoring.

Startup / product 9. Explain how constrained decoding on a small model gives reliable structure cheaply, and where value-accuracy risk remains.

14. Takeaways

1. Three enforcement levels: prompt-only < JSON mode < schema-constrained (prefer constrained).
2. JSON mode ≠ your schema; schema-constrained ≠ correct values.
3. Always validate in code and eval value accuracy separately.
4. Constrained decoding masks illegal tokens → ~100% conformance.
5. Extraction usually needs only a small model — escalate for value accuracy.
6. Self-hosted engines support constrained decoding too; tool calling is structured output underneath.

15. Artifact Checklist

• Schema + gold extraction eval set.
• (model × enforcement) conformance + value-accuracy table.
• Validation + repair wrapper.
• Selection (smallest model + enforcement meeting targets) + memo.
• (Extension) complex-schema stress test.
• Notes: valid ≠ correct; enforcement levels.

Next: 08 — Multimodal Models