Evaluation Terms — Benchmarks, Evals, and Measuring Quality

Phase 1 · Document 07 · LLM Vocabulary and Mental Models Prev: 06 — Local Model Terms · Next: 08 — Business and Pricing Terms

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

"Which model is best?" has no answer without evaluation — and the single most expensive mistake in LLM engineering is choosing a model by its leaderboard rank instead of by its performance on your task. Benchmarks are marketing-adjacent and contaminated; your eval set is ground truth. This vocabulary — benchmark vs eval, golden dataset, LLM-as-judge, pass@k, faithfulness, regression eval — is how you make defensible model decisions, catch quality regressions before users do, and answer "did that prompt change help?" with data instead of vibes. It is the foundation of Phase 13 — Evaluation.

2. Core Concept

Benchmark vs eval (the distinction that matters most)

• Benchmark: a standardized public test (MMLU, GSM8K, HumanEval, GPQA, SWE-bench, MT-Bench). Good for rough cross-model comparison and tracking the field. Weaknesses: may be in training data (contamination), rarely matches your task, and is heavily gamed in marketing.
• Eval: your test, built from your data and success criteria. The thing that actually predicts production quality. A benchmark is someone else's eval.

Rule: benchmarks shortlist; your eval decides.

Offline vs online

• Offline eval: run a fixed dataset through the model in a harness; deterministic, repeatable, runs in CI. Catches regressions before deploy.
• Online eval: measure quality on live traffic — A/B tests, user thumbs-up/down, implicit signals (edits, retries). Captures real-world behavior the offline set misses.

How outputs get scored

• Golden dataset: curated (input → known-good output) pairs; the gold standard for offline eval.
• Exact match / programmatic checks: deterministic correctness (does the code pass tests? is the JSON valid? does the number match?). Cheapest and most reliable — use wherever possible.
• LLM-as-judge: a strong model scores outputs against a rubric. Scales to subjective quality, but is biased (position, verbosity, self-preference) and must be calibrated against humans.
• Human eval: people rate or compare outputs. Most trustworthy, least scalable; reserve for calibration and high-stakes decisions.
• Pairwise / preference eval: judges pick A vs B (powers Arena-style rankings); robust to absolute-scale drift.

Task-specific eval families

Key quality words

• Hallucination: confident, fluent output that is false or unsupported by the context.
• Faithfulness / groundedness: the answer is supported by the provided sources (RAG).
• Calibration: does the model's confidence match its accuracy?

3. Mental Model

BENCHMARK (public, generic) ──shortlist──► YOUR EVAL (private, task-specific) ──decide──► SHIP
                                              │
                  ┌────────── scoring ladder (cheap→expensive, scalable→trustworthy) ──────────┐
                  │ programmatic checks  →  LLM-as-judge (calibrated)  →  human eval            │
                  └───────────────────────────────────────────────────────────────────────────┘

OFFLINE eval = repeatable, in CI, catches regressions BEFORE deploy
ONLINE eval  = live A/B + user signals, catches what offline missed AFTER deploy

Trust ladder: programmatic > human > calibrated LLM-judge > raw LLM-judge > a public benchmark for YOUR task

4. Hitchhiker's Guide

What to do first: build a small golden set (even 20–50 examples) from real or representative inputs with known-good answers. This beats any leaderboard for your decision.

What to ignore at first: chasing every public benchmark. Use one or two for shortlisting; don't optimize to them.

What misleads beginners:

• Trusting leaderboard rank as task quality (contamination + mismatch).
• Treating an uncalibrated LLM-judge as truth (it has biases).
• Reporting a single average and ignoring variance and failure cases.
• Evaluating on examples the model may have trained on.

How experts reason: they ask "what does good mean for this task, and how do I measure it cheaply and repeatably?" They prefer programmatic checks, escalate to a calibrated judge for subjective quality, keep a held-out golden set out of prompts, and gate deploys on a regression eval.

What matters in production: a CI eval that blocks regressions, online signals tied to user value, and a weighted score combining quality, cost, latency, reliability, and safety (see Phase 13).

Debug/verify: when a judge disagrees with your intuition, sample disagreements and check the rubric; when offline and online diverge, your golden set is unrepresentative — expand it from real traffic.

Questions to ask: Is this benchmark possibly in training data? How big/representative is the eval set? Is the judge calibrated against humans? Are we measuring variance, not just the mean?

What silently gets unreliable: evals that drift from real usage, judge prompts that reward verbosity, and "improvements" that help the eval set but hurt production (overfitting to the eval).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• A benchmark is someone else's eval — it rarely matches your task; your golden set decides.
• Benchmark contamination is real: public test items often leak into pretraining, inflating scores.
• Programmatic checks beat LLM-judges wherever a deterministic correctness signal exists (tests pass, JSON valid, exact match).
• LLM-as-judge is biased (position, verbosity, self-preference) and must be calibrated against humans.
• Offline catches regressions; online catches reality — you need both.
• pass@k is the standard code metric; report k and the test suite used.
• Report variance and failure cases, not just an average — one number hides the tail.
• A 20–50 example golden set built from real inputs often out-predicts any leaderboard for your decision.

9. Observations from Real Systems

• OpenAI Evals / Inspect AI structure evals as dataset + grader/scorer — exactly the offline-eval pattern, runnable in CI.
• Chatbot Arena (LMSYS) ranks models by pairwise human preference (Elo), sidestepping absolute-scale gaming — a benchmark designed against contamination.
• Ragas provides faithfulness/context-precision/recall metrics that production RAG teams gate on.
• SWE-bench / HumanEval use programmatic test execution (pass@k) — the most trustworthy code signal, mirrored in coding-tool evals.
• Cursor and other AI coding tools run internal evals (patch applies? tests pass? lint clean?) on every model/prompt change before shipping — the production form of regression eval.
• Model cards report benchmarks; experienced readers treat them as a shortlist input, then run their own eval (see Phase 3).

10. Common Misconceptions

11. Engineering Decision Framework

Choosing a model:
  1. Shortlist with 1–2 benchmarks + capability filters (Phase 4/5).
  2. Build a golden set from YOUR data (20–50+ examples).
  3. Score survivors: programmatic where possible; calibrated judge for subjective; human for high-stakes.
  4. Combine into a weighted score: quality + cost + latency + reliability + safety (Phase 13).
  5. Decide; record a model-selection memo.

Shipping a change (prompt/model/version):
  Run the OFFLINE regression eval → block on score drop → canary → ONLINE A/B → roll out.

Picking a scorer:
  Deterministic correctness exists?  → programmatic check (cheapest, most reliable).
  Subjective quality at scale?       → LLM-as-judge, calibrated vs a human sample.
  High-stakes / final decision?      → human eval on a sample.

12. Hands-On Lab

Goal

Build a minimal offline eval harness: a golden set, a programmatic grader, and a calibrated LLM-as-judge, then compare two models.

Prerequisites

• Python 3.10+, an API key (two model IDs to compare).

Setup

pip install openai
export OPENAI_API_KEY=sk-...

Steps

from openai import OpenAI
client = OpenAI()

# 1. Golden set (input → expected). Programmatic grading where possible.
golden = [
    {"q": "Return only the capital of France.", "expected": "Paris"},
    {"q": "What is 17 * 23? Return only the number.", "expected": "391"},
    {"q": "Is 'def f(): return 1' valid Python? Answer yes or no.", "expected": "yes"},
]

def ask(model, q):
    return client.chat.completions.create(
        model=model, messages=[{"role":"user","content":q}],
        temperature=0, max_tokens=20).choices[0].message.content.strip()

def programmatic_score(model):
    hits = sum(g["expected"].lower() in ask(model, g["q"]).lower() for g in golden)
    return hits / len(golden)

# 2. LLM-as-judge for a subjective prompt
def judge(answer, rubric):
    v = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role":"user","content":
            f"Rubric: {rubric}\nAnswer: {answer}\nScore 1-5, return only the number."}],
        temperature=0, max_tokens=3).choices[0].message.content.strip()
    return int("".join(c for c in v if c.isdigit()) or 0)

for m in ("gpt-4o-mini", "gpt-4o"):
    print(m, "programmatic:", programmatic_score(m))
    ans = ask(m, "Explain the KV cache in 2 sentences for a beginner.")
    print(m, "judge:", judge(ans, "clear, correct, beginner-friendly, <=2 sentences"))

Expected output

• A programmatic score (0–1) per model and a judge score (1–5) per model — a defensible comparison instead of vibes.

Debugging tips

• Judge returns non-numbers → tighten the prompt ("return only an integer 1–5").
• Suspiciously perfect benchmark numbers elsewhere → suspect contamination.

Extension task

Calibrate the judge: hand-score 10 answers, compare to the judge, and report agreement. Adjust the rubric until agreement is high.

Production extension

Wrap the harness so it runs in CI on a held-out golden set and fails the build if the score drops below baseline (a regression gate).

What to measure

Programmatic accuracy, judge score, judge-vs-human agreement, score variance across runs.

Deliverables

• A golden set (≥10 items) + grader.
• A two-model comparison table.
• A judge calibration note (agreement %).

13. Verification Questions

Basic

1. What's the difference between a benchmark and an eval?
2. Why is a programmatic check usually better than an LLM-judge when available?
3. What is benchmark contamination?

Applied 4. You need to choose between two models for invoice extraction. Outline the eval you'd build. 5. Define faithfulness and how you'd measure it for a RAG feature.

Debugging 6. Your offline eval improved but users complain quality dropped. What likely went wrong? 7. Your LLM-judge rates verbose answers higher regardless of correctness. What do you do?

System design 8. Design a CI + canary + online pipeline that prevents a prompt change from shipping a regression.

Startup / product 9. An investor asks how you'll prove and maintain quality as you swap models for cost. Describe your eval strategy as a moat.

14. Takeaways

1. Benchmarks shortlist; your eval decides — build a golden set from real data.
2. Prefer programmatic checks; escalate to a calibrated LLM-judge, then humans.
3. Offline catches regressions, online catches reality — use both.
4. Beware contamination and judge bias; report variance and failures, not just an average.
5. Gate every model/prompt change on a regression eval.
6. Quality decisions are measured, never assumed from leaderboard rank.

15. Artifact Checklist

• Golden dataset (≥10 real/representative examples).
• Code: eval harness with programmatic grader + LLM-judge.
• Benchmark result: two-model comparison table.
• Calibration note: judge-vs-human agreement.
• Regression gate: CI script that blocks on score drop.
• Model selection memo: the eval-backed decision.

Next: 08 — Business and Pricing Terms