Warmup Guide — Model Accuracy Evaluation & Benchmarking

Zero-to-expert primer for Phase 09. Builds rigorous evaluation from "what is a metric" through perplexity, log-likelihood benchmark scoring, statistical significance, Pareto analysis, and regression CI.

Table of Contents

• Chapter 1: Why Evaluation Is an Engineering Discipline
• Chapter 2: Perplexity — Done Correctly
• Chapter 3: Scoring Multiple-Choice Benchmarks
• Chapter 4: The Benchmark Landscape and Its Failure Modes
• Chapter 5: Sampling Variance — Why 0.8% Might Be Nothing
• Chapter 6: McNemar's Test — Paired Model Comparison
• Chapter 7: Multi-Objective Selection — Pareto Thinking
• Chapter 8: Accuracy Regression CI
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Why Evaluation Is an Engineering Discipline

The role context: "accuracy" is in the job title because every optimization in Phases 03–08 trades accuracy for performance, and someone must quantify the trade with enough rigor to ship against. The deliverable sentence of this entire phase is:

"INT4-GPTQ reduces MMLU by 0.7 ± 0.3 points at 95% confidence; latency improves 3.4×; recommendation: ship for tier-2 devices, hold tier-1 pending QAT."

Every clause requires machinery: a harness that computes the metric identically across model variants (Lab 01), statistics that separate signal from sampling noise (Lab 03, Ch. 5–6), and a decision framework when objectives conflict (Lab 02, Ch. 7).

The cardinal rule: evaluation code is production code. An off-by-one in prompt formatting or a tokenizer mismatch produces numbers that are precisely wrong — they look authoritative and misdirect a quarter's work. Treat eval bugs as P0s.

Chapter 2: Perplexity — Done Correctly

From zero: a language model assigns probability to text. Good models assign high probability to real text. Perplexity is the geometric-mean inverse probability:

$$\text{PPL} = \exp!\left(-\frac{1}{N}\sum_{i=1}^{N} \log p(w_i \mid w_{<i})\right)$$

Intuition: PPL = the effective branching factor — "the model is as uncertain as if choosing uniformly among PPL tokens." Lower is better; FP16 LLaMA-class models score ~5–6 on WikiText-2.

The details that make numbers comparable (where most public numbers go wrong):

• Tokenizer-dependent: PPL is per-token; different vocabularies make numbers incomparable across model families. Compare within a tokenizer, or normalize per-byte/ per-word.
• Context construction: documents longer than the context window are evaluated with a sliding window with stride — score only the tokens whose full context is present; scoring with truncated context inflates PPL. Stride choice changes the number; report it.
• What it's for: PPL is exquisitely sensitive to distribution-level damage — it's the best cheap canary for quantization harm (Phase 03 uses it as the fast gate). It is not a capability measure: two models 0.2 PPL apart can differ wildly on reasoning.

Chapter 3: Scoring Multiple-Choice Benchmarks

How does a generative model answer multiple choice? Not by generating "B". The standard (lm-evaluation-harness) method is log-likelihood scoring:

1. Format the question (+ few-shot examples) as a prompt.
2. For each choice $c_k$: compute $\log p(c_k \mid \text{prompt})$ — sum of token log-probs of the continuation only (one forward pass per choice, no sampling).
3. Predict $\arg\max_k$.

Normalization subtlety: longer continuations accumulate more negative log-prob — raw sum biases toward short answers. Variants: per-token normalization (acc_norm divides by length), per-byte normalization (robust across tokenizers). Which variant a leaderboard used changes scores by points — when reproducing a number, match the variant exactly.

Few-shot mechanics: the k examples, their order, the separator strings, and even whitespace measurably move scores (prompt sensitivity). The harness's job (Lab 01) is to fix every one of these degrees of freedom in code so that two runs differ only in the model. Validating your harness against the official one within 0.5% — Lab 01's acceptance bar — is what "the numbers are right" means.

Chapter 4: The Benchmark Landscape and Its Failure Modes

What each standard benchmark measures, and how it breaks:

Contamination is the field-level failure mode: benchmark text leaks into pretraining data and the score measures memory, not capability. For the optimization-engineering use case (comparing a model to its own quantized variant) contamination cancels out — one of the few places our job is statistically easier than the research job.

Chapter 5: Sampling Variance — Why 0.8% Might Be Nothing

A benchmark score is an estimate from $n$ sampled questions. For accuracy $p$ on $n$ items, the standard error is $\sqrt{p(1-p)/n}$:

• MMLU full (~14K): SE ≈ 0.4% → 95% CI ≈ ±0.8%
• ARC-Challenge (1.2K): SE ≈ 1.4% → 95% CI ≈ ±2.8%
• A 200-item CI subset: SE ≈ 3.5% → a "2% regression" is pure noise

The naive comparison error: treating two scores as independent and requiring CIs not to overlap. Both models answered the same questions — the comparison is paired, and pairing removes the shared per-question difficulty variance. Paired tests (Ch. 6) detect differences several times smaller than independent CIs suggest. This is the statistical fact that makes small-subset CI gating (Ch. 8) viable at all.

Chapter 6: McNemar's Test — Paired Model Comparison

For paired binary outcomes, only the discordant questions carry information:

Under $H_0$ (equal accuracy), each discordant item is a fair coin between $b$ and $c$. Test statistic (with continuity correction):

$$\chi^2 = \frac{(|b - c| - 1)^2}{b + c} \sim \chi^2_1 \quad\text{(use the exact binomial when } b + c < 25\text{)}$$

Worked example: 1000 questions; A correct on 720, B on 705. Discordant: $b = 45$ (A-only), $c = 30$ (B-only). $\chi^2 = (15-1)^2/75 = 2.61 \to p \approx 0.106$ — a 1.5- point gap on 1000 items is not significant. The same gap with $b=90, c=60$ would be. This calibration — gap size vs discordant counts — is what Lab 03 automates.

Multiple testing: gating on 10 tasks at α=0.05 each fires falsely ~40% of the time. Bonferroni (α/10) or, better, a hierarchical policy: gate on the aggregate, flag on per-task.

Chapter 7: Multi-Objective Selection — Pareto Thinking

After Phases 03–08 you hold dozens of (model × quant × config) variants with (accuracy, latency, memory, energy) each. There is no "best" — there is a Pareto front.

• Dominance: A dominates B if A is no worse on every objective and strictly better on at least one. Dominated configs are never the right choice; discard them.
• The front: the non-dominated set — the menu of rational tradeoffs. Everything Lab 02 computes flows from the dominance predicate (and its tests: irreflexive, transitive, antisymmetric).
• Choosing from the front requires preferences, made explicit: constraint satisfaction ("max accuracy subject to ≤50 ms and ≤4 GB" — the deployment-tier framing), scalarization (weighted score — sensitive to normalization; document it), or the knee point (closest to the normalized utopian point — the "best default" heuristic when stakeholders won't give you weights).
• Communication: the Pareto plot is the artifact that moves product decisions — engineers argue configs; a frontier chart lets a PM pick a point and own the tradeoff.

Chapter 8: Accuracy Regression CI

The capstone pattern (Lab 03, reused in Phase 11): make accuracy a gated property of the codebase, like tests.

Design:

1. Baseline storage: per-example results (not just the aggregate!) for the reference model — required for pairing (Ch. 6); content-address it by model+data+harness version.
2. Fast subset: a fixed, stratified ~200–500-item subset for PR gates (~minutes); the full suite runs nightly. Fixed means fixed — re-sampling per run adds variance.
3. Policy: fail if (drop > threshold) AND (McNemar p < α) on the aggregate; warn on per-task flags. Threshold from Ch. 5's SE math, not vibes.
4. Report: a markdown table per PR — aggregate delta with CI, per-task deltas, discordant counts, links to diffing examples. Humans override with recorded rationale.
5. The injection test: deliberately ship a known-bad change (e.g., a 2% degradation) and verify the gate fires — a regression gate that has never caught anything is untested code.

Failure modes to design against: baseline drift (harness change invalidates stored results — version everything), flaky gates eroding trust (calibrate α to observed noise), and silent eval-harness bugs (golden-output tests on the harness itself).

Lab Walkthrough Guidance

Order: Lab 01 (harness) → Lab 02 (Pareto) → Lab 03 (regression CI).

• Lab 01: build Task formatting and unit-test the prompt strings (golden outputs); implement log-likelihood scoring and verify on a hand-checkable toy (2-token continuations you can compute manually); only then add tasks; finish by reconciling with official lm_eval numbers — investigate any gap > 0.5% to its root cause (it's always formatting or normalization).
• Lab 02: dominance predicate first, with the property tests; then the front; then knee point and weighted ranking; sanity-check the front visually on synthetic clouds.
• Lab 03: McNemar on synthetic outcomes with known answers (Ch. 6's worked example as a test case); then baseline storage; then the policy; then the injection test.

Success Criteria

You are ready for Phase 10 when you can, from memory:

1. Write the PPL formula and name the three comparability traps (tokenizer, windowing, stride).
2. Explain log-likelihood scoring and why length normalization variants change scores.
3. Compute the SE of an accuracy on n items and the 95% CI; state why pairing beats it.
4. Run McNemar by hand on a 2×2 discordant table (the worked example).
5. Define Pareto dominance and defend the knee point as a default.
6. Sketch the regression-CI design including the injection test and why per-example baselines are stored.

Interview Q&A

Q: MMLU dropped 0.8% after our optimization. Ship or block? Don't answer from the aggregate. Get per-example pairs, run McNemar: on full MMLU (n≈14K), 0.8% with typical discordance is likely significant (block or investigate); on a 1K subset it's likely noise. Then decompose per-subject — a flat -0.8% is benign distribution shift; -8% on math with flat elsewhere is a targeted failure (often a quantized layer that matters for arithmetic). The answer is a procedure, not a verdict.

Q: Why validate a custom harness against lm-evaluation-harness within 0.5%? Because every formatting degree of freedom (separators, few-shot order, normalization variant, BOS handling) silently moves scores by points. Agreement within sampling noise on shared tasks is the only evidence your numbers live in the same universe as published ones. The 0.5% bound ≈ the SE of the tasks compared.

Q: How do you evaluate when the optimized model is on-device and slow to query? Stratified subsets sized by Ch. 5's SE math for the precision you need; pair with the host-side reference on identical examples; use PPL on a fixed shard as the cheap canary before paying for task evals; and cache per-example results — re-evaluation after a fix should only re-run affected configs. (This is the Phase 11 capstone's architecture.)

Q: Your CI gate fires on 1 of every 15 clean PRs. Diagnose. The gate's false-positive rate is miscalibrated: α too loose for the subset size, non-fixed eval subset adding sampling variance, nondeterminism in the model run (dropout left on? unseeded sampling? GPU nondeterminism?), or baseline drift. Measure run-to-run σ on identical code, set thresholds at 3σ, fix the seed and the subset, version the baseline. A flaky gate is worse than no gate — people stop believing it.

References

• lm-evaluation-harness — read docs/ on task formats and normalization
• Hendrycks et al., Measuring Massive Multitask Language Understanding (MMLU) (2020) — arXiv:2009.03300
• Zellers et al., HellaSwag (2019) — arXiv:1905.07830
• Cobbe et al., Training Verifiers to Solve Math Word Problems (GSM8K) (2021) — arXiv:2110.14168
• McNemar, Note on the sampling error of the difference between correlated proportions (1947)
• Dietterich, Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms (1998) — the paired-testing canon
• Biderman et al., Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) — arXiv:2405.14782
• HuggingFace: Perplexity of fixed-length models — the sliding-window mechanics
• DEEP-DIVE-EVALUATION-AT-SCALE.md — this phase's companion deep dive