Phase 09 — Model Accuracy Evaluation & Benchmarking
Difficulty: ⭐⭐⭐⭐☆
Estimated Time: 2 weeks (40–60 hours)
Roles Supported: All — the ability to set up rigorous accuracy evaluation and regression detection is what separates engineering from research
Why This Phase Exists
At Qualcomm, the word "accuracy" appears in the role title. A Senior Staff engineer in this space must be able to quantify accuracy changes with statistical rigor, set up automated regression detection, and communicate results to leadership ("our INT4 quantization scheme reduces MMLU by 0.7 ± 0.3% at 95% confidence").
The lm-evaluation-harness (EleutherAI) is the standard toolkit, but understanding its internals is required when you need to add a custom task (e.g., an industry-specific Q&A benchmark), debug unexpected results, or integrate it into CI.
This phase also addresses the Pareto frontier thinking that is essential for this role: there is no single "best" model. There is the best model given a latency budget, memory budget, or accuracy floor. Building tools to visualize and navigate this space is a core Senior Staff contribution.
Concepts
- Perplexity: $\text{PPL}(w_1 \ldots w_n) = \exp\left(-\frac{1}{n}\sum_{i=1}^n \log P(w_i | w_{1:i-1})\right)$; lower = better; measures how surprised the model is by the test corpus; sensitive to tokenization differences — always report tokenizer alongside PPL
- Few-shot evaluation: present $k$ input-output examples before the test question; measures in-context learning ability; standard benchmarks use 0-shot, 5-shot, or 25-shot
- MMLU (Massive Multitask Language Understanding): 57 subjects, 4-choice multiple choice; 14k test questions; measures broad knowledge; standard LLM eval from 2021
- HellaSwag: sentence completion requiring common-sense reasoning; 70k examples; robust to prompt sensitivity
- GSM8K: grade school math word problems; tests chain-of-thought reasoning; pass@1 vs majority voting
- ARC (AI2 Reasoning Challenge): science questions; ARC-Easy and ARC-Challenge
- WinoGrande: pronoun resolution requiring world knowledge; 44k questions
- TruthfulQA: measures factual accuracy and hallucination rate; adversarially designed to probe model weaknesses
- Calibration: a model is well-calibrated if its stated confidence matches empirical accuracy; ECE (Expected Calibration Error) = $\sum_m \frac{|B_m|}{n}|\text{acc}(B_m) - \text{conf}(B_m)|$
- McNemar's test: statistical test to determine if two models differ significantly on the same examples; more appropriate than t-test for paired accuracy comparisons
- Accuracy regression CI: automated comparison of a model's accuracy before/after a change; fails if accuracy drops by more than a threshold with statistical significance
Labs
Lab 01 — Eval Harness from Scratch
| Field | Value |
|---|---|
| Goal | Implement a model evaluation harness that supports MMLU, HellaSwag, and perplexity evaluation with pluggable task registry and 3 sampling strategies; produce JSON results compatible with the lm-evaluation-harness format |
| Concepts | Few-shot prompt formatting, log-likelihood scoring (for multiple-choice), generation scoring (for open-ended), task registry, statistical aggregation |
| Steps | 1. Implement Task base class with format_example(), get_metric(), get_doc_to_text(); 2. Implement MMLUTask and HellaSwagTask using log-likelihood scoring; 3. Implement PerplexityTask for WikiText-2; 4. Implement EvalHarness runner with batched log-likelihood computation; 5. Implement 3 sampling strategies: greedy, temperature, top-p; 6. Add multi-shot support (0, 1, 5 shots) with configurable separator format; 7. Validate: compare results to official lm_eval on 3 models — must agree within 0.5% |
| Stack | PyTorch 2.3+, transformers, datasets |
| Datasets | MMLU, HellaSwag, WikiText-2 (all via HuggingFace datasets) |
| Output | eval_harness.py with Task, EvalHarness classes; results JSON matching lm_eval schema; comparison table showing agreement with official harness |
| How to Test | pytest test_lab.py — checks: MMLU format matches expected few-shot template, log-likelihood computation matches model.generate reference, results within 0.5% of official lm_eval for 3 tasks |
| Talking Points | "How does log-likelihood scoring work for multiple-choice?" — sum log P(continuation |
| Resume Bullet | Implemented LLM evaluation harness from scratch supporting MMLU/HellaSwag/PPL; validated within 0.5% of EleutherAI's lm-evaluation-harness on 3 models; used for Qualcomm NPU deployment accuracy tracking |
| Extensions | Add custom task support (plug in domain-specific Q&A); add Math (GSM8K) with chain-of-thought; add calibration (ECE) reporting |
Lab 02 — Accuracy-Latency Pareto Frontier Tool
| Field | Value |
|---|---|
| Goal | Build a ParetoAnalyzer that sweeps multiple quantization configurations for a model family, measures accuracy and latency for each, and produces interactive Pareto frontier visualizations |
| Concepts | Pareto optimality, hypervolume indicator, configuration sweep, interactive plotting (plotly), multi-objective optimization, dominated vs non-dominated solutions |
| Steps | 1. Define configuration space: model_size × quant_method × bits × group_size (e.g., 30 configurations); 2. For each config: run Lab 03's quantization + Lab 01's eval; record MMLU %, PPL, latency (ms), memory (GB); 3. Implement is_pareto_dominant(a, b): a dominates b if a is better in all objectives; 4. Implement compute_pareto_front(configs): filter to non-dominated solutions; 5. Implement plot_pareto_2d(x_metric, y_metric) with plotly (interactive hover); 6. Add "recommendation engine": given constraints (max_latency=50ms, max_memory=4GB), return best-accuracy config |
| Stack | PyTorch 2.3+, plotly, pandas |
| Datasets | MMLU, WikiText-2 (for quick accuracy proxies) |
| Output | Interactive HTML Pareto plots (MMLU vs latency, PPL vs memory); recommendation engine; comparison table for 30 configs; best-config selection for 3 different constraint scenarios |
| How to Test | pytest test_lab.py — checks: Pareto front is a valid non-dominated set, dominance relation is transitive and irreflexive, recommendation satisfies constraints, plotly HTML is generated |
| Talking Points | "How do you present a quantization trade-off to a product team?" — Pareto frontier visualization; "What is the hypervolume indicator and why is it useful?" — aggregates multi-objective Pareto quality into a scalar |
| Resume Bullet | Built accuracy-latency Pareto analysis tool; swept 30 quantization configs for LLaMA-3.2-1B; produced interactive Pareto visualizations enabling product team to select optimal deployment config within latency/memory constraints |
| Extensions | Add energy consumption as a third axis; add per-task accuracy (not just aggregate); implement automated recommendation reporting |
Lab 03 — Automated Accuracy Regression Pipeline
| Field | Value |
|---|---|
| Goal | Build a CI-ready accuracy regression system that: (1) runs eval harness on a model before and after a change, (2) applies McNemar's test for statistical significance, (3) fails the build if accuracy drops beyond threshold with confidence |
| Concepts | McNemar's test, multiple testing correction (Bonferroni), accuracy threshold policy, CI integration (GitHub Actions YAML), baseline model checkpointing, regression report format |
| Steps | 1. Implement AccuracyBaseline that stores per-example results for a reference model; 2. Implement AccuracyRegressor that compares new results to baseline using McNemar's test; 3. Implement RegressionPolicy with configurable thresholds (e.g., fail if MMLU drops >0.5% with p<0.05); 4. Implement RegressionReport.to_markdown() with per-task analysis; 5. Write GitHub Actions YAML that runs the regression check on every PR; 6. Test: inject a bug that degrades MMLU by 2% — verify pipeline detects and blocks |
| Stack | PyTorch 2.3+, scipy (McNemar's test), PyYAML, GitHub Actions |
| Datasets | MMLU subset (200 examples per task for fast CI run ~5 min) |
| Output | regression_pipeline.py; .github/workflows/accuracy-ci.yml; sample regression report in Markdown; demo: deliberately broken model caught by CI |
| How to Test | pytest test_lab.py — checks: McNemar's test fires on 2% degradation (p<0.05), passes on 0.1% variation (p>0.05), CI YAML is valid, report Markdown is generated |
| Talking Points | "Why use McNemar's test instead of a t-test for accuracy?" — McNemar is designed for paired binary outcomes; accounts for correlation between test examples; "How do you choose the regression threshold?" — based on calibration variance and business requirement |
| Resume Bullet | Built accuracy regression CI pipeline using McNemar's test; integrated into PR workflow; successfully blocked 3 optimization changes that caused statistically significant accuracy regressions (>0.5% MMLU) |
| Extensions | Add Bonferroni correction for multi-task testing; add historical trend visualization; implement fast proxy eval (100 examples) vs full eval (14k) with calibration |
Deliverables Checklist
- Custom eval harness matching lm-eval-harness within 0.5%
- Pareto frontier tool with 30-config sweep and interactive plots
- Accuracy regression CI pipeline with GitHub Actions integration
-
All
test_lab.pysuites pass
Guides in This Phase
- HITCHHIKERS-GUIDE.md — evaluation taxonomy, perplexity, MMLU, statistical testing
- DEEP-DIVE-EVALUATION-AT-SCALE.md — multi-objective optimization, McNemar's test, CI regression gates, sample-size planning (supports Labs 02–03)
Interview Relevance
- "How would you set up evaluation for a new model before shipping to NPU?" — you describe: baseline FP16 eval on standard benchmarks → post-quantization eval → Pareto analysis → regression CI
- "What statistical test would you use to compare two quantization methods?" — McNemar's test, explain why
- "MMLU dropped 0.8% after our latest optimization. Is that significant?" — you apply McNemar's test, check sample size, report with confidence interval