Lab 01 — Evaluation Harness

Phase: 09 — Model Accuracy Evaluation | Difficulty: ⭐⭐⭐⭐☆ | Time: 3–4 hours

A production eval harness runs benchmarks, collects metrics, detects regressions, and generates reports suitable for CI/CD pipelines.

What you build

  • BenchmarkTask — wraps a dataset + metric function
  • EvalHarness — runs multiple tasks, aggregates per-task and overall scores
  • Integration with lm_eval format (few-shot benchmarks: MMLU, HellaSwag, ARC)
  • Regression detection: compare against baseline with configurable delta threshold
  • JSON report generation for CI artifacts

Key concepts

ConceptWhat to understand
lm-evaluation-harnessOpen-source framework for standardized LLM benchmarks
Few-shot promptingEvaluate with N in-context examples — tests generalization
Accuracy vs F1MMLU uses accuracy; NER/QA use F1; generation uses ROUGE/BLEU
CI integrationEval must be reproducible and deterministic for regression detection

Files

FilePurpose
lab.pyStubs with TODO markers
solution.pyComplete working implementation
test_lab.pypytest suite
requirements.txtDependencies

Run

pip install -r requirements.txt
python lab.py
pytest test_lab.py -v