The Hitchhiker's Guide — Phase 09: Model Accuracy Evaluation & Benchmarking

Section 1: The Evaluation Taxonomy

Model evaluation has three tiers:

Tier 1 — Intrinsic metrics (measure the model directly):

• Perplexity on a held-out corpus (WikiText-2, PTB, C4)
• Loss on validation set
• Fast, requires no task data, but weakly correlated with downstream performance

Tier 2 — Task benchmarks (measure performance on standardized tasks):

• MMLU, HellaSwag, ARC, WinoGrande, GSM8K, TruthfulQA
• Industry standard; slow but well-correlated with capability; allows comparison across models

Tier 3 — Human evaluation / production metrics:

• Human preference ratings (MT-Bench, Chatbot Arena)
• A/B test on production traffic
• Ground truth for deployed models; expensive, slow, necessary for final validation

For this role (model accuracy for hardware deployment), Tier 1 and Tier 2 are the daily tools. The key insight: quantization regression is primarily detectable at Tier 1 first, then Tier 2.

Section 2: Perplexity — The Right Way to Compute It

Definition

$$\text{PPL}(w_1 \ldots w_T) = \exp\left(-\frac{1}{T} \sum_{t=1}^{T} \log P(w_t \mid w_1, \ldots, w_{t-1})\right)$$

Lower PPL = better (model is less surprised by the test text).

Stride Trick for Long Documents

A transformer has a fixed context window. For a 2048-token document evaluated with a 1024-token window, naive evaluation would split into chunks, missing cross-chunk context. The stride trick overlaps chunks:

def compute_perplexity_strided(model, tokenizer, text, stride=512, max_length=1024):
    encodings = tokenizer(text, return_tensors="pt")
    seq_len = encodings.input_ids.size(1)
    
    nlls = []
    prev_end_loc = 0
    for begin_loc in range(0, seq_len, stride):
        end_loc = min(begin_loc + max_length, seq_len)
        trg_len = end_loc - prev_end_loc  # tokens for which we compute NLL
        input_ids = encodings.input_ids[:, begin_loc:end_loc].to(device)
        target_ids = input_ids.clone()
        target_ids[:, :-trg_len] = -100  # mask prefix tokens (they're context only)
        
        with torch.no_grad():
            outputs = model(input_ids, labels=target_ids)
            neg_log_likelihood = outputs.loss * trg_len  # unnormalized
        
        nlls.append(neg_log_likelihood)
        prev_end_loc = end_loc
        if end_loc == seq_len:
            break
    
    ppl = torch.exp(torch.stack(nlls).sum() / seq_len)
    return ppl.item()

PPL sensitivity to tokenization: PPL is only comparable between models using the same tokenizer. LLaMA-2 and GPT-2 tokenizers produce different token counts for the same text → different PPL denominators → incomparable values. Always report the tokenizer alongside PPL.

Section 3: MMLU — How the Benchmark Works

MMLU (Massive Multitask Language Understanding) is a 4-choice multiple-choice benchmark across 57 subjects (STEM, humanities, social sciences, professional exams).

Log-Likelihood Scoring

For each question, the model is scored on which of the 4 completions it assigns highest log-probability to:

def score_multiple_choice(model, tokenizer, context, choices):
    """
    context: "Question: ... A. ... B. ... C. ... D. ... Answer:"
    choices: [" A", " B", " C", " D"]
    Returns: index of highest log-likelihood choice
    """
    scores = []
    for choice in choices:
        input_text = context + choice
        input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to(device)
        choice_ids = tokenizer(choice, return_tensors="pt").input_ids[:, 1:]  # skip BOS
        
        with torch.no_grad():
            outputs = model(input_ids)
            logits = outputs.logits  # [1, seq_len, vocab_size]
        
        # Score only the choice tokens
        choice_start = input_ids.shape[1] - choice_ids.shape[1]
        log_probs = torch.nn.functional.log_softmax(logits[0, choice_start-1:-1, :], dim=-1)
        score = log_probs.gather(1, choice_ids[0].unsqueeze(1)).sum().item()
        scores.append(score)
    
    return torch.tensor(scores).argmax().item()

Why MMLU is the Standard

1. Diverse: 57 subjects reduces task-specific overfitting
2. Cheap: multiple-choice = no generation, fast to evaluate
3. Well-calibrated: results are stable across runs (no sampling variance)
4. Established: every major model reports it → direct comparison possible

Typical Numbers

After INT8 quantization: typically -0.5 to -1.5% absolute
After INT4 quantization: typically -1 to -3% absolute
After extreme INT2: typically -5 to -15% — may be unacceptable

Section 4: Statistical Testing for Accuracy Comparisons

Why You Can't Just Compare Percentages

If model A gets 73.4% and model B gets 73.7% on MMLU (14k questions), is B significantly better?

With 14,000 examples, the standard error of an accuracy estimate is: $$\text{SE} = \sqrt{\frac{p(1-p)}{n}} = \sqrt{\frac{0.734 \times 0.266}{14000}} = 0.0037 = 0.37%$$

A difference of 0.3% is less than 1 standard error — not statistically significant.

McNemar's Test

For paired binary comparisons (both models evaluated on the same test examples), McNemar's test is the correct choice:

The test statistic (with continuity correction): $$\chi^2 = \frac{(|n_{01} - n_{10}| - 1)^2}{n_{01} + n_{10}}$$

Under the null hypothesis that the two models perform equally, $\chi^2 \sim \chi^2(1)$. Reject null if $\chi^2 > 3.84$ ($p < 0.05$).

from scipy.stats import chi2

def mcnemar_test(results_a: list[bool], results_b: list[bool]):
    n01 = sum(not a and b for a, b in zip(results_a, results_b))  # B right, A wrong
    n10 = sum(a and not b for a, b in zip(results_a, results_b))  # A right, B wrong
    
    if n01 + n10 == 0:
        return 1.0  # no disagreements → identical → p=1.0
    
    # McNemar's test with continuity correction
    chi2_stat = (abs(n01 - n10) - 1) ** 2 / (n01 + n10)
    p_value = 1 - chi2.cdf(chi2_stat, df=1)
    return p_value

# Usage:
# results_baseline[i] = True if baseline model got example i correct
# results_new[i] = True if new model got example i correct
p = mcnemar_test(results_baseline, results_new)
print(f"McNemar p-value: {p:.4f}")
print("Significantly different" if p < 0.05 else "Not significantly different")

When to Use What Test

Section 5: Building a Custom Eval Task

The lm-evaluation-harness task registry:

from lm_eval.api.task import Task
from lm_eval.api.instance import Instance
import datasets

class MyCustomTask(Task):
    DATASET_PATH = "my_org/my_dataset"
    DATASET_NAME = None
    
    def has_training_docs(self): return True
    def has_validation_docs(self): return True
    def has_test_docs(self): return False
    
    def training_docs(self): return self.dataset["train"]
    def validation_docs(self): return self.dataset["validation"]
    
    def doc_to_text(self, doc) -> str:
        """Format the prompt."""
        return f"Question: {doc['question']}\nAnswer:"
    
    def doc_to_target(self, doc) -> str:
        """The expected completion."""
        return f" {doc['answer']}"
    
    def construct_requests(self, doc, ctx, **kwargs):
        """Return list of request objects."""
        # For multiple choice: one loglikelihood request per choice
        return [
            Instance(
                request_type="loglikelihood",
                doc=doc,
                arguments=(ctx, f" {choice}"),
                idx=i,
            )
            for i, choice in enumerate(doc["choices"])
        ]
    
    def process_results(self, doc, results):
        """Given results from model, return metrics."""
        gold = doc["answer_idx"]
        pred = max(range(len(results)), key=lambda i: results[i][0])
        return {"acc": int(pred == gold)}
    
    def aggregation(self): return {"acc": mean}
    def higher_is_better(self): return {"acc": True}

Section 6: Interview Pitfalls

Section 7: Resources

1. MMLU paper — Hendrycks et al., "Measuring Massive Multitask Language Understanding" (ICLR 2021)
2. lm-evaluation-harness — https://github.com/EleutherAI/lm-evaluation-harness
3. HellaSwag — Zellers et al., "HellaSwag: Can a Machine Really Finish Your Sentence?" (ACL 2019)
4. GSM8K — Cobbe et al., "Training Verifiers to Solve Math Word Problems" (arXiv 2021)
5. McNemar's test — Wikipedia has a clean derivation; also see: Dietterich, "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms" (Neural Computation 1998)
6. Open LLM Leaderboard — https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard — see canonical evaluation numbers
7. BIG-bench — Srivastava et al., "Beyond the Imitation Game" (TMLR 2023) — harder diverse benchmark