Deep Dive — Evaluation at Scale (Phase 09)

"You cannot ship a model you cannot measure. You cannot measure a model you cannot compare. This phase is about comparison done right — from the mathematics of dominance to the statistics of significance."

Table of Contents

Section 1: Why Evaluation at Scale Is Hard

1.1 The Naive Approach and Why It Fails

Imagine you train two versions of a model:

  • v1: 84.6% accuracy on your test set
  • v2: 83.0% accuracy on the same test set

Your instinct says: "v1 is better by 1.6 points, ship v1." But this is wrong for at least two independent reasons:

  1. The difference might be noise — with a finite test set, there's sampling uncertainty around every accuracy number. The true difference could be anywhere from −2% to +5%.
  2. Accuracy is not the only metric — v2 might be 3× faster, use half the memory, and run on a mobile chip where v1 doesn't fit. "Accuracy is best" is not the same as "v1 should ship."

Both failures are common in production ML teams. This phase gives you the tools to avoid both.

1.2 Two Compounding Problems

ProblemWrong answerRight answer
Multiple metrics (accuracy, latency, memory, power)Weighted sumPareto analysis
Statistical significance (is the accuracy delta real?)Compare raw numbersMcNemar's test

These two tools are orthogonal and complementary. A model evaluation at scale uses both.

Section 2: Multi-Objective Optimization — From First Principles

2.1 What Is an Objective?

In optimization, an objective is a function you want to maximize or minimize. When you train a neural network, you minimize a loss function — that's a single-objective optimization problem.

Real-world deployment involves multiple objectives that you care about simultaneously:

ObjectiveDirectionWhy you care
Accuracy (F1, top-1, mAP, etc.)MaximizeCore task quality
Inference latency (ms per token)MinimizeUser experience, SLA
Peak memory (MB RAM)MinimizeFits on device
Model size (MB on disk)MinimizeDownload / OTA update cost
Power draw (mW)MinimizeBattery life on mobile
Throughput (tokens/sec)MaximizeServer cost efficiency

These objectives conflict. Larger models tend to be more accurate and slower. Quantized models are faster and less accurate. You cannot simultaneously maximize accuracy and minimize latency without making tradeoffs.

2.2 Why You Cannot Collapse Multiple Objectives Into One

The obvious hack: create a single composite score.

$$S = w_1 \cdot \text{accuracy} + w_2 \cdot \text{speed_normalized} + w_3 \cdot \text{memory_efficiency}$$

This is called a weighted sum scalarization. It is tempting because it reduces a hard multi-objective problem to a single-objective one. But it has deep problems:

Problem 1 — Arbitrary weights. Where did $w_1 = 0.7$ come from? No principled justification. Different product teams would pick different weights and reach different conclusions from the same data.

Problem 2 — Scale sensitivity. If latency is measured in milliseconds (range: 50–500) and accuracy is measured as a fraction (range: 0.8–0.95), then latency numerically dominates the sum regardless of weights unless you normalize first. And normalization choices are also arbitrary.

Problem 3 — Non-convex tradeoff surfaces. The actual relationship between accuracy and latency is often non-linear and non-convex. A weighted sum can only find solutions on the convex hull of the tradeoff surface — it will miss entire regions of good solutions.

Problem 4 — Incomparable units. How do you compare 1 percentage point of accuracy against 10ms of latency? There is no universal conversion rate. Any weight encodes an implicit and arbitrary conversion.

The correct solution: do not collapse the objectives. Find all non-dominated solutions first, then apply business constraints to pick among them.

2.3 Pareto Optimality — The Origin Story

Vilfredo Pareto (1848–1923) was an Italian economist and engineer studying income distribution. He observed that 80% of Italy's land was owned by 20% of the population — the famous "Pareto principle" — but more importantly, he developed the concept of optimal resource allocation in settings where improving one person's situation necessarily worsens another's.

His key insight: in an economic system with multiple agents and multiple goods, you can often improve some allocation without making anyone worse off. Once you reach a state where no such improvement exists, you have reached a Pareto efficient state.

This concept migrated to engineering, operations research, and eventually ML: when you have N candidate solutions and M conflicting objectives, the Pareto-efficient solutions are those where you cannot improve one objective without worsening at least one other.

The mathematics generalizes completely from economics to model evaluation.

2.4 Formal Definition: Pareto Dominance

Let $\mathbf{a} = (a_1, a_2, \ldots, a_M)$ and $\mathbf{b} = (b_1, b_2, \ldots, b_M)$ be two models' scores across $M$ objectives, where all objectives are framed as "higher is better" (maximization).

Definition: Model $\mathbf{a}$ Pareto-dominates model $\mathbf{b}$ (written $\mathbf{a} \succ \mathbf{b}$) if and only if:

$$\mathbf{a} \succ \mathbf{b} \iff \left( \forall i \in {1, \ldots, M}: a_i \geq b_i \right) \land \left( \exists i \in {1, \ldots, M}: a_i > b_i \right)$$

In plain English:

  • Model $\mathbf{a}$ is no worse than model $\mathbf{b}$ on every objective, AND
  • Model $\mathbf{a}$ is strictly better than model $\mathbf{b}$ on at least one objective

If $\mathbf{a} \succ \mathbf{b}$, then there is no rational reason to ever pick $\mathbf{b}$ — $\mathbf{a}$ is unconditionally better.

Definition: A solution $\mathbf{a}$ is Pareto-optimal (a.k.a. non-dominated) if no other solution dominates it. The set of all Pareto-optimal solutions is the Pareto front (or Pareto frontier).

Converting minimization to maximization: Before applying Pareto analysis, convert all objectives to "higher is better":

  • Latency (minimize) → throughput proxy: $1 / \text{latency_ms}$
  • Memory (minimize) → memory efficiency: $1 / \text{memory_mb}$
  • Power (minimize) → power efficiency: $1 / \text{power_mw}$

2.5 The Pareto Front — Visualized and Intuited

Consider 6 models scored on 2 objectives (accuracy ↑, speed ↑):

Model   Accuracy  Speed (tok/s)  Status
───────────────────────────────────────────────────────
M1      0.92      1.0            Pareto-optimal
M2      0.88      2.5            Pareto-optimal
M3      0.85      3.0            Pareto-optimal
M4      0.88      2.0            Dominated by M2 (M2: same acc, strictly faster)
M5      0.80      1.5            Dominated by M3 (M3: more accurate AND faster)
M6      0.92      0.8            Dominated by M1 (M1: same acc, strictly faster)

Why is M4 dominated by M2?

  • M2 accuracy = 0.88, M4 accuracy = 0.88 → M2 no worse (tied)
  • M2 speed = 2.5, M4 speed = 2.0 → M2 strictly faster → M2 dominates M4. M4 is never the right choice.

Pareto front: {M1, M2, M3}

Accuracy
 0.93 |  M1 ●
 0.90 |
 0.88 |        M4 ○  M2 ●
 0.85 |                    M3 ●
 0.80 |     M5 ○
 0.77 |
      +─────────────────────────→ Speed
         0.8  1.0  1.5  2.0  2.5  3.0

● = Pareto-optimal (on the frontier)
○ = dominated (below/left of the frontier curve)

The Pareto front traces the "efficient frontier" — the tradeoff curve between accuracy and speed. Above and to the right of the frontier is impossible (no model exists there). Below and to the left is suboptimal (some dominated model sits there). The frontier itself represents all rationally defensible choices.

When you have 3+ objectives, the Pareto front becomes a surface (or hyperplane) in M-dimensional space. The intuition is identical: no point on the frontier can be improved in all dimensions simultaneously.

2.6 Computing the Pareto Front

The naive algorithm: for each model $i$, check if any other model $j$ dominates it.

import torch

def is_pareto_optimal_naive(scores: torch.Tensor) -> torch.Tensor:
    """
    scores: [N, M] — N models, M objectives, all to maximize (higher = better)
    Returns: [N] bool tensor, True if model i is Pareto-optimal

    Time complexity: O(N² × M)
    - Outer loop: N iterations (one per model)
    - Inner loop: N-1 comparisons per model
    - Each comparison: M element-wise comparisons
    Acceptable for N < 1000 models and M < 20 metrics.
    """
    N = scores.shape[0]
    is_optimal = torch.ones(N, dtype=torch.bool)

    for i in range(N):
        if not is_optimal[i]:
            # Already known to be dominated — skip for efficiency
            continue

        for j in range(N):
            if i == j:
                continue

            # Does model j dominate model i?
            # j dominates i iff: j >= i on ALL metrics AND j > i on SOME metric
            no_worse     = (scores[j] >= scores[i]).all()   # bool scalar
            strictly_better = (scores[j] > scores[i]).any() # bool scalar

            if no_worse and strictly_better:
                is_optimal[i] = False
                break  # No need to check further for model i

    return is_optimal

Step-by-step trace for our example:

scores = [
    [0.92, 1.0],   # M1
    [0.88, 2.5],   # M2
    [0.85, 3.0],   # M3
    [0.88, 2.0],   # M4
    [0.80, 1.5],   # M5
]

Check M4 (i=3):
  vs M2 (j=1): scores[1] = [0.88, 2.5], scores[3] = [0.88, 2.0]
    no_worse:      [0.88>=0.88, 2.5>=2.0] = [True, True] → .all() = True
    strictly_better: [0.88>0.88, 2.5>2.0] = [False, True] → .any() = True
    → M2 dominates M4. is_optimal[3] = False. Break.

Check M5 (i=4):
  vs M3 (j=2): scores[2] = [0.85, 3.0], scores[4] = [0.80, 1.5]
    no_worse:      [0.85>=0.80, 3.0>=1.5] = [True, True] → True
    strictly_better: [0.85>0.80, 3.0>1.5] = [True, True] → True
    → M3 dominates M5. is_optimal[4] = False. Break.

Final: is_optimal = [True, True, True, False, False]

2.7 Vectorized Pareto Front (Production Implementation)

For N=1000 models, the naive O(N²·M) loop in Python is too slow. Vectorized using broadcasting:

def pareto_front_vectorized(scores: torch.Tensor) -> torch.Tensor:
    """
    Vectorized Pareto front computation using PyTorch broadcasting.
    scores: [N, M] — higher is better for all M objectives

    Returns: [N] bool tensor

    How it works — broadcasting trick:
      scores.unsqueeze(0) has shape [1, N, M]  — "potential dominators" axis
      scores.unsqueeze(1) has shape [N, 1, M]  — "models being checked" axis

      After broadcasting, both have effective shape [N, N, M]:
        result[j, i, k] = scores[j, k] vs scores[i, k]

      no_worse[j, i]     = True iff model j >= model i on ALL metrics
      strictly[j, i]     = True iff model j >  model i on SOME metric
      dominated_by[j, i] = True iff model j dominates model i

      A model i is NOT Pareto-optimal iff ANY j dominates it:
        dominated_by.any(dim=0)[i] = True
    """
    N = scores.shape[0]

    # [N, N, M]: candidate[j] vs model[i] on each metric k
    s_j = scores.unsqueeze(0)   # [1, N, M] → broadcast to [N, N, M]
    s_i = scores.unsqueeze(1)   # [N, 1, M] → broadcast to [N, N, M]

    no_worse     = (s_j >= s_i).all(dim=2)   # [N, N]: no_worse[j, i]
    strictly     = (s_j >  s_i).any(dim=2)   # [N, N]: strictly[j, i]

    dominated_by = no_worse & strictly        # [N, N]: dominated_by[j, i]

    # Remove self-dominance (i dominates itself vacuously — remove those)
    dominated_by.fill_diagonal_(False)

    # Model i is Pareto-optimal iff no other model j dominates it
    return ~dominated_by.any(dim=0)           # [N]: True if NOT dominated

Memory cost: The intermediate [N, N, M] tensor uses N² × M × 4 bytes. For N=1000, M=5: 20 MB — fine. For N=10,000: 2 GB — problematic. In that case, use batched chunk comparisons.

Time complexity: O(N²·M) but fully vectorized on GPU/CPU SIMD — in practice 100–1000× faster than the Python loop.

2.8 From Pareto Front to a Decision: The Knee Point

The Pareto front gives you a set of equally valid, non-dominated models. You still need to pick one (or rank them) for deployment. How?

The Knee Point method — pick the Pareto-optimal model that is geometrically "closest to the ideal" when all objectives are normalized to [0, 1].

Intuition: The "utopian" point $(1, 1, \ldots, 1)$ represents a model that is the best possible on every metric simultaneously. It doesn't exist on the Pareto front, but the model closest to it in normalized space represents the best balanced tradeoff — you don't sacrifice too much on any single axis.

Algorithm:

def find_knee_point(pareto_scores: torch.Tensor) -> int:
    """
    pareto_scores: [K, M] — scores of K Pareto-optimal models
    Returns: index into pareto_scores of the knee point

    Steps:
    1. Normalize each objective to [0, 1] using min-max normalization
    2. Compute Euclidean distance from each model to utopian point (1,...,1)
    3. Return the model with minimum distance
    """
    # Step 1: Normalize each objective independently
    min_vals = pareto_scores.min(dim=0).values    # [M]
    max_vals = pareto_scores.max(dim=0).values    # [M]
    range_   = (max_vals - min_vals).clamp(min=1e-8)  # avoid division by zero

    normalized = (pareto_scores - min_vals) / range_  # [K, M], all in [0, 1]

    # Step 2: Utopian point in normalized space
    utopian = torch.ones(pareto_scores.shape[1])  # [M] = (1, 1, ..., 1)

    # Step 3: Euclidean distance to utopian
    distances = ((normalized - utopian) ** 2).sum(dim=1).sqrt()  # [K]

    return distances.argmin().item()

Worked example with 3 objectives:

Pareto models (raw scores):
  M1: [accuracy=0.92, speed_norm=0.33, mem_efficiency=0.80]
  M2: [accuracy=0.88, speed_norm=0.83, mem_efficiency=0.70]
  M3: [accuracy=0.85, speed_norm=1.00, mem_efficiency=0.90]

Step 1 — Min-max normalization:
  accuracy:        min=0.85, max=0.92, range=0.07
  speed_norm:      min=0.33, max=1.00, range=0.67
  mem_efficiency:  min=0.70, max=0.90, range=0.20

  M1_norm: [(0.92-0.85)/0.07, (0.33-0.33)/0.67, (0.80-0.70)/0.20]
          = [1.000,            0.000,             0.500]
  M2_norm: [(0.88-0.85)/0.07, (0.83-0.33)/0.67, (0.70-0.70)/0.20]
          = [0.429,            0.746,             0.000]
  M3_norm: [(0.85-0.85)/0.07, (1.00-0.33)/0.67, (0.90-0.70)/0.20]
          = [0.000,            1.000,             1.000]

Step 2 — Distance to utopian (1, 1, 1):
  M1: sqrt((1-1)² + (1-0)² + (1-0.5)²) = sqrt(0 + 1 + 0.25) = 1.118
  M2: sqrt((1-0.429)² + (1-0.746)² + (1-0)²) = sqrt(0.326 + 0.064 + 1) = 1.148
  M3: sqrt((1-0)² + (1-1)² + (1-1)²) = sqrt(1 + 0 + 0) = 1.000  ← MINIMUM

Knee point: M3 (fastest, most memory-efficient, acceptable accuracy drop)

M3 wins because it's best on 2 out of 3 objectives. Its accuracy deficit (0.85 vs 0.92) is outweighed by being optimal on speed and memory, and the normalized distance captures this balance objectively.

2.9 Constrained Selection: When Product Has Hard Requirements

Sometimes "knee point" is not right — the product team has hard constraints ("we cannot ship below 90% accuracy, period"). Apply constraints first, then choose from the feasible Pareto-optimal models:

def constrained_pareto_select(
    pareto_scores: torch.Tensor,
    metric_names: list[str],
    constraints: dict,   # e.g., {'accuracy_idx': 0, 'min_value': 0.90}
    weights: dict,       # e.g., {1: 0.7, 2: 0.3} — weights for unconstrained metrics
) -> int:
    """
    1. Apply hard constraints (eliminate models below threshold)
    2. Among feasible models, use weighted score

    constraints: list of (metric_index, minimum_value) pairs
    weights: dict {metric_index: weight} for weighted sum among feasible models
    """
    valid_mask = torch.ones(len(pareto_scores), dtype=torch.bool)

    for metric_idx, min_val in constraints.items():
        valid_mask &= (pareto_scores[:, metric_idx] >= min_val)

    if not valid_mask.any():
        raise ValueError(
            f"No Pareto-optimal model satisfies constraints: {constraints}. "
            f"Consider relaxing the constraints."
        )

    valid_scores = pareto_scores[valid_mask]  # [K', M]

    # Weighted score among valid models
    weighted = torch.zeros(valid_scores.shape[0])
    for metric_idx, weight in weights.items():
        weighted += weight * valid_scores[:, metric_idx]

    local_best = weighted.argmax().item()
    global_idx = valid_mask.nonzero().squeeze(1)[local_best].item()
    return global_idx

2.10 Real-World Multi-Metric Evaluation (Qualcomm Context)

For on-device models deployed via SNPE / QNN / QAI Hub, the standard evaluation matrix:

MetricDirectionTool
Accuracy (task-specific)MaximizeYour eval harness
Latency (ms/inference)MinimizeQAI Hub profiler, SNPE benchmark
Memory (peak RAM, MB)MinimizeQAI Hub, runtime instrumentation
Power (mW)MinimizeQualcomm profiler
Model size (MB)MinimizeFile size
Throughput (fps)MaximizeDerived from latency + batch size

Convert all to "higher is better" before Pareto analysis:

scores_maximized = torch.stack([
    accuracy,              # already ↑
    1.0 / latency_ms,     # throughput proxy ↑
    1.0 / memory_mb,      # memory efficiency ↑
    1.0 / power_mw,       # power efficiency ↑
], dim=1)   # [N, 4]

Important: 1/x transforms are monotone-decreasing, which means they preserve the dominance relationships. Model A with lower latency than B will have higher 1/latency than B — the Pareto analysis gives identical results.

Section 3: Statistical Regression Detection — From First Principles

3.1 What Is Statistical Inference?

When you run 1000 test examples through a model and observe 846 correct, you have measured a sample. The true accuracy of the model — what you'd observe if you ran it on all possible inputs — is unknown. Your observed 84.6% is an estimate with uncertainty.

Statistical inference is the discipline of reasoning from samples to populations under uncertainty. It answers: "Given what I measured, what can I conclude about the underlying truth?"

Every inference is probabilistic. We cannot say "v2 is definitely worse than v1." We can only say "if v1 and v2 were equally accurate (identical underlying distributions), the probability of observing a gap this large or larger is 16%." Whether 16% is low enough to act on is a human decision — but at least we know what we're deciding.

3.2 The Null Hypothesis and Alternative Hypothesis

In classical frequentist hypothesis testing, you formulate two competing hypotheses:

  • H₀ (null hypothesis): The simplest, most conservative claim. "Models v1 and v2 have the same true accuracy." The null hypothesis is what you assume to be true until the data convinces you otherwise.

  • H₁ (alternative hypothesis): The claim you're trying to gather evidence for. "Models v1 and v2 have different true accuracies" (two-sided) or "v2 is less accurate than v1" (one-sided).

The logic of hypothesis testing is falsification: you compute how likely your observed data is assuming H₀ is true. If the data is very unlikely under H₀, you reject H₀ in favor of H₁.

This is philosophically important: you never "prove" H₁. You only fail to disprove H₀ (which is different from H₀ being true).

3.3 What Is a p-value, Really?

The p-value is the probability of observing a test statistic as extreme as (or more extreme than) the one you measured, assuming the null hypothesis is true.

$$p = P(\text{statistic} \geq \text{observed} \mid H_0 \text{ is true})$$

What p-value is NOT:

  • It is NOT the probability that H₀ is true
  • It is NOT the probability that your result was due to chance
  • It is NOT the probability that H₁ is false

What p-value IS:

  • A measure of how compatible your data is with H₀
  • Small p-value → data is incompatible with H₀ → evidence to reject H₀

The 0.05 threshold: By convention (Fisher, 1925), p < 0.05 is called "statistically significant." This threshold is arbitrary — it means "we would reject H₀ even if 5% of the time we're wrong." The actual threshold should reflect the cost of false alarms vs. missed regressions in your specific context.

3.4 Type I and Type II Errors — The Cost of Being Wrong

When you make a decision about H₀, there are four outcomes:

                    Reality
                   H₀ true    H₁ true
Decision  Reject   Type I     Correct!
H₀        Fail     Correct!   Type II
  • Type I error (false positive): You reject H₀ (claim regression exists) but it doesn't. You blocked a valid model deployment. Rate = α (significance level).
  • Type II error (false negative): You fail to reject H₀ (claim no regression) but one exists. You shipped a regressed model. Rate = β.

Power = 1 − β = probability of correctly detecting a real regression.

In ML deployment:

  • Type I error (false alarm) → unnecessary engineering work, delayed shipping
  • Type II error (missed regression) → user-facing quality drop, potential production incident

Setting α = 0.05 means you accept a 5% false alarm rate. If you require 80% power (standard), you need a large enough test set.

3.5 Statistical Power — Detecting Real Differences

Power depends on four quantities:

  1. Effect size (δ): The magnitude of the true difference. Smaller differences require more data.
  2. Sample size (N): More data = more power.
  3. Significance level (α): Lower α = harder to reject H₀ = less power.
  4. Test design: Paired tests (like McNemar's) have higher power than unpaired tests for correlated data.

You must decide on the minimum effect size you care about before collecting data. If you only care about regressions ≥ 2%, you need far fewer samples than if you care about ≥ 0.5% regressions.

3.6 The Chi-Squared (χ²) Distribution — From Scratch

The chi-squared distribution is the key mathematical object underlying McNemar's test. Understanding it from scratch is essential.

Step 1: The standard normal distribution.

If $Z \sim \mathcal{N}(0, 1)$ (standard normal), then $Z^2$ follows a chi-squared distribution with 1 degree of freedom, written $\chi^2_1$:

$$Z \sim \mathcal{N}(0,1) \implies Z^2 \sim \chi^2_1$$

Why? The PDF of $Z^2 = X$ is the transformation of the normal PDF. For $x > 0$: $$f_{X}(x) = \frac{1}{\sqrt{2\pi x}} e^{-x/2}$$

This is the chi-squared PDF with 1 degree of freedom.

Step 2: Summing squared normals.

If $Z_1, Z_2, \ldots, Z_k$ are independent standard normals, then:

$$\sum_{i=1}^k Z_i^2 \sim \chi^2_k$$

The parameter $k$ is called degrees of freedom (df). Adding more independent squared normals increases the df.

Step 3: Why this matters for hypothesis testing.

Many test statistics are constructed as normalized squared differences. When H₀ is true, those statistics follow a chi-squared distribution. We can then look up the probability of seeing a value "this extreme or larger" — that's the p-value.

For McNemar's test specifically, the test statistic measures the squared imbalance between regressions and improvements, normalized by the total number of disagreements. Under H₀ (models are equally accurate), this statistic is approximately $\chi^2_1$.

Step 4: Reading the chi-squared table.

For $\chi^2_1$ at $\alpha = 0.05$ (two-sided), the critical value is 3.84:

P(χ²₁ ≥ 3.84) = 0.05

If your computed statistic > 3.84 → p < 0.05 → significant
If your computed statistic ≤ 3.84 → p ≥ 0.05 → not significant

In Python:

from scipy.stats import chi2
critical_value = chi2.ppf(0.95, df=1)   # = 3.841
p_value = 1 - chi2.cdf(my_statistic, df=1)

3.7 Why Accuracy Delta Alone Is Insufficient

Return to our example: v1 = 84.6%, v2 = 83.0% on N=500 examples.

The observed delta is -1.6%. But how uncertain is each estimate?

The standard error of a proportion estimator is: $$\text{SE}(\hat{p}) = \sqrt{\frac{\hat{p}(1-\hat{p})}{N}}$$

For v1: $\text{SE} = \sqrt{0.846 \times 0.154 / 500} = \sqrt{0.000261} \approx 0.016 = 1.6%$

So v1's accuracy estimate has a 95% CI of roughly $0.846 \pm 2 \times 0.016 = [81.4%, 87.8%]$.

The observed delta of -1.6% is exactly 1 standard error — barely visible above the noise floor. Without a proper test, you would block a valid model deployment.

But there's more: v1 and v2 were evaluated on the same test examples. This creates a correlation between their accuracy estimates. Most examples are easy for both models — both get them right. Most failures cluster on hard examples — both fail on the same ones. This correlation means the effective uncertainty of the delta is much smaller than $\sqrt{2} \times SE$ (which would assume independence).

Ignoring this correlation throws away statistical power. McNemar's test captures exactly this structure.

3.8 Why Not a t-test? The Dependency Problem

A two-sample t-test assumes the two sets of measurements are independent. But v1 and v2's per-example outcomes are not independent — they are evaluated on the same inputs, and the difficulty of each example affects both models' outcomes.

Specifically: whether example $i$ is correctly classified by v1 and by v2 are correlated random variables. A t-test ignores this correlation and uses the wrong variance formula, leading to incorrect p-values (usually too conservative — less power).

For paired binary outcomes (correct/wrong per example), McNemar's test is the correct choice. It uses exactly the information that matters: the examples where the two models disagree.

3.9 McNemar's Test — Derivation and Intuition

Setup: For N test examples, each model either gets it right (1) or wrong (0). Build a 2×2 contingency table:

              Model v2 correct (1)   Model v2 wrong (0)
Model v1      a (both right)         b (v1 right, v2 wrong)   ← REGRESSIONS
correct (1)
Model v1      c (v1 wrong, v2 right) d (both wrong)           ← IMPROVEMENTS
wrong (0)

Note: a + b + c + d = N (total examples)
      a + b = total correct for v1
      a + c = total correct for v2

Key insight: Cells $a$ and $d$ tell you nothing about which model is better. They represent examples where both models agree — trivially consistent with either H₀ or H₁. Only cells $b$ and $c$ — the discordant pairs — carry information about relative performance.

Under H₀ (models are equally accurate):

  • Among the discordant pairs, each discordant example is equally likely to be in cell $b$ (regression) or cell $c$ (improvement)
  • The number of regressions $b$ follows a binomial distribution: $b \sim \text{Binomial}(b+c, 0.5)$
  • By the normal approximation to the binomial: $\frac{b - (b+c)/2}{\sqrt{(b+c)/4}} \approx \mathcal{N}(0,1)$

Squaring this z-score gives a chi-squared statistic:

$$\chi^2 = \frac{(b - (b+c)/2)^2}{(b+c)/4} = \frac{(b-c)^2}{b+c}$$

Under H₀: $\chi^2 \sim \chi^2_1$ (approximately, for large $b+c$)

3.10 Yates' Continuity Correction — Why and When

The chi-squared distribution is continuous but the contingency table counts ($b$, $c$) are discrete. For small counts (b+c < 25), the approximation is poor — the p-value is underestimated, increasing Type I error (false positives).

Yates' continuity correction adjusts the discrete statistic toward the continuous distribution by subtracting 0.5 from the absolute difference:

$$\chi^2_{\text{Yates}} = \frac{(|b - c| - 1)^2}{b + c}$$

The $-1$ (not $-0.5$) comes from: $|b-c| - 1$ is the continuity-corrected version of $|b-c| - 0.5$, squared. The $-1$ is conservative — it reduces the test statistic, increasing the p-value slightly, which reduces false positives at the cost of slightly less power.

When to use it: Always when $b + c$ is small (< 25). For large $b+c$ (> 50), the correction has minimal effect. The convention is to always use it for safety.

3.11 Worked Example: From Raw Numbers to Decision

Test set: N = 1000 examples
v1 correct: 846/1000  (84.6%)
v2 correct: 830/1000  (83.0%)
Delta:     -1.6%

Assume we can collect the full contingency table:
  a = 780  (both correct)
  b = 66   (v1 right, v2 wrong — regressions)
  c = 50   (v1 wrong, v2 right — improvements)
  d = 104  (both wrong)

Verify:
  a + b = 780 + 66 = 846 ✓ (v1 correct)
  a + c = 780 + 50 = 830 ✓ (v2 correct)
  a + b + c + d = 1000 ✓

McNemar statistic (with Yates):
  χ² = (|66 - 50| - 1)² / (66 + 50)
     = (16 - 1)² / 116
     = 225 / 116
     = 1.94

p-value:
  p = 1 - CDF(χ²₁, 1.94) = P(χ²₁ ≥ 1.94) = 0.164

Conclusion: p = 0.164 > 0.05. Not statistically significant. We cannot conclude v2 regressed.

Now suppose the models are more similar (agree on more examples):

  a = 810, b = 36, c = 20, d = 134

  χ² = (|36 - 20| - 1)² / (36 + 20)
     = (15)² / 56
     = 225 / 56
     = 4.02

  p = 1 - CDF(χ²₁, 4.02) = 0.044

Conclusion: p = 0.044 < 0.05. Statistically significant regression.

The paradox: Same raw accuracy delta (-1.6%), but opposite conclusions. How?

When models agree on more examples (a=810 vs a=780), there are fewer discordant pairs (b+c = 56 vs 116). Each discordant pair carries more information. With only 56 examples where the models disagree, observing 36 regressions vs. 20 improvements is a 16:20 imbalance that is very unlikely under H₀ (50:50 would be expected). With 116 discordant pairs, the same delta (66 vs 50) is much less extreme relative to the uncertainty.

Principal-level insight: McNemar's power scales with the number of discordant pairs ($b+c$), not the test set size $N$. For highly similar models (agree on 95% of examples), you need larger test sets to get enough discordant pairs for sufficient power.

3.12 Full Implementation with Diagnostics

import torch
from scipy import stats
import numpy as np

def mcnemar_test(
    y_true:   torch.Tensor,   # [N] int — ground truth labels
    y_pred_1: torch.Tensor,   # [N] int — model 1 predictions
    y_pred_2: torch.Tensor,   # [N] int — model 2 predictions
    alpha: float = 0.05,
) -> dict:
    """
    Paired comparison of two classifiers on the same test set.

    Returns dict with full diagnostics:
      b:           count of regressions (m1 right, m2 wrong)
      c:           count of improvements (m1 wrong, m2 right)
      a:           count of both correct
      d:           count of both wrong
      chi2:        McNemar test statistic (with Yates correction)
      p_value:     two-sided p-value
      significant: True if p < alpha
      net_delta:   (c - b) / N — positive means m2 improved overall
      acc_m1:      m1 accuracy
      acc_m2:      m2 accuracy
      power_note:  qualitative note on statistical power
    """
    correct_1 = (y_pred_1 == y_true)   # [N] bool
    correct_2 = (y_pred_2 == y_true)   # [N] bool

    a = ( correct_1 &  correct_2).sum().item()   # both right
    b = ( correct_1 & ~correct_2).sum().item()   # m1 right, m2 wrong (regression)
    c = (~correct_1 &  correct_2).sum().item()   # m1 wrong, m2 right (improvement)
    d = (~correct_1 & ~correct_2).sum().item()   # both wrong

    N = len(y_true)
    assert a + b + c + d == N, "Contingency table does not sum to N"

    if b + c == 0:
        return {
            'a': a, 'b': 0, 'c': 0, 'd': d,
            'chi2': 0.0, 'p_value': 1.0, 'significant': False,
            'net_delta': 0.0, 'acc_m1': correct_1.float().mean().item(),
            'acc_m2': correct_2.float().mean().item(),
            'power_note': 'Models have identical predictions — no test possible.'
        }

    # McNemar statistic with Yates continuity correction
    chi2     = (abs(b - c) - 1) ** 2 / (b + c)
    p_value  = float(1 - stats.chi2.cdf(chi2, df=1))

    # Power note: b+c < 10 → very low power; b+c < 50 → moderate
    if b + c < 10:
        power_note = f"Very low power: only {b+c} discordant pairs. Consider larger test set."
    elif b + c < 50:
        power_note = f"Moderate power: {b+c} discordant pairs."
    else:
        power_note = f"Adequate power: {b+c} discordant pairs."

    return {
        'a':           a,
        'b':           b,
        'c':           c,
        'd':           d,
        'chi2':        float(chi2),
        'p_value':     p_value,
        'significant': p_value < alpha,
        'net_delta':   (c - b) / N,
        'acc_m1':      correct_1.float().mean().item(),
        'acc_m2':      correct_2.float().mean().item(),
        'power_note':  power_note,
    }

# ── Usage example ────────────────────────────────────────────────────────────
torch.manual_seed(42)
N = 1000

y_true   = torch.randint(0, 10, (N,))
y_pred_1 = (y_true + torch.randint(0, 3, (N,))) % 10   # ~70% accurate
y_pred_2 = (y_true + torch.randint(0, 4, (N,))) % 10   # ~65% accurate

result = mcnemar_test(y_true, y_pred_1, y_pred_2)

print(f"Contingency table:")
print(f"  a (both correct):   {result['a']}")
print(f"  b (regressions):    {result['b']}")
print(f"  c (improvements):   {result['c']}")
print(f"  d (both wrong):     {result['d']}")
print(f"Model 1 accuracy:     {result['acc_m1']:.3f}")
print(f"Model 2 accuracy:     {result['acc_m2']:.3f}")
print(f"χ² statistic:         {result['chi2']:.3f}")
print(f"p-value:              {result['p_value']:.4f}")
print(f"Significant (α=0.05): {result['significant']}")
print(f"Power note:           {result['power_note']}")

3.13 Extension: Multi-Class and Regression Problems

Multi-class classification: McNemar's test extends directly. "Correct" means the predicted class equals the true class, regardless of how many classes exist. The test remains binary (correct/wrong per example):

# Works for any number of classes — "correct" is still binary
correct_1 = (y_pred_1 == y_true)
correct_2 = (y_pred_2 == y_true)
# Apply mcnemar_test() as above

Regression (continuous outputs): Binary correct/wrong doesn't apply. Use the Wilcoxon signed-rank test instead — a non-parametric paired test for continuous paired differences:

from scipy.stats import wilcoxon

# Error per example (lower is better)
errors_1 = (y_pred_float_1 - y_true_float).abs().numpy()
errors_2 = (y_pred_float_2 - y_true_float).abs().numpy()

# H0: both models have the same error distribution
statistic, p_value = wilcoxon(errors_1, errors_2, alternative='greater')
# alternative='greater' tests H1: errors_1 > errors_2 (model 2 regressed)
print(f"Wilcoxon p-value: {p_value:.4f}")

Wilcoxon vs paired t-test: The paired t-test assumes errors are normally distributed. Neural network errors on real data often aren't (long-tailed). The Wilcoxon test is distribution-free (non-parametric) and more appropriate.

Section 4: CI/CD Integration — Automated Regression Gates

4.1 What Is a Deployment Gate?

A deployment gate is an automated check that must pass before a new model version can be deployed to production. Without gates, regressions slip through whenever:

  • Engineers are in a hurry
  • The test set isn't run consistently
  • Reviewers look at absolute accuracy but not statistical significance

A gate codifies the decision criteria in code, runs on every PR/model checkpoint, and blocks deployment automatically if criteria fail.

4.2 Multi-Criterion Gate Implementation

class RegressionGate:
    """
    Multi-criterion deployment gate for model accuracy.

    Policy — block deployment if ANY of these fail:
      1. Absolute accuracy below floor (absolute safety)
      2. Accuracy delta exceeds threshold (raw change check)
      3. McNemar's test: significant regression in the statistical sense

    All three criteria are necessary:
      - Criterion 1 alone: misses gradual degradation over many releases
      - Criterion 2 alone: blocks on noise (not significant changes)
      - Criterion 3 alone: allows continuous slow drift if each step is insignificant
    """

    def __init__(
        self,
        threshold_delta: float = -0.01,  # max allowed accuracy drop (absolute)
        min_accuracy:    float = 0.80,   # floor — never ship below this
        alpha:           float = 0.05,   # significance level
    ):
        self.threshold_delta = threshold_delta
        self.min_accuracy    = min_accuracy
        self.alpha           = alpha

    def evaluate(
        self,
        y_true:          torch.Tensor,
        y_pred_baseline: torch.Tensor,
        y_pred_candidate: torch.Tensor,
        task_name:       str = "",
    ) -> dict:
        mc       = mcnemar_test(y_true, y_pred_baseline, y_pred_candidate, self.alpha)
        acc_base = mc['acc_m1']
        acc_new  = mc['acc_m2']
        delta    = acc_new - acc_base

        checks = {
            'floor':            acc_new >= self.min_accuracy,
            'accuracy_delta':   delta >= self.threshold_delta,
            'statistical':      not (mc['significant'] and delta < 0),
            # Only block on stats if the test confirms a real regression AND
            # the direction is indeed a regression (not an improvement)
        }

        return {
            'passed':          all(checks.values()),
            'checks':          checks,
            'delta':           delta,
            'acc_baseline':    acc_base,
            'acc_candidate':   acc_new,
            'mcnemar_p':       mc['p_value'],
            'regressions':     mc['b'],
            'improvements':    mc['c'],
            'power_note':      mc['power_note'],
            'task':            task_name,
        }

# ── Usage in CI pipeline ──────────────────────────────────────────────────
gate   = RegressionGate(threshold_delta=-0.005, min_accuracy=0.85)
result = gate.evaluate(y_true, baseline_preds, candidate_preds, "imagenet_val")

if not result['passed']:
    failed_checks = [k for k, v in result['checks'].items() if not v]
    raise SystemExit(
        f"Deployment blocked for task '{result['task']}'!\n"
        f"  Accuracy:    {result['acc_baseline']:.4f} → {result['acc_candidate']:.4f}"
        f" (Δ={result['delta']:+.4f})\n"
        f"  Regressions: {result['regressions']}, Improvements: {result['improvements']}\n"
        f"  McNemar p:   {result['mcnemar_p']:.4f}\n"
        f"  Power:       {result['power_note']}\n"
        f"  Failed:      {failed_checks}"
    )

4.3 Combining Pareto Analysis with Statistical Testing

Complete end-to-end evaluation pipeline:

def full_evaluation_pipeline(
    models:     list,
    eval_data,
    metrics_fn,   # returns [accuracy, latency_ms, memory_mb, ...]
) -> list[dict]:
    """
    1. Evaluate all candidate models on all metrics
    2. Compute Pareto front (multi-objective, non-dominated set)
    3. For each Pareto-optimal model, run McNemar vs baseline (model 0)
    4. Identify the knee point among Pareto-optimal models
    5. Return ranked recommendations

    Returns list of recommendation dicts, sorted by knee-point distance.
    """
    N = len(models)

    all_preds  = {}
    all_scores = {}   # [accuracy_norm, 1/latency, 1/memory] — all ↑

    for i, model in enumerate(models):
        preds, latency_ms, memory_mb = run_inference(model, eval_data)
        acc = (preds == eval_data.labels).float().mean().item()
        all_preds[i]  = preds
        all_scores[i] = torch.tensor([acc, 1.0/latency_ms, 1.0/memory_mb])

    scores_tensor = torch.stack([all_scores[i] for i in range(N)])   # [N, 3]

    # Step 2: Pareto front
    pareto_mask = pareto_front_vectorized(scores_tensor)
    pareto_ids  = pareto_mask.nonzero().squeeze(1).tolist()

    # Step 3: Knee point within Pareto front
    pareto_scores = scores_tensor[pareto_mask]                        # [K, 3]
    knee_local    = find_knee_point(pareto_scores)                    # index in pareto_scores
    knee_global   = pareto_ids[knee_local]

    # Step 4: McNemar vs baseline for each Pareto-optimal model
    baseline_preds = all_preds[0]
    recommendations = []

    for model_id in pareto_ids:
        if model_id == 0:
            mc = {'b': 0, 'c': 0, 'p_value': None, 'significant': False,
                  'acc_m1': all_scores[0][0].item(), 'acc_m2': all_scores[0][0].item()}
        else:
            mc = mcnemar_test(eval_data.labels, baseline_preds, all_preds[model_id])

        recommendations.append({
            'model_id':        model_id,
            'accuracy':        all_scores[model_id][0].item(),
            'latency_proxy':   all_scores[model_id][1].item(),
            'memory_proxy':    all_scores[model_id][2].item(),
            'mcnemar':         mc,
            'is_knee_point':   (model_id == knee_global),
            'is_baseline':     (model_id == 0),
        })

    return sorted(recommendations, key=lambda x: not x['is_knee_point'])

Section 5: Sample Size Planning

5.1 Why Sample Size Matters Before You Collect Data

The worst time to discover your test set is too small is after running the evaluation. If you have 200 test examples and your models agree on 95% of them, you have only ~10 discordant pairs — far too few for a reliable McNemar's test.

Sample size planning is done before data collection:

  1. Decide the minimum accuracy delta worth detecting (e.g., 1%)
  2. Estimate the expected disagreement rate between models (e.g., 10%)
  3. Choose desired power (80% is standard)
  4. Solve for the required N

5.2 Deriving the Formula

Under H₁ (true accuracy delta = δ):

  • Expected discordant pairs: $(b + c) \approx N \times r$ where $r$ is the disagreement rate
  • Expected net imbalance: $(b - c) \approx N \times \delta$ (because δ extra examples are regressions)

For the McNemar statistic to exceed the critical value $z_{\alpha/2}$ with probability $1-\beta$:

$$\frac{|b-c|}{\sqrt{b+c}} \approx z_{\alpha/2} + z_\beta$$

Substituting: $$\frac{N\delta}{\sqrt{Nr}} \approx z_{\alpha/2} + z_\beta$$ $$\frac{\sqrt{N} \cdot \delta}{\sqrt{r}} \approx z_{\alpha/2} + z_\beta$$ $$\sqrt{N} \approx \frac{(z_{\alpha/2} + z_\beta) \sqrt{r}}{\delta}$$ $$N \approx \left(\frac{z_{\alpha/2} + z_\beta}{\delta}\right)^2 \cdot r$$

For $\alpha = 0.05$ (two-sided), $z_{\alpha/2} = 1.96$. For 80% power, $z_\beta = 0.84$. So $z_{\alpha/2} + z_\beta = 2.80$.

5.3 Implementation

from scipy.stats import norm
import numpy as np

def sample_size_for_mcnemar(
    delta_acc:                 float,   # minimum delta to detect (e.g., 0.01 = 1%)
    expected_disagreement_rate: float = 0.10,  # expected b+c as fraction of N
    power:                     float = 0.80,   # 0.80 = 80% power
    alpha:                     float = 0.05,   # significance level
) -> int:
    """
    Compute minimum N for McNemar's test.

    Example interpretation:
      delta_acc=0.02, expected_disagreement_rate=0.15, power=0.80:
      → Need N examples such that with 80% probability, we detect a 2% regression
        when ~15% of examples differ between models.

    Typical values:
      - Similar models (int8 PTQ of fp16): disagreement rate ~5-15%
      - Different architectures: disagreement rate ~20-40%
    """
    z_alpha = norm.ppf(1 - alpha / 2)   # 1.96 for α=0.05
    z_power = norm.ppf(power)           # 0.84 for 80% power

    N = ((z_alpha + z_power) / delta_acc) ** 2 * expected_disagreement_rate
    return int(np.ceil(N))

# ── Example calls ─────────────────────────────────────────────────────────
print("Sample sizes needed:")
for delta in [0.005, 0.01, 0.02, 0.05]:
    for disagree_rate in [0.05, 0.10, 0.20]:
        N = sample_size_for_mcnemar(delta, disagree_rate)
        print(f"  δ={delta:.1%}, disagreement={disagree_rate:.0%} → N={N:,}")

# Typical output:
#   δ=0.5%, disagreement=5%  → N=15,680
#   δ=0.5%, disagreement=10% → N=31,360
#   δ=1.0%, disagreement=5%  → N=3,920
#   δ=1.0%, disagreement=10% → N=7,840
#   δ=2.0%, disagreement=5%  → N=980
#   δ=2.0%, disagreement=10% → N=1,960
#   δ=5.0%, disagreement=5%  → N=157
#   δ=5.0%, disagreement=10% → N=314

Rule of thumb for production:

  • Aim for ≥ 1000 test examples (detects ~2% delta at 10% disagreement with 80% power)
  • For safety-critical models: 5000+ examples
  • For detecting sub-1% regressions: 10,000+ examples

Section 6: End-to-End: Evaluating Quantized Model Variants

A common scenario: compare FP16 baseline against INT8-PTQ, INT4-GPTQ, and INT4-QAT variants.

def evaluate_quantization_tradeoffs(models_dict: dict, test_data) -> None:
    """
    models_dict: {'fp16': model_fp16, 'int8_ptq': ..., 'int4_gptq': ..., 'int4_qat': ...}

    Produces:
    - Full Pareto analysis (accuracy vs latency vs memory)
    - McNemar test for each variant vs FP16 baseline
    - Knee point recommendation
    """
    results = {}

    for name, model in models_dict.items():
        preds, latency_ms, memory_mb = benchmark_model(model, test_data)
        acc = (preds == test_data.labels).float().mean().item()
        results[name] = {
            'preds':      preds,
            'accuracy':   acc,
            'latency_ms': latency_ms,
            'memory_mb':  memory_mb,
        }

    # Score matrix — all higher = better
    names  = list(results.keys())
    scores = torch.tensor([
        [results[n]['accuracy'],
         1.0 / results[n]['latency_ms'],
         1.0 / results[n]['memory_mb']]
        for n in names
    ])

    pareto_mask = pareto_front_vectorized(scores)
    knee_idx    = find_knee_point(scores[pareto_mask])

    fp16_preds = results['fp16']['preds']

    print(f"\n{'Model':<16} {'Acc':>8} {'Lat(ms)':>10} {'Mem(MB)':>10} "
          f"{'Pareto':>8} {'Knee':>6} {'p-val':>8}")
    print('─' * 72)

    pareto_ctr = 0
    for i, name in enumerate(names):
        is_pareto = pareto_mask[i].item()
        is_knee   = is_pareto and (pareto_ctr == knee_idx)
        if is_pareto:
            pareto_ctr += 1

        if name != 'fp16':
            mc    = mcnemar_test(test_data.labels, fp16_preds, results[name]['preds'])
            p_str = f"{mc['p_value']:.3f}{'*' if mc['significant'] else ' '}"
        else:
            p_str = "baseline"

        print(f"{name:<16} {results[name]['accuracy']:>8.4f} "
              f"{results[name]['latency_ms']:>10.1f} "
              f"{results[name]['memory_mb']:>10.0f} "
              f"{'✓' if is_pareto else '✗':>8} "
              f"{'◉' if is_knee else ' ':>6} "
              f"{p_str:>8}")

# Typical output:
#
# Model               Acc    Lat(ms)    Mem(MB)   Pareto   Knee    p-val
# ────────────────────────────────────────────────────────────────────────
# fp16            0.9234     124.0       6300        ✓             baseline
# int8_ptq        0.9218      62.5       1580        ✓             0.612
# int4_gptq       0.9147      35.2        810        ✓             0.031*
# int4_qat        0.9201      35.2        810        ✓      ◉      0.274
#
# Interpretation:
#   - int8_ptq: no significant regression (p=0.612), 2× faster → good candidate
#   - int4_gptq: significant regression (p=0.031*, p<0.05) → risky
#   - int4_qat:  no significant regression (p=0.274), same speed as gptq → knee point

Section 7: Interview Questions — Principal-Level Answers

Q: "Why McNemar's and not a paired t-test for accuracy comparison?"

McNemar's is designed for paired binary outcomes — right/wrong per example. A paired t-test assumes your measured values are approximately normally distributed continuous quantities. Accuracy per example is 0 or 1, not normally distributed. More importantly, McNemar's uses only the discordant pairs (examples where models disagree) as the effective sample. When comparing two classifiers on the same test set, the outcomes are correlated — both models tend to get easy examples right and hard examples wrong. This correlation means that if both models get 990 out of 1000 examples right in the same way, the effective evidence for comparison comes from only those 10 disagreements. McNemar's uses 10 as the denominator (not 1000), giving the correct answer. A t-test would use 1000 and underestimate uncertainty.

Q: "Model A has 85% accuracy, model B has 83% accuracy, model B is 3× faster. Which do you ship?"

Neither yet — there are two separate questions to answer. First: is the 2% delta statistically real? Run McNemar's test. If p > 0.05, the delta is noise and model B is strictly better (faster, same real accuracy). If p < 0.05, the delta is real — then it becomes a business decision. Is 83% above the product floor? Is the 3× speedup worth the 2% accuracy cost? If the product requires ≥ 85%, model B is eliminated. If 83% meets the floor, model B is Pareto-optimal (dominates everything between 83-85% accuracy at 3× latency). The decision cannot be made unilaterally by the ML engineer — it requires explicit sign-off from the product owner on the accuracy vs. speed tradeoff.

Q: "What's the minimum test set size for a reliable evaluation?"

It depends on what you want to detect. Use the formula: $N \approx (z_{\alpha/2} + z_\beta)^2 \times r / \delta^2$ where $r$ is the expected disagreement rate and $\delta$ is the smallest regression worth catching. For 80% power, α=0.05, 10% disagreement rate: detecting 2% delta needs ~2000 examples; detecting 1% delta needs ~8000. The key insight is that statistical power scales with the number of discordant pairs, not total test set size. Two models that are very similar (agree on 99% of examples) need a huge test set to get enough disagreements for a reliable test.

Q: "Explain Pareto dominance without using the word 'dominate'."

Model A makes model B irrelevant if A is at least as good as B on every single metric, and strictly better on at least one. The Pareto front is the set of models where none of them makes any other one irrelevant — every model on the front offers a distinct, genuine tradeoff. If you want maximum accuracy and don't care about speed, you pick from one end. If you need maximum speed and can accept lower accuracy, you pick from the other end. The knee point is the model that seems to offer the best deal when you weigh all metrics equally — geometrically, it's the model closest to the impossible ideal of being best at everything simultaneously.

Q: "Your team just increased test set size from 500 to 5000 examples and suddenly 3 previously non-significant regressions became significant. Did your model suddenly get worse?"

No — the model didn't change. What changed is your ability to detect real effects. With 500 examples, you had low statistical power — real regressions of 1-2% were invisible in the noise. With 5000 examples, you now have enough statistical power to reliably detect those same real differences that always existed. This is the correct outcome. The 3 regressions were always there; you just couldn't see them. This illustrates why sample size planning before evaluation is critical — if you had run with 5000 examples from the start, you would have made better shipping decisions all along.

Q: "How do you handle evaluation when you have 50+ candidate models from a hyperparameter sweep?"

Two-step process. First, Pareto analysis: compute the Pareto front across all 50+ models on all metrics. Typically only 3-10 models will be Pareto-optimal — the rest are dominated and can be eliminated immediately without any statistical testing. Second, for the Pareto-optimal models, run McNemar's test against the current production baseline. This controls Type I error: if you ran 50 McNemar tests without correction, you'd expect 2-3 false positives at α=0.05. But since Pareto filtering reduces the candidate set to ~5-10 models, the multiple testing problem is manageable. If you need strict control, apply Bonferroni correction: use α' = 0.05 / (number of Pareto-optimal models tested).

Section 8: Quick Setup and Tests

pip install torch>=2.1.0 scipy numpy

Test: Pareto analysis

import torch

# Paste the pareto_front_vectorized and find_knee_point functions above

scores = torch.tensor([
    [0.92, 0.33, 0.80],   # M1
    [0.88, 0.83, 0.70],   # M2
    [0.85, 1.00, 0.90],   # M3
    [0.88, 0.67, 0.70],   # M4 — dominated by M2
    [0.80, 0.50, 0.60],   # M5 — dominated by M3
])

pareto = pareto_front_vectorized(scores)
assert pareto.tolist() == [True, True, True, False, False], f"Got: {pareto.tolist()}"
print("Pareto-optimal models:", pareto.nonzero().flatten().tolist())  # [0, 1, 2]

knee = find_knee_point(scores[pareto])
print(f"Knee point: model index {pareto.nonzero().flatten()[knee].item()}")  # 2 (M3)

Test: McNemar's test

import torch

# Paste the mcnemar_test function above

torch.manual_seed(0)
N      = 1000
y_true = torch.randint(0, 5, (N,))
y_old  = (y_true + torch.randint(0, 3, (N,))) % 5   # ~70% accurate
y_new  = (y_true + torch.randint(0, 3, (N,))) % 5   # similar accuracy

result = mcnemar_test(y_true, y_old, y_new)
print(f"Contingency: a={result['a']}, b={result['b']}, c={result['c']}, d={result['d']}")
print(f"χ² = {result['chi2']:.3f}")
print(f"p  = {result['p_value']:.4f}")
print(f"Significant: {result['significant']}")
print(result['power_note'])

Test: Sample size planning

# Paste the sample_size_for_mcnemar function above

# Verify: to detect a 2% delta with 10% disagreement rate and 80% power
N = sample_size_for_mcnemar(delta_acc=0.02, expected_disagreement_rate=0.10)
print(f"Need {N} examples to detect 2% delta (10% disagreement rate, 80% power)")
# Expected: ~1960 examples

References

Foundational Papers

  1. McNemar, Q. (1947). "Note on the sampling error of the difference between correlated proportions or percentages." Psychometrika, 12(2), 153–157. — The original paper introducing the test.

  2. Dietterich, T. G. (1998). "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms." Neural Computation, 10(7), 1895–1923. — Essential comparison of McNemar's vs other tests for ML classifiers; shows McNemar's is best for paired evaluation.

  3. Demšar, J. (2006). "Statistical Comparisons of Classifiers over Multiple Data Sets." Journal of Machine Learning Research, 7, 1–30. — Definitive guide to comparing classifiers; covers Wilcoxon, Bonferroni corrections, multi-dataset comparison.

Multi-Objective Optimization

  1. Pareto, V. (1896). Cours d'économie politique. — Original formulation of Pareto efficiency.

  2. Deb, K., Pratap, A., Agarwal, S., & Meyarivan, T. (2002). "A fast and elitist multiobjective genetic algorithm: NSGA-II." IEEE Transactions on Evolutionary Computation, 6(2), 182–197. — NSGA-II: the standard Pareto-front algorithm in evolutionary computation; basis for many ML hyperparameter search frameworks.

  3. Loshchilov, I., & Hutter, F. (2016). "CMA-ES for Hyperparameter Optimization of Deep Neural Networks." ICLR Workshop. — Pareto analysis applied to DNN hyperparameter search.

Statistical Testing for ML

  1. Efron, B., & Hastie, T. (2016). Computer Age Statistical Inference. Cambridge University Press. — Chapter 3 covers hypothesis testing; Chapter 15 covers bootstrap methods for ML evaluation.

  2. Kohavi, R., & Provost, F. (1998). "Glossary of Terms." Machine Learning, 30, 271–274. — Definitive ML terminology for evaluation metrics.

  3. Yates, F. (1934). "Contingency tables involving small numbers and the χ² test." Journal of the Royal Statistical Society, 1(Suppl.), 217–235. — Original continuity correction paper.

Practical ML Evaluation

  1. Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020). "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList." ACL 2020. — How to build rigorous eval beyond single-number accuracy.

  2. Liang, P., et al. (2022). "Holistic Evaluation of Language Models (HELM)." arXiv:2211.09110. — Canonical multi-metric, multi-objective evaluation framework for LLMs.

  3. Hooker, S., et al. (2019). "A Benchmark for Interpretability Methods in Deep Neural Networks." NeurIPS 2019. — Statistical rigor in model comparison for interpretability research.

On-Device / Production Evaluation

  1. Qualcomm AI Hub Documentation — aihub.qualcomm.com — Model profiling, on-device benchmarking, latency/memory/power metrics. The practical reference for Qualcomm NPU evaluation.

  2. Reddi, V. J., et al. (2020). "MLPerf Inference Benchmark." arXiv:1911.02549. — Industry standard for multi-metric, hardware-aware model evaluation.

Python / PyTorch References

  1. SciPy documentation — scipy.stats.chi2 — CDF, PPF, and PDF for chi-squared distribution used in McNemar's p-value computation.

  2. PyTorch documentation — Broadcasting semantics — pytorch.org/docs/stable/notes/broadcasting — Explains the unsqueeze + broadcast trick used in the vectorized Pareto front.