m05 — The Evaluation Harness

A fully worked design. The system that decides whether a model ships. It has to produce numbers that are comparable across weeks, teams, and model versions — from a stack that is non-deterministic at every layer.

The number that reframes this design: a 500-item benchmark can only resolve accuracy differences larger than 6.7 percentage points. Most reported model improvements are smaller than that. Most eval harnesses are measuring noise and reporting it as progress, and being the person who says so — with the arithmetic — is the whole value of this round.

Table of Contents

• The Prompt
• 1. Requirements and Scope
• 2. Scale Numbers
• 3. API Surface
• 4. Data Model
• 5. High-Level Architecture
• 6. Deep Dive A: Reproducibility in a Non-Deterministic Stack
• 7. Deep Dive B: When Is a Difference Real?
• 8. Failure and Recovery
• 9. Bottlenecks and Evolution
• 10. Tradeoffs Explicitly Rejected
• The Hostile Critique
• The Revision
• References

The Prompt

"Design our model evaluation system. Every training run produces checkpoints, every checkpoint needs to be evaluated, and we need to be able to say confidently whether model B is better than model A."

"Say confidently" is the requirement, and it is a statistics requirement, not an infrastructure one. The infrastructure is not hard: running 500,000 prompts through a model is a batch job that costs $123 (§2). What is hard is that the answer must mean something.

Three things make it not mean anything, and naming them in the first two minutes is the strongest possible opening:

1. The stack is non-deterministic. The same prompt against the same weights gives different logits depending on batch composition. Run the eval twice, get two numbers.
2. The measurement has error bars nobody computes. 82.4 vs 82.1 on a 500-item benchmark is noise, and it will be reported as an improvement.
3. The benchmark may be in the training data. A contaminated score is not a low-quality measurement, it is a measurement of the wrong thing.

So the design's job is to produce numbers with known error bars, from a reproducible procedure, on uncontaminated data. Everything else is a batch scheduler.

1. Requirements and Scope

Clarifying questions asked

"What decisions does this drive?" The framing question. Assumed three, with very different requirements:

• Training telemetry — is the run healthy? Cheap, frequent, noisy is fine.
• Model selection — which checkpoint ships? Expensive, rare, must be precise.
• Release reporting — the public number. Must be reproducible by a third party.

Three consumers, three different cost/precision points, one system. A design that treats them identically is either too expensive for the first or too imprecise for the third.

"Generative or likelihood-based scoring?" Both, and they are different systems. Multiple-choice benchmarks can be scored from logprobs (no generation, ~50× cheaper, fully deterministic given fixed batching). Open-ended tasks need generation and a grader. Assumed: ~70% logprob-scored, ~30% generative.

"Who grades the generative tasks?" Assumed: exact-match/regex where possible, a model grader otherwise, with human spot-checks. The model grader is itself a model that changes, which makes it a versioned dependency of every number — this is the subtlest reproducibility hazard in the design and it must be stated, not discovered.

"How fast after a checkpoint lands?" Assumed: telemetry evals within 10 minutes, or they do not affect the run they are measuring and become archaeology.

"Can we trust benchmarks we didn't build?" Assumed no, not fully — public benchmarks are of unknown contamination status and often have label errors. Held-out internal sets are the decision-grade instrument; public benchmarks are for comparability with the outside world. Different purposes, and conflating them is how teams optimize for a leaderboard.

Functional

1. Register benchmarks with a pinned version: items, prompt template, scoring function.
2. Run a suite against a checkpoint; emit per-item results, not just aggregates.
3. Compare two models with a stated confidence interval.
4. Detect and report contamination against the training corpus.
5. Reproduce any historical result exactly from its recorded configuration.

Non-functional

Explicitly out of scope

• Human preference evaluation / arena-style ranking. Different system, different latency, worth naming so nobody assumes it is covered.
• Red-teaming and adversarial safety evaluation — related, but the workflow is exploratory rather than batch, and merging them produces a system that serves neither.
• Training-time loss curves. That is the trainer's telemetry, not evaluation.
• The inference engine — m01.

2. Scale Numbers

The suite. ~50 benchmarks, ~500,000 items total. Prompt p50 ~500 tokens; generative outputs ~200 tokens.

Compute for one full run:

generative portion:  500k items x 200 out-tok           = 100M output tokens
per replica (4xH100, 70B):                                ~2,259 out-tok/s
1 replica:  12.3 hours     |     10 replicas:  1.23 hours
cost: 10 replicas x 4 GPU x $2.50/hr x 1.23 h            = $123

$123 for a full evaluation of a model that cost millions to train. That ratio is the design's licence to be thorough — but it inverts immediately when you eval every checkpoint:

30-day run, checkpoint every 30 min = 1,440 checkpoints
1,440 x $123                                             = $177,000

So the tiering is forced by arithmetic, not preference:

Logprob scoring is the reason telemetry is affordable. No generation, one forward pass per item: ~50× cheaper than generative scoring and deterministic given a fixed batch composition. Using it for the frequent tier and generation for the rare tier is the single decision that makes the cost work.

And now the number that matters more than any cost figure. For a benchmark of n items at accuracy p ≈ 0.82, the standard error of the accuracy is sqrt(p(1-p)/n):

Most public benchmarks have 500–2,000 items. At 500 items, two models must differ by nearly 7 points before an unpaired comparison can distinguish them. Every reported "+1.2 on MMLU" at that scale is within noise.

The requirement "resolve 1 pp" therefore demands ~10,000 items per decision-grade benchmark — which most benchmarks do not have, and which is a fact about the instruments, not about the harness. §7 shows how to recover most of that power without more items.

3. API Surface

POST /benchmarks             {name, version, items_uri, template, scorer_ref} -> {bench_id}
POST /suites                 {name, version, bench_ids[], tier}               -> {suite_id}

POST /runs                   {checkpoint_uri, suite_id, seed, engine_config}  -> {run_id}
GET  /runs/{id}              -> {status, per_bench: {score, n, ci95}, config_digest}
GET  /runs/{id}/items        -> per-item: prompt, output, score, logprobs  (the ground truth)

POST /compare                {run_a, run_b}
  -> { per_bench: [ {name, delta, ci95, p_value, n_discordant, verdict} ],
       verdict: "A better" | "B better" | "INDISTINGUISHABLE" }

GET  /contamination?bench=..&corpus=..  -> {overlap_rate, contaminated_item_ids[]}

Four decisions:

/runs/{id}/items is not optional and not a debug endpoint. Per-item results are what make paired comparison possible (§7), what make a regression diagnosable, and what let a human check whether the grader is sane. An eval system that stores only aggregates has thrown away the data that makes its aggregates trustworthy — and it is the most common shortcut in real harnesses.

/compare returns INDISTINGUISHABLE as a first-class verdict. Not a low confidence score, not a small delta — an explicit refusal to call it. The system must be able to say "this difference is noise", because if it cannot, someone will read a delta and act on it. Making it a verdict rather than a caveat is a design decision about what the system asserts.

benchmark carries a version, and the template is part of it. The same items with a different prompt template score differently — often by more than the model differences being measured. The template is part of the instrument, so a template change is a new benchmark version and the old numbers are not comparable. This is the mistake that silently invalidates a quarter of historical results.

engine_config is recorded on the run, not defaulted. Batch size, dtype, TP degree, attention kernel — all of them can change the numbers (§6). Recording them is what makes reproduction possible.

4. Data Model

benchmark  (bench_id, name, version, items_digest, template_digest,
            scorer_ref, n_items, license, created_at)
item       (bench_id, item_id, prompt_fields{}, target, metadata{})
suite      (suite_id, name, version, bench_ids[], tier)
run        (run_id, checkpoint_digest, suite_id, seed, engine_config{},
            harness_version, grader_model_ref, started_at, config_digest)
result     (run_id, bench_id, item_id, output, score, logprobs[], latency_ms)
aggregate  (run_id, bench_id, score, n, se, ci_lo, ci_hi)
contam     (bench_id, corpus_id, item_id, overlap_ngrams, checked_at)

config_digest is a hash of everything that could change the numbers — checkpoint, benchmark versions, template, engine config, harness version, grader model. Two runs with the same config_digest must produce identical result rows. That is the testable invariant the whole design is built to satisfy, and stating it as an invariant (rather than a goal) is what makes it enforceable: §8 runs it as a continuous check.

result is per-item and retained. 500k items × ~1 KB × a few hundred runs is a few TB. Cheap, and it is the only way to do paired comparison (§7), diagnose regressions, or audit a published number.

grader_model_ref on the run. The model grader is a model; when it is upgraded, every generative score shifts by an unknown amount. Recording it makes the discontinuity visible in the data rather than mysterious in a chart.

contam is per-item, not per-benchmark. Because the useful operation is "recompute the score excluding contaminated items", which needs item granularity. A per-benchmark contamination rate tells you there is a problem and not what to do about it.

5. High-Level Architecture

  checkpoint lands
        │
  ┌─────▼───────────────────────────────────────────────┐
  │ TRIGGER: tier by checkpoint index (telemetry/select) │
  └─────┬───────────────────────────────────────────────┘
  ┌─────▼───────────────────────────────────────────────┐
  │ PLANNER: expand suite -> items; shard by cost;       │
  │          FIX BATCH COMPOSITION (deep dive A)         │
  └─────┬───────────────────────────────────────────────┘
        │
  ┌─────▼──────────────┐        ┌──────────────────────┐
  │ LOGPROB WORKERS    │        │ GENERATION WORKERS   │
  │ 1 fwd pass/item    │        │ decode, greedy       │
  │ deterministic      │        │ + model grader       │
  └─────┬──────────────┘        └──────────┬───────────┘
        └──────────┬───────────────────────┘
  ┌────────────────▼────────────────────────────────────┐
  │ SCORING: per-item scores -> results store            │
  └────────────────┬────────────────────────────────────┘
  ┌────────────────▼────────────────────────────────────┐
  │ AGGREGATION: score, SE, CI, contamination-excluded   │
  │              variant of every number                 │
  └────────────────┬────────────────────────────────────┘
  ┌────────────────▼────────────────────────────────────┐
  │ COMPARISON: paired test vs baseline -> verdict       │
  └─────────────────────────────────────────────────────┘

  SIDE: contamination service (n-gram index from m04) — queried, not inline

Five decisions:

1. Evaluation runs on a dedicated engine configuration, not the production serving fleet. Production optimizes throughput with dynamic batching, which makes batch composition vary, which makes results non-deterministic (§6). The eval fleet trades throughput for determinism — a deliberate inversion of every other design in this track, and worth saying so explicitly.

2. Logprob and generation are separate worker pools. Different cost profiles, different determinism properties, different scaling. Merging them means the cheap path inherits the expensive path's problems.

3. Every aggregate ships with n, se, and ci95. Not available on request — in the same record as the score. A number without its error bar is a number that will be over-interpreted, and the defence is to make them inseparable.

4. Contamination is a side service, queried at aggregation time. So every score has a score_decontaminated companion computed from the same per-item results. Both are reported, always — reporting only the clean number hides a problem, and reporting only the raw one publishes a wrong one.

5. The planner fixes batch composition. Deep dive A. It is one line in the diagram and it is the difference between an eval system and a random number generator.

6. Deep Dive A: Reproducibility in a Non-Deterministic Stack

Every layer that breaks determinism

The one people miss is batch composition, and it is the most important. It is not a bug and it cannot be patched away: floating-point addition is not associative, and a matmul's reduction order depends on the shapes it is given. The same prompt in a batch of 4 and a batch of 60 produces logits that differ in the last few bits.

Usually irrelevant. Not irrelevant when the top two multiple-choice options are within 1e-6 — then the last bits decide the answer, the answer flips, and the benchmark score moves. On a 500-item benchmark, a handful of flipped items is several tenths of a point: the same size as the differences people report as progress.

The fix: fix everything that can be fixed, and pin the rest

EvalEngineConfig = {
    "batch_size":       "FIXED — 32, padded; never dynamic",
    "item_order":       "FIXED — sorted by item_id, not by length",
    "tp_degree":        "PINNED — recorded in the run",
    "attn_impl":        "PINNED — one kernel, recorded",
    "dtype":            "PINNED — bf16 weights, fp32 logit accumulation",
    "sampling":         "greedy (temperature=0) for scored generation",
    "cublas_workspace": "CUBLAS_WORKSPACE_CONFIG=:4096:8 (deterministic GEMM)",
    "seed":             "recorded; only affects sampling-based evals",
}

item_order sorted by item_id, deliberately not by length. Length-sorting is the obvious throughput optimization — it minimizes padding — and it makes batch composition depend on the item set, so adding one item to a benchmark reshuffles every batch and changes results for items that did not change. A benchmark version bump would silently move every score. Sorting by ID costs padding waste and buys stability. That is the trade this system exists to make, and it is the clearest single example of eval infrastructure being the opposite of serving infrastructure.

Fixed batch size with padding, so the last partial batch is padded rather than being a different shape. Costs a few percent of throughput; makes the run reproducible.

fp32 logit accumulation narrows the tie region substantially. The comparison that decides a multiple-choice answer happens in fp32 even though the weights are bf16 — cheap, and it removes a large fraction of the flip cases.

What cannot be fixed, and what to do about it

Cross-hardware determinism is not achievable. A100 and H100 select different kernels; results will differ in the last bits. Do not promise it. Pin the hardware class per suite and record it. If a comparison spans hardware, say so and treat it as a lower-confidence comparison — which the paired test in §7 will show directly, because the discordant-pair count will rise.

The honest position, which is the answer to "so is it reproducible?":

"Bit-identical on the same hardware, same engine config, same batch plan — and we assert it in CI. Across hardware generations, no; nobody can. What we do instead is measure the size of that variation and require model differences to exceed it."

That last clause is the real answer, and it moves the problem from determinism to statistics, which is where it belongs and where §7 handles it.

The determinism test, run continuously

Nightly: rerun a fixed suite against a fixed checkpoint.
  Assert: per-item results are IDENTICAL to the recorded baseline.
  On mismatch: the harness is broken, or a dependency changed silently.
                Block all evals until explained.

This catches the silent-dependency-change class of bug — a CUDA update, a kernel library bump, a framework upgrade — before it contaminates a quarter of results. It is cheap (one small suite), and it is the single highest-value test in the system. Without it, you discover the change months later as an unexplained discontinuity in a chart, and you cannot tell which side of it is right.

7. Deep Dive B: When Is a Difference Real?

The problem, in one number

From §2: a 500-item benchmark at 82% accuracy has SE = 1.72 pp. To distinguish two models with 95% confidence, the difference must exceed roughly 6.7 pp.

Model B scores 83.1, model A scored 82.4. Is B better? No — the data cannot say. And every eval system that reports "+0.7" without an interval will be read as saying yes.

Paired comparison: the free 2.4× improvement

The unpaired calculation throws away the most useful fact available: both models were evaluated on the same items. Most of the variance is item difficulty, which is identical for both models and cancels in a paired test.

Only discordant items — where one model is right and the other wrong — carry information. This is McNemar's test:

              B correct   B wrong
A correct        a          b        <- b: A right, B wrong
A wrong          c          d        <- c: A wrong, B right

Only b and c matter. Under H0, b ~ Binomial(b + c, 0.5).

A 2.4–3.4× improvement in resolving power, for free, from the same data. No extra items, no extra compute — just a test that uses the per-item results the system already stores.

This is why /runs/{id}/items is not optional (§3). A harness that stores only aggregates cannot do this and is stuck at the unpaired numbers. That connection — a schema decision determining statistical power — is exactly the kind of cross-layer reasoning this round rewards.

The multiple-comparisons problem, which is worse than it looks

50 benchmarks per suite. At α = 0.05, you expect 2.5 false positives per comparison run even if the models are identical.

And the failure mode is not statistical, it is human: someone scans 50 numbers, finds the three that moved, and builds a story. The story is about noise, and it is persuasive because the numbers are real.

Fixes, and the third is the one that works:

1. Report the family-wise picture. Benjamini–Hochberg across the suite, and report how many benchmarks would be expected to move by chance. A comparison view that does not show the null expectation invites the story.
2. Pre-register the primary metric. One benchmark, or a defined composite, designated before the run as the decision metric. Everything else is exploratory and labelled as such.
3. Require a pre-registered direction for shipping decisions. "We ship if the composite improves by ≥1 pp with p < 0.01, regardless of what else moved." A decision rule written before seeing the data is the only real defence against post-hoc storytelling.

Variance from the model itself

Even with perfect determinism, a retrained model with a different seed will score differently. Seed-to-seed variance on a benchmark is often 0.5–1.5 pp — comparable to the effects being measured.

Consequence, stated plainly: comparing one checkpoint of A to one checkpoint of B measures checkpoint difference, not method difference. Claiming a method improvement from a single pair of runs is a claim the data does not support.

What the harness does about it: it cannot fix the experiment, but it can make the ambiguity visible.

/compare returns:
  delta:            +0.7 pp
  ci95:             [-1.3, +2.7]
  p_value:          0.42
  n_discordant:     47
  seed_variance:    ±0.9 pp   (from the archive of same-config runs)
  verdict:          INDISTINGUISHABLE
  note: "Delta is within seed-to-seed variance for this benchmark.
         3+ seeds per arm needed to resolve a 0.7 pp effect."

seed_variance comes from the run archive — the system has been storing per-item results all along, so it knows empirically how much identical configurations vary. That number is the honest noise floor of the entire measurement apparatus, and almost no harness reports it. Reporting it, and refusing to call differences below it, is what "say confidently" actually requires.

8. Failure and Recovery

The last row deserves emphasis because it is the most common real failure: a template change or a parsing bug makes the scorer fail to extract answers, and the benchmark reports ~0. That looks like a catastrophic model regression and triggers a fire drill.

The guard is a per-benchmark plausible range recorded with the benchmark, plus an automatic sample of per-item outputs on any excursion. The system should show you five raw outputs before you page anyone. Nine times out of ten the answer is visible immediately — the model wrote "Answer: B" and the regex expected "(B)".

On "block all evals" for a determinism failure — it is deliberately aggressive and worth defending. The alternative is continuing to produce numbers whose comparability is unknown, which contaminates the archive that seed_variance and every historical comparison depend on. A halted eval system costs a day. A silently non-comparable archive costs the ability to make any historical comparison at all, and you will not know when it started.

9. Bottlenecks and Evolution

Now: the bottleneck is statistical power, not compute (§2: a full run is $123). More GPUs do not make the numbers more trustworthy; more items and better tests do.

Interventions in order:

1. Grow decision-grade benchmarks to ~10,000 items. The only way to resolve 1 pp unpaired (§2). Expensive in human effort, not compute — which is why it does not happen and why it is the highest-leverage item.
2. Paired testing everywhere (§7). Free 2.4–3.4× power gain from data already stored.
3. Multi-seed evaluation for release decisions. 3 seeds per arm turns "checkpoint difference" into "method difference". 3× cost on the rare tier = ~$400. Cheap relative to shipping the wrong model.
4. Item-response-theory weighting. Items vary enormously in discriminative power; many are too easy or mislabelled and contribute noise. Weighting by discrimination raises effective power at fixed item count. Standard in psychometrics, rare in ML, and a genuinely differentiating thing to bring up.
5. Continuous contamination monitoring. New benchmarks appear after the corpus is built; the n-gram index from m04 R4 makes retroactive checks possible, and every published number should carry its contamination rate.
6. Human evaluation for the tasks automatic grading cannot score. Different system entirely (§1) — but the harness should reserve a hook so that human labels land in the same result table and participate in the same paired tests.

10. Tradeoffs Explicitly Rejected

Rejected: evaluating on the production serving fleet. Dynamic batching makes batch composition vary, which makes results non-reproducible (§6). Dedicated fixed-batch eval workers.

Rejected: length-sorted batching in eval. The obvious throughput win; it makes results depend on the item set, so adding one item changes every score. Sort by ID.

Rejected: storing only aggregate scores. Kills paired comparison (§7 — a 2.4× power loss), regression diagnosis, and auditability. Per-item, retained.

Rejected: reporting a delta without an interval. The single most damaging thing this system could do, because the number will be acted on.

Rejected: temperature > 0 for scored generation. Adds sampling variance on top of everything else for no benefit to the measurement. Greedy, with sampling reserved for evals specifically about diversity.

Rejected: a single "overall score". Compresses 50 benchmarks into one number whose movement cannot be attributed, and invites optimizing the aggregate. A pre-registered composite for the decision rule, with all components always visible.

Rejected: full-suite evaluation on every checkpoint. $177k per run (§2) for numbers that are mostly noise at that frequency. Tiered.

Rejected: trusting public benchmark scores for internal decisions. Unknown contamination, label errors, and heavy optimization pressure. Public benchmarks for external comparability, held-out internal sets for decisions.

Rejected: promising cross-hardware bit-determinism. Not achievable. Pin the hardware class, record it, and handle the residual statistically.

The Hostile Critique

C1. "You fix batch size at 32 with padding for determinism. Multiple-choice logprob scoring depends on the exact token positions, and padding changes attention masks. Are you certain a padded batch gives identical logits to an unpadded one? Have you tested that, or assumed it?"

C2. "seed_variance comes 'from the archive of same-config runs'. Same config means same checkpoint — that's your determinism test, which by construction gives zero variance. Seed variance requires retraining, which you do a handful of times a year. Where does the number actually come from?"

C3. "The determinism test blocks all evals on failure. A CUDA driver update rolls out across the fleet over six hours. Half your workers are on the new driver. Your nightly test passes or fails depending on which worker it lands on. Then what?"

C4. "You require paired comparison for power. Paired requires both models to have been evaluated on the same benchmark version. Benchmarks get versioned when templates change, which you said is often. So how many of your historical models can actually be compared to today's candidate?"

C5. "Model-graded generative evals: you record grader_model_ref and 'establish an offset' when it changes. An offset is a single number for a whole benchmark. If the new grader is stricter about one category of answer, the offset is wrong for every model that answers differently in that category. What have you actually corrected?"

C6. "Telemetry tier: 2,000 items, every checkpoint, to tell whether the run is healthy. From §2, 2,000 items resolves about 3.4 pp. Training progress between adjacent checkpoints is far smaller than that. What exactly is the telemetry tier detecting?"

The Revision

R1 — Padding must be validated, not assumed, and the test belongs in CI (answers C1)

The critique is right to challenge the assumption, and the honest answer is that padding is only safe if the implementation is correct, and that is testable rather than assumable.

The mechanism: with a correct attention mask, padded positions contribute zero to attention weights and cannot affect real positions' outputs — mathematically. In practice, softmax over -inf-masked positions, kernels that ignore the mask on some paths, and left- vs right-padding interacting with position IDs all break it. Left padding with absolute position IDs is a known correctness bug, not a numerical subtlety.

Change: an explicit invariant test, run in CI:

def test_padding_invariance():
    for item in SENTINEL_ITEMS:                       # ~200 items, all shapes
        alone  = model.logprobs([item])               # batch of 1, no padding
        padded = model.logprobs([item] + FILLERS)[0]  # batch of 32, item padded
        assert torch.equal(alone, padded), f"padding changes logits for {item.id}"

Bitwise equality, not allclose. If padding perturbs the last bits, the multiple-choice tie-breaking is affected and the eval is not reproducible — allclose would pass and hide exactly the failure that matters.

And what to do if it fails, which is the more useful half: it likely will on some kernels. Then:

• Right-pad with explicit position_ids so real tokens keep their positions.
• Bucket by length into a small fixed set of bucket sizes (e.g. 128/512/2048) — the bucket boundary is a function of the item, not of the item set, so it preserves the §6 property that adding an item does not change other items' batches. This recovers most of the padding efficiency without reintroducing set-dependence, and it is a better design than the original fixed-32 rule.

Cost: the invariance test is a real gate that will block on kernel upgrades. That is the point.

R2 — Two different variances, measured two different ways (answers C2)

The critique catches a genuine conflation. There are three distinct sources of variance and the design named one and measured a different one:

Change: report the first two directly, and treat the third as a prior, clearly labelled.

# Engine variance: cheap, measured continuously.
engine_var = rerun_with_perturbed_batch_plan(checkpoint, suite)     # nightly

# Training-seed variance: rare, from a deliberate program.
# Twice a year, train 3 small models with identical config, different seeds.
# Their spread, per benchmark, is the SEED VARIANCE PRIOR.
seed_var_prior = archive.seed_study(benchmark, model_scale)

And the report says which is which, because the distinction changes what the reader should do:

delta: +0.7 pp   ci95: [-1.3, +2.7]
engine_variance:     ±0.05 pp   (measured, this checkpoint, nightly)
seed_variance_prior: ±0.9  pp   (from the 2026-03 seed study at 7B; EXTRAPOLATED to 70B)
verdict: INDISTINGUISHABLE

The word EXTRAPOLATED is the important one. A seed study at 7B is affordable; at 70B it is not, so the prior is transferred across scale — which is an assumption, and labelling it as such is the difference between an honest instrument and a confident wrong one.

Cost: the seed study is a real training expenditure (3 small runs, twice a year) charged to the eval budget rather than to research. Justified by what it buys: without it, no comparison between two independently-trained models has a stated noise floor, which means every such comparison is an opinion.

R3 — The determinism test must be per-environment, and environment is part of identity (answers C3)

The critique identifies a real and common operational failure: a heterogeneous fleet during a rollout makes a global determinism assertion both flaky and meaningless.

Change: determinism is asserted within an environment fingerprint, and the fingerprint is part of the run's identity.

env_fingerprint = H(
    gpu_model, driver_version, cuda_version, torch_version,
    flash_attn_version, harness_version, kernel_lib_digest,
)

Then:

Baselines are stored PER (suite, checkpoint, env_fingerprint).
A new fingerprint appears        -> not a failure; a NEW BASELINE to establish.
Same fingerprint, different result -> a REAL failure. Block.
Comparing runs across fingerprints -> allowed, but flagged, and the
    cross-fingerprint delta on a fixed checkpoint is MEASURED and reported
    as an additional error term.

The last line is the useful part. Instead of forbidding cross-environment comparison, the system measures what the environment change is worth: rerun the fixed baseline checkpoint under both fingerprints and the difference is a directly observed environment effect. If it is 0.05 pp, comparisons are fine; if it is 0.8 pp, every cross-environment comparison inherits that error bar.

This turns an operational nuisance into a measured quantity, which is the same move as §7 — when you cannot eliminate a source of variation, measure it and put it in the interval.

Cost: baselines multiply by the number of live environments, and a rolling upgrade means carrying two for a while. Small: a baseline is one small suite. And scheduling eval jobs with node-selector affinity to a single fingerprint per run is required, or a single run straddles two environments and is internally inconsistent — a bug the critique implies and that the fingerprint alone would not have caught.

R4 — Benchmark versioning needs a compatibility relation, not just a version (answers C4)

The critique lands on a real consequence I had not followed through: if every template change is a new benchmark version and old results are "not comparable", then the archive fragments and paired comparison — the design's main source of statistical power — becomes unavailable exactly when it is most needed.

Change: versions carry a compatibility relation, and the item set is versioned separately from the template.

benchmark_version = (items_digest, template_digest, scorer_digest)

PAIRED-COMPARABLE  iff items_digest matches           <- same items = pairing works
SCORE-COMPARABLE   iff all three match                <- same instrument = scores comparable

Two runs with the same items but different templates can be paired per item — the pairing is over items, and item difficulty still cancels — but the absolute scores are not comparable. That distinction recovers most of the archive, because template changes are far more common than item-set changes.

And a bridging mechanism for the item-set case: when items are added, the intersection is still paired-comparable. So:

compare(run_old, run_new):
    common = items(old) & items(new)
    report BOTH:
      - paired test on `common`      (n = |common|, high power, valid)
      - full-set scores              (not directly comparable, labelled)

Cost: more bookkeeping and a compare endpoint that returns two numbers with a careful explanation. Worth it — the alternative is throwing away history every time a template is fixed, which is a strong disincentive to fixing templates, which is how bad templates survive.

R5 — Grader changes require re-scoring, not an offset (answers C5)

The critique is correct and the offset idea was wrong. An offset assumes the grader change is a uniform shift; the critique's example — a grader stricter about one answer category — is a model-dependent shift, and a single scalar cannot correct it. Applying an offset would make the numbers look comparable while being differently wrong for each model.

Change: a grader change triggers re-scoring, not adjusting.

Generative outputs are STORED (§4, `result.output`).
Re-scoring = run the new grader over stored outputs. NO model inference needed.

grader upgrade:
  1. re-score the last N runs' stored outputs with the new grader
  2. every comparison uses a SINGLE grader version across both arms
  3. old scores retained, labelled with their grader version, never mixed

This is why storing raw outputs matters — it makes the grader a post-processing step that can be replayed at will. Re-scoring 500k stored outputs with a grader model costs one eval's worth of inference, ~$123, and it is exact rather than approximate.

And the grader needs its own evaluation, which the critique implies:

GRADER AGREEMENT SET: ~1,000 outputs with human labels, held fixed.
Every grader version is scored on it: agreement rate, and PER-CATEGORY agreement.
A grader that loses agreement in any category is not adopted, however
good its aggregate number is.

Per-category is the point — the critique's failure mode is invisible in an aggregate agreement rate and obvious in a per-category one. The grader is an instrument and needs calibration like any other, and a harness that versions its grader without evaluating it has an unmeasured dependency at the centre of 30% of its numbers.

R6 — The telemetry tier measures liveness, not quality, and should say so (answers C6)

The critique is right and the tier was mislabelled, which is worse than mis-sized: it invites exactly the over-interpretation §7 exists to prevent. At 2,000 items resolving ~3.4 pp, adjacent checkpoints are indistinguishable and always will be.

Change: rename it and re-scope it to what it can actually detect.

HEALTH tier (was "telemetry") — 2,000 items, every checkpoint.
Detects, all of which are LARGE effects:
  * divergence / loss spike aftermath   (score falls 10+ pp)   <- resolvable
  * tokenizer or data pipeline bug      (score -> chance)      <- resolvable
  * catastrophic forgetting on a domain (score falls 5-15 pp)  <- resolvable
Does NOT detect:
  * "is the model getting better"                              <- NOT resolvable. Do not chart it
    as a progress curve; the trend is noise at this n.

And a better instrument for the actual question, which the critique's framing points at: per-item loss on a held-out set is far more sensitive than accuracy, because it uses the full probability rather than a thresholded decision. Loss on 2,000 held-out items resolves changes an order of magnitude smaller than accuracy on the same items — it does not throw away the model's confidence, which is where the signal is.

HEALTH tier = held-out LOSS (sensitive, continuous, cheap: one forward pass)
              + accuracy on 2,000 items (insensitive, but catches catastrophic breakage)

The lesson, and it generalizes past this design: match the metric's sensitivity to the effect size you need to detect. Accuracy thresholds a continuous quantity into a binary one and throws away most of the information; that is affordable for a decision-grade comparison with 10,000 items and wasteful for a 2,000-item health check. The critique found a tier sized for the wrong metric, and the fix was to change the metric rather than the size.

References

• m04-training-data-pipeline.md — the corpus and the contamination index this queries
• m01-llm-api-platform.md — why the serving fleet's dynamic batching is unusable here
• ../WARMUP.md — the inference mechanics behind logprob vs generative scoring cost
• ../../systems-design/designs/README.md#what-the-critiques-found — the defect taxonomy; "arithmetic never done" is what §2's CI table prevents
• Gao, L. et al. A Framework for Few-Shot Language Model Evaluation (lm-evaluation-harness) — prompt-template sensitivity in practice
• Liang, P. et al. Holistic Evaluation of Language Models (HELM). — multi-metric reporting and the case against a single score
• Dietterich, T. Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. 1998 — McNemar's test for exactly this comparison
• Benjamini, Y. & Hochberg, Y. Controlling the False Discovery Rate. 1995 — the multiple-comparisons correction in §7
• Zheng, L. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 — model graders and their biases
• PyTorch docs, Reproducibility — CUBLAS_WORKSPACE_CONFIG, deterministic algorithms, and their limits