Model-Selection Eval Harness

Phase 12 · Document 08 · Evaluation Prev: 07 — Safety and Policy Evals · Up: Phase 12 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

This is the capstone of Phase 12 and the payoff of the whole evaluation discipline: a reusable harness that runs candidates (models / prompts / routes) over your golden set (01), scores them on all axes — quality (02–05), latency, cost (06), safety (07) — and produces a single, defensible decision: which to deploy, and a regression gate for every future change. It operationalizes the Phase 5 model-selection framework and the cost-quality-latency trade-off into running code. Without it, "which model?" is an argument; with it, it's a measured answer you can show stakeholders and re-run when models change. The harness is the moat-in-action — the compounding asset that lets you adopt new models in hours, not weeks.

2. Core Concept

Plain-English primer: run candidates, score all axes, decide

A model-selection harness is a program that, for each candidate (a model, prompt, or routing config):

1. Runs it over the golden set (01),
2. Scores each output on the relevant quality metric(s) (programmatic / judge — 02–05) and captures latency + cost (06),
3. Gates on safety (07) — disqualify failures,
4. Combines the surviving axes into a weighted score, and
5. Reports a ranked comparison → a deployment decision + a re-runnable gate.

It's the synthesis of everything in this phase (and Phase 5): golden set + scorers + metric-per-task + the weighted trade-off, in one tool.

The weighted scoring formula (the decision math)

Quality/latency/cost/reliability are weighted trade-offs (Phase 5.09); safety is a gate (07):

if not passes_safety(candidate):  DISQUALIFY            # safety = hard gate, BEFORE scoring [07]
score = w_q·q + w_l·l + w_c·c + w_r·r                   # weighted, normalized 0–1, weights sum to 1
   q = quality (your metric, normalized)   [02–05]
   l = latency score (1 − normalized latency, p95)   [06]
   c = cost score (1 − normalized cost per RESOLVED TASK)   [06]
   r = reliability (1 − error rate)

• Normalize each axis to 0–1 (higher = better) so they're comparable, then weight.
• Weights are a product decision, set before scoring — autocomplete weights latency; legal analysis weights quality; batch weights cost (Phase 5.09). Setting weights after seeing scores = rationalizing a favorite.
• Safety is not in the weighted sum — it's a pass/fail filter applied first (07).

def evaluate(candidate, golden, weights):
    rows = [run_and_score(candidate, ex) for ex in golden]      # quality + latency + cost + safety per item
    if not all(r.safe for r in rows): return Disqualified(candidate)   # safety gate [07]
    q = mean(r.quality for r in rows)                            # your metric [02–05]
    l = 1 - normalize(p95(r.latency for r in rows))              # [06]
    c = 1 - normalize(cost_per_resolved_task(rows))              # NOT $/token [06, 5.09]
    r_ = 1 - error_rate(rows)
    return weights["q"]*q + weights["l"]*l + weights["c"]*c + weights["r"]*r_

Build it once, reuse forever (the moat)

The harness is reusable infrastructure: golden set + scorers + the runner. Once built, evaluating a new model (or prompt, or route) is re-running it — turning "should we adopt model X?" from a multi-week investigation into a few hours (01: the compounding moat). This is why mature teams adopt new models faster: their harness already encodes "good for our task."

It's also the regression gate (CI)

The same harness is your offline regression gate (00): run it in CI on every model/prompt/routing change; block the merge if quality, safety, or the weighted score regresses vs the baseline. This is how you safely answer "is the cheaper routed model good enough?" (Phase 11.06) and how RAG/agent/code changes get gated (03–05). Build the harness once; it serves both selection and regression-gating.

Beyond a single model: routing simulation

The harness extends to evaluate a routing policy, not just single models (Phase 5.09, Phase 8.05): label golden items easy/hard, simulate routing (easy→cheap, hard→premium), and compute the blended quality/latency/cost — often beating any single model. The harness should be able to score a system (router + caching), not only a model.

Online closes the loop

Offline selection isn't the end (00): the chosen candidate is monitored online (quality drift, latency, cost — Phase 7.08), and production failures feed back into the golden set (01), so the harness gets sharper over time. Selection → deploy → monitor → feed back → re-select.

3. Mental Model

   THE CAPSTONE: a reusable HARNESS over the GOLDEN SET [01] scoring candidates on ALL axes → ONE decision + a GATE

   per candidate (model/prompt/route):  run over golden set →
     SAFETY GATE [07] (pass/fail; DISQUALIFY failures — BEFORE scoring)  →
     WEIGHTED SCORE = w_q·quality[02–05] + w_l·latency[06] + w_c·(cost per RESOLVED TASK)[06] + w_r·reliability
        normalize axes 0–1; WEIGHTS = product decision set BEFORE scoring [5.09]
   → ranked comparison → DEPLOY decision

   BUILD ONCE, REUSE: new model = RE-RUN (hours not weeks) → the compounding MOAT [01]
   SAME harness = CI REGRESSION GATE (block on quality/safety/score drop) [00, 11.06]
   extends to ROUTING simulation (system, not just model) [5.09/8.05] ; ONLINE monitor + feed failures back [7.08]

Mnemonic: the harness runs candidates over the golden set, gates on safety, then weighted-scores quality + latency + cost-per-resolved-task + reliability → one defensible decision. Build it once: it's both your selection tool and your CI regression gate, and it compounds (new model = re-run).

4. Hitchhiker's Guide

What to look for first: does the harness gate safety before weighted scoring, use cost per resolved task (not $/token), and set weights before seeing scores? Those three keep the decision honest.

What to ignore at first: elaborate routing simulation. Start by scoring 2–3 single models on quality + latency + cost + safety; add routing/online later.

What misleads beginners:

• Folding safety into the weighted sum. Safety is a gate — a leak isn't offset by accuracy (07).
• Weighting by $/token. Use cost per resolved task (06, Phase 5.09).
• Setting weights after scoring. That rationalizes a favorite — set weights from product priorities first (Phase 5.09).
• Building it as a one-off. The value is reuse — make it infrastructure (golden set + scorers + runner) so it's the regression gate too.
• Quality-only ranking. Capture latency + cost + safety or you'll pick an unshippable winner.

How experts reason: they build a reusable harness (golden set + scorers + runner), gate safety first, weighted-score quality/latency/cost-per-resolved-task/reliability with pre-set weights, rank candidates, and wire it into CI as the regression gate. They extend it to routing simulation and monitor online, feeding failures back (01). They treat it as the compounding asset that makes model adoption fast.

What matters in production: the harness being trusted (golden set representative [01], judge calibrated [02], safety gated [07]), runnable in CI, covering all axes, and extensible to routing — plus online monitoring closing the loop (Phase 7.08).

How to debug/verify: if the harness picks a model that feels wrong → check weights (match product?), cost metric ($/token vs resolved-task?), safety gate (applied before scoring?), and golden-set representativeness (01). If "passes harness, fails prod" → the golden set isn't representative or you're not monitoring drift.

Questions to ask: does it score all axes (quality/latency/cost/safety)? safety gated before scoring? cost per resolved task? weights set first from product priorities? reusable + in CI? extends to routing? online feedback?

What silently gets expensive/unreliable: safety-in-the-weighted-sum (ships a leak), $/token weighting (wrong winner), post-hoc weights (rationalized favorite), one-off harness (no regression gate), and unrepresentative golden set (passes harness, fails prod).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• The harness runs candidates over the golden set, scores all axes, and produces one defensible decision + a regression gate — the synthesis of Phase 12 (and Phase 5).
• Weighted score = w_q·quality + w_l·latency + w_c·(cost per resolved task) + w_r·reliability, axes normalized 0–1 (Phase 5.09).
• Safety is a hard gate applied before scoring (disqualify failures) — not a weighted term (07).
• Cost uses per-resolved-task, not $/token (06); latency uses p95 under load.
• Set weights from product priorities BEFORE scoring — post-hoc weights rationalize a favorite (Phase 5.09).
• Build it once; reuse forever — new model = re-run (hours, not weeks); it's the compounding moat (01).
• The same harness is the CI regression gate (00) — selection and gating share infrastructure.
• Extend it to routing simulation (score a system, not just a model, Phase 5.09/Phase 8.05); monitor online + feed failures back (Phase 7.08).

9. Observations from Real Systems

• OpenAI Evals, Inspect AI, promptfoo, Braintrust, Langfuse are harness frameworks — the value is your golden set + scorers + weights, not the framework (01).
• Teams with a harness adopt new models in hours (re-run) while others take weeks — the compounding moat in action.
• The same harness gates CI — RAG/agent/code/prompt/routing changes all run through it (03–05, Phase 11.06).
• Routing simulation (easy→cheap, hard→premium) in the harness routinely beats any single model on blended cost/quality/latency (Phase 5.09).
• The classic mistake — folding safety into the weighted score, or weighting $/token — has shipped leaky or secretly-expensive models; gating safety + cost-per-resolved-task fixes it (07/06).

10. Common Misconceptions

11. Engineering Decision Framework

BUILD THE MODEL-SELECTION HARNESS (the capstone):
 1. INPUTS: golden set [01] + scorers (quality per task [02–05], latency+cost [06], safety [07]).
 2. PER CANDIDATE (model/prompt/route): run over golden set → capture quality, p95 latency, cost/resolved-task, reliability, safety.
 3. SAFETY GATE [07]: disqualify any candidate failing safety thresholds — BEFORE scoring (not weighted).
 4. WEIGHTED SCORE: normalize axes 0–1; weights set FIRST from product priorities [5.09]; rank survivors.
 5. DECIDE: pick the top, write a memo (Phase 3.07); EXTEND to routing simulation (score the system) [5.09/8.05].
 6. REUSE: wire the SAME harness into CI as the REGRESSION GATE (block on quality/safety/score drop) [00].
 7. ONLINE: monitor the chosen candidate (drift/latency/cost [7.08]); feed failures back into the golden set [01].

12. Hands-On Lab

Goal

Build a reusable model-selection harness: score 2–3 candidates on quality + latency + cost-per-resolved-task, gate safety, produce a weighted ranking + decision, and wire it as a CI regression gate.

Prerequisites

• The golden set (01) + scorers (02–07); 2–3 candidate models; pip install openai.

Steps

1. Runner: for each candidate, run over the golden set, capturing per-item quality (your metric, 02–05), latency (p95, 06), cost (per resolved task, 06), reliability (error rate), and safety (the 07 attack subset).
2. Safety gate: disqualify any candidate failing safety thresholds (attack-success/leakage > 0) before scoring (07).
3. Weights first: pick weights from your product's priorities (e.g., autocomplete: latency 0.4, quality 0.3, cost 0.2, reliability 0.1) before looking at scores (Phase 5.09).
4. Score + rank: normalize axes 0–1, compute the weighted score, output a ranked comparison table + a one-line decision.
5. Routing simulation (stretch): label items easy/hard; simulate easy→cheap, hard→premium; show the blended score can beat any single model (Phase 5.09).
6. CI gate: wire the harness to run on a change (new prompt/model) and fail if the weighted score or safety regresses vs a stored baseline (00).

Expected output

A reusable harness producing a ranked, safety-gated, weighted comparison + a deployment decision, a routing-simulation result, and a working CI regression gate — the operationalized model-selection discipline.

Debugging tips

• The "winner" feels wrong → check weights (match product?), cost metric (resolved-task?), safety gate (applied first?), golden-set representativeness (01).
• Passes harness, fails prod → unrepresentative golden set or no drift monitoring (01/Phase 7.08).

Extension task

Add online monitoring of the chosen model (drift/latency/cost) feeding failures back into the golden set (Phase 7.08, 01); write the selection memo (Phase 3.07).

Production extension

Adopt a framework (OpenAI Evals/Inspect/promptfoo/Braintrust) for the harness, run it in CI on every change, extend to routing/system evaluation (Phase 8.05), and close the loop with online feedback.

What to measure

Per-candidate quality/p95-latency/cost-per-resolved-task/reliability/safety; weighted score + ranking; routing blended score; CI-gate firing.

Deliverables

• A reusable selection harness (all axes, safety-gated, weighted).
• A ranked comparison + decision memo.
• A CI regression gate + (stretch) a routing-simulation result.

13. Verification Questions

Basic

1. What does the model-selection harness do, end to end?
2. Why is safety a gate rather than a weighted term?
3. Why must weights be set before scoring?

Applied 4. Write the weighted-score formula and explain each axis (and which cost metric). 5. How does the same harness serve both selection and regression-gating?

Debugging 6. The harness picks a model that feels wrong. Four things to check. 7. A candidate passes the harness but fails in production. Likely cause?

System design 8. Design a reusable harness: golden set, scorers, safety gate, weighted scoring, CI gate, routing simulation, online feedback.

Startup / product 9. Why is the eval harness a compounding moat (and how does it speed model adoption + de-risk changes)?

14. Takeaways

1. The harness is the capstone — run candidates over the golden set, score all axes, and produce one defensible decision + a regression gate.
2. Weighted score (quality + latency + cost-per-resolved-task + reliability) with weights set first; safety is a hard gate before scoring (07).
3. Use cost per resolved task and p95-under-load latency, not $/token / averages (06).
4. Build it once; reuse forever — new model = re-run (the compounding moat); the same harness is the CI regression gate.
5. Extend to routing simulation and close the loop online (monitor + feed failures back) (Phase 5.09/Phase 7.08).

15. Artifact Checklist

• A reusable selection harness (golden set + scorers + runner, all axes).
• A safety gate applied before weighted scoring.
• A weighted, ranked comparison with weights set first + a decision memo.
• A CI regression gate (same harness).
• (Stretch) a routing simulation + online-feedback loop.

Up: Phase 12 Index · Next: Phase 13 — Fine-Tuning and Adaptation