Interview Prep 08 — LLM Evaluation Engineer

Owns measurement: golden sets, judges, per-domain metrics, regression gates — the empirical backbone. Phase: 12. Cheatsheet: 16.

1. Core concepts

Evaluate YOUR task, not benchmarks; a metric per task; benchmark contamination.
Golden datasets (representative + edge + unanswerable; never train; versioned) = the moat.
Scoring: programmatic > calibrated LLM-judge > human.
LLM-judge biases (position/verbosity/self) + calibration.
Domain metrics: RAG (retrieval vs generation), agents (outcome/trajectory), code (pass@k/apply-rate).
Latency (p95 under load) + cost (per resolved task) evals; safety as a gate.
Model-selection harness + CI regression gate.

2. System design questions

Design an eval harness that gates every prompt/model change in CI (Phase 12.08).
Design a RAG eval that separates retrieval from generation (Phase 12.03).
Design an LLM-as-judge you can trust (calibration + bias mitigation).

eval pipeline diagram.

3. Debugging questions

Eval scores great, prod is bad → contamination/leakage or unrepresentative golden set.
Judge disagrees with humans → uncalibrated; position/verbosity bias; add rubric + pairwise.
A model "improved" but users complain → wrong metric; missing edge cases.
Flaky eval results → temperature/seed nondeterminism; pin them.

4. Coding exercises

Build an eval harness: golden set + programmatic scorers + a calibrated judge + a weighted report (labs/lab-05).
Implement retrieval metrics (recall@k, MRR) + a faithfulness judge.
Implement a CI regression gate (fail the build if score drops).

5. Architecture exercises

Draw the model evaluation pipeline (golden → score → weighted + safety gate → CI).
Design the golden-set growth loop (production failures → new cases).

6. Model selection questions

Design the bake-off harness combining quality/cost/latency with a safety gate (template 01).
Why is safety a gate, not a weighted axis?

7. Production incident questions

A regression shipped despite "passing" → golden set too small/unrepresentative; expand; tighten gate.
Judge drift over time → re-calibrate; version the judge + rubric.
Eval cost too high → sample, programmatic where possible, tiered evals.

A reusable eval harness: versioned golden set (representative + edge + unanswerable), programmatic + calibrated-judge scoring, per-domain metrics, latency/cost axes, a safety gate, and a CI regression gate — applied to a real feature with a decision memo. README with the methodology + numbers.

9. 30-minute drill

5 min: evaluate-your-task + metric-per-task + golden-set-as-moat.
10 min: design a CI-gating eval harness (whiteboard).
10 min: debug "great eval, bad prod" + judge miscalibration.
5 min: LLM-judge biases + mitigations; safety as a gate.

10. 2-hour take-home

Build an eval harness for a provided feature: assemble a small golden set, implement programmatic + judge scoring, report a weighted score with a safety gate, and wire a pass/fail regression check. Writeup: how you'd prevent contamination and grow the set.

11. Senior-level expectations

Junior: runs a benchmark / eyeballs outputs.
Mid: builds a golden set + basic scorers.
Senior: designs metric-per-task evals, calibrates judges, prevents leakage, gates CI, and treats the golden set as a compounding asset.
Principal: sets the org's eval standards and the model-selection harness; makes eval the gate on every change; quantifies cost/latency/safety as first-class.

Next: 09 — AI Startup CTO · Index: interview-prep/

LLM Inside-Out Mastery