A new model writes a better-looking answer in a demo. The question in the release meeting is harder: does it improve the work people actually do, and what gets worse if we switch?
An offline harness should produce the evidence for that decision. A leaderboard of average scores is only one small part of it. This note describes the evaluation structure I use to reason about the choice, with an illustrative record rather than private workload data or unpublished performance results.
Write the release rule before the run
Decide which task families the candidate could serve, what quality failures would rule it out, and which latency and cost budgets apply. Keep these dimensions separate. A faster answer does not compensate for missing a required field in an extraction task.
The candidate does not need to win everywhere. A gateway can send one workload to a new model while leaving another on the baseline. Define that option before the results arrive, so partial routing is an explicit decision rather than a way to explain away a disappointing score.
Keep the cases and the system attached to the result
A run should identify the corpus version, model identifier, provider, prompt, gateway revision, generation settings, and scoring configuration. Store raw outputs before scoring. If the judge changes later, you can rescore those outputs without pretending a fresh model response reproduces the old run.
{
"case_id": "example-extraction-0042",
"task_family": "structured_extraction",
"input_ref": "synthetic-fixtures/manual-0042.txt",
"expected_ref": "synthetic-fixtures/manual-0042.expected.json",
"checks": ["valid_schema", "required_fields", "source_support"],
"provenance": "hand-authored synthetic example"
}A useful corpus contains the awkward inputs too: conflicting passages, a table split across pages, missing information, long irrelevant sections. Keep a held-out set so prompt tuning does not gradually turn the benchmark into a memorized test. Report the distribution being tested; overrepresenting rare failures is useful, but the resulting average is not a traffic-weighted production estimate.
Run the comparison through the path you will ship
Measure both baseline and candidate through the gateway and tooling the application uses. Otherwise the evaluation can miss the costs of retrieval, validation, retries, and adapters.
flowchart LR
cases["Pinned cases"] --> run["Baseline + candidate"]
run --> raw["Outputs, timings, usage, failures"]
raw --> checks["Deterministic checks"]
checks --> review["Rubric scoring + human audit"]
review --> slices["Compare by task family"]
slices --> decision["Keep / switch / route selectively"]Use repeated trials where generation varies. Interleave or randomize candidate runs when shared load could skew timing. Log failures and retries rather than dropping them from the denominator.
| Dimension | Keep visible |
|---|---|
| Quality | Per-case failures, task-family slices, uncertainty, and the size of the evaluated sample |
| Latency | Time to first useful output, completion time, and tail behavior |
| Cost | Recorded usage multiplied by the applicable dated rates, including tools, caching, and retries where relevant |
Token counts are an input to a cost calculation, not a bill by themselves. A cheaper per-token model can cost more for a task if it emits more tokens or needs more recovery attempts.
Test the judge too
Start with deterministic checks where they fit: valid output shape, required fields, exact reference values, and legal tool arguments. Use a model judge for questions that need judgment, with a rubric specific to the task.
Position, verbosity, and self-enhancement biases are documented limitations of LLM judges, including in the original MT-Bench and Chatbot Arena study. A judge's preference is evidence about its assessment, not ground truth about the answer.
Hide model identities from the judge, swap answer order in pairwise comparisons, and record inconsistent verdicts. Audit a human-labeled sample that includes the hard cases, not just obvious successes. Check agreement by task family; a good aggregate can hide a judge that is unreliable on the task you most need to trust.
Changing to a different model family does not automatically remove bias. Neither does averaging several judges. Both need their own calibration against the criterion the system is supposed to satisfy.
Make the report answer the meeting's question
A useful decision record can be short:
- What configuration is being replaced, and for which workloads?
- Which pinned run supports the change?
- Which predefined criteria passed or failed?
- What remains on the baseline, and why?
- How will rollout be observed and reversed?
Read the failure slices before the overall score. If a critical task regresses, a headline average should not hide it. If there are too few examples to tell, record that uncertainty and collect more evidence.
Offline results also need an operational follow-through: a limited rollout, observable failure signals, and a way to return to the prior configuration. The harness describes behavior on its cases; users will supply cases it has never seen.
A regression is a lead, not a diagnosis
Re-running a stable subset can reveal that something changed. It does not prove the provider changed the model. Sampling variation, rate limits, source drift, network conditions, and infrastructure changes can all move the result. Pin what you can, retain the run metadata, then investigate the difference.
I have a more personal example of why that discipline matters. The memorial fighting-game bot looked broken partly because its evaluation interface decoded the wrong input bits and its control layer starved the policy. The model was only one suspect. That audit is in When Your Eval Harness Lies.