How Do You Pick Which Model to Ship?

Every few weeks a new model drops and someone asks the same question: should we switch? The honest answer, without infrastructure, is "nobody knows." Public benchmarks tell you how a model performs on public benchmarks. They don't tell you how it performs on your tasks, at your latency budget, at your token volumes. The gap between those two things is where production incidents live.

While building GenAI platform infrastructure, I worked on the version of this problem that I want to write about generically here: an offline evaluation harness that compares frontier models against the current production baseline — on latency, token cost, and output quality — to inform which model actually ships. The harness sits in front of a multi-provider LLM gateway spanning Anthropic, OpenAI/Azure, Google/Vertex, and AWS Bedrock, which means the question is never "is this model good" in the abstract. It's "is this model better than what we're running, for these workloads, at a price and speed we can live with."

This post is the technique, not the deployment: why offline evals have to come before shipping, how to build a corpus that resembles production, what the latency/cost/quality tradeoff surface actually looks like when you plot it, how automated hallucination scoring works and where LLM-as-judge quietly lies to you, and how to turn all of it into a ship/no-ship decision someone can defend six months later.

Why offline, why before shipping

The tempting shortcut is online experimentation: route a slice of traffic to the new model and watch the metrics. Sometimes that's the right final step. It is never the right first step, for three reasons.

Production feedback is slow and confounded. User-visible quality signals — thumbs, retries, escalations — arrive at low volume and get tangled with everything else that changed that week. An offline harness gives you thousands of scored comparisons in an afternoon, on identical inputs, with the only variable being the model.

Some failures are asymmetric. A model that's slightly better on average but hallucinates confidently on a specific task family is a downgrade you will not detect from aggregate online metrics until someone acts on a fabricated answer. Offline evals let you slice by task type and hunt for exactly this shape of failure before anyone is exposed to it.

A gateway multiplies the blast radius. When one gateway serves many downstream applications, "let's just try it" means "let's just try it on everyone." The platform team owes its consumers a documented, reproducible reason for every default-model change. The harness is that reason.

The mental model I use: offline evals are unit tests for model selection. They don't prove production will be fine — no offline artifact can — but they catch the failures that were catchable, cheaply, before the expensive kind of learning starts.

The corpus is the product

The decision that matters most in an eval harness isn't the judge, the metrics, or the dashboard. It's the task corpus. A harness scoring a corpus that doesn't resemble production is a very precise answer to the wrong question.

What "resembles production" means in practice:

• Sample from real traffic shapes, not imagined ones. If the gateway's workloads skew toward long-context summarization and structured extraction, a corpus full of short creative prompts measures nothing you ship. Pull the task distribution from what actually flows through the system (with whatever privacy handling your context demands — dedaction, synthesis from real templates, or hand-authored cases that mirror observed patterns).
• Weight by consequence, not just frequency. Rare-but-dangerous task families — anything where a hallucinated fact gets acted on — deserve overrepresentation relative to their traffic share. The corpus is a portfolio; allocate by risk.
• Include the ugly inputs. Truncated documents, mixed languages, adversarially formatted tables, prompts that are really two questions. Models differentiate on the ugly tail far more than on the clean median.
• Pin ground truth where it exists, define rubrics where it doesn't. Extraction and classification tasks get reference answers and exact or fuzzy matching. Open-ended generation gets a rubric per task family — what does "good" mean here — because a single global rubric flattens exactly the differences you're trying to measure.

Each case in the corpus is a small, boring, versioned artifact. Something like:

{
  "case_id": "summarize-long-report-0042",
  "task_family": "summarization",
  "input": {
    "system": "You are a careful analyst. Summarize the document.",
    "document_ref": "corpus/docs/report_0042.txt",
    "max_words": 200
  },
  "ground_truth": null,
  "rubric_ref": "rubrics/summarization/v3",
  "grading": "llm_judge",
  "checks": [
    { "type": "claim_support", "note": "every claim traceable to source" },
    { "type": "length_bound", "max_words": 220 }
  ],
  "weight": 1.0,
  "added": "2026-06-14",
  "provenance": "sampled-from-traffic"
}

The provenance and added fields matter more than they look. A corpus is a living thing — cases go stale, task mixes drift — and you can't reason about eval results over time if you can't reason about what the corpus was when the run happened. Version the corpus like code, because it is.

The pipeline

The harness itself is a pipeline with four honest stages: run every case through every candidate, score the outputs on every dimension, aggregate along the three axes, and force a decision.

Two implementation notes that save pain later. First, the runner records latency and token counts as measured through the gateway, not from provider pricing pages — the number you'll pay and the latency your users feel include your own overhead, and per-provider token accounting differs in annoying ways. Second, raw outputs are stored verbatim before any scoring touches them. When a judge disagrees with a human later (it will), you want to re-score history with the improved judge, not re-run history with a model that may no longer exist.

The scoring loop is deliberately boring:

for case in corpus:
    for model in [baseline, *candidates]:
        result = gateway.complete(model, case.input)   # measured through the real path
        record(case, model,
               output=result.text,
               latency_ms=result.latency_ms,
               tokens_in=result.tokens_in,
               tokens_out=result.tokens_out)
 
for record in records:
    scores = {}
    if record.case.ground_truth is not None:
        scores["accuracy"] = deterministic_match(record.output, record.case.ground_truth)
    for dim in record.case.rubric.dimensions:        # e.g. groundedness, completeness,
        scores[dim] = judge.score(                    # instruction-following, formatting
            output=record.output,
            source=record.case.input,
            rubric=record.case.rubric[dim])
    persist(record, scores)

Deterministic checks first, judge second, everything persisted. The boring loop is a feature: when a result looks wrong, you want the pipeline to be too simple to be the suspect.

The tradeoff surface has three axes, and "best" isn't a point on it

Latency, token cost, output quality. The mistake is collapsing them into one score. A weighted composite hides exactly the information the decision needs, because different workloads live in different corners of the surface:

• An interactive assistant is latency-dominated. A quality win that costs a visible pause at the time-to-first-token boundary is frequently a net loss.
• A batch enrichment pipeline is cost-dominated. It will happily trade seconds for a cheaper token bill, and quality only needs to clear the bar, not lead the pack.
• A task where outputs get acted on without review is quality-dominated — specifically hallucination-dominated — and no latency or cost win buys back a fabrication.

So the harness reports the three axes separately, sliced by task family, always as candidate versus baseline deltas rather than absolute scores. Absolute scores flatter whoever wrote the rubric; deltas against the thing you're actually running answer the question you're actually asking. And because the gateway can route per-workload, the decision isn't forced to be global — a candidate can win the batch corner of the surface and lose the interactive corner, and the right answer is to route it accordingly rather than crown it everywhere.

One more habit worth stealing: report latency as a distribution, not a mean. Two models with the same median can have wildly different tails, and the tail is what your users experience as "sometimes it just hangs."

Automated hallucination scoring, and how the judge lies

Quality scoring at corpus scale can't be human-first — the whole point is running thousands of cases per candidate, repeatedly. So the quality axis is scored automatically across multiple dimensions: groundedness against the source material, completeness, instruction-following, format compliance, with hallucination checks as the dimension that gets the most paranoid treatment. Where a claim-level check is possible — does every factual claim in the output trace back to the provided context — it beats a holistic "rate this 1–5" prompt, because it fails loudly and specifically.

The workhorse for the non-deterministic dimensions is LLM-as-judge, and it works — with caveats that the field has documented well enough that ignoring them is negligence, not innocence:

• Family bias. Judges tend to prefer outputs from models in their own family — similar phrasing, similar structure, similar "voice" reads as similar quality. In a multi-provider comparison this is disqualifying if unmanaged. Mitigations: use a judge from a family not under test where possible, or cross-check contested slices with a second judge from a different family and treat disagreement as a flag for human review.
• Position bias. In pairwise comparisons, judges systematically favor one position (often the first). The fix is mechanical: score every pair in both orders and treat inconsistent verdicts as ties or escalations, never as data.
• Rubric drift. The subtle one. As you edit rubrics and judge prompts across weeks of runs, scores stop being comparable over time — a "4" in March and a "4" in June may not mean the same thing. Version rubrics and judge prompts like code, pin them per run, and keep a small human-labeled anchor set that every judge version gets calibrated against before its scores count.

I'm describing these generically, as known pitfalls of the technique — not as findings I'm reporting measurements on. The meta-point is the same one that governs the rest of the harness: the judge is a model, and models get evaluated, not trusted. A judge whose agreement with human labels you've never measured is a random number generator with good grammar.

Structuring the ship/no-ship decision

The harness produces evidence. Someone still has to decide. The failure mode here is deciding by scoreboard glance — "it's greener, ship it" — so the decision itself gets structure:

1. Pre-commit the bar. Before the run, write down what "ship" requires: quality at parity or better overall, no regression beyond an agreed threshold on any high-consequence task family, latency and cost within the budgets of the workloads it would serve. Deciding the bar after seeing the results is how motivated reasoning gets a login.
2. Read the slices before the summary. An aggregate win with a hallucination regression on one critical task family is a no-ship, or at most a partial routing decision. The slice view exists precisely to stop the average from laundering the tail.
3. Make "partially ship" a first-class outcome. Behind a gateway, routing is a dial. "New model for summarization workloads, baseline stays for extraction" is often the evidence-shaped answer, and a binary framing hides it.
4. Write the decision down. Corpus version, model versions, scores, the bar, the verdict, the date. When someone asks in October why the default changed in July, the answer should be a document, not archaeology.

Regression evals: providers change models under your feet

The uncomfortable operational truth of a multi-provider gateway: the model behind an API name is not immutable. Providers update, re-tune, and occasionally silently swap what a model alias points to. Your carefully evidenced July decision describes the model as it existed in July.

The same harness closes this loop cheaply. A regression track — a stable, pinned subset of the corpus — runs against the shipping configuration on a schedule, scores flowing into the same store as selection runs. What you're watching for isn't a specific number but a discontinuity: a step change in any quality dimension, latency distribution, or token accounting that isn't explained by anything you changed. When the corpus, rubric, judge version, and gateway code are all pinned and the scores still move, the remaining suspect is the provider — and now you know before your users file the ticket.

This reframes the harness from a one-shot selection tool into standing infrastructure: the same corpus, the same scoring layer, the same reports, running in two modes — compare candidates and watch the champion.

What's hard, and what's next

Honest ledger, because the harness is a tool with edges, not a solved problem.

The corpus decays silently. Production traffic drifts; the corpus doesn't drift with it unless someone does the unglamorous work of re-sampling and re-weighting. A stale corpus produces confident, precise, wrong answers — the most dangerous kind. I don't have a fully automated fix; the current state of the art in my hands is a scheduled human-in-the-loop refresh, and I'd like something better.

Judge calibration is a treadmill, not a milestone. Every rubric edit and judge-model upgrade re-opens the comparability question. The anchor set of human labels helps, but growing and maintaining that set is real ongoing cost that eval enthusiasm consistently underestimates.

Offline evals bound risk; they don't eliminate it. The harness answers "is this model better on a production-shaped corpus." Production will still find inputs the corpus never imagined. The right posture is offline evals as the gate, followed by cautious online validation as the confirmation — not either one pretending to be sufficient alone.

Multi-provider fairness is unfinished. Prompt formatting, system-prompt conventions, and tokenizer differences mean "the same case" is never perfectly the same case across four provider families. The harness normalizes what it can through the gateway, but I hold the residual asymmetry as an open caveat on every cross-provider comparison, and I'd rather admit that than false-precision my way past it.

Next: tightening the judge-versus-human agreement loop, making the regression track's discontinuity detection less eyeball-driven, and treating corpus refresh as a first-class pipeline instead of a calendar reminder. If you're building one of these: start with the corpus, pin everything, compare against your baseline instead of a leaderboard, and never trust a judge you haven't measured.