The first version took an afternoon. You wired in an LLM, wrote a prompt, and had something useful working by dinner.

Six months later: forty prompt edits, three model swaps, one regression you can't trace. Is the current version better than the old one? Eyeballing a few outputs only gets you so far.

At some point, you make a judgment call and ship.

The problem: the uncertainty moved

Traditional software keeps its uncertainty at the edges: user input, networks, flaky dependencies. Tests mock those away and assert against a deterministic core. Same input, same output.

LLM applications invert that. The code around the model is deterministic; the model is not. It decides what to do, which tool to call, how to phrase the answer. The same request can take a different path next run, and one prompt tweak moves outputs everywhere. There is no single expected result for a green check to prove.

The statisticians got here first: all models are wrong, but some are useful. You can't guarantee every output, but you can measure how often the system behaves as intended. Accuracy is how often it gets the task right; precision is how much to trust that number. Improving the system raises the first, more samples the second. Instead of a pass/fail assertion, you get a score, a confidence interval, and a baseline.

Large AI labs have teams dedicated to evaluation. Most product teams do not. Whether you are building a support tool, a contract summarizer, or a product built around a model, every change raises the same questions:

• Can we ship this?
• Which model should we use?
• Will we notice when it gets worse?

Answering them properly takes real infrastructure: a harness, isolated sandboxes, reliable graders, secure key handling, limits on what a runaway task can spend. That is its own roadmap item, so it gets postponed while the product keeps changing.

The solution: eval-driven development

Eval-driven development applies test-driven discipline to probabilistic software: you still define correct behavior, but measure it across a dataset instead of checking one exact output. A demo shows the system can work; an eval says how reliably.

The useful part is the loop. Change a prompt, model, or retrieval step, then run the eval against a pinned baseline. Clear the gate and it becomes the new baseline; miss it and the change never ships. Run that on every pull request and evaluation becomes part of development: regressions tie to specific changes, improvements come with evidence.

A single score can mislead you. Optimize one number hard enough and the system finds ways to raise it without improving the product: a support bot judged on ticket resolution learns to close tickets by deflecting customers, and churn gets worse.

This is Goodhart's law: once a measure becomes a target, it stops being a good measure. A cleverer metric doesn't fix that; evaluating from several angles does, so a gain on one axis can be checked against the others.

Start with accuracy, cost, and latency. Retries and longer reasoning may raise accuracy, but the effect shows up in cost and response time; a cheaper model lowers the bill, and the quality score says whether that was worth it. Together they make the compromises visible. A test suite, not any one test, defines correctness, and Benchwright applies that discipline to AI on every change.

The product: an AI benchmarking factory

Benchwright is a factory for evals. You describe what you want to measure; it finds the benchmark, creates the sandboxes, runs the tasks, and grades the outputs.

For a standard benchmark, a run takes one call. The catalog includes hundreds: MMLU for broad knowledge, GPQA Diamond for expert reasoning, AIME for competition math, LiveCodeBench for coding, WebArena for live environments, MMMU for images and documents. Launch any of them from the CLI or a POST request:

benchwright run "GPQA Diamond" --model qwen/qwen3-32b@openrouter

curl https://benchwright.ai/v1/runs \
  -H "Authorization: Bearer $BW_KEY" \
  -d '{
    "request": "GPQA Diamond",
    "model": {
      "model": "my-model",
      "base_url": "https://your-endpoint.example.com/v1/chat/completions",
      "api_key": "$MODEL_KEY"
    }
  }'

The AI workforce

Operators do the work inside the factory: AI agents that take a plain-English goal, find the right benchmark, run it against your model, and investigate failures along the way.

Public benchmarks tell you about a model in general. Your product succeeds or fails on a much narrower set of scenarios. A minimum viable evaluation covers that gap: hand the operator the cases that matter most and it turns them into a small graded dataset. It won't cover everything, but it runs on every change and grows with the product.

Repeat a run on every change and you get a record: what is safe to ship, which model performs best, whether the product is drifting.

Benchwright is live, and the free tier covers the full workflow: pick a benchmark that resembles your workload, point it at your model, and run your first evaluation.

Prompts will keep changing, and so will the models behind them. What matters is knowing whether each change made the product better, before you ship.