Stop Chasing Leaderboards: Measure Cost Per Solved Task
Public leaderboards rank models on other people's tests; measure quality and cost per solved task on your own work instead.
In this essay
The leaderboard trap
Every week a new model tops a public leaderboard. The chart goes up. The headline says "the best model yet."
A leaderboard answers one question: which model wins these tests? Someone else wrote those tests. They reward the work the authors chose to measure. Your product does different work, with your own prompts, your own users and your own budget.
So the real question is different:
Which setup gives enough quality, for this job, at the lowest cost?
No leaderboard answers that. You have to measure it yourself.
A small story about a big bill
A feature grades student essays. It runs thousands of times a month. The team picks the strongest model available, because it feels safe. The quality is excellent.
Three months later, the monthly bill is far above the plan. Nobody set a cap. Nobody asked whether a smaller model would pass the same rules. Nobody measured.
This story is an example, not a real case. The pattern is common. Teams pay for quality they do not need, because they never measure what each correct answer costs.
Two questions for every AI feature
- 01Quality. Does the output follow the rules that matter? Does a tutor avoid writing the essay for the student? Does a grader quote the draft exactly?
- 02Price. What does each correct answer cost?
A model can score well on quality and still be the wrong choice, if a cheaper model passes the same bar. You need both numbers, side by side.
Cost per call lies. Cost per solved task does not.
Cost per call is easy to read, and it misleads. Compare two setups below. Move the sliders and watch the result change.
Move the sliders for two setups. Cost per solved task = cost per run Γ· pass rate. Lower is better.
Setup A looks cheaper on the price list. But you pay for every failed run too. Divide the total by the runs that pass, and the order can flip.
Quality against cost: the frontier
Put every setup on one chart. Quality goes up. Cost goes right. Some setups sit on a line that no other setup beats on both axes. That line is the Pareto frontier.
Off the frontier, a setup is rarely worth choosing. Something is cheaper, or something is better, or both.
Seven example setups, illustrative numbers only. The blue line joins the setups that no other setup beats on both price and pass rate.
Frontier setups are the candidates. Choose among them by how much quality you need, and how much you will pay for it.
One answer per job
Different jobs need different answers:
| Job | What matters most | Likely answer |
|---|---|---|
| A tutor that must not write the essay | Strict rule following | A strong model |
| A rule checker on short inputs | Cost per check | A cheap model, if it passes the bar |
| A coding agent on hard bugs | Pass rate on hidden tests | The setup with the lowest cost per solved task |
Your list will differ. That is the point. There is no single right answer for everyone. There is one right answer for each use case, and you find it by measuring your own work.
What measuring takes
- Real tasks. Use work from your own history, not generic puzzles.
- Hidden checks. Grade results with tests or rules the model never sees.
- Honest costs. Record the billed cost of each call. Set a hard cap before a run starts.
- Repeat runs. One run proves little. A wide interval means you need more runs before you decide.
The EduDojo benchmark has two tracks. The dev track tests coding agents on 51 real tasks from the product history. The product track tests AI features on real prompts, with synthetic inputs in English and Hindi, on any model in OpenRouter. Both record cost, time, tokens, errors and whether the rules held.
Why this changes your work
- 01You stop guessing. A prompt change or a model change shows what breaks and what it costs.
- 02You stop overpaying. The cheapest setup that passes your bar is often the right one.
- 03You can explain the choice. When someone asks why, you show the frontier, not a feeling.
The short version
- A public leaderboard shows who wins their tests. It does not show what your product needs.
- Measure cost per solved task, not cost per call.
- Look at the frontier: the setups that no other setup beats on both price and quality.
- The goal is the most efficient setup for your job, not the best model.
There is no single right answer for everyone. There is one right answer for each use case. Measure yours.
The numbers in both interactive figures are illustrative. The EduDojo benchmark has no published results yet. Cost per solved task follows the benchmark's method notes: total cost divided by the number of passing runs.
Thanks for reading.