The same bug fix cost $0.11 on Haiku and $0.93 on Opus
One task, three models, twenty-six runs, success checked by running the test suite. Haiku finished it in 7 turns and 290,010 tokens. Sonnet 5 took the most tokens of the three, 74% more than Opus, on identical work.
Every model comparison is priced per token. That is not the number you pay. You pay for however many tokens the model needs to finish the job, and a model that takes twice as many turns can cost more than a model with double the sticker price.
So I measured the thing that actually matters: the same task, the same prompt, three models, and success verified by running the tests rather than by reading the reply.
The task
A small JavaScript module with a failing test suite. One genuine bug, and a trap: the obvious fix makes the failing test pass and breaks a different one, because the function must not mutate the caller’s array. Only a non-mutating fix reaches ALL PASS.
Each run got a clean copy of the fixture and this prompt:
Run
node test.js. Some tests fail. Fix the code in window.js so that every test passes. Do not modify test.js.
Success was then checked by running node test.js and reading the exit code. test.js was checksummed after every run, because a suite that can be edited is not a pass/fail signal. All 26 runs left it untouched.
The fixture is published so this is reproducible.
What it cost to finish
Four instrumented runs per model, every one from a cold start in a fresh directory.
| Haiku 4.5 | Opus 5 | Sonnet 5 | |
|---|---|---|---|
| Median cost | $0.109 | $0.925 | $0.764 |
| Median prompt tokens | 290,010 | 622,508 | 1,084,272 |
| Median turns | 7 | 12 | 16 |
| Median wall clock | 46s | 92s | 99s |
| Token range across runs | 239k–382k | 555k–1,010k | 526k–2,525k |
Haiku finished the same task for 12% of what Opus cost. That is an 8.5x difference in dollars and a 2.1x difference in tokens, on identical work with identical instructions.
The mid-tier model used the most tokens
Sonnet 5 took a median of 1,084,272 prompt tokens against Opus 5’s 622,508, which is 74% more to do the same job. It also took more turns, 16 against 12, and more wall-clock time.
It still costs less in dollars than Opus because its per-token price is lower, but the gap is nothing like the price list suggests: $0.764 against $0.925, a 17% saving, not the multiple you would expect from comparing rates. Most of the per-token discount is consumed by needing more tokens.
Sonnet was also by far the least predictable. On identical input its runs ranged from 525,797 to 2,525,231 tokens, a 4.8x spread. Haiku’s widest spread was 1.6x.
Cost per token is the wrong unit
This is the general point and it survives the specific numbers. Two models advertised at the same rate can differ 2x in what they consume to finish, and a model advertised at a fraction of the rate can be the cheapest by a much wider margin than the rate implies, because the discount compounds with needing fewer turns.
The same effect appears one level down. The same file is a different number of tokens on different models, so even the input side of the price is not comparable. Two conversion steps sit between a published rate and your bill, and neither is on the price list.
What I am not claiming
Not a capability ranking. This is one small, well-specified, objectively checkable task. A harder problem could reverse the order entirely, and the fact that a model can fix a six-line bug cheaply says nothing about a multi-file refactor.
No reliability difference is being claimed. Pooled over all 26 runs, Opus finished 7 of 7, Haiku 7 of 7, and Sonnet 9 of 12. That looks like a difference and it does not survive a test: one-tailed Fisher exact gives p = 0.085. With numbers this small that is a coin landing heads a few extra times. It is reported here because leaving it out would be selective, not because it is a finding. Every Sonnet failure was the same one, leaving the original unsorted-input bug unfixed rather than falling into the mutation trap.
Dollar figures are what the CLI reported, and reported cost depends on cache state. Every run here started cold in a fresh directory, so the comparison is like for like, but do not carry these dollar amounts into a warm-session context. The token counts are the sturdier number.
The practical read
The standard advice is to default to Sonnet and drop to Haiku only for mechanical work. On this task that advice cost 7x more than necessary, and Haiku completed it every time.
The useful habit is not picking a favourite model. It is measuring cost-to-complete on your own recurring tasks, because the ranking depends on the task and the price list will not tell you.
Method
26 runs total on Claude Code 2.1.226, headless, --permission-mode acceptEdits, each in a fresh temporary project with an empty settings file. Token totals from modelUsage, which counts subagent tokens that usage omits. Success determined by the suite’s exit code, never by the model’s own report, with the suite checksummed against the original after every run. Medians are quoted rather than means because two Sonnet runs were extreme enough to drag an average around.