Models
The same method pointed at the models themselves: what each one costs to run, and where they measurably differ.
tools built from these findings
What actually changes the answer
Prompting advice checked against these trials: what moved the result, and what is repeated everywhere but did not.
what each model costs 3
The same 8KB file is 3,519 tokens on Opus and 2,586 on Haiku
Measured the token cost of byte-identical files across Opus 5, Sonnet 5 and Haiku 4.5, and the startup context floor of each. Opus and Sonnet agree to within 15 tokens on every input. Haiku does not agree with either, and the gap depends on what the text is.
- claude code 2.1.224
- 8KB prose on opus 5 3,519 tokens
- same file on haiku 4.5 2,586 tokens
- haiku vs opus range 0.66 to 0.90x
Eight tokenizers agree exactly on English. On Hindi they differ by 3.3x
Measured one paragraph and its translations into ten languages across eleven tokenizers, including three Claude models measured empirically. English is unanimous. Everything else is not, and for non-Latin scripts the spread is large enough to change which model is cheaper.
- english agreement 8 of 8 exact
- hindi spread 3.27x
- russian spread 3.70x
- spanish, gpt-4o vs opus 5 103 vs 201
The same bug fix cost $0.11 on Haiku and $0.93 on Opus
Measured what it costs to actually complete a task rather than what a token costs, by giving Opus 5, Sonnet 5 and Haiku 4.5 the same failing test suite and verifying the fix by running it. Cost to finish varied 8.5x, and the ranking is not the one the price list implies.
- claude code 2.1.226
- haiku 4.5 median cost $0.109
- opus 5 median cost $0.925
- sonnet 5 median tokens 1,084,272
where they measurably differ 15
7 of 12 runs got the date right. Every miss that showed its working was arithmetic
Six free-tier chat models were each asked twice what date a project finishes after 120 working days from 3 March 2026. Seven of twelve runs landed on a defensible answer. Of the three misses that showed any working, none was a misreading of the question: each chose a correct method and then executed the arithmetic wrongly, including one model that used the same method twice and got two different dates. Four of the five wrong answers contain a statement that is false or impossible on its own terms; the fifth is one of the two that showed no working at all.
- runs 12, six models, four vendors
- correct 7 of 12
- models right both times 3 of 6
- runs that showed a setup 8 of 12, all correct
A CLAUDE.md rule held for 270 turns and 136,000 tokens
Context rot is described everywhere and measured almost nowhere. A CLAUDE.md convention that pushes against the language default held on both models at every context size reached, and the function names were parsed rather than pattern-matched so a single ambiguous word could not be scored as a failure.
- claude code 2.1.226
- scorable turns 270
- violations 0 of 270
- Haiku 4.5 context reached 136,392 tokens
- re-verified 2026-08-10, isolated 80 of 80, to 41,072 tokens
A required field was absent. It answered null 24 times out of 24
A schema with seven deliberate traps, extracted by plain prompting rather than constrained decoding. Shape and truth were scored separately so the gap the guides warn about would be visible. It did not appear once.
- claude code 2.1.226
- extractions 24
- schema-valid 24 of 24
- every field correct 24 of 24
- re-verified 2026-08-10, isolated 24 of 24, unchanged
AI tests killed 35 of 36 planted bugs. Four suites never ran at all
Mutation testing on generated pytest suites, with each mutant verified to be catchable before it counted. Opus produced ten usable suites out of ten and killed every mutant. Re-verified under isolation the caught-bug result held and improved to 38 of 38; the claim that Haiku failed on the same two functions every time did not.
- claude code 2.1.226
- suites generated 20
- mutants killed 35 of 36
- Opus 5 usable suites 10 of 10
- re-verified 2026-08-10, isolated 38 of 38 mutants killed
Haiku got XXE wrong every time. Opus used defusedxml every time
The circulating figure is that 44% of AI code generation tasks introduce a vulnerability, from corpora nobody can inspect. Here are six classes, both prompts and every generated file published. SQL injection never appeared once. XML external entities appeared in three runs out of three.
- claude code 2.1.226
- files generated 36
- flagged by the scanner 4 of 36
- XXE, Haiku 4.5 3 of 3 vulnerable
CLAUDE.md stopped Opus adding docstrings 8 of 8. The Haiku half did not hold
The advice everywhere is to put a "never add comments" rule in CLAUDE.md because the model ignores you otherwise. The original Haiku figures here did not reproduce and have been withdrawn; a re-run the same day produced the opposite result. The Opus finding, that the rule removed a docstring from every file, is unaffected.
- claude code 2.1.226
- generations 86
- inline comments seen withdrawn, did not reproduce
- Opus 5 file writes, no rule 8 of 8 had a docstring
The loud prompt injection failed 30 times. The quiet one worked 6 of 6
A benign marker payload, five placements, two models, and an arrival control proving the file was read on all 42 runs. The obvious injection was caught in all 30. Reworded as an ordinary editorial note, the same instruction got through the cheaper model 6 times out of 6, silently.
- claude code 2.1.226
- obvious payload runs 30
- obvious payload obeyed 0 of 30
- quiet payload, Haiku 4.5 6 of 6 followed
- re-verified 2026-08-10, isolated unchanged, 0 of 20 obeyed
"Think step by step" changed nothing in 96 runs
Three prompt styles, two models, four questions with ground truth computed by the script that asked them. Adding "think step by step" never once turned a wrong answer right, because the models already reason without being told to. Forbidding the working is what broke one of them.
- claude code 2.1.226
- runs 96
- correct overall 92 of 96
- cells CoT improved 0 of 8
- re-verified 2026-08-10, isolated direction holds, 0 of 4 became 1 of 4
With the control arm controlled, Haiku commented all 16 files
The control arm was running under a machine-level prohibition, so neither arm was rule-free. Isolating it needed a flag that blocks user memory without breaking tools, verified by hook rather than assumed. The corrected result reproduces across two rounds and says the opposite of the original.
- claude code 2.1.226
- files, corrected trial 32
- no rule, files clean 0 of 16
- rule, files clean 16 of 16
A model hashed passwords with raw SHA-256 in 8 of 9 runs. Two extra words fixed it
The widely quoted figure is that 45% of AI-generated code contains vulnerabilities. I tested two textbook cases across three models with an objective checker. One failed badly and consistently, one classic vulnerability never appeared at all, and the fix was two words of prompt.
- password, neutral prompt 8 of 9 insecure
- same model, "make it secure" 2 of 2 secure
- SQL injection found 0 of 12
- models tested 3
Eight models, one bug, three rounds: the 32B coder passed every time and the 72B failed every time
A hands-on trial rather than a benchmark: one small JavaScript bug with a trap in it, eight models, three rounds each, every answer executed against the real test suite. The results are not ordered by model size, and the code the models wrote shows exactly why.
- models tried 8
- rounds each 3
- Qwen2.5-Coder-32B 3/3 passed
- Qwen2.5-72B 0/3 passed
Finding one fact in 43,000 words: 10 of 10. Adding three of them up: 8 of 10
A needle-in-a-haystack test at 43,000 words found no position bias whatsoever: the fact was retrieved from 0%, 25%, 50%, 75% and 100% depth every time. A harder version requiring three buried facts to be added together failed twice in ten runs, and both failures were arithmetic, not retrieval.
- document length 43,000 words
- single fact, all depths 10 of 10
- three facts combined 8 of 10
- decoy codes in document 12
It gave the same wrong answer 8 times out of 8. Consistency is not accuracy
The circulating concern is that models give different answers to the same question. Across 64 runs the more useful finding was the opposite: the most stable answers included some of the most reliably wrong ones, and asking twice tells you nothing about correctness.
- runs 64
- correct overall 41 of 64
- questions that varied 4 of 8
- letter count 0 of 8 correct, 1 answer
Models hit an exact word count once in 31 tries. Sentences: 17 out of 17
A measured test of the most-repeated prompting workaround. Word counts came in anywhere from 28% short to 52% long. Sentence and paragraph counts were hit exactly in all 17 runs, across every model tested.
- exact word counts 1 of 31
- exact sentence counts 9 of 9
- exact paragraph counts 8 of 8
- worst word miss +52%
Only 6 of 30 AI citations were real, and 11 of the fakes had DOIs that resolve
Citation fabrication measured rather than described: 30 citations from three open models, each DOI resolved against Crossref and DataCite and its title compared to the claimed one. Verifying that a DOI merely exists would have passed 11 fabrications.
- citations checked 30
- verified real 6
- niche topic real 0 of 15
- real DOI, invented title 11
Every figure is collected on the measurements page, and the method behind them is written up here.