Eight models, one bug, three rounds: the 32B coder passed every time and the 72B failed every time

I wanted to know something simple: given one small bug and no chance to iterate, which models actually fix it?

Not which one sounds most confident. Which one produces code that passes when you run it.

The task, and the trap in it

A little JavaScript module that groups timestamped events into windows, plus a test suite. One test fails: the function assumes its input is sorted and the test passes it unsorted.

The obvious fix is to sort. That is also a trap, because another test asserts the function must not mutate the caller’s array, so sorting in place fixes the visible failure and breaks a different one. Only a non-mutating sort reaches ALL PASS.

Every model got the module, the test suite, and one instruction: return the corrected file, nothing else. No tools, no second attempt, no feedback. I extracted the code from the reply, wrote it to disk, and ran the real suite. Nothing here is judged by reading an answer. The fixture is published.

What happened

Model	Passed	Typical output tokens	Typical time
DeepSeek V3	3/3	193	4s
Qwen2.5-Coder-32B	3/3	185	4s
Claude Haiku 4.5	3/3	1,756–4,183	20–39s
Claude Sonnet 5	3/3	2,141–5,504	25–69s
Claude Opus 5	3/3	2,267–6,132	42–77s
Llama 3.3 70B	2/3	180	3s
Qwen2.5-72B	0/3	174–191	5–21s
Llama 3.1 8B	0/3	163–176	10–11s

Five models were perfect. Two never managed it. One was inconsistent.

The ordering has very little to do with size.

The 32B beat the 72B, and the difference is three characters

This is the part I did not expect. Two models from the same family, the larger more than twice the size of the smaller, and the results are opposite and perfectly consistent.

Here is what Qwen2.5-72B wrote, all three times:

function rollingWindows(events, windowMs) {
  const out = [];
  let current = null;
  events.sort((a, b) => a.at - b.at); // Ensure events are sorted by time

And Qwen2.5-Coder-32B, all three times:

function rollingWindows(events, windowMs) {
  const sortedEvents = [...events].sort((a, b) => a.at - b.at);

That is the whole difference. [...events] copies before sorting; events.sort() reorders the caller’s array in place. The 72B even annotates the line that breaks the test with a confident comment about what it is achieving.

It is not that the bigger model failed to understand the visible bug. It understood it perfectly and fixed it. It just did not consider that sorting has a side effect, three times out of three, while the smaller model specialised for code did consider it, three times out of three.

On this task, specialisation beat scale, and it was not close.

The two failure modes are not the same failure

Qwen2.5-72B failed identically every time: fix the bug, spring the trap. That is a competent answer with one blind spot, and it reproduces exactly.

Llama 3.1 8B failed three different ways across three rounds: once by mutating, once by not fixing the sorting at all, once by breaking the boundary test that had been passing. That is not a blind spot, it is not really tracking the problem.

Both score 0/3. They are not remotely the same result, and a leaderboard would hide that.

Terse models did as well as verbose ones

The open models answered in about 180 tokens: the corrected file, no commentary. The Claude models spent between 1,756 and 6,132 output tokens on the same one-shot question, working through it before answering, and took ten to twenty times as long.

Both approaches got to 3/3. DeepSeek V3 matched Claude’s reliability here with roughly a twentieth of the output and a response in four seconds.

For a bug this size, the thinking did not buy accuracy that terseness lacked. I would not extend that to a harder problem, and this task is deliberately small.

What this is not

It is one task. A single small bug with one trap in it. A model that nails this could fall apart on a multi-file refactor, and the reverse is entirely possible too.

Three rounds is three rounds. Enough to see that Qwen2.5-72B’s failure is systematic rather than unlucky, since it produced the same wrong line every time. Not enough to put a percentage on anything, and I am not going to. Llama 3.3 70B’s 2/3 in particular is just “sometimes”, not 67%.

One-shot is not how you would actually use these. Given the test output and a second attempt, most of the failures here are trivially fixable. That is a different experiment, and the agentic version where models could run the tests themselves tells a different story.

What it does show is that a trap the careless answer falls into separates models that a straightforward bug does not, and that “bigger” told you almost nothing about who cleared it.

Method

Eight models, three rounds each, identical prompt, no tools and no retries. Replies were parsed for a code block, written to disk alongside the untouched original test suite, and executed with node; success is the suite’s exit code. Open models ran through hosted inference, Claude models through Claude Code 2.1.226 headless with no file access so the protocol matched. Timings are wall clock and include queueing, so treat them as rough.