AI tests killed 35 of 36 planted bugs. Four suites never ran at all

Re-verified and partly corrected, 2026-08-10. These runs loaded this machine’s user-level CLAUDE.md, which carries six testing rules including TDD and acceptance criteria, so the suites might have been the machine’s doing. The trial was re-run with user memory excluded via --setting-sources project, the isolation method described here. The headline held and improved: 38 of 38 mutants killed, with Opus again 10 of 10 usable and 22 of 22. One claim did not survive and is withdrawn: that Haiku’s failures fell on the same two functions every time. Details at the end.

The standard complaint about generated tests is that they are theatre: they pass, they look thorough, and they would not notice if the code were broken.

So I broke the code. Each function got deliberate mutations, and a test suite only counts if it fails when the implementation is wrong. Across 16 scorable suites, 35 of 36 mutants were caught. The theatre complaint did not reproduce.

What did go wrong was something else entirely.

The suites that ran were strong

Model	Usable suites	Mutants killed	Test cases per suite
Opus 5	10 of 10	22 of 22	70 to 875
Haiku 4.5	6 of 10	13 of 14	8 to 13

One mutant survived in the whole trial: Haiku’s second clamp suite missed a boundary shifted by one, so clamp(11, 0, 10) returning 11 instead of 10 went unnoticed. Everything else, including dropped input validation, swapped bounds, a discarded final chunk and a wrong denominator, was caught.

Opus wrote a great deal more, and the two do not overlap at all: the smallest Opus suite has 70 test cases and the largest usable Haiku suite has 13. One Opus round wrote 875 test cases for a single seven-line clamp. Whether that is thoroughness or noise is a judgement I am not making from this data, but it caught everything.

An earlier version of this table gave a median of 196 test cases for Opus against 11 for Haiku. Both figures are withdrawn, corrected 2026-08-21. 196 is a phantom: with ten runs the median is the midpoint of the fifth and sixth values, 145 and 247, and no run produced it. The Haiku figure was not a phantom but simply wrong, because the median of the six usable Haiku suites is 11.5. Neither error changes the finding, and the ranges above replace them because a non-overlap needs no estimator to be true. The full per-run counts and the arithmetic are in the corpus.

The real failure was a suite that did not run

Four of Haiku’s ten suites were never scored, because a suite that fails on correct code cannot tell you anything about broken code. All four were the same two functions, in both rounds. That last clause is withdrawn: across four rounds the failing subjects were none, two, two and three, and a third function joined them once. The failures are real but which function they land on is run-to-run variance, not a property of the function.

The clearest one did not even parse:

def test_with_floats():
    assert percent_change(50.5, 60.6) == pytest.approx(19.801980198...)

That is a literal ... inside a number. The file raised a SyntaxError at collection and pytest never ran a single test.

The rest asserted behaviour the function does not have. On percent_change, it expected a negative-to-positive swing to be positive:

assert -200.0 == 200.0

On parse_range, it wrote tests for negative inputs, but the function splits on "-", so "-5" was never going to parse as one number. The tests were written from what the function’s name implies rather than from the code sitting directly above them, and where intent and implementation diverge, the test is simply wrong.

Why this is better news than the complaint, and worse

A weak suite that passes everything is dangerous because it is invisible: green means nothing and nobody looks again. That is what people fear and it is not what happened.

A suite that fails on correct code is loud. You see it immediately, you lose ten minutes, you move on. It is a worse experience and a smaller risk.

But there is a trap in the loud failure. When a generated test fails against your existing code, the natural reading is that the test found a bug. Twice here it was the test that was wrong, and in one of those cases the “bug” it reported was a sign convention the function had deliberately chosen. Changing the code to satisfy that test would have introduced the very defect the suite appeared to catch.

What this does not establish

Small pure functions. Five functions of three to eight lines with no I/O, no state and no dependencies. Tests for a class with mocked collaborators are a different problem and are untested here.

My mutants, not real bugs. Twelve hand-written mutations, of which one was excluded as uncatchable. They are the sort of defect that mutation testing targets: boundary shifts, dropped validation, swapped operands. A subtle concurrency or ordering bug is not represented.

Counts, not rates. Twenty suites, two rounds, two models. Nothing here supports a percentage for either model.

Python and pytest only.

The practical version

1. Run the generated suite against your correct code first. That single step separates the two failure modes and takes seconds.
2. When a generated test fails, suspect the test before the code. Half the failures here were the test asserting something the implementation never claimed.
3. If the suite runs, it is probably doing real work. On these functions the caught-bug rate was 35 of 36, which is better than the discourse suggests.

Method

Claude Code 2.1.226 headless, --model claude-haiku-4-5-20251001 and --model claude-opus-5, two rounds, fresh generation each time. The model saw only the function source and was asked for a pytest suite. Suites were run with pytest 9.1.1 against the original and then against each mutant.

Three controls had to hold before any miss could mean anything.

Every mutant had to be catchable. Each was run against the original over a probe set, and one that behaved identically everywhere was excluded as an equivalent mutant rather than counted as a miss. One of twelve was dropped that way.

That control initially misfired, and it is the mistake worth naming: my first probe sets contained no input that reached the mutated path, so a mutant that removes input validation looked identical to the original and was marked equivalent. My own thorough hand-written suite then killed it, which is what exposed the contradiction. A probe set that never exercises the change cannot detect the change, and three real mutants were nearly discarded on that basis.

The apparatus had to discriminate. A hand-written thorough suite killed 3 of 3 clamp mutants; a hand-written trivial one killed 0 of 3. Both passed the original. Without that spread, a good score would not have meant anything.

A suite had to pass the original and contain at least one test, otherwise it is recorded unusable rather than scored.

All 20 runs, with every generated suite, every mutant and the pytest output for the unusable ones, are published at ai-test-mutation-trial.json. The 20 isolated re-verification runs are at ai-test-mutation-isolated.json.

Re-verification in detail, 2026-08-10

	Original	Isolated
Mutants killed, scorable suites	35 of 36	38 of 38
Opus 5 usable suites	10 of 10	10 of 10
Opus 5 mutants killed	22 of 22	22 of 22
Haiku 4.5 usable suites	6 of 10	7 of 10
Haiku failing subjects, per round	2, 2	0, 3

The result this post is built on is unaffected: generated suites that run do catch planted bugs, and on the isolated runs they caught every one.

What does not hold is the inference I drew from the pattern of Haiku’s failures. Seeing the same two functions fail in both of two rounds, I called it a reproducible per-subject failure rather than noise. Two more rounds produced zero failures and then three, one of them on a function that had never failed before. Four rounds is enough to say the failures happen and not enough to say where, and the original wording claimed the second thing on the strength of two rounds.