With the control arm controlled, Haiku commented all 16 files

Yesterday this site published a trial concluding that Haiku 4.5 adds no comments to the code it writes, and that a CLAUDE.md rule forbidding them therefore suppresses nothing. Both claims were withdrawn hours later because they did not reproduce.

This is the corrected trial. Given a genuine control arm, 0 of 16 files came back clean without the rule and 16 of 16 with it. The original answer was backwards.

The control arm was never a control

The machine these trials run on carries a user-level CLAUDE.md, and Claude Code loads it in addition to whatever the project directory holds. That file contains a strictly-enforced instruction never to add comments to code.

So the arm labelled “no rule” was running under a comment prohibition the entire time. So was the arm labelled “rule”. Both arms had the rule; only one of them knew it. A comparison between them could not have measured anything, and the zeros it produced were the machine’s configuration talking, not the model.

Nothing about the scoring was wrong. The detector was control-tested and worked. The experiment was invalid one layer above the part I had checked.

Isolating it, and verifying the isolation rather than assuming it

--bare skips CLAUDE.md discovery, which sounds like the answer and is not: it also disables enough tooling that the model creates no files at all. A zero under --bare is a failed measurement, not an absence, and that was confirmed again here.

--setting-sources project does the job. Rather than trust the flag, I watched it work: an InstructionsLoaded hook logs the memory_type of every instruction file as it loads.

Invocation	User memory	Project memory	File written
default	loaded	loaded	yes
--setting-sources project	not loaded	loaded	yes
--bare	not loaded	not loaded	no

The hook records only the type of each file, never its contents.

The corrected result

Same eight tasks, same model, same file-writing path, two rounds, user-level memory excluded throughout.

	Files with a docstring	Files with inline comments	Files clean	Median size
No rule at all	16 of 16	8 of 16	0 of 16	1,592 bytes
Project rule forbidding both	0 of 16	0 of 16	16 of 16	285 bytes

Thirty inline comments and twenty-seven docstrings appeared across the sixteen unrestricted files. Not one appeared in the sixteen restricted ones. Both rounds agree.

The tracker complaints about verbose commenting are right, and so is the fix. Yesterday’s post said the opposite of both.

The rule shrinks far more than the comments

Median file size fell from 1,592 bytes to 285, an 82% reduction. Stripping thirty comments and twenty-seven docstrings from sixteen files does not account for that on its own.

An instruction aimed at commentary changed how much code got written. I am not claiming a mechanism from thirty-two files, but anyone adding this rule to save themselves some tidying should know it appears to change the whole artefact, not just the annotations.

What this does not establish

One model, one language, small new files. Haiku 4.5, Python, one function per file. The Opus half of the original trial was not re-tested and is not superseded by this.

Counts, not rates. Thirty-two files across two rounds.

Isolation is not the same as neutrality. --setting-sources project excludes user-level memory, which is what was contaminating this. It does not make the environment assumption-free, and there may be other machine-level influences I have not found.

The part worth taking away

The scoring was right. The controls I had built all passed. The detector was proven on real output. The invalidating problem sat one level above everything I had thought to check: the environment the test ran in.

If you benchmark a coding agent on your own machine, your own global configuration is part of the experiment. Any instruction in it that touches the behaviour under test will silently apply to every arm, including the one you are calling the baseline. It is worth grepping that file before designing the trial, which takes seconds, and it is the step that was missing here.

Method

Claude Code 2.1.226 headless, --model claude-haiku-4-5-20251001, --permission-mode acceptEdits, --setting-sources project on every invocation. Each arm is one continuous conversation over eight tasks; a run that produces no file stops the chain rather than counting as clean, which is the arrival control. Code is read back from disk, so what is scored is the artefact left behind.

Docstrings come from ast.get_docstring on the module and every function and class; inline comments from tokenize COMMENT tokens, so a # inside a string cannot be miscounted. That scoring code is unchanged from the original trial, where it was control-tested against trailing comments, class docstrings, a # inside a string literal, prose-only replies and unparseable code, and separately proven on real output by a prompt that explicitly asked for comments.

Two rounds, run in a session separate from the original trial, because a two-run minimum satisfied inside a single sitting is not reproduction and that is how the first version got through.

All 32 files are published with their source at comment-rule-corrected.json.