CLAUDE.md stopped Opus adding docstrings 8 of 8. The Haiku half did not hold

There are two open issues on the Claude Code tracker about verbose comments, and the framing in both is that the habit is strong enough to override explicit instructions: a mandatory rule in CLAUDE.md does not reliably suppress it, and the model regresses on the next turn.

Correction, 2026-08-10. The Haiku figures below did not reproduce and are withdrawn. The trial has since been re-run with the contamination removed, and the corrected answer is the opposite of the one here: with the control arm controlled, Haiku commented all 16 files. Re-running the same design, the same model and the same tasks a few hours after publishing produced 8 of 8 files carrying a docstring where this post reported 0 of 8, and inline comments appeared in a second re-run where this post reported none in 86 generations. The Opus result, that the rule removed the docstring from every file, was not re-tested and is not withdrawn. What the two runs disagree about, and the confound that makes the original control invalid, are set out at the end.

The only claim here that still stands unqualified is the Opus one: given a CLAUDE.md forbidding comments and docstrings, it wrote none in 8 files out of 8, where it had put a docstring in all 8 without the rule.

Three paths, because the first two found nothing

Path	Generations	Inline comments	Docstrings
Snippet, reply constrained to a code block	44	0	0
Snippet, no output-format instruction	10	0	0
Files created on disk	32	0	9

The first path found nothing, and my first thought was that my own prompt was the cause: asking for “a single code block and nothing else” reads as keep it minimal. So I removed that instruction entirely and asked plainly. Still nothing.

The complaints describe real work, so the third path asks the agent to create files, and the code is read back from disk rather than from a reply. That is where the only real behaviour showed up.

Only Opus, only in files, and the rule stops it

	Files with a docstring	Inline comments	Median file size
Haiku 4.5, no rule	0 of 8	0	369 bytes
Haiku 4.5, rule	0 of 8	0	432 bytes
Opus 5, no rule	8 of 8	0	3,411 bytes
Opus 5, rule	0 of 8	0	1,948 bytes

Opus put a docstring in every file it wrote, unprompted, nine in total across eight files. Given a CLAUDE.md forbidding comments and docstrings, it wrote none, across the same eight tasks in one continuous session. No regression on later turns: the eighth file was as clean as the first.

Haiku never did it in the first place. Withdrawn. On re-run Haiku put a docstring in 8 files out of 8 without any project rule, so the row above showing 0 of 8 does not describe stable behaviour. A CLAUDE.md rule is still charged on every session, but the claim that it suppresses nothing on Haiku is not supported.

The side effect is bigger than the docstrings

Opus’s files came out 43% smaller under the rule, a median of 1,948 bytes against 3,411.

Removing docstrings alone does not halve a file. The instruction appears to have shifted how much it wrote in general, and I am not going to claim a mechanism from eight files per arm. It is worth knowing that a rule aimed at comments changed the size of the whole artifact, because that is not what someone adding it expects.

The control that proves the detector works

A sweep of zeros is exactly what a broken detector looks like. So the same detector was pointed at the same path and the same model, with a prompt asking for comments:

POSITIVE CONTROL: comments=10 docstrings=1 status=ok bytes=1720

Ten inline comments and a docstring, detected end to end on real generated output. The zeros are absence, not blindness.

What this does not establish

It does not say the tracker issues are wrong. They may describe an earlier version, interactive rather than headless use, other languages, or larger refactors of existing files. After the correction below, the only thing this shows is that on 2.1.226 the prohibition worked on Opus.

Small files, written from scratch. Every task creates one new file with one function. Editing a large existing file is untested and is the likelier home of the complaint.

Python only, two models, counts not rates. Eight files per arm on the path that mattered.

The practical version

1. Check whether you have the problem before adding the rule, and check it more than once. The Haiku answer here swung from 0 of 8 to 8 of 8 between two runs on the same day, which is the real lesson: a single run of your own is not enough to conclude you do not have the behaviour.
2. If you use Opus and do not want docstrings, the rule works. Eight out of eight, with no regression across a session.
3. Expect it to change more than comments. The files got substantially smaller, which may or may not be what you wanted.

Method

Claude Code 2.1.226 headless, --model claude-haiku-4-5-20251001 and --model claude-opus-5. Each arm is one continuous conversation via --continue, so a late-turn regression would show. The file path uses --permission-mode acceptEdits and reads the artifact from disk; a run where the file was not created stops the chain rather than being scored as “wrote no comments”, which is the arrival control.

Scoring is structural. Docstrings come from ast.get_docstring on the module and on every function and class; inline comments come from tokenize COMMENT tokens, so a # inside a string literal cannot be miscounted. The control suite covered exactly that case, along with trailing comments, class docstrings, prose-only replies and unparseable code, and passed before any generation was scored.

All 86 generations are published at comment-rule-trial.json, including the full source of every file written to disk.

The first design was abandoned rather than published: with the output-format instruction in place, both arms sat at zero, so nothing could be distinguished. A control arm that reads zero means the experiment cannot answer the question, and the fix was to keep changing the path until the baseline showed the behaviour the claim is about.

Correction in detail, 2026-08-10

Two separate problems, found within hours of publishing.

The Haiku result does not reproduce. Re-running the identical continuous-chain design on the identical eight tasks gave 9 docstrings across 8 files, against the 0 reported here. A second re-run using a fresh invocation per task rather than one chain gave 5 docstrings across 8 files and 2 inline comments, against the 0 inline comments in 86 generations reported here. Same model, same version, same day. A figure that swings from 0 of 8 to 8 of 8 on a re-run is not a measurement, and the Haiku rows are withdrawn rather than restated, because two runs disagreeing does not tell you which is right.

The control arm was never rule-free, which is the more instructive error. The machine these trials run on carries a user-level CLAUDE.md containing a strictly-enforced instruction never to add comments to code. Claude Code loads user-level memory in addition to any project file, so the arm labelled “no rule” was in fact running under a machine-wide prohibition the entire time. It was never a control. That alone invalidates the comparison this post was built on, independently of whether the numbers reproduce.

Isolating that turned out to be harder than expected: --bare does skip CLAUDE.md discovery, but it also disables enough tooling that the model created no files at all, so it is not usable for a file-writing trial. A correct version of this experiment needed a genuinely clean user scope. --setting-sources project turned out to provide one, verified with an InstructionsLoaded hook rather than assumed, and the re-run is published separately: 0 of 16 files clean without the rule, 16 of 16 with it.