"Think step by step" changed nothing in 96 runs

Re-verified 2026-08-10. The original runs loaded this machine’s user-level CLAUDE.md, which contains no rule about showing working but does carry two brevity rules, so the suppressed arm deserved re-checking. The load-bearing question was re-run with user memory excluded via --setting-sources project, the isolation method described here. The direction holds: forbidding the working still broke Opus while both other arms were perfect, and “think step by step” still added nothing over simply not forbidding it. One figure moved: Opus scored 1 of 4 correct with its working forbidden, not 0 of 4. Same conclusion, slightly softer number, and a reminder that a 4-run cell is a 4-run cell.

Every prompting guide still tells you to add “think step by step”. The 2026 versions hedge it slightly, saying the phrase matters less now that models reason on their own, but that it still helps on smaller or weaker models.

So I ran it. Two models, four questions, three prompt styles, four rounds each. 96 runs. Adding the instruction changed the outcome in none of the eight model-and-question cells.

The result

Every answer is computed by the same script that builds the prompt, so the ground truth cannot itself be wrong.

Model	Question	Working forbidden	Free to reason	“Think step by step”
Haiku 4.5	“n” in “unnecessarily”	4/4	4/4	4/4
Haiku 4.5	9.11 vs 9.9	4/4	4/4	4/4
Haiku 4.5	words in a sentence	4/4	4/4	4/4
Haiku 4.5	“s” in “Mississippi”	4/4	4/4	4/4
Opus 5	“n” in “unnecessarily”	0/4	4/4	4/4
Opus 5	9.11 vs 9.9	4/4	4/4	4/4
Opus 5	words in a sentence	4/4	4/4	4/4
Opus 5	“s” in “Mississippi”	4/4	4/4	4/4

92 of 96 correct. Going from a plain question to an explicit “think step by step” moved nothing, in either direction, anywhere in the table.

The instruction is redundant because the model already does it

This is the whole explanation, and it is visible in the raw output. Asked the plain question with no instruction to reason, Sonnet 5 replied:

"unnecessarily" — letters: u-n-n-e-c-e-s-s-a-r-i-l-y. The "n" appears twice (positions 2 and 3).

ANSWER: 2

Nobody asked it to spell the word out. It decomposed the problem unprompted, which is exactly what “think step by step” is supposed to elicit. You cannot elicit a behaviour that is already the default, and the advice survives from an era when it was not.

What it costs

The instruction is not free. Median output tokens on the same questions:

Model	Free to reason	“Think step by step”	Increase
Haiku 4.5	277	427	1.54x
Opus 5	166	275	1.66x

Half again as many output tokens for an identical answer. On these questions it bought nothing at all, so that is the price of the phrase, not the price of the reasoning.

The one prompt that changed a result made it worse

There is exactly one failure in the whole trial, and it belongs to the most expensive model.

Told to reply with the answer and nothing else, Opus 5 answered the letter count wrong four times out of four, at 10 output tokens per run:

ANSWER: 3

The word has two. Given the identical question with no ban on showing its working, the same model got it right four times out of four, at a median of 189 output tokens.

So the lever is not whether you ask for reasoning. It is whether you forbid it.

But this is not a general law, and that matters

It would be neat to conclude that suppressing the working breaks models. It does not, and I checked before writing it.

Model, working forbidden	“n” in “unnecessarily”	Output tokens
Opus 5	3 (wrong), 4 times	10
Sonnet 5	2 (right), 4 times	10
Haiku 4.5	2 (right), 4 times	225

Sonnet 5 answers correctly at the same 10 tokens Opus fails at, so it is not spending hidden effort, it simply gets it right. Haiku 4.5 spends 225 tokens under an instruction that told it not to, and lands on the right answer.

Three models, three different behaviours, same prompt. The failure is specific to one model on one question, and anyone who tells you that suppressing the working reliably breaks letter counting has not tested the other two.

The practical version

1. Adding “think step by step” to a current Claude model is a no-op on this kind of task, and costs about 1.6x the output tokens. If it is in your prompt template out of habit, it is doing nothing.
2. “Answer with only the number” is the instruction with a real effect, and the effect is negative. It cost Opus 5 four out of four on a question it otherwise gets right every time.
3. If you want a bare value for parsing, let the model work and extract the answer afterwards, rather than forbidding the working to save yourself the parsing.

What this does not establish

Two models plus a spot check, four questions, four rounds. These are counts of what happened, not rates. Four rounds cannot support a reliability claim about any model, which is why none is made here.

Deliberately awkward questions. Letter counting, decimal comparison and word counting are chosen because they are checkable by script and known to be hard. None of this tests world knowledge, reasoning over documents, or code, and the effect of CoT on those is untouched by this trial.

Claude models only. The circulating advice is specifically that the phrase still helps weaker models, and the weakest model here still did not need it. A genuinely small open model might behave differently, and I could not test one: the free multi-vendor route available today sits behind a terms acceptance that is not mine to click.

The answer tag is in every arm. Every prompt ends by asking for a final ANSWER: line, so extraction is structural rather than a regex hunting numbers in prose. That instruction is identical across all three arms, so the comparison is paired, but it is not a plain unadorned question.

Method

Claude Code 2.1.226 headless, --model claude-haiku-4-5-20251001 and --model claude-opus-5, fresh invocation per run with no shared conversation, four rounds per cell. Expected answers are produced by the same script that builds the prompts. Output tokens are summed from modelUsage, not .usage.

The three arms differ only in one clause: nothing, Think step by step before answering., or Reply with nothing except that final line, no working, no explanation.

The extractor reads the tagged final line structurally and never hunts for numbers in prose. It was control-tested against twelve reply shapes before any model output was scored, including a reply whose prose contradicts its own tag, a bolded **ANSWER: 4**, a comma-grouped number, a missing tag and an unparseable one. The bolded case failed on the first attempt and the anchor was fixed before any scoring ran.

A control killed this post’s first headline. The suppressed Haiku runs reported 234 output tokens while returning only ANSWER: 2, which looked like proof that models reason invisibly and bill you for it. Running a three-point ladder on the same model showed Haiku emits a median 196 output tokens for a bare OK, so its floor is simply high and the 234 was near noise. Opus, measured the same way, went 4 tokens for OK and 10 for the hidden-working count: no hidden reasoning at all. The striking version of the finding was an artifact of comparing two points without a zero-work baseline.

The suppressed arm ran concurrently with the other two, so no latency comparison is drawn from these runs.

Every run is published at cot-trial.json: all 104 of them, each with its exact prompt, its exact reply, the computed truth and the extracted answer, including the eight Sonnet 5 spot-check runs. Nothing here has to be taken on trust.