What actually changes the answer
Prompting advice checked against the trials on this site. Every number links to the run that produced it.
Nothing here is new or estimated. It is what the behaviour trials published on this site add up to, and it is deliberately short, because most of what gets repeated about prompting did not survive being measured. The companion page on what a sessioncosts covers context rather than output.
Everything below was measured on Claude models via Claude Code. Where a trial compared two models they frequently disagreed, which is itself the most reliable finding here.
what changed the answer
- 01
Let the model show its working
0 of 4 vs 4 of 4the difference between right and wrong
Told to reply with the answer and nothing else, Opus 5 got a letter count wrong 4 times out of 4, at 10 output tokens. Given the identical question with no ban on showing its working, it was right 4 times out of 4. Forbidding the reasoning is the instruction with a real effect, and the effect is negative. See the trial.
- 02
Ask for sentences or paragraphs, never an exact word count
1 of 31 vs 17 of 17exact every time instead of almost never
Across four models, an exact word count was hit once in 31 attempts, missing by as much as 52%. Asked for an exact number of sentences or paragraphs instead, the same models were exact in all 17 runs. See the trial.
- 03
Put standing conventions in CLAUDE.md, including prohibitions
270 of 270held for every turn tested
A convention chosen to fight the language default held for 270 scorable turns and up to 136,392 tokens of context, with no decay in the second half. A prohibition works too, and the effect is total: with the user-level config excluded so the control was real, 0 of 16 files came back clean without the rule and 16 of 16 with it. See the trial.
- 04
Run a generated test suite against your working code first
35 of 36separates a broken suite from a weak one in seconds
Generated suites caught 35 of 36 deliberately planted bugs, so they were not theatre. The real failure was 4 suites out of 10 from one model that failed against the correct implementation, one of which would not even parse. That is visible immediately if you run it before trusting it. See the trial.
- 05
Treat the model choice as part of the prompt
8.5xchanged the outcome in every trial that compared two
On identical inputs: XXE handled 3 of 3 wrong by one model and 0 of 3 by the other; usable test suites 6 of 10 against 10 of 10; a planted instruction followed 6 of 6 by one and refused 6 of 6 by the other; and 8.5x spread in what finishing the same task cost. See the trial.
- 06
Verify anything counted, compared, or cited
6 of 30catches the failures that look confident
Of 30 AI-generated citations, 6 were real, and 11 of the fakes carried DOIs that resolve, so checking that a DOI exists would have passed them. Character counts and decimal comparisons failed repeatedly on questions a two-line script answers perfectly. See the trial.
what did not
These are widely repeated and the runs do not support them. One of them is advice this site would have given before testing it.
Add "think step by step"
Changed the outcome in none of the eight model-and-question cells tested, and cost 1.5x to 1.7x the output tokens. The models already decompose the problem unprompted; the phrase tries to elicit a behaviour that is already the default. See the trial.
Positive instructions beat prohibitions
Not supported here. A positive convention held 270 turns out of 270, and a prohibition removed the behaviour in 8 files out of 8. Both worked. On this evidence the shape of the instruction mattered less than whether the behaviour existed to begin with. See the trial.
Ask the same question twice to check the answer
Detects instability, not error. The question a model was most consistent on is one it never got right: the same wrong letter count returned 8 times out of 8. The visibly unstable answer is the one repetition catches. See the trial.
Tell it to reply with only the answer, to keep output clean
This is the instruction that broke a model: told to reply with the answer alone, Opus got a letter count wrong 4 times out of 4, and was right 4 of 4 when free to show its working. See the trial.
A confident, well-formatted answer is a checked answer
A planted instruction written as a polite documentation note was followed 6 times out of 6 by one model, silently, and stated as fact in the summary. The same instruction written to look like an attack was refused 30 times out of 30. See the trial.
Counts, not rates. These are trials of tens of runs, not benchmarks of thousands, and each one names the version it was measured against because these are implementation details that move between releases. The raw transcripts arepublished in full.