It gave the same wrong answer 8 times out of 8. Consistency is not accuracy
Asked the same eight questions eight times each, with every answer computed by script rather than asserted. The question it was most consistent on is one it never once got right, and the two failure modes look identical if you only check whether the answer changed.
The complaint doing the rounds is that models give you a different answer every time, with a widely quoted figure that the same prompt repeated ten times produces consistent answers only about 73% of the time.
So I asked eight questions eight times each. Every answer is computed by the script that asks, so the ground truth cannot itself be a mistake, which is the usual way these tests go wrong.
What came back
| Question | Expected | Correct | Distinct answers | What it said |
|---|---|---|---|---|
| 4173 + 2896 | 7069 | 8/8 | 1 | 7069 x8 |
| 218 x 47 − 1996 | 8250 | 7/8 | 2 | 8250 x7, 9250 x1 |
| “s” in Mississippi | 4 | 8/8 | 1 | 4 x8 |
| Weekday of 9 Aug 2026 | Sunday | 8/8 | 1 | Sunday x8 |
| 100 days after 14 Mar 2026 | 2026-06-22 | 7/8 | 2 | correct x7, input date x1 |
| “n” in “unnecessarily” | 2 | 0/8 | 1 | 3 x8 |
| Words in a 20-word sentence | 20 | 1/8 | 3 | 18 x5, 19 x2, 20 x1 |
| Larger: 9.11 or 9.9 | 9.9 | 2/8 | 2 | 9.11 x6, 9.9 x2 |
41 of 64 correct. Four of the eight questions produced more than one distinct answer.
The row that matters is the one with perfect consistency
Look at "n" in "unnecessarily". Eight runs, one distinct answer, zero correct. It said 3 every single time. The word has two.
By any consistency metric that is a flawless result. Ask it twice, get the same answer, conclude it is reliable. It is confidently, stably, reproducibly wrong.
Now compare it with the word count, which returned 18 five times, 19 twice and 20 once. That one is visibly unreliable. You would catch it by asking twice, and you would know not to trust it.
The unstable answer is the safer one, because it announces itself. The stable wrong answer does not, and there were two of those here: the letter count at 0 of 8, and the decimal comparison at 2 of 8 which returned the same wrong answer six times.
Consistency is being used as a proxy for reliability across a lot of writing on this. On this evidence it is not one. It measures whether the model is confident, not whether it is right.
Where the variation actually lives
The two arithmetic-style questions that did vary are worth separating.
The multi-step calculation returned 9250 once against a correct 8250, an error of exactly 1000. The date question returned the input date once instead of the computed one. Both look like a single slipped step rather than confusion about the task, and both were right the other seven times.
The word count is different in kind. Three distinct answers clustered below the truth, mostly 18. That is not a slip, it is a systematic undercount with noise on top, and averaging those eight answers would land you on 18.4 rather than 20. Repetition does not rescue it because the errors are not centred on the right answer.
The two clean failures reproduce what everyone already suspects
9.11 versus 9.9 came back as 9.11 six times out of eight. The decimal comparison problem is well known and it is still there.
Character counting inside a word failed on one case and passed on another: "s" in Mississippi was 8 out of 8 correct, while "n" in “unnecessarily” was 0 out of 8. Same task shape, opposite outcome, and I would not have predicted which way round it went.
What this does not establish
One model, eight questions, eight runs each. These are counts of what happened, not rates for Claude Sonnet 5 or anything else. Eight questions chosen partly because some of them are known-hard is not a representative sample of anything.
Deliberately computational questions. Every answer is exactly checkable by script, which rules out my ground truth being wrong but also means none of this tests world knowledge, where a stable wrong answer would be much harder to detect and considerably more dangerous.
Nothing here isolates a cause. Sampling temperature, server batching and prompt phrasing have all been offered as explanations for variation, and this trial separates none of them. It only reports what varied and what did not.
The practical version
- Asking twice detects instability, not error. It caught the word count. It would have told you nothing about the letter count.
- A confident, repeated answer is not evidence. The most consistent wrong answer here was returned identically eight times.
- Verify anything a model counts or compares, especially characters within words and decimals against each other. Both failed here and both are trivially checkable by other means.
- If you can compute the answer another way, do that instead of asking. Every failure in this table was a task a two-line script does perfectly.
Method
Eight questions, eight runs each, Claude Sonnet 5 via Claude Code 2.1.226 headless, fresh invocation per run with no shared conversation. Expected answers are produced by the same script that builds the prompts, so the truth is computed rather than asserted. Replies are normalised by extracting the final number, the ISO date, or the weekday name, never by substring matching, and the normaliser was control-tested against eight hand-written reply formats before any run was scored.