Finding one fact in 43,000 words: 10 of 10. Adding three of them up: 8 of 10
The lost-in-the-middle effect did not appear at all for single-fact retrieval, at any depth, with twelve decoy codes in the document. It appeared as soon as the task required combining three facts buried at different depths.
“Lost in the middle” is the best-known long-context finding: bury a fact halfway through a long document and the model misses it, with accuracy said to drop by 30 percentage points or more compared to the start or end.
It is repeated constantly, including in 2026 write-ups claiming it is still real on million-token models. So I planted a fact at five depths in a 43,000-word document and asked for it back.
Single-fact retrieval showed no position effect at all
| Depth of the planted fact | Claude Sonnet 5 |
|---|---|
| 0% (very start) | found, found |
| 25% | found, found |
| 50% (the middle) | found, found |
| 75% | found, found |
| 100% (very end) | found, found |
Ten out of ten. No U-shape, no middle penalty, nothing.
And the document was not made easy. It contains twelve decoy codes for other pump stations, formatted identically to the real one, scattered evenly throughout. Finding a code was not enough; the model had to find the code for station 7 specifically. It did, every time, in six to eleven seconds.
Llama 3.3 70B retrieved it correctly too, at 0%, 25% and 50% depth, at both 11,000 and 43,000 words, before inference credits ran out and ended its run.
Then I made the task require three facts
Same document length, same construction. This time three codes carry a service-hours figure, planted at 10%, 50% and 90% depth, and the question is:
Add together the last service hours for pump stations 3, 7 and 11. Reply with ONLY the total number.
The right answer is 90. Ten runs:
| Result | Count | Replies |
|---|---|---|
| Correct | 8 | 90 |
| Wrong | 2 | 3960, 3390 |
The same model that never once failed to find a single fact got this wrong twice in ten attempts.
The failures are arithmetic, not retrieval
This is the part worth noticing. Both wrong answers, 3960 and 3390, are not “I could not find the codes”. They are numbers in the wrong range entirely, of the shape you get from combining the wrong quantities, while the correct answer of 90 is a sum of 12, 47 and 31.
So the model located the buried facts and then did something wrong with them. On this evidence the weak link at 43,000 words was not attention over long distance, it was holding three retrieved values and operating on them.
That distinction matters because of what it implies about testing. A needle-in-a-haystack benchmark is the easiest possible long-context task, and it is the one nearly everyone runs. It passed here perfectly while a genuinely trivial follow-on step, adding three small integers, failed 20% of the time. If you are choosing a model on the strength of its needle-in-a-haystack score, you are measuring the thing that no longer breaks.
What this does not establish
One model has the full matrix. Claude Sonnet 5 completed all 20 runs. Llama 3.3 70B has four data points before its credit ran out, and DeepSeek V3 has none, because 58,000-token inputs exhaust a free inference allowance very quickly. Two of the three intended models are missing, and that is a gap rather than a result.
Two failures in ten is not a 20% error rate. It is two failures in ten. With numbers this small the honest statement is that the harder task failed sometimes and the easier one never did.
One document, one topic, one length. 43,000 words is long but it is not 500,000. The published claims often concern much longer inputs and multi-hop reasoning across documents, neither of which is tested here.
Above 43,000 words I could not test at all. A 112,000-word document was rejected outright by the hosted endpoint, so the ceiling in this setup is between the two.
A scoring bug worth confessing
My first pass scored the multi-fact task by checking whether the reply contained the string "90". One failure replied 3390, which contains “90” as a substring, and was recorded as a pass.
I noticed because a printed run said FAIL in the log line and PASS in the tally. The fixed scorer extracts the final integer and compares numbers, and was control-tested against seven cases including 3390, 3960 and 190 before rescoring.
Substring matching on numbers is silently wrong in exactly the cases you care about, because a wrong answer that shares digits with the right one is the most likely kind of wrong answer.
Method
A 43,000-word document of 1,000 generated maintenance-log sections, with one target fact planted at a controlled depth and twelve identically-formatted decoy codes distributed evenly. The needle’s presence in the exact text sent was asserted programmatically before every call. Single-fact runs: five depths, two rounds. Multi-fact runs: three facts at 10%, 50% and 90% depth, ten rounds, scored by extracting the final integer from the reply. Claude Sonnet 5 via Claude Code 2.1.226 headless; Llama 3.3 70B via hosted inference until credits were exhausted.