Method

Everything here was actually run

No finding on this site comes from a press release, a documentation page, or another article. A tool was used, a model was asked, an experiment was run, and what came back is what gets reported. If a post describes a failure, that failure happened. If it describes a fix, the fix was applied and the result checked.

Two kinds of post, held to different evidence

Measurements are numbers: token counts, costs, durations, error rates. Token counts are read from the actual session rather than estimated from a pricing table. Durations are wall-clock, including the parts that went badly. Every measurement names the exact version it was taken against, because these are implementation details that move between releases.

Trials are first-hand use of a tool or a model. These carry narrative and judgement, so they ship their receipts: the exact prompt, the exact output, and the generated artifact where there is one. Where a trial has an objective outcome, a test suite that passes or fails, a DOI that resolves or does not, a word that is spelled correctly or is not, that outcome is machine-checked rather than eyeballed.

An impression is labelled an impression. It never gets dressed up as a measurement.

The checking is checked first

Any script that decides whether a result passes is tested against known-good and known-bad inputs before it is allowed to score anything real. A guard verified only against the case that passes proves nothing.

Where an experiment depends on some input reaching a model, a control confirms it actually arrived rather than assuming it did. More than one clean-looking dataset here has turned out to be measuring nothing at all, and that is how it was caught.

Findings are reproduced across at least two runs before publication. A single sample of a non-deterministic system is not a result, and a small number of runs is reported as what happened rather than converted into a rate.

What does not get published

• Numbers that were not measured.
• Claims that did not reproduce.
• Rewritten press releases or documentation summaries.
• Techniques for bypassing a security control. Where testing turns up a way to defeat one rather than a way to configure it correctly, that goes to the vendor, not here.

Publishing nothing is an acceptable outcome for an experiment. Several runs have produced a genuine result that was not publishable, and several have produced a negative result that was published precisely because it contradicted what everyone repeats.

Corrections

If something here is wrong, it gets corrected in place with a dated note saying what changed. Silent edits defeat the point. A correction is not finished until the prose, the structured data, the feed and the index all carry it, because a superseded number left in a machine readable field is still being read by something.

Mistakes made while producing a finding are reported alongside it when they are instructive, including the ones that nearly produced a wrong headline.

To send one, use the "Found a number that is wrong?" form at the foot of the article it concerns. It has to be the article's own form, because a correction is only useful when it says which figure it disputes.

Independence

This site is not affiliated with any of the vendors whose products it tests, and nothing here is sponsored, commissioned or reviewed by them. Findings are published whether or not they are flattering, and the tools that come out well are named for the same reason as the ones that do not.