A required field was absent. It answered null 24 times out of 24

Re-verified 2026-08-10. The original runs loaded this machine’s user-level CLAUDE.md, which carries nine separate anti-guessing instructions. Since the headline result is a model declining to invent a value, that file could plausibly have produced it rather than the model. The whole trial was re-run in a later session with user-level memory excluded via --setting-sources project, the isolation method described here. The result is identical: 24 of 24 schema-valid and every field correct, customer_vat_number null in all 24, and no decoy value returned. The finding is the model’s, not the machine’s.

Every guide to getting JSON out of a model finishes with the same warning: the output can validate perfectly and still be wrong, so validate the shape and then check the values. It is good advice. None of the guides shows how often it actually happens.

So I measured both at once. 24 extractions, two models, and the shape was valid and every field correct in all of them.

The schema was built to be failed

Seven fields, each a different trap, extracted from a short invoice:

Field	The trap
total_amount	written £1,284.50, and a thousands separator has broken this site’s own parsers twice
currency	only the symbol £ appears; the enum contains GBP, not a symbol
status	never stated as a status, only as prose: payment received in full
line_item_count	never stated at all, has to be counted
due_date	written 30 June 2026, the schema demands ISO
customer_vat_number	required by the schema and absent from the document
any extra key	additionalProperties: false

The last two are the ones that matter. A required field with no answer in the source is exactly where a model is supposed to invent something plausible, and additionalProperties: false means any helpful extra key fails.

Then I put a decoy next to it

A clean document is a soft test, so half the runs used the same facts wrapped in the sort of noise a real invoice carries: a superseded proforma number, a quotation in USD before the contract was agreed in GBP, an unrelated delivery date, and a VAT number belonging to the supplier rather than the customer the schema asks about.

That noisy arm was written before the first result came back, not added afterwards to rescue a clean one. The correct answer is identical in both.

	Schema-valid	Every field correct
Haiku 4.5, clean	6 of 6	6 of 6
Haiku 4.5, with decoys	6 of 6	6 of 6
Opus 5, clean	6 of 6	6 of 6
Opus 5, with decoys	6 of 6	6 of 6

customer_vat_number came back null in all 24. The supplier’s VAT number, sitting three lines above the invoice reference in half the runs, was never returned. Neither was the superseded invoice number, and the USD figure never displaced GBP.

What that is worth knowing

This was plain prompting, not constrained decoding. No API schema enforcement, no grammar, no retry loop: the schema was pasted into the prompt and the reply was validated afterwards. That is what you get from a CLI or any plain call, and on this task it did not need the machinery.

The absent-field result is the useful one. A required field with no answer available is the classic invitation to hallucinate, and a null is a well-behaved answer that a downstream validator accepts and a human can act on. Twenty-four out of twenty-four is not a rate, but it is not nothing either.

What this does not establish

One document, one schema, seven fields. A schema with nested objects, arrays of items, or twenty fields is a different problem, and long extractions are where the reported failures cluster.

The traps are the ones I thought of. A gap I did not construct would not have shown up. The scorer was proven able to detect a schema-valid falsehood before any run was scored, but that only proves the detector works, not that the task was hard enough.

Counts, not rates. Twenty-four runs on two models from one vendor. Nothing here supports a percentage, and it does not contradict the reported failures in other people’s pipelines; it says this particular shape of task did not produce one.

Nothing here tests a long document. The invoice is a few hundred bytes.

Method

Claude Code 2.1.226 headless, --model claude-haiku-4-5-20251001 and --model claude-opus-5, six rounds per arm, fresh invocation each time. The document and the expected values are written in the same file, so the ground truth cannot itself be wrong.

Shape is checked with jsonschema against a Draft 2020-12 validator. Truth is checked field by field against the expected values, with floats compared to a tolerance rather than by equality. JSON is taken from a fenced block if present and otherwise from the outermost braces, structurally, never by pattern-matching values out of prose.

The scorer was control-tested on eight synthetic objects first, including the case that matters most: an object with an invented VAT number, which must score shape-valid and truth-wrong. Without that, a clean sweep would be indistinguishable from a scorer that cannot report a falsehood.

One control expectation of mine was wrong rather than the code: I had assumed an object with an extra key should count as untrue, when every field it was asked for was in fact correct and the extra key is a shape violation already caught by the validator. The expectation was corrected, not the scorer.

All 24 runs, both documents, the schema and every returned object are published at json-extraction-trial.json. The 24 isolated re-verification runs are at json-extraction-isolated.json.