Planning helped least on exactly the work its advice recommends it for

The two measurements already published here put plan mode against straight auto-accept on a bug with one right answer, and both ended on the same admission: the advice specifies “open-ended prompts”, and this task is not one. That is a fair objection and it is the obvious next experiment, so here it is.

The prompt this time says only that a module is not robust and to improve it. What “robust” means, how many failure modes to cover and how far to go are the model’s call. Success is still objective, because the rubric is hidden from the model and applied afterwards, identically to both arms.

Medians, 7 paired rounds	Auto-accept	Plan then execute	Ratio
Tokens	146,179	334,425	2.29x
Dollars	$0.2910	$0.6600	2.27x
Turns	7	15	2.14x
Tool calls	6	13	2.17x
Wall clock	60.5s	142.8s	2.36x
Rubric	11 of 11	11 of 11	—

No overlap on any of the five cost measures. The worst auto-accept round beat the best plan-then-execute round every time: 212,012 tokens against 276,851, $0.4257 against $0.5660, 9 turns against 13, 8 tool calls against 11, 105.1s against 122.3s. Every one of those ratios is an upper bound, because the plan arm also paid for a broken tool it needed and the other arm did not. That is quantified below, and the separations survive it.

The interesting part is not the ratio

The ratio barely moved: 2.18x on the specified task, 2.27x here. What moved is where the money went.

On the specified task, with the plan in hand, the execute call was 29% cheaper than doing the whole job in one shot, in 8 of 8 rounds with no overlap, and the plan that bought that saving cost about five times it. That split is not clean, and the companion corpus says so in its own limits: in 6 of 8 rounds the plan call ran the failing suite before proposing anything while the auto-accept arm never ran it before editing, so the execute call inherited a round of diagnosis it had not paid for. Both the 29% and the 1.47x are upper bounds on their own effect.

On open-ended work that saving is gone.

	Specified task	Open-ended task
Whole workflow, dollars	2.18x	2.27x
Execute call against one-shot	29% cheaper, 8 of 8, separated (upper bound)	2.7% cheaper, 5 of 7, overlapping
Plan call against the whole one-shot job	1.47x, 8 of 8, separated (upper bound)	1.33x, 5 of 7, overlapping

Median cost per call here: auto-accept $0.2910 for the whole job, the plan call $0.3856, the execute call $0.2833. The execute call is within 3% of just doing it, and two of the seven rounds have it more expensive.

The action records say why, and they say it without interpretation. On the specified task the plan call ran the failing suite in 6 of 8 rounds, and the execute call then skipped discovery entirely, going straight to Read, Edit, pytest in 8 of 8. Here no plan call ran the suite at all: every non-ls command in the plan phase is an interpreter or pytest version probe. And all seven execute calls re-ran ls -la and re-read both config.py and test_config.py, exactly as the auto-accept runs did. There was no diagnosis to inherit and discovery was repeated in full, so the second call paid close to full price.

So part of the collapse is that transfer disappearing rather than planning ceasing to work. That still inverts the advice’s own scoping, because the transfer is what made planning cheaper, and it happened on the task with one right answer, not on the open-ended one. A plan collapses a search. When there is a single correct fix to find, finding it is most of the work and the plan carries it forward. When the work is deciding what to build and then writing a lot of code, the plan does not remove the writing.

What the rubric can and cannot say

Every run in both arms scored 11 of 11, all fifteen of them. So this experiment detects no quality difference, and it is worth being exact about what that means.

The rubric is not a formality. It was control-tested three ways before any model call: the untouched module scores 5 of 11, leaving six genuine failures to fix; a blanket except: return None clears every handling item and fails the distinguishability item, scoring 10; a careful implementation scores 11. So it separates broken from lazy from good, and it would have caught either arm shipping a swallow-everything wrapper or breaking the existing contract.

What it cannot do is rank two implementations that both clear it. Above full marks it is blind. The honest statement is therefore narrow and still useful: on fifteen runs, neither arm ever produced the failure the advice warns about, and auto-accept did not need a plan to avoid it.

Nor did planning produce less code. config.py came out at a median 141 lines from auto-accept and 130 from plan-then-execute, ranges 137 to 187 and 97 to 185. Fully overlapping.

What this does not settle

A rubric both arms max cannot measure quality above its ceiling. Everything above is a cost result plus a floor on quality, not a quality comparison. A harder task, or a rubric with headroom, could still find a difference this one cannot see.

One task, one shape of open-ended. “Make this module robust” is open in scope but narrow in surface: one file, one function, no architecture to choose. The wrong turns plan mode is supposed to prevent are cheapest exactly here. A multi-file feature with a real design decision is a different case and is untested.

ExitPlanMode was unavailable in every plan call, and the plan arm paid for the workaround. Five of the seven plan replies say so outright, all seven spent at least one ToolSearch hunting for it, ten in the corpus and every one inside a plan call, and all seven fell back to writing the plan to a file. The auto-accept arm never touched ToolSearch. In the plan phase each extra tool call costs about 30,473 tokens and $0.0616 here, so removing one from every plan call takes the median workflow from $0.6600 to $0.5984 and the dollar ratio from 2.27x to 2.06x; dropping the dead turns takes the turn ratio from 2.14x to 1.86x and tool calls from 2.17x to 1.83x. Every separation still holds, so the direction is safe and the ratios are upper bounds. The same artifact is present in the specified-task corpus, so the comparison between the two is not distorted by it.

The plan arm carries one genuine outlier. One round’s execute call ran to 334,335 tokens and 13 turns against 147,550 to 174,784 and 7 or 8 turns for the other six, taking that round to $0.98. It is the maximum in its arm, so it is not the median and no figure here rests on it. Removing it makes the plan arm cheaper and the gap smaller, not larger.

Controls

The rubric is hidden from the model and applied afterwards, identically to both arms, so the prompt stays genuinely open-ended while scoring stays mechanical. Every item, every input and every per-run outcome is published.

The rubric was strengthened once, and only before the measured rounds. A single smoke round on a 6-item version had both arms at full marks, so the instrument had no headroom; three further malformed inputs were added and it was not touched again once measuring began. The smoke round is not pooled into any figure. Both facts are recorded in the corpus.

Both arms held the same tool permissions. --allowedTools Bash on every call, asserted onto the command line and proven to have arrived by requiring a Bash tool use in every call’s action record: 22 of 22 calls, one per auto-accept run and two per plan-then-execute run. An earlier version of this experiment was defeated by exactly this asymmetry, so it is now a hard control rather than an assumption. Equal permissions did not produce equal capability, though: the plan arm could not exit plan mode, which is quantified above.

Every figure is computed over rounds where both arms ran. One auto-accept run lost its partner when the harness crashed parsing a stream event whose message field was a string rather than an object. It is published unchanged, marked paired: false, and excluded from every median, ratio and separation claim above. Seven paired rounds remain.

The test file was checksummed before and after every run and was byte-identical in all 15. Isolation on every call, --setting-sources project, asserted. Tokens from modelUsage, not .usage.

Every run, with its full action record and per-item rubric outcome, is published in plan-mode-open-ended.json. The specified-task companion is plan-mode-equal-permissions.json.