{
 "measurement": "Does plan mode cost less than letting Claude Code edit straight away",
 "claude_code_version": "2.1.228 (Claude Code)",
 "claim_under_test": "Plan mode 'is more cost-efficient than pure auto-accept on open-ended prompts', because the model 'takes more actions per task when it doesn't pause for approval, increasing both tool call count and session length'.",
 "method": {
  "why_two_calls": "Plan mode cannot finish a task non-interactively: with --permission-mode plan the model produces a plan and stops, and 4 of the 5 plan replies say no ExitPlanMode tool is available. The fair comparison is therefore the workflow the advice recommends: plan, then execute carrying that plan. Cost for that arm is the SUM of both calls, because that is what the user pays.",
  "task": "A three-test pytest suite over one small module. One genuine bug plus a trap: the obvious fix sorts in place, which makes the failing test pass and breaks a different one, so only a non-mutating fix reaches ALL PASS.",
  "success": "objective, the suite must exit 0.",
  "controls": {
   "fixture_discriminates": "Control-tested before any model call: unfixed FAILS, the correct fix PASSES, the in-place-sort trap FAILS. Without that spread a pass is indistinguishable from a rig that cannot fail anything.",
   "test_integrity": "test_work.py is sha256-checksummed before and after every run. A suite the model can edit is not a pass/fail signal. 8 of 8 runs left it byte-identical.",
   "isolation": "--setting-sources project on every invocation, asserted onto the command line.",
   "tokens": "modelUsage, not .usage."
  },
  "redaction": "Model replies are stored truncated to their first 400 characters, so the longer ones stop mid-sentence; 8 of the 15 calls hit that cap. Within the stored text, any path under a home directory is replaced with <redacted-local-path>. Nothing else in the runs is altered, and no measured value (context, output, turns, cost, wall) is affected by either.",
  "known_limits": {
   "permissions_are_not_symmetric": "The largest limit in this corpus. Edits were auto-accepted; bash was not. All 10 calls made outside plan mode (every autoaccept run and every execute phase) report their attempts to run the suite being denied by the permission layer. None of the 5 plan calls reports a denial, and 2 of them show the suite actually run: measured_4_rounds r0 plan quotes pytest output, and the pilot plan says the diagnosis was confirmed by running the suite. The plan phase therefore got empirical diagnosis the autoaccept arm was structurally denied, which inflates the plan arm and truncates the other. Both push the published ratios up, so 1.69x tokens and 1.95x dollars are upper bounds. A re-run with identical tool permissions in both arms is pending.",
   "plan_phase_wrote_files": "3 of the 5 plan calls wrote the plan out to a file of their own, outside the working tree. Plan mode was not inert. Only test_work.py is checksummed, so work.py's state after the plan call is not recorded; the claim that the plan call left it alone rests on the execute replies, which all describe applying the fix themselves.",
   "per_call_fields": "Each call publishes six aggregate counters plus the truncated reply. There is no tool-call list, no timestamps and no per-call wall clock, so the turn counts cannot be audited against an action record."
  }
 },
 "fixture_control": {
  "unfixed_fails": true,
  "correct_fix_passes": true,
  "in_place_sort_fails": true
 },
 "runs": {
  "measured_4_rounds": [
   {
    "arm": "autoaccept",
    "round": 0,
    "status": "ok",
    "suite_intact": true,
    "passed": true,
    "calls": [
     {
      "phase": "execute",
      "context": 161309,
      "output": 1507,
      "turns": 8,
      "cost": 0.1912835,
      "reply_excerpt_400": "I made the fix \u2014 `top_scores` now uses `sorted(scores, reverse=True)[:n]` instead of sorting the caller's list in place, so it returns `[9, 7]` without mutating the input, and `average` is untouched.\n\nI could not run the suite to confirm: both `python3 -m pytest test_work.py -q` and a direct `python3 -c` check were denied by the permission prompt. Approve a run and I'll verify, and then reply DONE"
     }
    ],
    "total_context": 161309,
    "total_output": 1507,
    "total_turns": 8,
    "total_cost": 0.1913,
    "wall": 36.5,
    "pytest_tail": "...                                                                      [100%]\n3 passed in 0.00s\n"
   },
   {
    "arm": "plan_exec",
    "round": 0,
    "status": "ok",
    "suite_intact": true,
    "passed": true,
    "calls": [
     {
      "phase": "plan",
      "context": 150370,
      "output": 2129,
      "turns": 7,
      "cost": 0.22467849999999998,
      "reply_excerpt_400": "ExitPlanMode isn't available in this session's toolset, so here's the plan for approval.\n\n**Diagnosis:** One test fails. `work.py:3` uses `scores.sort(reverse=True)`, which sorts the caller's list in place \u2014 contradicting the function's own docstring.\n\n```\ntest_top_scores_does_not_mutate_caller\nassert [9, 7, 3, 1] == [3, 9, 1, 7]\n1 failed, 2 passed\n```\n\n**Fix** \u2014 one line in `work.py`, swapping th"
     },
     {
      "phase": "execute",
      "context": 138883,
      "output": 1474,
      "turns": 7,
      "cost": 0.15071299999999999,
      "reply_excerpt_400": "I applied the fix \u2014 `work.py:3` now uses `sorted(scores, reverse=True)[:n]`, which builds a new list instead of reordering the caller's.\n\nI could not run the suite to confirm: every attempt to execute pytest was denied by the permission layer (`python3 -m pytest`, `pytest -q`). So I'm not saying DONE, since that was conditioned on a green run I haven't seen.\n\nBy inspection all three tests should p"
     }
    ],
    "total_context": 289253,
    "total_output": 3603,
    "total_turns": 14,
    "total_cost": 0.3754,
    "wall": 69.6,
    "pytest_tail": "...                                                                      [100%]\n3 passed in 0.00s\n"
   },
   {
    "arm": "autoaccept",
    "round": 1,
    "status": "ok",
    "suite_intact": true,
    "passed": true,
    "calls": [
     {
      "phase": "execute",
      "context": 209378,
      "output": 1925,
      "turns": 10,
      "cost": 0.23114050000000003,
      "reply": "Fix applied \u2014 `work.py:3` now uses `sorted(scores, reverse=True)[:n]` instead of `scores.sort(...)`, so the caller's list isn't reordered.\n\nI couldn't verify by running it: every `python3` invocation (pytest and a direct call of the three test functions) was denied by the permission prompt. Approve a `python3 -m pytest` run and I'll confirm the suite green."
     }
    ],
    "total_context": 209378,
    "total_output": 1925,
    "total_turns": 10,
    "total_cost": 0.2311,
    "wall": 38.1,
    "pytest_tail": "...                                                                      [100%]\n3 passed in 0.01s\n"
   },
   {
    "arm": "plan_exec",
    "round": 1,
    "status": "ok",
    "suite_intact": true,
    "passed": true,
    "calls": [
     {
      "phase": "plan",
      "context": 150101,
      "output": 2516,
      "turns": 7,
      "cost": 0.235568,
      "reply": "ExitPlanMode is disabled in this session, so I can't hand off through it. The plan is written to `<redacted-local-path>`:\n\n**Diagnosis:** `work.py:3` calls `scores.sort(reverse=True)`, which sorts the caller's list in place \u2014 contradicting the function's own docstring and failing `test_top_scores_does_not_mutate_caller`. The other two tests ("
     },
     {
      "phase": "execute",
      "context": 187261,
      "output": 1860,
      "turns": 9,
      "cost": 0.1918865,
      "reply_excerpt_400": "Fix applied \u2014 `work.py:3` now uses `sorted(scores, reverse=True)[:n]` instead of `scores.sort(reverse=True)`, so the caller's list is no longer reordered. `average` was already correct and is untouched; `test_work.py` unmodified.\n\nI can't say DONE, because I was not able to verify: every attempt to run the suite (`pytest`, direct `python3 -c`, even `python3 -V`) was blocked as requiring approval. "
     }
    ],
    "total_context": 337362,
    "total_output": 4376,
    "total_turns": 16,
    "total_cost": 0.4275,
    "wall": 76.2,
    "pytest_tail": "...                                                                      [100%]\n3 passed in 0.00s\n"
   },
   {
    "arm": "autoaccept",
    "round": 2,
    "status": "ok",
    "suite_intact": true,
    "passed": true,
    "calls": [
     {
      "phase": "execute",
      "context": 185230,
      "output": 1606,
      "turns": 9,
      "cost": 0.2080755,
      "reply_excerpt_400": "I made the fix \u2014 `top_scores` now uses `sorted(scores, reverse=True)[:n]` instead of the in-place `scores.sort()`, so the caller's list is no longer reordered. `average` was already correct.\n\nI couldn't confirm by running the suite: the Bash calls to run `pytest` (and a plain `python3 -c` check) were denied by the permission prompt, so I have no test output. The change satisfies all three tests by"
     }
    ],
    "total_context": 185230,
    "total_output": 1606,
    "total_turns": 9,
    "total_cost": 0.2081,
    "wall": 34.3,
    "pytest_tail": "...                                                                      [100%]\n3 passed in 0.01s\n"
   },
   {
    "arm": "plan_exec",
    "round": 2,
    "status": "ok",
    "suite_intact": true,
    "passed": true,
    "calls": [
     {
      "phase": "plan",
      "context": 125744,
      "output": 2103,
      "turns": 6,
      "cost": 0.21121250000000003,
      "reply": "The plan is written to `<redacted-local-path>`.\n\n**Summary:** `work.py:3` uses `scores.sort(reverse=True)`, which sorts in place and reorders the caller's list \u2014 contradicting the function's own docstring. Fix is a one-liner: replace the two lines with `return sorted(scores, reverse=True)[:n]`. Same return value, no mutation. `average` and `te"
     },
     {
      "phase": "execute",
      "context": 138720,
      "output": 1072,
      "turns": 7,
      "cost": 0.1402025,
      "reply": "I couldn't run the suite \u2014 the `pytest` commands needed approval and were declined, so the fix is unverified by execution.\n\nThe change in `work.py:3`: `scores.sort(reverse=True)` / `return scores[:n]` \u2192 `return sorted(scores, reverse=True)[:n]`. Same return value, no mutation of the caller's list. `average` untouched."
     }
    ],
    "total_context": 264464,
    "total_output": 3175,
    "total_turns": 13,
    "total_cost": 0.3514,
    "wall": 55.1,
    "pytest_tail": "...                                                                      [100%]\n3 passed in 0.01s\n"
   },
   {
    "arm": "autoaccept",
    "round": 3,
    "status": "ok",
    "suite_intact": true,
    "passed": true,
    "calls": [
     {
      "phase": "execute",
      "context": 185090,
      "output": 1492,
      "turns": 9,
      "cost": 0.20431950000000001,
      "reply_excerpt_400": "I made the fix \u2014 `top_scores` now uses `sorted(scores, reverse=True)[:n]` instead of the in-place `scores.sort()`, so the caller's list is no longer reordered.\n\nI could not verify it, though: every attempt to run the suite (`python3 -m pytest test_work.py`, `pytest -q`) was blocked by the permission system. So I can't honestly reply \"DONE\" \u2014 that would claim a passing run I never saw. Please appro"
     }
    ],
    "total_context": 185090,
    "total_output": 1492,
    "total_turns": 9,
    "total_cost": 0.2043,
    "wall": 27.0,
    "pytest_tail": "...                                                                      [100%]\n3 passed in 0.01s\n"
   },
   {
    "arm": "plan_exec",
    "round": 3,
    "status": "ok",
    "suite_intact": true,
    "passed": true,
    "calls": [
     {
      "phase": "plan",
      "context": 160153,
      "output": 2109,
      "turns": 7,
      "cost": 0.3176755,
      "reply_excerpt_400": "`ExitPlanMode` isn't available in this session, so here's the plan for approval:\n\n**Problem:** `work.py:3` calls `scores.sort(reverse=True)`, which sorts the caller's list **in place** \u2014 contradicting the function's own docstring. `test_top_scores_does_not_mutate_caller` fails because `data` becomes `[9, 7, 3, 1]`.\n\n**Fix \u2014 one line in `work.py`:**\n\n```python\ndef top_scores(scores, n):\n    \"\"\"Retu"
     },
     {
      "phase": "execute",
      "context": 187024,
      "output": 1731,
      "turns": 9,
      "cost": 0.18622950000000002,
      "reply_excerpt_400": "I can't run the suite \u2014 the `python3 -m pytest` call needs approval and is being blocked, so I have not verified it passing.\n\nThe fix is in place (`work.py:3`):\n\n```python\ndef top_scores(scores, n):\n    \"\"\"Return the n highest scores, without disturbing the caller's list.\"\"\"\n    return sorted(scores, reverse=True)[:n]\n```\n\n`sorted()` returns a new list instead of reordering the caller's, so `data`"
     }
    ],
    "total_context": 347177,
    "total_output": 3840,
    "total_turns": 16,
    "total_cost": 0.5039,
    "wall": 68.9,
    "pytest_tail": "...                                                                      [100%]\n3 passed in 0.01s\n"
   }
  ],
  "pilot_separate_session": [
   {
    "arm": "autoaccept",
    "round": 0,
    "status": "ok",
    "suite_intact": true,
    "passed": true,
    "calls": [
     {
      "phase": "execute",
      "context": 161490,
      "output": 1293,
      "turns": 8,
      "cost": 0.1869085,
      "reply": "I fixed `top_scores` to sort a copy (`sorted(scores, reverse=True)[:n]`) instead of mutating the caller's list; `average` already satisfies its test. I could not run the suite \u2014 both the `pytest` run and a direct Python check were denied approval, so this is verified by reading only, not by execution."
     }
    ],
    "total_context": 161490,
    "total_output": 1293,
    "total_turns": 8,
    "total_cost": 0.1869,
    "wall": 28.1,
    "pytest_tail": "...                                                                      [100%]\n3 passed in 0.01s\n"
   },
   {
    "arm": "plan_exec",
    "round": 0,
    "status": "ok",
    "suite_intact": true,
    "passed": true,
    "calls": [
     {
      "phase": "plan",
      "context": 189479,
      "output": 3458,
      "turns": 8,
      "cost": 0.37932049999999995,
      "reply": "ExitPlanMode isn't available in this session's toolset, so I can't call it \u2014 here's the plan for approval instead. It's written to `<redacted-local-path>`.\n\n**Diagnosis (confirmed by running the suite):** 2 of 3 tests pass. The one failure is a genuine bug in `work.py`, not a test issue:\n\n```\nassert [9, 7, 3, 1] == [3, 9, 1, 7]\n```\n\n`top_scores`"
     },
     {
      "phase": "execute",
      "context": 187306,
      "output": 1790,
      "turns": 9,
      "cost": 0.18925999999999998,
      "reply": "I fixed the bug \u2014 `top_scores` now uses `sorted(scores, reverse=True)[:n]` instead of the in-place `scores.sort()`, so the caller's list is no longer reordered.\n\nI can't confirm the suite passes: every attempt to run it (`pytest`, and a direct Python invocation of the three test functions) was blocked pending approval, so I'm not reporting DONE. Approve a test command and I'll verify."
     }
    ],
    "total_context": 376785,
    "total_output": 5248,
    "total_turns": 17,
    "total_cost": 0.5686,
    "wall": 96.6,
    "pytest_tail": "...                                                                      [100%]\n3 passed in 0.00s\n"
   }
  ]
 }
}