AI every minute

A book costs 41% more in Claude Code than tiktoken says

Fri, 21 Aug 2026 00:00:00 GMT

I ran book-to-skill v1.4.0 on the 501-page Pro Git PDF, then measured what the extracted text actually costs in Claude Code context against its tiktoken count. The gap is 1.41x on a technical book and 1.41x on Moby-Dick, and because the two books have different cl100k rates, solving both together puts any per-byte framing charge at or below zero, which bounds an artefact of how Claude Code wraps the file without ruling one out. Anthropic publishes ~15-20% for typical text and an unbounded "much more" for code, and only the novel is the typical-text case; both books measured the same 1.41x regardless, which that split does not predict. It leaves book-to-skill's headline ratio underdetermined rather than refuted, and makes its dollar table low. Running it also surfaced three things a README cannot tell you, including a CLI that reports 93 chapters in a ten-chapter book because it counts shell comments inside code samples.

tool: book-to-skill v1.4.0, commit 3a97a71, 23,485 stars
book: Pro Git, 501 pages, 17.97 MB PDF
this trial: 28 runs, six cells of four rounds plus a floor per round
extraction: 501 pages in 0.55-0.64s
claude vs cl100k: 1.41x on both books
anthropic publishes: typical text 15-20% (1.18-1.25x); code unbounded
every segment: between 1.382x and 1.452x
their CLI vs their docs: 198K against 229K on one book

Measured on Claude Code 2.1.238.

Your skill description is cut at 1,536 characters, and the cap is a setting you can raise

Thu, 20 Aug 2026 00:00:00 GMT

A rare marker planted at a known character offset inside a skill description, then asked for. Text ending at 1,530 characters is answered from context with no tool calls, four rounds of four; text ending at 1,536 is never answered from context, and the model shells out to read the file instead. That lands within six characters of the documented skillListingMaxDescChars default of 1,536. A 51-character increase in the skill name moves it not at all, and the tail past the cut is text you wrote that Claude never sees when deciding whether your skill is relevant. Corrected 2026-08-21: an earlier version of this post framed the cut as undocumented and cited a 1,024-character maximum.

cut between: 1,530 and 1,536 characters
at 1,400: in context 4 of 4 rounds
at 1,550: in context 0 of 4 rounds
documented cap: 1,536, skillListingMaxDescChars default
skill name length: 51 more characters moved nothing
corrected 2026-08-21: the cut is documented, and configurable
cut moves with skill count: no, marker at 1,400 held at 1, 10 and 40
counts characters not tokens: dense text at 2.06x still in at 1,400

Measured on Claude Code 2.1.235.

A trending 817-skill pack costs about 56,600 tokens before you type anything

Thu, 20 Aug 2026 00:00:00 GMT

An 817-skill cybersecurity pack from GitHub Trending costs about 56,600 tokens of startup context in an empty isolated project, 69.30 per skill, from 24 readings of which 21 sit inside a 48-token band. The synthetic skills curve behind this site's own calculator interpolates to 13.46 per skill at that count, 5.15x low, and its 1,000-skill cell still re-measures correctly. Description uniqueness and the pack's extra frontmatter were each tested here and refuted; deleting 338KB of frontmatter moved the reading by 8 tokens. The cause is not established: neither the synthetic nor the real fixture carries its descriptions whole, which rules the comparison out as a test of what the words cost.

pack: mukul975/Anthropic-Cybersecurity-Skills, commit 4c0b700
skills: 817, counted off disk on every arm run
measured: 56,622 tokens, 69.30 per skill
published: 30 rounds, 80 runs, five arms
synthetic curve predicts: 13.46 per skill at 817
extra frontmatter: not charged, +8 tokens on 338KB

Measured on Claude Code 2.1.236.

Anthropic ships five skill plugins, not one, and its own ~100-token figure fits the one I could measure

Thu, 20 Aug 2026 00:00:00 GMT

Anthropic documents ~100 tokens per Skill for the metadata loaded at startup. Its own repository ships five plugins rather than one, and only one of them is large enough for this instrument to give a figure. example-skills, twelve skills with ordinary 288-character descriptions, measured 102 tokens per skill. The three single-skill plugins cannot be measured: two carry ranges 198 tokens wide and the third spans 897, from minus 578 to plus 319, which contains zero. A separate sweep then knocked out the story I had been about to tell about long descriptions: above 500 characters they are close to free. Which turns out not to mean what it sounds like, because a later measurement found the text stops reaching the model entirely past about 1,400 characters.

pack: anthropics/skills, commit f379e5a
documented: ~100 tokens per Skill
measurable plugin: 102 per skill, 288-char median
no figure published: 4 of 5, one range spans zero
description above 500 chars: 0.015 tokens per character

Measured on Claude Code 2.1.234, description sweep on 2.1.235.

7 of 12 runs got the date right. Every miss that showed its working was arithmetic

Mon, 17 Aug 2026 00:00:00 GMT

Six free-tier chat models were each asked twice what date a project finishes after 120 working days from 3 March 2026. Seven of twelve runs landed on a defensible answer. Of the three misses that showed any working, none was a misreading of the question: each chose a correct method and then executed the arithmetic wrongly, including one model that used the same method twice and got two different dates. Four of the five wrong answers contain a statement that is false or impossible on its own terms; the fifth is one of the two that showed no working at all.

correct: 7 of 12
models right both times: 3 of 6
runs that showed a setup: 8 of 12, all correct
misses that showed working: 3 of 5, all arithmetic
wrong answers refutable from their own text: 4 of 5
ground truth: computed, rubric fixed before running
surface: duck.ai free tier, no account

A 19,700-star diagram skill drew once in four unattended runs. One file fixes it

Mon, 17 Aug 2026 00:00:00 GMT

A trending Claude Code skill for editorial diagrams, installed and measured on 2.1.233. What it costs to sit on disk unused turned out not to be measurable: twelve readings of the same pack span 70 to 967 tokens. Run unattended on an ordinary request it delivered a file in 1 of 4 rounds, spending 733,824 tokens across the three that produced nothing, because SKILL.md tells the agent to pause and ask about brand colours before the first diagram in a project. Dropping a one-line .diagram-design marker in the project root took it to 4 of 4. No cost multiplier is quoted, because the plain arm alone varied 6.7x and the skill's arm has one delivering round. The marker effect is established at p=0.018; the skill-versus-plain comparison is underpowered and is not claimed.

pack: cathrynlavery/diagram-design, commit 09df49d
stars: 19,728 on 2026-08-17
installed, never used: not measurable, 70 to 967
delivered a diagram: 1 of 4 unattended
with the marker file: 4 of 4
spent producing nothing: 733,824 tokens
type references read: 2 to 3, never 1

Measured on Claude Code 2.1.233, re-verified on 2.1.234.

"A 5,000-token CLAUDE.md is a 5,000-token tax per turn." Half of that is true.

Sat, 15 Aug 2026 00:00:00 GMT

An 18KB CLAUDE.md adds 4,967 tokens to every turn of a session on Claude Code 2.1.233, and it never decays. But it costs $0.04966 and $0.04967 on turns one and two and $0.00248 on turns three and four, because the file stops being written to cache and starts being read from it. The context claim is true and the cost claim is true for about two turns.

CLAUDE.md: 18,000 bytes, 4,967 tokens
added to every turn: 4,967, all 4 of 4 turns
cost, turns 1-2: $0.04966 then $0.04967
cost, turns 3-4: $0.00248 per turn
drop: 20x
on 2.1.232: 4,969 tokens, 2 apart

Measured on Claude Code 2.1.233.

google/skills costs 9,020 tokens. Every session, before you use any of it.

Sat, 15 Aug 2026 00:00:00 GMT

Six trending skill packs, cloned and installed on Claude Code 2.1.233, measured against an empty floor, then re-measured after I found I had priced one of them wrong. 111 skills cost 9,020. The 25 skills Matt Pocock actually ships cost 852, not the 1,209 I first published for 35 files sitting in his repository. Per skill that is 34 to 96 tokens, and multiplying a published rate by a file count would have got two of the three badly wrong.

clean floor: 22,283 then 22,287, each round paired
google/skills, 111 skills: 9,020 tokens
addyosmani/agent-skills, 24: 2,302 tokens
mattpocock/skills, 25 shipped: 852 tokens, corrected from 1,209
per skill: 34.1 to 95.9
my own model: 8% low, 63% high, 64% high
re-verified 2026-08-18: diagram-design 70 to 967 on 2.1.234

Measured on Claude Code 2.1.233.

The docs say ~100 tokens a skill. I measured 34 to 96.

Sat, 15 Aug 2026 00:00:00 GMT

Anthropic documents ~100 tokens per Skill for the metadata loaded at startup. Measured across the three trending packs where a measurement is reliable, per-skill cost ran 34.1 to 95.9, all below the documented figure, over-predicting by 4% on one pack and 193% on another, and the size of the miss does not order by description length. Then I asked six model runs the same question: three repeated the documented number, one gave a different one, and two declined, including the model that had given a different one moments earlier.

documented: ~100 tokens per Skill
measured, 3 packs: 34.1 to 95.9 per skill
worst over-prediction: 193%, mattpocock/skills
closest: 4%, addyosmani/agent-skills
source runs: 56, two harnesses
model runs: 6, four models, three vendors
re-verified 2026-08-18: one-skill pack 70 to 967 on 2.1.234

Measured on Claude Code 2.1.233.

400 commands cost 9,989 tokens. 400 agents cost 20,404.

Fri, 14 Aug 2026 00:00:00 GMT

One widely shared write-up prices a plugin at about 100 words of metadata each and tells you to keep four to six. Measured on 2.1.231, a plugin has no single price: a skill or a slash command costs about 40 tokens and collapses in bulk, while a subagent definition costs 51 and never does.

re-verified on 2.1.232: reproduced, within one 198-token component
a slash command: 40.0 tokens, then 5.6
a subagent definition: 51 tokens, never less to 400
400 commands: 9,989 tokens, or 10,187 when the component fires
400 agents: 20,404 tokens, or 20,602
re-verified on 2.1.238: 8 rounds, marginal rates reproduced
200 commands beside 40 skills: 25.7 tokens each
200 agents beside 40 skills: 39.0, unchanged

Measured on Claude Code 2.1.231.

The same forty skills cost 1,558 tokens, or 152

Fri, 14 Aug 2026 00:00:00 GMT

A skill is not a fixed price. On a clean project the marginal cost falls from 39.0 tokens to 3.9 between 200 and 400 skills; on a configured machine it is 3.8 straight away. That is where this site measured 1.5 tokens a skill from, and why that figure never reproduced in isolation.

clean project, below 200 skills: 39.0 tokens each
clean project, past 400 skills: 3.9 tokens each
40 skills, clean project: 1,558 tokens
40 skills, config loaded: 152 tokens
description, at 40 skills: 0.276 per char, to 300 chars, English prose only
description, at 600 skills: 300 chars costs no more than 30
body size, the control: no effect across 140x
re-verified on 2.1.233: unchanged, 39.05 per skill at 200

Measured on Claude Code 2.1.231.

A 7.9MB node_modules cost 2 tokens, and .claudeignore does nothing

Thu, 13 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.229. A lean project cost 22,401 tokens at startup in four runs out of four, identically. The same project carrying 7.9MB and 1,960 files of node_modules cost 22,403. Across 24 runs, including 12 that invited exploration, node_modules was read zero times. A .claudeignore excluding it did not stop a file inside it being read, 3 of 3, while the documented deny rule blocked it 3 of 3.

lean project, startup: 22,401 tokens, 4 of 4 identical
plus 7.9MB node_modules: 22,403 tokens (+2)
node_modules read: 0 times in 24 runs
.claudeignore blocks a read: no, canary returned 3 of 3
deny rule blocks a read: yes, 3 of 3
.claudeignore in the docs: 0 of 193 pages

Measured on Claude Code 2.1.229.

The 150-rule limit for CLAUDE.md measures something a CLAUDE.md is not

Thu, 13 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.229 across 35 runs, no exclusions. A rule at the top of the file and a rule at the very bottom were both obeyed in 15 of 15 function definitions at 2, 10, 50, 150, 300 and 500 rules. The negative control scored 0 of 15 on both. This does not refute the benchmark behind the 150 figure, which measures a different thing. What 500 rules did cost was 23,427 tokens of context.

rule counts tested: 2, 10, 50, 150, 300, 500, plus a control
top rule: 15 of 15 at every count
bottom rule: 15 of 15 at every count
negative control: 0 of 15 on both rules
what 500 rules cost: 23,427 tokens of context
functions written: 3 of 3 in 35 of 35 runs

Measured on Claude Code 2.1.229.

Fanning out was about 1.7x slower on work built to favour it

Thu, 13 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.229 across 24 runs with no exclusions. Eight agents took 29.6s against 16.8s inline, a 12.8-second gap; four took 20.0s against 11.7s. Tokens ran 1.82x and 2.17x. Every arm is separated. The delegation control confirms the parent made zero file operations in all twelve fan-out runs, which took a field called parent_tool_use_id to establish.

eight agents: 29.6s against 16.8s inline
four agents: 20.0s against 11.7s inline
wall clock ratio: 1.76x and 1.72x, no overlap
tokens: 1.82x and 2.17x, no overlap
parent file operations: 0 in all 12 fan-out runs
markers found: 24 of 24

Measured on Claude Code 2.1.229.

Halving the output saved 6.8% of the bill

Thu, 13 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.229 across 30 runs in three batches, no exclusions. Output is 1.45% of billed tokens and 10.2% of cost, seven times its token weight, because the input side is mostly cache reads at a tenth of the output rate. A terseness instruction halved the output and saved 6.8%. Every terse run cost less than every default run.

reply length: 4,733 chars to 1,969
output tokens: 1,821 to 842 median, a 54% cut
what it saved: $0.4489 to $0.4184, 6.8%
output share of tokens: 1.45%
output share of cost: 10.2%
arms separated: on output and on cost
markers echoed: 30 of 30, all three turns

Measured on Claude Code 2.1.229.

Forcing the cache expiry everyone warns about made the session 7.6% cheaper

Thu, 13 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.229 across 24 runs with no exclusions. After an eight-minute break the next turn cost $0.017 on the default cache and $0.118 with the five-minute TTL forced on, 6.8x. Summed over all three turns the same forced arm cost $0.370 against $0.400, and every forced run beat every default run. It wrote 50% more cache tokens and still cost less. A separate bracket found the default cache alive at 55 minutes and gone by 70.

turn after an 8-minute break: $0.017 default, $0.118 forced 5m
whole 3-turn session: $0.400 default, $0.370 forced 5m
session arms: separated, 5 of 5 each way
cache written, forced arm: 52,731 tokens against 35,139
write tier, default: 1-hour tier, 0 to the 5-minute tier
kept at 55 min, lost at 70: 3 of 3 each way
markers echoed: 24 of 24 across both runs

Measured on Claude Code 2.1.229.

The third step cost less than the first, in the session that was never cleared

Thu, 13 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.228, ten rounds per arm, on three steps where each builds on the last. Keeping one session cost 274,436 tokens against 344,408 for clearing between every step, with the fresh arm dearer in 10 of 10 paired rounds. Both lose to asking once: the worst single-prompt run beat the best run of either multi-prompt arm on every measure. All 30 runs passed the hidden suite.

rounds: 10 per arm, 30 runs on 2.1.228
one prompt: 103,828 tokens, $0.1697
one session, three prompts: 274,436 tokens, 2.64x
fresh session each step: 344,408 tokens, 3.32x
fresh dearer: 10 of 10 paired, sign p=0.0010
resumed step 3 vs its own step 1: -25,312 tokens, 10 of 10
success: 42 of 42 passed, both versions
re-verified: 2.1.229, every claim held

Measured on Claude Code 2.1.228.

A deny rule on .env leaked nothing in 20 runs, and 14 of 24 got through another way

Thu, 13 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.229 across 88 runs with a fabricated credential as the canary. With permissions.deny Read(./.env) in place, the Read tool, cat, head, sed and grep leaked 0 times in 20. A python one-liner, a node one-liner and dd leaked 14 times in 24. Every result matches the documented boundary; what the documentation does not convey is that the uncovered side has no enforcement, only judgement.

rule tested: deny: Read(./.env)
Read tool, cat, head, sed, grep: 0 of 20 leaked, 18 blocked at the layer
python, node, dd: 14 of 24 leaked
control, no rule at all: 20 of 20 and 20 of 24 leaked
declined with no rule present: 4 runs
matches the documentation: yes, boundary and mitigation

Measured on Claude Code 2.1.229.

Compaction kept all 64 planted values, across 24 compactions

Wed, 12 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.228. Eight unguessable values were planted at spaced turns, the session was driven past the compaction threshold, and every value was asked back with files off-limits. 64 of 64 survived across 24 compactions and two bulk types. This tests recall of discrete stated values, which is not the same thing as preserving nuance, and the post says so.

values recalled: 64 of 64
compactions survived: 24
dense arm: 32 of 32, 4 compactions per run
sparse arm: 32 of 32, 2 compactions per run
dense bulk compressibility: 25.03% vs 0.79% under gzip -9, 31.7x
positive control: 8 of 8 without compaction
compaction window: forced to 100k

Measured on Claude Code 2.1.228.

Planning made the execution cheaper, and cost more than the job it was planning

Wed, 12 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.228. Plan-then-execute and straight auto-accept were run four times each on the same bug fix with an objective pass/fail. Both passed 4 of 4. Planning genuinely made the doing cheaper, $0.1685 against $0.2062, but the plan call cost $0.2301 on its own, six times the saving it bought. The workflow cost 1.69x the tokens and 1.95x the dollars.

auto-accept, one call: $0.2062 median
execute call, plan in hand: $0.1685, cheaper 4 of 4
plan call alone: $0.2301, dearer 4 of 4
whole workflow: 1.95x dollars, no overlap
turns: 9 against 15
success: 4 of 4 in both arms
known limit: bash denied outside plan mode
corrected 2026-08-12: withdrew the "mechanism is backwards" claim and the title asserting it
corrected again, same day: withdrew "upper bounds"; the equal-permission re-run is larger

Measured on Claude Code 2.1.228.

Planning helped least on exactly the work its advice recommends it for

Wed, 12 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.228 over seven paired rounds, with an open-ended prompt and a hidden rubric applied afterwards. Plan-then-execute cost 2.29x the tokens, 2.27x the dollars, 2.14x the turns and 2.17x the tool calls, with no overlap on any of them. Every run in both arms scored full marks on the rubric, which is a real limit on what the quality comparison can say, and the plan arm paid for a broken exit tool, which makes the ratios upper bounds.

paired rounds: 7, bash control 22/22
auto-accept: $0.2910, 146,179 tokens
plan then execute: $0.6600, 334,425 tokens
dollar ratio: 2.27x, no overlap
tool calls: 6 against 13, no overlap
rubric: 11 of 11 in all 15 runs
execution saving: 29% specified, 3% open-ended

Measured on Claude Code 2.1.228.

The plan cost 1.47x the whole job, and made the job 29% cheaper

Wed, 12 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.228, eight runs per arm, with --allowedTools Bash on every call in both arms and a full tool-call record behind every turn count. Plan-then-execute cost 2.01x the tokens and 2.18x the dollars of straight auto-accept, with no overlap. Split by call: the execute call is cheaper than doing it in one shot in 8 of 8, and the plan call alone is dearer than the entire one-shot run in 8 of 8.

auto-accept, whole job: $0.1493, 114,268 tokens
plan call alone: $0.2192, dearer 8 of 8
execute call, plan in hand: $0.1057, cheaper 8 of 8
whole workflow: 2.18x dollars, no overlap
tool calls: 5 against 9.5, no overlap
success: 8 of 8 in both arms

Measured on Claude Code 2.1.228.

The same fan-out costs 53% more, about a third of the time

Wed, 12 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.228 across 26 runs. An eight-agent fan-out has two execution modes separated by num_turns: nine turns costs 207,102 tokens, one turn costs 317,786, for the same eight files. It happened in 3 of 10 runs, which means a single measured figure for per-agent cost is not one number.

cheap mode, 8 agents: 207,102 tokens (turns=9)
expensive mode, 8 agents: 317,786 tokens (turns=1)
difference: 1.534x, +53.4%
frequency: 3 of 10 runs
overlap between modes: none
per agent, cheap mode: 20,293 to 20,325 tokens
per agent, expensive mode: 36,105 to 36,137 tokens
corrected 2026-08-22: four phantom medians replaced with observed ranges

Measured on Claude Code 2.1.228.

Each parallel subagent costs 20,398 tokens, or 92,304 with your config loaded

Tue, 11 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.227 across 44 runs, of which 4 sit behind the configured figure. Cost per parallel subagent is flat at about 20,400 tokens on a minimal config, so eight agents cost 3.2x one rather than 8x. On a machine carrying 37,808 tokens of user-level configuration, the per-agent cost rises to 92,304, which is about twice that configuration rather than most of it. The 75,466 figure this post originally used was the same quantity counted twice; see the correction. Re-measured on 2.1.231 the same slope is 20,690 and 103,397, about twice the configuration again.

re-verified on 2.1.232: unchanged at matched turn count
per parallel subagent: 20,398 tokens
per subagent, configured machine: 92,304 tokens (4 runs)
8 subagents vs 1: 3.20x
8 subagents vs inline: 4.44x
same work inline: 299 tokens each
corrected 2026-08-12: the 288,040 outlier was a second execution mode, 3 of 10 on 2.1.228
corrected 2026-08-13: the config figure was double-counted; 37,808 not 75,466

Measured on Claude Code 2.1.227.

The env var sold as a 90% saving cost 8% more

Tue, 11 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.227 across 28 runs. The flag is real and it engages in print mode, but on both a cold and a genuinely warm parent it increased cost rather than reducing it, because each child inherits the parent conversation instead of skipping the prefix. Every flag-on run used more context than every flag-off run; cache creation barely moved and 97% of the increase arrived as cache reads.

cold session cost, 4 agents: +8.1% with the flag on
warm session cost, 4 agents: +15.6% with the flag on
cold total context: +38.2%
warm total context: +65.6%
cache creation change: +1.7% to +5.4%
per agent, flag off: 92,525 tokens
per agent, flag on: 135,232 tokens

Measured on Claude Code 2.1.227.

A CLAUDE.md rule held for 270 turns and 136,000 tokens

Mon, 10 Aug 2026 00:00:00 GMT

Context rot is described everywhere and measured almost nowhere. A CLAUDE.md convention that pushes against the language default held on both models at every context size reached, and the function names were parsed rather than pattern-matched so a single ambiguous word could not be scored as a failure.

scorable turns: 270
violations: 0 of 270
Haiku 4.5 context reached: 136,392 tokens
Opus 5 context reached: 100,832 tokens
ambiguous names excluded: 11
turns excluded, rate limit: 74
re-verified 2026-08-10, isolated: 80 of 80, to 41,072 tokens

Measured on Claude Code 2.1.226.

A required field was absent. It answered null 24 times out of 24

Mon, 10 Aug 2026 00:00:00 GMT

A schema with seven deliberate traps, extracted by plain prompting rather than constrained decoding. Shape and truth were scored separately so the gap the guides warn about would be visible. It did not appear once.

extractions: 24
schema-valid: 24 of 24
every field correct: 24 of 24
absent field invented: 0 of 24
decoy value returned: 0 of 24
scorer control-tested: 8 of 8
re-verified 2026-08-10, isolated: 24 of 24, unchanged

Measured on Claude Code 2.1.226.

AI tests killed 35 of 36 planted bugs. Four suites never ran at all

Mon, 10 Aug 2026 00:00:00 GMT

Mutation testing on generated pytest suites, with each mutant verified to be catchable before it counted. Opus produced ten usable suites out of ten and killed every mutant. Re-verified under isolation the caught-bug result held and improved to 38 of 38; the claim that Haiku failed on the same two functions every time did not.

suites generated: 20
mutants killed: 35 of 36
Opus 5 usable suites: 10 of 10
Haiku 4.5 usable suites: 6 of 10
test cases per suite, Opus: 70 to 875
test cases per suite, Haiku: 8 to 13
corrected 2026-08-21: phantom medians 196 and 11 withdrawn
re-verified 2026-08-10, isolated: 38 of 38 mutants killed
corrected 2026-08-10: per-subject failure claim withdrawn

Measured on Claude Code 2.1.226.

Haiku got XXE wrong every time. Opus used defusedxml every time

Mon, 10 Aug 2026 00:00:00 GMT

The circulating figure is that 44% of AI code generation tasks introduce a vulnerability, from corpora nobody can inspect. Here are six classes, both prompts and every generated file published. SQL injection never appeared once. XML external entities appeared in three runs out of three.

files generated: 36
flagged by the scanner: 4 of 36
XXE, Haiku 4.5: 3 of 3 vulnerable
XXE, Opus 5: 0 of 3
SQL injection found: 0 of 6
scanner control-tested: 6 of 6 classes

Measured on Claude Code 2.1.226.

CLAUDE.md stopped Opus adding docstrings 8 of 8. The Haiku half did not hold

Mon, 10 Aug 2026 00:00:00 GMT

The advice everywhere is to put a "never add comments" rule in CLAUDE.md because the model ignores you otherwise. The original Haiku figures here did not reproduce and have been withdrawn; a re-run the same day produced the opposite result. The Opus finding, that the rule removed a docstring from every file, is unaffected.

generations: 86
inline comments seen: withdrawn, did not reproduce
Opus 5 file writes, no rule: 8 of 8 had a docstring
Opus 5 file writes, with rule: 0 of 8
Haiku 4.5, either way: withdrawn, did not reproduce
corrected 2026-08-10: Haiku control re-ran as 8 of 8
detector positive control: 10 comments, 1 docstring

Measured on Claude Code 2.1.226.

The loud prompt injection failed 30 times. The quiet one worked 6 of 6

Mon, 10 Aug 2026 00:00:00 GMT

A benign marker payload, five placements, two models, and an arrival control proving the file was read on all 42 runs. The obvious injection was caught in all 30. Reworded as an ordinary editorial note, the same instruction got through the cheaper model 6 times out of 6, silently.

obvious payload runs: 30
obvious payload obeyed: 0 of 30
quiet payload, Haiku 4.5: 6 of 6 followed
quiet payload, Opus 5: 0 of 6 followed
arrival control failures: 0 of 42
scorer bugs caught: 2
re-verified 2026-08-10, isolated: unchanged, 0 of 20 obeyed

Measured on Claude Code 2.1.226.

"Think step by step" changed nothing in 96 runs

Mon, 10 Aug 2026 00:00:00 GMT

Three prompt styles, two models, four questions with ground truth computed by the script that asked them. Adding "think step by step" never once turned a wrong answer right, because the models already reason without being told to. Forbidding the working is what broke one of them.

correct overall: 92 of 96
cells CoT improved: 0 of 8
CoT output token cost: 1.5x to 1.7x
opus 5, working forbidden: 0 of 4
opus 5, free to reason: 4 of 4
re-verified 2026-08-10, isolated: direction holds, 0 of 4 became 1 of 4

Measured on Claude Code 2.1.226.

With the control arm controlled, Haiku commented all 16 files

Mon, 10 Aug 2026 00:00:00 GMT

The control arm was running under a machine-level prohibition, so neither arm was rule-free. Isolating it needed a flag that blocks user memory without breaking tools, verified by hook rather than assumed. The corrected result reproduces across two rounds and says the opposite of the original.

files, corrected trial: 32
no rule, files clean: 0 of 16
rule, files clean: 16 of 16
inline comments, no rule: 30
median size, no rule vs rule: 1,592 vs 285 bytes
isolation verified: user=0, project=1

Measured on Claude Code 2.1.226.

720 of 720 table cells read correctly. My scorer was the thing that kept failing

Sun, 09 Aug 2026 00:00:00 GMT

Every value in the test tables was unique, so a misplaced number could be told apart from a misread one. The model got all 720 cells right. The scorer got it wrong three times, always for the same reason, and twice it would have produced a confident and completely false headline.

cells checked: 720
read correctly: 720
misplaced cells: 0
table sizes: 5x4, 10x6, 20x8
scorer bugs caught: 3
currency and negatives: read correctly

A 2MB screenshot costs the same as a 623-byte one: 3,135 tokens

Sun, 09 Aug 2026 00:00:00 GMT

Measured the context cost of reading images at eleven sizes, twice each. A 3,366x range in file size produced a 5-token difference. Above 2000px the cost stops growing entirely, and a retina screenshot costs 1.69x a normal one for exactly the same picture.

1568x1568 image: 3,135 tokens
same image at 2MB: 3,140 tokens
client-side resize cap: 2000 px
retina vs logical: 1.69x
3072x3072 vs 2048x2048: no extra cost

Measured on Claude Code 2.1.224.

A model hashed passwords with raw SHA-256 in 8 of 9 runs. Two extra words fixed it

Sun, 09 Aug 2026 00:00:00 GMT

The widely quoted figure is that 45% of AI-generated code contains vulnerabilities. I tested two textbook cases across three models with an objective checker. One failed badly and consistently, one classic vulnerability never appeared at all, and the fix was two words of prompt.

password, neutral prompt: 8 of 9 insecure
same model, "make it secure": 2 of 2 secure
SQL injection found: 0 of 12
models tested: 3
checker control-tested: 7 of 7
insecure pattern: salt + single SHA-256

A skill costs 1.5 tokens, an MCP tool 15, and a subagent 1.72x the whole task

Sun, 09 Aug 2026 00:00:00 GMT

Skills, MCP servers, subagents and CLAUDE.md do overlapping jobs at wildly different prices: 1.5 tokens, 15 tokens, 1.72x the entire task, and 0.26 tokens per byte on every single round trip. The right choice is usually obvious once you can see the numbers.

one skill: ~1.5 tokens
one MCP tool, deferred: ~15 tokens
one MCP tool, loaded: ~304 tokens
subagent delegation: 1.72x total
CLAUDE.md: 0.26 tokens/byte, always

Measured on Claude Code 2.1.226.

The same 8KB file is 3,519 tokens on Opus and 2,586 on Haiku

Sun, 09 Aug 2026 00:00:00 GMT

Measured the token cost of byte-identical files across Opus 5, Sonnet 5 and Haiku 4.5, and the startup context floor of each. Opus and Sonnet agree to within 15 tokens on every input. Haiku does not agree with either, and the gap depends on what the text is.

8KB prose on opus 5: 3,519 tokens
same file on haiku 4.5: 2,586 tokens
haiku vs opus range: 0.66 to 0.90x
opus vs sonnet agreement: within 15 tokens
context floor spread: 24,070 tokens

Measured on Claude Code 2.1.224.

I gave two AI detectors text from 1859 and 1996. Both called it human, but the newer one scored 23% AI

Sun, 09 Aug 2026 00:00:00 GMT

A hands-on trial of free AI detectors using text that cannot be disputed: an 1859 book, a 1996 internet standard, and two essays generated on the spot. Every verdict is shown. The detectors were right, and the margins say something the pass/fail does not.

texts tested: 4
GPTZero verdicts correct: 4 of 4
Darwin 1859: 100% human
RFC 1958, 1996: 77% human / 23% AI
AI essays: 100% AI, both
detectors tried: GPTZero, QuillBot
re-verified 2026-08-09: model label matters

Claude Code's auto-compaction fails with `too_few_groups` when one turn fills your context

Sun, 09 Aug 2026 00:00:00 GMT

Measured when auto-compaction actually fires and whether it works. It reports a status event carrying compact_result and an undocumented compact_error. Filling the context in one turn produces failed / too_few_groups, reproducibly, which is precisely the case people hit.

single turn, 122,575 tokens: failed / too_few_groups
multi-turn, 115,471 tokens: success
--autocompact range: auto or 100k–1M only
values that disable it: none
visible in json output: no

Measured on Claude Code 2.1.226.

A detector called an 1859 book AI-generated while its own numbers said 85% human

Sun, 09 Aug 2026 00:00:00 GMT

A hands-on trial of GPTinf, a free unlimited no-signup AI detector, using a public domain passage from 1859. The tool contradicted itself three times out of three, and the number most people would read is the one that moved.

text tested: Darwin, 1859
verdict sentence shown: "seems to be AI generated"
its own breakdown: 0% AI, 85% human
contradiction observed: 3 of 3 runs
headline on identical text: 18% then 0%
punctuation-only edit: human 85% to 65%

Eight models, one bug, three rounds: the 32B coder passed every time and the 72B failed every time

Sun, 09 Aug 2026 00:00:00 GMT

A hands-on trial rather than a benchmark: one small JavaScript bug with a trap in it, eight models, three rounds each, every answer executed against the real test suite. The results are not ordered by model size, and the code the models wrote shows exactly why.

models tried: 8
rounds each: 3
Qwen2.5-Coder-32B: 3/3 passed
Qwen2.5-72B: 0/3 passed
DeepSeek V3: 3/3 passed
Llama 3.1 8B: 0/3 passed

Eight tokenizers agree exactly on English. On Hindi they differ by 3.3x

Sun, 09 Aug 2026 00:00:00 GMT

Measured one paragraph and its translations into ten languages across eleven tokenizers, including three Claude models measured empirically. English is unanimous. Everything else is not, and for non-Latin scripts the spread is large enough to change which model is cheaper.

english agreement: 8 of 8 exact
hindi spread: 3.27x
russian spread: 3.70x
spanish, gpt-4o vs opus 5: 103 vs 201
claude latin-script gap: 23 to 36%
identical tokenizers found: Phi-4 = OLMo-2 = GPT-4

Finding one fact in 43,000 words: 10 of 10. Adding three of them up: 8 of 10

Sun, 09 Aug 2026 00:00:00 GMT

A needle-in-a-haystack test at 43,000 words found no position bias whatsoever: the fact was retrieved from 0%, 25%, 50%, 75% and 100% depth every time. A harder version requiring three buried facts to be added together failed twice in ten runs, and both failures were arithmetic, not retrieval.

document length: 43,000 words
single fact, all depths: 10 of 10
three facts combined: 8 of 10
decoy codes in document: 12
depths tested: 0, 25, 50, 75, 100%
failure type: arithmetic, not retrieval

A free AI detector switched to an older model when my scans ran out, then called an 1859 book 100% AI

Sun, 09 Aug 2026 00:00:00 GMT

A hands-on trial that nearly produced a spectacular false finding. GPTZero classified an 1859 public domain passage as 100% AI-generated, with high confidence. The text had not changed. The model that judged it had.

text tested: Darwin, 1859
verdict on Model 4.8b: 100% human
verdict after quota: 100% AI
model after quota: 3.3b / 2025-03-13-base
text changed between runs: not at all
warning shown about the swap: none

I tried to make a free image model misspell a word. It got all six right

Sun, 09 Aug 2026 00:00:00 GMT

Text in AI images is supposed to be the thing that still breaks. I gave FLUX.1-schnell six words to render, from OPEN up to a rare twelve-letter genus, at a fixed seed on a free public Space. All six spelled correctly.

model: FLUX.1-schnell
words attempted: 6
spelled correctly: 6
hardest word: ZEPHYRANTHES
seed: 12345
cost: free, no account

It gave the same wrong answer 8 times out of 8. Consistency is not accuracy

Sun, 09 Aug 2026 00:00:00 GMT

The circulating concern is that models give different answers to the same question. Across 64 runs the more useful finding was the opposite: the most stable answers included some of the most reliably wrong ones, and asking twice tells you nothing about correctness.

correct overall: 41 of 64
questions that varied: 4 of 8
letter count: 0 of 8 correct, 1 answer
9.11 vs 9.9: 2 of 8 correct
ground truth: computed, not asserted

Models hit an exact word count once in 31 tries. Sentences: 17 out of 17

Sun, 09 Aug 2026 00:00:00 GMT

A measured test of the most-repeated prompting workaround. Word counts came in anywhere from 28% short to 52% long. Sentence and paragraph counts were hit exactly in all 17 runs, across every model tested.

exact word counts: 1 of 31
exact sentence counts: 9 of 9
exact paragraph counts: 8 of 8
worst word miss: +52%
models tested: 4
counter control-tested: 7 of 7

Only 6 of 30 AI citations were real, and 11 of the fakes had DOIs that resolve

Sun, 09 Aug 2026 00:00:00 GMT

Citation fabrication measured rather than described: 30 citations from three open models, each DOI resolved against Crossref and DataCite and its title compared to the claimed one. Verifying that a DOI merely exists would have passed 11 fabrications.

citations checked: 30
verified real: 6
niche topic real: 0 of 15
real DOI, invented title: 11
models that refused: 1
verifier: Crossref + DataCite

The same bug fix cost $0.11 on Haiku and $0.93 on Opus

Sun, 09 Aug 2026 00:00:00 GMT

Measured what it costs to actually complete a task rather than what a token costs, by giving Opus 5, Sonnet 5 and Haiku 4.5 the same failing test suite and verifying the fix by running it. Cost to finish varied 8.5x, and the ranking is not the one the price list implies.

haiku 4.5 median cost: $0.109
opus 5 median cost: $0.925
sonnet 5 median tokens: 1,084,272
cost spread to finish: 8.5x

Measured on Claude Code 2.1.226.

Tesseract hit 53% word error at 6px text. Claude vision hit 0%

Sun, 09 Aug 2026 00:00:00 GMT

A ladder of deliberately degraded document images with exact ground truth, read by tesseract and by Claude vision. JPEG quality turned out to be irrelevant. Text height turned out to be everything, with a cliff between 7 and 6 pixels that one engine fell off and the other did not notice.

tesseract at 6px text: 53.3% word error
Claude vision at 6px: 0.0% word error
JPEG quality 5: 0.0% for both
tesseract cliff: between 7px and 6px
unguessable-text control: passed
ground truth: exact, 45-54 words

Trimming CLAUDE.md saves under 4%. Batching your tool calls saves 49%

Sun, 09 Aug 2026 00:00:00 GMT

The advice circulating about Claude Code token costs is mostly unmeasured. Here is each common tip with the measurement next to it: what actually saves tokens, what saves a rounding error, and the two claims the numbers contradict outright.

batching tool calls: saves 49%
trimming an 8KB CLAUDE.md: saves 3.6%
200 MCP tools, default: 3,054 not 60,854
subagent delegation: costs 1.72x
cache tier requested: 1 hour, not 5 min

Measured on Claude Code 2.1.226.

A 28KB CLAUDE.md costs 7,379 tokens and no measurable time

Sat, 08 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.224: adding a 28KB CLAUDE.md produced a median time-to-first-token difference of 178ms, against run-to-run variance ranging from 1,527ms to 8,337ms on identical configurations. A sign test over 20 paired samples gives p = 0.263.

28KB slower in: 13 of 20
sign test p: 0.263
median difference: 178 ms
warm-run spread: 1,527-8,337 ms

Measured on Claude Code 2.1.224.

The same 28KB costs 7,379 tokens in CLAUDE.md and 4 tokens in a skill

Sat, 08 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.224: a CLAUDE.md costs about 0.26 tokens per byte plus 106 tokens of fixed overhead, and that cost lands on every single session. The same bytes parked in a skill body cost 4 tokens until invoked.

28KB in CLAUDE.md: +7,379 tokens
28KB in a skill body: +4 tokens
ratio: 1,845x
CLAUDE.md rate: 0.26 tokens/byte
fixed overhead: 106 tokens
re-verified on 2.1.227: mechanism holds; ratio is fragile
re-verified on 2.1.228: +8,137 vs +5; ratio now ~1,620x, third value in three versions

Measured on Claude Code 2.1.224.

Delegating to a subagent costs 1.72x and saves 6.4%

Sat, 08 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.224: delegation buys a smaller parent context at a large one-off cost. The saving is exactly the content you kept out of the parent, so the trade only works if the session continues long enough to bank it. Also: .usage does not count subagent tokens at all.

inline: 126,825 tokens
via subagent: ~218,000 tokens
cost multiplier: 1.72x
parent context saved: 8,175 tokens (6.4%)
break-even: ~11 round trips

Measured on Claude Code 2.1.224.

Trying a new --effort level costs 40,054 tokens. Switching back is free.

Sat, 08 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.224: changing --effort does not change your context size at all, but the first run at each new level writes about 40,054 tokens of cache at $0.41 against $0.029 warm. Once a level is warm, alternating between levels is free.

first run at a new level: 40,054 tokens created
cost cold vs warm: 13.9x
switching back: 0 tokens
prompt size change: none
trivial task output: 4 tokens at every level

Measured on Claude Code 2.1.224.

Adding one 5-token skill costs 47,544 tokens

Sat, 08 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.224: a warm session reads 66,078 cached tokens and reports $0.033. Change a single character of CLAUDE.md, or add one small skill, and the next session recreates 47,544 tokens of cache and reports $0.485. The file size barely matters; the invalidation does.

warm session: 0 created, 66,078 read
after any change: 47,544 created
cost multiplier: 14.6x
skill added: 5 tokens
invalidation: content-hashed

Measured on Claude Code 2.1.224.

Reading a 6-byte file costs 58,851 tokens

Sat, 08 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.224: a tool call costs approximately your whole current context, regardless of what it returns. File content is only about 0.35 tokens per byte, so on a 40KB read the overhead is four times the file itself. Batching tool calls halves the total.

one tool round trip: +58,851 tokens
3 reads batched: 121,732 tokens
3 reads sequential: 240,073 tokens
sequential penalty: 1.97x
file content rate: 0.35 tokens/byte

Measured on Claude Code 2.1.224.

Resuming a session costs 43,350 tokens once, then 293 a turn

Sat, 08 Aug 2026 00:00:00 GMT

Measured on Claude Code 2.1.224: continuing a conversation adds exactly 293 tokens of cache write per trivial turn, with everything else served from cache. Across six turns the context grew 1,465 tokens while reported cost rose 2%.

first resume: 43,350 tokens created
each turn after: 293 tokens created
cost over 6 turns: +2%
context growth: 293 tokens/turn

Measured on Claude Code 2.1.224.

In Claude Code permissions, deny always wins

Fri, 07 Aug 2026 00:00:00 GMT

Eight headless runs across four settings configurations show a deny rule blocks a tool call regardless of which scope it sits in, even when a higher-precedence file explicitly allows the same pattern, and even inside the same file.

deny lost: never

Measured on Claude Code 2.1.223.

Three ways to write a dead deny rule, one of them silent

Fri, 07 Aug 2026 00:00:00 GMT

Eighteen headless runs across five rule forms. A misspelled tool name warns, a raw-string matcher warns, and a misplaced colon fails completely silently while looking correct.

dead rules found: 4
silent failures: 1

Measured on Claude Code 2.1.223.

Nothing in a project can unset an inherited Claude Code env var

Fri, 07 Aug 2026 00:00:00 GMT

Measured how Claude Code combines the env setting across user, project, local and --settings. It deep-merges across all four, an inherited key cannot be removed from a higher-precedence scope, non-string values are silently coerced by JavaScript string conversion, and OTEL_ variables never reach subprocesses at all.

scopes env merges across: 4 of 4
ways to unset that work: none cleanly
null becomes: the string "null"
{"a":1} becomes: [object Object]
OTEL_ vars reaching subprocesses: none

Measured on Claude Code 2.1.224 (8a2a469b68f9).

Three Claude Code hook fields the docs name wrong

Fri, 07 Aug 2026 00:00:00 GMT

Wiring 15 hook events to a capture script and running two headless sessions turned up three field names that did not match the documentation on 2026-08-07, plus several useful fields undocumented at the time, including per-tool duration_ms. Re-checked on 2026-08-21: every one of those discrepancies has since been fixed in the hooks reference, so this now stands as a record rather than as current guidance.

hook events wired: 15
events captured: 22
field names wrong: 3
rechecked 2026-08-21: all discrepancies now fixed in the docs

Measured on Claude Code 2.1.223.

Subagents load CLAUDE.md invisibly

Fri, 07 Aug 2026 00:00:00 GMT

A general-purpose subagent provably reads your CLAUDE.md, and the hook meant to observe instruction loading never fires for it. If you use InstructionsLoaded to audit what reaches your agents, subagents are a blind spot.

subagents spawned: 5
InstructionsLoaded events: 8
events with agent_id: 0

Measured on Claude Code 2.1.223.

Turning off MCP tool search costs you 17,214 tokens before you connect anything

Fri, 07 Aug 2026 00:00:00 GMT

Measured context cost of MCP tools with tool search on and off, across 5, 50 and 200 tools. Deferring saves about 95%, an MCP tool costs roughly 200 times what a skill costs, and disabling tool search adds 17,214 tokens to an empty session.

200 tools deferred: +3,054 tokens
200 tools loaded: +60,854 tokens
tool search off, no servers: +17,214 tokens
per tool definition: 798 bytes
re-verified on 2.1.224: +17,795 tokens
re-verified on 2.1.233: +14,644 tokens, down 17% from 2.1.232
per-tool curve on 2.1.233: unchanged, 652 loaded and 16 deferred

Measured on Claude Code 2.1.223.

Every measurement re-run on Claude Code 2.1.224: two were identical to the token

Fri, 07 Aug 2026 00:00:00 GMT

Claude Code 2.1.223 became 2.1.224. Re-measuring every figure this site has published shows the deltas are remarkably stable across the version bump, while the absolute baselines drift, which is precisely why paired baselines matter.

figures re-verified: 11
unchanged: 9
moved: 2
MCP deltas: identical to the token
re-verified on 2.1.226: all figures hold
re-verified on 2.1.227: batching 2.0x, mechanisms hold
re-verified on 2.1.228: batching 2.0x, mechanisms hold, ratio moved again
re-verified on 2.1.231: floor +64 tokens, tool-search delta held

Measured on Claude Code 2.1.224.

Deploying a static site to Hostinger shared hosting

Fri, 07 Aug 2026 00:00:00 GMT

Both candidate folders carried a DO_NOT_UPLOAD_HERE marker and the domain resolved to an IP that was not the SSH host. Here is how to find the real document root in one command instead of guessing.

host: shared, no root
node on PATH: none
node available: v24.6.0
time to find docroot: one probe

What 1,000 skills actually cost you: 1,474 tokens

Fri, 07 Aug 2026 00:00:00 GMT

Measured prompt tokens across skill counts from 1 to 1,000, with body sizes from 200 bytes to 28KB and descriptions from 90 to 1,500 characters. Every configuration paired against a baseline measured in the same batch, because the baseline drifts.

1000 skills cost: 1,474 tokens
on disk: 3.9 MB
28KB body cost: +5 tokens
desc length effect: none
re-verified on 2.1.224: 1,461 tokens

Measured on Claude Code 2.1.223.