# AI every minute > First-hand trials of AI tools and models, with the receipts. Everything here was installed and run: what it cost, what broke, and where a published claim did not survive being checked. Nothing here is a summary of someone else’s claim. First-hand trials of AI tools and models, produced by actually running the thing and recording what happened, rather than restating what its documentation or its vendor claims. Trials are run against inputs whose correct answer is known in advance, so results are machine-checked rather than judged. Every finding below came off a real run, was reproduced across at least two runs before publication, and names the exact version tested. Negative results are published as readily as positive ones. This site is not affiliated with any vendor whose products it tests, and nothing here is sponsored or reviewed by them. The site is organised as 3 verticals: - Tools (https://aieveryminute.com/tools/): AI tools installed, run and measured first-hand, including what they cost you and what they do badly. - Claude Code (https://aieveryminute.com/claude-code/): What the tool actually costs and how it actually behaves, measured against its own documentation. - Models (https://aieveryminute.com/models/): The same method pointed at the models themselves: what each one costs to run, and where they measurably differ. These are implementation details and they move between releases. Always check the version a figure was measured against before relying on it. ## Most consequential findings Everything published is listed further down. These are the findings that change what someone should actually do, ranked by that rather than by recency. Each one is a measured result with its full run linked. ### 8 of 8 — Given a task that never mentioned a command line, Claude Code built one in 8 of 8 runs https://aieveryminute.com/ponytail-architectural-2-1-243.json The task was to read a CSV of orders and report revenue per customer. It never mentioned a command line. Claude Code with no instructions built an argparse interface in 8 of 8 runs, wrote two files every time, and came in at 73 to 88 lines. With a standing terseness instruction installed through a SessionStart hook, the same task produced an argparse interface in 0 of 8 runs and 37 to 60 lines. The arms do not overlap, with a gap of 13 lines. Eight rounds per arm, none excluded, and an arrival control confirmed the instruction reached the model in every round. The instruction is a third party plugin and the point here is not the plugin: it is that the DEFAULT adds structure nobody asked for, reliably enough to separate cleanly across sixteen runs, and that saying what you do not want changes it. What is NOT shown is that shorter is better. Neither program was run against real data or reviewed for correctness. ### 78% — Crossing the documented hook output cap silently drops 78% of the context you sent https://aieveryminute.com/hook-threshold-2-1-238.json Anthropic caps SessionStart hook output at 10,000 and documents it twice. Forty runs on 2.1.238 measure the price of crossing: 10,016 tokens of context arrive below the cap, identical in all four rounds, against 2,172 to 2,181 above it. The arms do not overlap. The payload is not destroyed, it is written to a file the model must spend a tool call to read, so a plugin that injects standing instructions silently stops delivering most of them. obra/superpowers at 3,334 bytes is well under. The cap counts UTF-16 code units, not characters, so every emoji counts twice; this corpus RETIRES its own earlier characters claim rather than rewording it. ### 1 of 5 — Four of the five AI crawlers this site allows never fetched robots.txt or llms.txt in 14 days https://aieveryminute.com/crawler-consumption-2026-08-25.json Over 14 days and 19 hours a logging shim on robots.txt and llms.txt recorded 278 non-curl requests. robots.txt names seven AI agents but only five of them crawl: Google-Extended and Applebot-Extended are training-use control tokens that make no requests at all, so they are counted separately rather than as absent crawlers, and the crawlers they govern both showed up, Googlebot 26 times and Applebot 38. Of the five that do crawl, only OAI-SearchBot appeared, 50 times. GPTBot, ClaudeBot, Claude-SearchBot and PerplexityBot returned zero, including both crawlers their vendors document as collecting training data. llms.txt was fetched 25 times and exactly one came from any of the five; the rest were ordinary search bots, SEO tools and unidentified clients. The shim serves only those two paths, so a zero means never fetched robots.txt or llms.txt here, not never crawled the site. One young low-authority domain, one window, self-reported user agents. ### 1.41x — A book costs 41% more than tiktoken says, so a tiktoken estimate runs about 29% under https://aieveryminute.com/a-book-costs-41-percent-more-than-tiktoken-says/ The same bytes written as a project-scope CLAUDE.md and read back from modelUsage cost 1.41x what tiktoken cl100k_base counts, on a 501-page technical book and on Moby-Dick, across 28 runs. The published figure is the slope between slices, so the CLAUDE.md framing constant cancels rather than inflating it, and every segment measured falls between 1.382x and 1.452x. The two books have different cl100k rates, so solving both for a tokeniser factor and a per-byte framing charge puts the framing term at -0.0133 tokens per byte, indistinguishable from zero and unable to be positive, which bounds a size-proportional overhead without ruling it out. Anthropic's own guidance puts this gap at 15-20% on typical text, implying 1.18x to 1.25x, and at an unstated 'much more' on code. Only the novel is the typical-text case, and at 29.3% it sits above the published band; the code-heavy book cannot contradict a clause that names no number. What the guidance does not predict is that both books landed at the same ratio, 29.08% and 29.28%, with the prose one marginally higher. A book tiktoken prices at 227,000 tokens is nearer 320,000 of Claude Code context, and any dollar figure computed that way is low by the same margin. ### 56,600 — A trending 817-skill pack costs about 56,600 tokens before you type anything https://aieveryminute.com/an-817-skill-pack-costs-56622-tokens-before-you-type/ mukul975/Anthropic-Cybersecurity-Skills ships 817 Agent Skills and adds about 56,600 tokens of startup context in an empty isolated project, 69.30 per skill, from 24 readings of which 21 sit inside a 48-token band. The synthetic curve behind this site’s own calculator interpolates to 13.46 per skill at that count, 5.15x low, while its real-pack measurements never disagreed. Description uniqueness and the pack’s extra frontmatter were tested and refuted, and deleting 338KB of frontmatter moved the reading 8 tokens, which matches what Anthropic documents. Why a real pack costs so much more than the synthetic fixture is not established. ### 1,536 — Your skill description is cut at 1,536 characters, and the cap is a setting you can raise https://aieveryminute.com/a-skill-description-stops-being-read-at-1400-characters/ A rare marker planted at a known offset inside a skill description, then asked for. Text ending at 1,530 characters is answered from context with no tool calls, four rounds of four; text ending at 1,536 never is, and the model shells out to Bash to open the file instead. That brackets the documented skillListingMaxDescChars default of 1,536 to within six characters, and the cap is a setting you can raise. Nothing errors: the words past the cut are simply absent when Claude decides whether your skill is relevant, so trigger conditions written last silently stop working. A 51-character increase in the skill name moves the cut not at all, and it does not move as more skills are installed, tested at 1, 10 and 40. ### 1 of 4 — A trending diagram skill drew once in four unattended runs. One file fixes it https://aieveryminute.com/a-diagram-skill-drew-once-in-four-unattended-runs/ cathrynlavery/diagram-design carries two pause points that a headless run has nobody to clear, so it delivered a file in 1 of 4 rounds and spent 733,824 tokens across the three that produced nothing. A one-line .diagram-design marker in the project root took it to 4 of 4 across eight further rounds, which is the comparison that reaches significance at p=0.018. What it costs to sit installed and unused is not measurable at all: twelve readings of the same pack span 70 to 967 tokens. ### 7 of 12 — Six models, one date question, and every visible miss was arithmetic https://aieveryminute.com/7-of-12-runs-got-the-date-right/ Six free-tier models from four vendors, asked twice each what date falls 120 working days after 3 March 2026. Seven of twelve runs landed on a defensible answer. All eight runs that showed a route to their answer set the problem up correctly and the wrong ones then slipped on the arithmetic, including one model that used the same method twice and got two different dates. Four of the five wrong answers contradict themselves in their own text. ### ~100 vs 34 — The docs put a skill at ~100 tokens. Three measured packs came in at 34 to 96 https://aieveryminute.com/the-docs-say-100-tokens-a-skill-i-measured-34-to-96/ Anthropic documents ~100 tokens per Skill for the metadata loaded at startup. Across the three trending packs where a measurement is reliable, per-skill cost ran 34.1 to 95.9 and every one came in below it, over-predicting by 4% on one and 193% on another, and the size of the miss does not order by description length. It is NOT a ceiling: this site later measured the example-skills pack Anthropic itself ships, twelve skills with 288-character descriptions, at 102 per skill, above the documented figure. So ~100 over-predicts for most packs and under-predicts for at least one. Measure the pack if the number matters. ### 9,020 — The biggest trending skill pack costs 9,020 tokens on every session https://aieveryminute.com/google-skills-costs-9020-tokens-every-session/ Six Agent Skill collections from GitHub Trending, installed and measured rather than estimated. google/skills is 111 skills and 40% of a clean startup context, paid whether or not you invoke any of them. Per skill it is 34.1 to 95.9, and it does not order by description length: the pack with the longest descriptions is not the dearest one. Check the plugin manifest before pricing a pack: the SKILL.md files in a repository are not all the ones you get. ### 2.04x — Subagent definitions never get cheaper in bulk, and everything else does https://aieveryminute.com/400-commands-cost-10088-tokens-400-agents-cost-20503/ A skill or a slash command costs about 40 tokens and collapses to under 6 once you pass a few hundred. A subagent definition costs 51 at every count measured, so 400 of them cost 20,404 to 20,602 tokens against 9,989 to 10,187 for the same number of commands. ### 49% — Batching your tool calls saves about half a session https://aieveryminute.com/reading-a-6-byte-file-costs-58851-tokens/ Three files read in one batch cost 121,732 tokens. The same three read one at a time cost 240,073, because every round trip re-sends the whole conversation. ### 20,581 — A parallel subagent costs 20,600 to 28,500 tokens before it does anything, and far more with config loaded https://aieveryminute.com/each-parallel-subagent-costs-20398-tokens/ The spread has a cause: a four-way fan-out takes six or seven parent turns and the total moves 16% between them, and which one you get is not under your control. On 2.1.238 six turns is the common case at 20,581 per agent, against 28,504 at seven; at matched turn counts those figures moved 1.0% and 0.9% since 2.1.232. On a configured machine every subagent re-pays about twice your CLAUDE.md and tools. ### 64 of 64 — Compaction kept every planted value, across 24 compactions https://aieveryminute.com/compaction-kept-all-64-planted-values/ Arbitrary values survived auto-compaction intact, including when the surrounding bulk was 31.7x less compressible and competing for room in the summary. Discrete stated facts survive; the loss people report is somewhere else. ### 7,379 vs 4 — The same 28,000 bytes costs 7,379 tokens in CLAUDE.md and 4 as a skill body https://aieveryminute.com/claude-md-costs-1845x-more-than-a-skill/ A skill body only loads when it is invoked, so reference material belongs there. Move it, do not trim it. The gap is the durable part; the headline ratio it produces is not. ### 20x — Your CLAUDE.md is re-sent every turn, and after turn two it costs 20x less https://aieveryminute.com/a-claude-md-is-a-tax-per-turn-half-of-that-is-true/ An 18KB CLAUDE.md added 4,967 tokens to every turn of a session, identical on all four turns with zero spread, so the context cost is permanent. The money is not: the same tokens billed at $0.04966 and $0.04967 on turns one and two and $0.00248 from turn three, because the file stops being written to cache and starts being read from it. Trim it for the window you get back, not for the bill. ### 2.2x — Planning first cost about 2.2x, on a specified task and on an open-ended one https://aieveryminute.com/planning-helped-least-on-open-ended-work/ Plan mode is recommended as the cheaper way to work, specifically on open-ended prompts. It did make the execution cheaper, but only on the task with one right answer, and the plan cost about five times the saving it bought. ### +8% — The env var sold as a 90% saving cost 8% more https://aieveryminute.com/the-fork-subagent-flag-cost-8-percent-more/ CLAUDE_CODE_FORK_SUBAGENT raised total context 38% cold and 66% warm across 28 runs. Cache creation, the thing it should remove, moved upward. ### 6 of 30 — Only 6 of 30 AI-generated citations were real, and 11 fakes had working DOIs https://aieveryminute.com/only-6-of-30-ai-citations-were-real/ Checked against Crossref and DataCite. On a niche topic, zero of fifteen were real. Verifying that a DOI merely resolves would have passed eleven fabrications. ### 8.5x — The same bug fix cost $0.11 on Haiku and $0.93 on Opus https://aieveryminute.com/same-bug-fix-cost-11-cents-on-haiku-93-on-opus/ Cost per token is the wrong unit. Sonnet used 74% more tokens than Opus to do identical work, so most of its per-token discount disappears. ### 8 of 8 — It gave the same wrong answer eight times out of eight https://aieveryminute.com/it-gave-the-same-wrong-answer-8-times-out-of-8/ Consistency is not accuracy. The question it was most stable on is one it never got right, while the visibly unstable answer is the one you actually catch by asking twice. ### 1 of 31 — Exact word counts were hit once in 31 tries. Sentences and paragraphs, 17 out of 17 https://aieveryminute.com/models-hit-an-exact-word-count-once-in-31-tries/ The workaround every guide repeats turns out to be right, across four models. If the number has to be exact, ask for sentences or paragraphs, never words. ### 2.2x per char — A Chinese skill description costs 2.2x per character, and the documented cap counts characters https://aieveryminute.com/skillpack-language-2-1-247.json ConardLi/garden-skills reached GitHub Trending at 11,112 stars with five skills whose descriptions are roughly half Chinese. Nine skills were installed one at a time on 2.1.247, four rounds each, every delta against a floor re-measured in the same round. Per description CHARACTER, descriptions with no CJK cost 0.363 to 0.482 tokens and majority-Chinese ones cost 0.887 to 1.048: the arms do not overlap, and the character-length ranges DO overlap, so length is not the explanation. Per BYTE of UTF-8 the arms OVERLAP (0.363 to 0.482 against 0.459 to 0.513), which is the honest qualification: a Chinese character is about 2.3 bytes, and that accounts for most of the per-character gap. It matters anyway, because the description cap Anthropic documents counts CHARACTERS and this site prices skills per character, so a Chinese description reaches that cap having spent roughly twice the tokens, and the published model, 39 tokens plus 0.276 per description character and declared valid only from 30 to 300 characters, lands 2.48x under on the Chinese arm while landing within 5 percent on the English one. What is NOT supported: that the tokeniser handles Chinese badly per unit of encoded text. Skill BODIES are unmeasured. ### 1.12x — The same CLAUDE.md in Chinese costs 12 percent more than in English, not two to three times https://aieveryminute.com/claudemd-language-2-1-247.json One translation pair, the same project instructions written in English and in Chinese, with byte sizes matching within 1.4 percent while character counts differ by 2.8x. That is what lets a single pair separate the two units. Sixteen self-paired rounds on 2.1.247, floor re-measured every round, one constant working-directory path because this site has measured the startup floor moving with the path. English is 604 tokens at the mode and Chinese 678: 1.12x, or 0.388 against 0.441 tokens per byte. Three predictions were pre-registered before anything ran and all are published. The one that matters most is refuted decisively: that Chinese is CHEAPER because it uses fewer characters predicted 183 to 248 tokens against 678 measured. What is NOT claimed: separated arms. Both arms wander in quantised 198-token steps and their ranges overlap, so 1.12x is a modal difference across 16 rounds, not a resolved per-run effect. One pair, one author, no native-speaker check. ### 6 of 6 — A vendor skill rewrote the old Go idiom in 6 of 6 runs, and tripled the cost of the task https://aieveryminute.com/gomg-skill-idiom-2-1-247.json JetBrains publishes go-modern-guidelines, 1,916 stars, as a Claude Code plugin with a single skill whose body tells the model to run a pinned Go CLI before editing Go code. Two tasks were measured on 2.1.247, paired, same constant working directory. WHAT IT DOES: on a task the tool own ruleset covers, the arm without the skill wrote the pre-1.21 sort.Slice in 6 of 6 runs and the arm with it wrote slices.SortFunc with cmp.Compare in 6 of 6. No run went the other way, and the rule was taken from the CLI own published list rather than invented. WHAT IT COSTS: 45,582 to 45,960 tokens without against 146,181 to 146,872 with, roughly 3.2x, arms not overlapping. It does not inject 100,000 tokens, it adds tool round trips and each tool call re-sends the whole conversation. Sitting installed and unfired it costs only tens of tokens, so the listing cost usually quoted for skills is the wrong number here by three orders of magnitude. On a second task that had no modern idiom to apply, the same 3.2x bought code identical apart from one word of a doc comment, and the skill description fires on any Go edit at all. Neither arm was compiled or tested, and one rule of roughly two dozen was exercised. ### -24.5% — Claude Code startup floor fell 5,594 tokens between 2.1.246 and 2.1.251 https://aieveryminute.com/baseline-floor-2-1-251.json The startup floor is what a session costs before you add anything, and it is the base every other figure here is measured against. Re-measured on 2.1.251: 17,207 tokens with tool search on and 32,400 with it off, against 22,801 and 37,870 on 2.1.246. That is a 24.5 percent fall and the largest move recorded on this site. Eight rounds with every cell measured twice, and both floor arms returned a SINGLE value across all sixteen repetitions, histograms 17207 x16 and 32400 x16. An independent four-round invocation run minutes earlier returned 17,205 and 32,398 with an IDENTICAL 15,193 tool-search delta; it is reported separately rather than pooled because its working directory was one character shorter, and this site has measured that the floor moves with the path, so pooling would have made an exact cell look bimodal. The paired tool-search delta moved only 15,069 to 15,193, so whatever shrank is common to both arms rather than specific to tool search. What is NOT claimed: any cause. Nothing here establishes what Anthropic removed. ### 0 tokens — A 229 KB skill body costs nothing at startup, 8x beyond where this site had tested https://aieveryminute.com/bigbody-2-1-251.json This site published that a skill body is free at any size up to 28KB. mvanhorn/last30days-skill, 60,605 stars and MIT, ships one skill whose SKILL.md is 229,438 bytes: about 8.2x beyond that range, so the claim was being applied outside where it had been measured. Twelve pooled rounds on 2.1.251, floor re-measured every round, skill installed and NEVER invoked. The whole thing costs 101 tokens, which is just the listing: one skill plus a 246-character description. The body contributes zero, and the published model predicts 106.9 against 101 measured. Three predictions were recorded before the run; the site own position held and its validated range now extends 8.2x on a real pack rather than a synthetic fixture. One round read 998 instead of 101 because its FLOOR was 897 low, this machine known intermittent component, and that is published rather than dropped. What is NOT measured: what the skill costs when it FIRES. The tool was deliberately not run, because doing so auto-installs third-party CLIs via npx and can extract browser cookies behind an opt-in. ## Method https://aieveryminute.com/method/ How the numbers are produced and what is never published. ## How to cut Claude Code context cost https://aieveryminute.com/reduce/ The practical guide, derived entirely from the measurements below and ranked by measured impact. What helps: - Keep a SessionStart hook under 10,000 UTF-16 code units (saves the 78% of your payload that otherwise never arrives) - Ask for the whole task in one prompt, not step by step (saves 2.6x on a three-step task) - Batch your tool calls (saves up to 2x the whole session) - Leave MCP tool search on (saves 15,069 tokens minimum) - Count your subagent definitions, not your plugins (saves 51 tokens per definition, and it never falls) - Measure your own skill pack; the documented ~100 tokens a skill is not a ceiling (saves up to two thirds of what the figure implies) - Price a skill pack before you install it (saves up to 9,020 tokens per session) - Move reference material out of CLAUDE.md and into skills (saves 0.26 to 0.42 tokens per byte moved, depending on how the file is written) - Trim CLAUDE.md for the window, not for the bill (saves context on every turn, but almost no money after turn 2) - Batch your config edits (saves 47,544 tokens per edit avoided) - Capture screenshots at logical resolution, not retina (saves 41% of the image cost, for free) - Front-load discovery, then read in one batch (saves one whole context per call avoided) - Keep your fan-out narrow, and trim CLAUDE.md before you widen it (saves 20,581 to 28,504 tokens per agent you do not spawn) - Count a non-Latin skill description in tokens, not characters, against the 1,024 cap (saves roughly half your description budget on a CJK description) - Price a skill by what it DOES when it fires, not by what it costs sitting installed (saves up to 100,000 tokens on a single task) - Price an always-on output-style hook at what it injects every session, not once (saves 1,475 tokens per session if you decide it is not worth it) - Count what a skill pack MANIFEST installs, not the SKILL.md files on disk (saves a 3.1x error in your own estimate before you measure anything) - Extract PDFs to markdown, not plain text, if the document has tables (saves the column-to-value association, for about 9 percent more tokens) - Price a long skill description at 0.363 to 0.482 tokens a character, not 0.276 (saves a 16 percent under-budget on every description past 300 characters) - Budget parallel subagents from a current measurement, not from the release that made your session cheaper (saves the false expectation that a 23 percent floor drop applies to fan-out) - Write a deny rule with two leading slashes for an absolute path, or none at all (saves a rule you believe protects a secret and does not) What does not help, despite being commonly optimised: - Add a .claudeignore so Claude Code stops reading node_modules - Clear the context between tasks to keep costs down - Plan mode is the cheaper way to work - Long conversations get expensive as context grows - Leaving --effort high is quietly expensive - Delegating to a subagent keeps your context small for free - A big CLAUDE.md makes Claude slower - Keep to four to six plugins and you will be fine - Long skill bodies are expensive - Installing lots of skills bloats your context - Long skill descriptions cost more context - Compressing an image before sending it saves context - A cold cache makes the next run slower - Keep CLAUDE.md under about 150 rules or it starts ignoring them - Make Claude Code less verbose to cut your token bill - The five-minute cache expiry is draining your quota - Set CLAUDE_CODE_FORK_SUBAGENT=1 to cut parallel subagent costs by 90% - Rewrite your CLAUDE.md in English, or in Chinese, to save tokens ## What actually changes the answer https://aieveryminute.com/prompting/ Prompting advice checked against the trials here, derived entirely from them. What changed the answer: - Let the model show its working (0 of 4 vs 4 of 4) - Ask for sentences or paragraphs, never an exact word count (1 of 31 vs 17 of 17) - Put standing conventions in CLAUDE.md, including prohibitions (270 of 270) - Run a generated test suite against your working code first (35 of 36) - Treat the model choice as part of the prompt (8.5x) - Verify anything counted, compared, or cited (6 of 30) Widely repeated and not supported by these runs: - Add "think step by step" - Positive instructions beat prohibitions - Ask the same question twice to check the answer - Tell it to reply with only the answer, to keep output clean - A confident, well-formatted answer is a checked answer ## Context cost calculator https://aieveryminute.com/cost/ Estimate startup context tokens for a given number of skills and MCP tools. Per-unit figures, each measured on the release named: an empty session is 17,207 tokens with tool search on and 32,400 with it off (2.1.251); a skill costs about 39.3 tokens below roughly 200 skills (that is the 40-skill cell divided by 40; /cost/ rounds the same quantity to 39) and 3.9 past 400, and its body costs nothing (2.1.251); an MCP tool costs about 16 tokens deferred and about 652 loaded upfront (2.1.251). ## Measure your own project https://aieveryminute.com/measure/ The shell script that produces the figures on this site, published in full so any claim here can be independently reproduced. It measures a project's startup context by diffing it against an empty baseline, and measures its own noise floor first so it never reports a number smaller than its own uncertainty. ## Raw data behind the trials Where a trial produced a transcript, the whole thing is published rather than summarised, so any figure here can be checked rather than trusted. - Skills curve on 2.1.246: unchanged, and one cell that shows why the estimator is a mode: https://aieveryminute.com/skills-cost-2-1-246.json A re-verification that found nothing moved, on the curve the calculator interpolates every skill estimate from. Eighty runs, four rounds, none failed. Ten of the eleven count cells returned the SAME value in all four rounds and every one matches 2.1.238 exactly: 400 tokens at 10 skills, 7,810 at 200, 11,711 at 1,000, with the two-regime collapse from about 39 tokens per skill below 200 to about 3.9 between 400 and 1,000. THE ONE EXCEPTION IS WHY THE ESTIMATOR IS A MODE RATHER THAN A MEDIAN. The 300-skill cell returned [8966, 8966, 8966, 8069], three rounds at 8,966 and one at 8,069, a spread of 897 which is this harness’s known intermittent component failing to fire. Reading only the last round would have reported the curve as changed at that point; the mode is 8,966 and the published value is correct. That intermittent component is NOT retired here, unlike the MCP curve measured the same day where every cell came back single-valued and its noise limit could be retired. The description-length and body-size arms are published as their own cells and never pooled into the count curve, because pooling makes the 40-skill cell appear to disagree with itself. Measured under --setting-sources project from zero pre-existing skills, so it is the clean-project case only. - MCP per-tool curves on 2.1.246: unchanged in all six cells, and one control that failed: https://aieveryminute.com/mcp-curve-2-1-246.json A re-verification that found nothing moved, published because a null on a load-bearing calculator curve is worth as much as a change. Every one of the six cells is BYTE-IDENTICAL to 2.1.238 across eight releases: deferred costs 90, 810 and 3,210 tokens at 5, 50 and 200 tools, loaded costs 3,270, 32,610 and 130,410, and every cell returned the SAME value in all four rounds with zero spread. That zero spread RETIRES a limit the previous corpus carried, which said the tool-search-ON floor had an intermittent component giving every deferred cell a 719-token spread as large as the measurement itself at 5 and 50 tools. THE ISOLATION CONTROL COULD NOT BE RUN AND SAYS SO RATHER THAN BEING OMITTED: its unisolated arm loads this machine’s own configuration, it lost two of four rounds outright, and the two that survived disagreed by 5,271 tokens, so every delta computed against it inherits that movement. What moved that baseline is NOT established and no cause is asserted. The consequence is stated plainly in the corpus: the per-tool figures rest on the isolated arm alone, which is what the calculator uses, and the claim that control would have supported, that per-tool cost is the same whether or not user config is loaded, is NOT carried forward to this release. The fixture is one shape, 798 bytes of a repeated character, so the cross-version comparison is sound while the absolute per-tool figure is fixture-specific. - Startup floor on 2.1.246: the previous release-boundary move reverted: https://aieveryminute.com/baseline-floor-2-1-246.json Four rounds, every cell measured twice, so eight readings per floor arm. Both floor arms came back single-valued with a zero noise floor in every round: 22,801 tokens with tool search on and 37,870 with it off. THE HEADLINE IS THAT THE PREVIOUS RELEASE REVERTED. The 2.1.243 corpus reported the tool-search delta moving to 15,251 after being bit-identical at 15,069 across three consecutive measured releases, and framed that as a boundary that retired an earlier note. It went straight back: this run measures 15,069 again, the exact prior value. So 15,251 was a ONE-RELEASE EXCURSION rather than a new level, and the delta reads 15,069 on 2.1.236, 2.1.238, 2.1.241 and 2.1.246 with a single release out of line between them. What that does not license is a rule: two observations, one out and one back, do not establish that such moves are always transient, and no mechanism is claimed in either direction. Both ends moved DOWN across this boundary by different amounts, on by 158 and off by 340, and their difference still landed exactly on the prior value, which is recorded as an observation and not explained. The delta_series block is computed from every published floor corpus at assembly time rather than typed, so the four-releases-at-15,069 claim is checkable inside the file. The loaded arm reached 64,083 and is deliberately NOT attributed to the release, because it mixes the version change with this machine’s own configuration; it is also the only arm that was not single-valued, mode on 5 of 8. Twelve rows, none excluded. - On real work the lazy-coder plugin did shorten the code, and skipped the CLI 8 of 8: https://aieveryminute.com/ponytail-architectural-2-1-243.json The one-liner corpus at /ponytail-behaviour-2-1-243.json named its own limit: a task with real architecture to choose, which is where a do-not-over-build instruction would matter most, was untested. This runs it, and it REVERSES BOTH HALVES of that finding. Reading a CSV of orders and reporting revenue per customer, eight rounds per arm, 16 usable runs, none excluded. THE ARMS DO NOT OVERLAP: ponytail wrote 37 to 60 lines against 73 to 88, a gap of 13. The single clearest difference is an argparse command-line interface: the baseline built one in 8 of 8 runs and ponytail in 0 of 8, total separation on a feature nobody asked for. And the docstring result INVERTS: on the trivial task ponytail dropped the docstring 8 of 8, here it keeps it 8 of 8, same as the baseline. So the effect is real in both trials but expresses differently. On a trivial task the only thing left to cut is cosmetic, so the docstring goes; on a task with structure the docstring stays and the unrequested CLI is what does not get built. The one-liner was the misleading case. Five rounds were run and then three more before anything was drafted, the step that reversed the headline on the previous task; here it held. Arrival control 8 of 8. What is NOT shown: that shorter is better. Neither program was run against real data or reviewed for correctness, and a 40 percent shorter program that drops a needed guard would score identically here. The baseline is also not a neutral control, it is Claude Code with no instructions at all, which builds a CLI every single time. - A lazy-coder plugin dropped the docstring 8 of 8, but did not reliably shorten the code: https://aieveryminute.com/ponytail-behaviour-2-1-243.json SUPERSEDED IN PART by /ponytail-architectural-2-1-243.json, which runs the harder task this corpus named as untested and reverses both halves of the result below. ponytail claims the best code is the code you never wrote. The cost corpus at /ponytail-session-cost-2-1-243.json measured what it costs to load and said plainly that its actual claim was untested; this closes that line. One small task with room to over-build, a Celsius-to-Fahrenheit function, eight rounds per arm, 16 usable runs and none excluded. THE CLEAN RESULT IS CATEGORICAL: the baseline wrote a docstring in 8 of 8 runs and ponytail in 0 of 8, a total separation. THE LINE COUNTS DO NOT SEPARATE, and that matters because the tempting summary is that ponytail makes the model write less code. It wrote 2 lines in seven of eight runs against a baseline of 3 or 6, but one run came in at 7 and overlaps. That run added assert-based self-checks rather than a demo block, which ponytail’s own ruleset prescribes for non-trivial logic, so it is the tool following a different rule of its own rather than failing. AFTER FIVE ROUNDS THE ARMS DID NOT OVERLAP and that was going to be the headline; three more rounds produced the 7-line run and made the five-round claim false. That reversal is published in the corpus rather than quietly dropped. The algorithm was identical in every run of both arms, so the difference is what was written around the code, not the code. Arrival control: 8 of 8 probes named the active level, so a null could not be confused with a hook that never ran. Whether shorter is BETTER is not measured, and one arithmetic one-liner says nothing about a task with real architecture to choose, which is where such an instruction would matter most. - A trending lazy-coder plugin costs 831 to 1,109 tokens a session: https://aieveryminute.com/ponytail-session-cost-2-1-243.json ponytail was trending at 944 stars on 2026-08-25 and injects a standing ruleset through a SessionStart hook, so it is paid on every session. Its payload measures 5,229 UTF-16 units, comfortably under the 10,000-unit cap established at /hook-unit-2-1-238.json, so the instructions are delivered in full rather than filed. Eight rounds, two readings per cell, 24 usable rows and none excluded. THE ARMS DO NOT OVERLAP: every run with the hook cost 23,854 to 24,302 tokens, every run without it 22,885 to 23,027. The paired per-round difference ranges 831 to 1,109 tokens. NO single number is published, and that is the point: the baseline arm is bimodal, splitting into two clusters about 138 tokens apart, and BOTH of the largest differences are exactly the rounds whose baseline sat in the low cluster. So the spread is largely the baseline’s own intermittency rather than variation in what the hook costs, and a mode over that would describe neither cluster. Four rounds were run first, came back multimodal with the treated cell tied 3-3, and four more were run before anything was drafted; they confirmed the bimodality rather than resolving it. The arrival control is what makes the difference mean anything: each round also asked, with the hook installed, which ponytail level was active, and 16 of 16 replies named a level, so a zero difference could not have been confused with a hook that never ran. A security scan of the runtime path passed. Nothing here tests whether ponytail changes what the model writes, which is its actual claim. - Two lockdown mechanisms, 16 runs, no leak: a bypass that would not reproduce: https://aieveryminute.com/deny-mechanisms-2-1-243.json A follow-up that failed to reproduce its own starting point, published as one. /hook-threshold-2-1-238.json records a model reaching a file with Read, Bash, Glob and Grep all disallowed, by finding a tool that accepts a command and running tail through it. This trial tried to reproduce that and could not: 0 of 16 locked-down runs leaked the canary, against 8 of 8 unlocked controls, so the arms do not overlap and the fixture is proven reachable. TWO hypotheses were tested and BOTH refuted, and both are published rather than deleted. First, that a lockdown holds for a secret-looking file and fails for an ordinary one: the same canary string was placed in a .env as a credential and in a notes.txt as a build id, and both framings blocked identically. Second, that the CLI flag behaves differently from the settings rule, since removing a tool leaves missing capability while denying a path returns an explicit policy error: --disallowedTools and permissions.deny each held 8 of 8, though the flag arm visibly went looking, calling ToolSearch nine times across the runs before giving up. Tool calls are read from the session transcript rather than inferred from the reply. This closes an open line on /what-a-read-deny-rule-on-env-actually-stops/, which said only Read(./.env) had been tested. It is NOT an enforcement guarantee: one task shape, one machine, the vendor documents that deny rules do not cover arbitrary subprocesses, and failing to reproduce a bypass on a different task does not show it cannot happen. - Crossing the hook output cap costs 78 percent of the context you sent: https://aieveryminute.com/hook-threshold-2-1-238.json A SessionStart hook returning additionalContext is how a plugin injects standing instructions. Anthropic caps that output at 10,000 and says so twice in the hooks reference. Forty runs on Claude Code 2.1.238, four rounds per rung of a ladder, measure what crossing it costs: below the cap 10,016 tokens of context arrive, identical in all four rounds; above it 2,172 to 2,181 arrive. The arms do not overlap, every inlined round sits above every persisted round, and about 78 percent of what was sent is not delivered. It is not destroyed: the full payload is written to a file the model can read with a tool call, so this is a change of delivery rather than deletion. obra/superpowers, at 3,334 bytes, sits comfortably under and was inlined in every round at 1,271 tokens. Two things this corpus is careful about. Its own conclusion that the cap counts CHARACTERS was refuted by a later trial and is RETIRED in a superseded_claims block rather than reworded away; the cap counts UTF-16 code units, established at /hook-unit-2-1-238.json, and the cost figures are unaffected because this ladder is repeated ASCII where all three units are the same number. And the four probes that asked a model to recite a codeword are scored INVALID, because with Read, Bash, Glob and Grep all disallowed the model still reached the file: it searched for a tool, found one that accepts a command, and read the file with it. The threshold rests on the filesystem observable instead. The estimator refuses to report a mode on any rung where rounds did not repeat. - Startup floor on 2.1.243: a delta move that did NOT persist, see 2.1.246: https://aieveryminute.com/baseline-floor-2-1-243.json Four rounds, every cell measured twice, so eight readings per floor arm. Both floor arms came back single-valued with a zero noise floor in every round: 22,959 tokens with tool search on and 38,210 with it off. This is the boundary where the tool-search delta finally moved. It had been bit-identical at 15,069 across three consecutive measured releases, 2.1.236, 2.1.238 and 2.1.241, and here it goes to 15,251, because the two ends moved by DIFFERENT amounts for the first time since 2.1.233: on by +166 and off by +348. That retires the qualification written at 2.1.241, which had observed both ends moving together by exactly six tokens and concluded the deferred-tools saving does not move. No mechanism is claimed for either observation. The loaded arm went to 62,357 and is deliberately NOT attributed to the release, because it mixes the version change with this machine’s own user-scope configuration; it is also the only arm that was not single-valued, mode on 6 of 8 with a spread of 62,078 to 62,627. The harness was given the shared result_guard before this run, so a 429 or 529 that parses cleanly and reports an empty modelUsage can no longer be recorded as a floor of zero; the guard was control-tested against a synthetic 529 first. All 12 rows are usable, none excluded. - Which AI crawlers actually showed up: 1 of 5 over 14 days: https://aieveryminute.com/crawler-consumption-2026-08-25.json robots.txt and llms.txt are served through a shim that logs one line per request, because this hosting plan exposes no access logs. Over 14 days and 19 hours it recorded 278 non-curl requests. robots.txt names seven AI agents, but only five of them actually crawl: Google-Extended is a standalone product token and Applebot-Extended, in Apple’s words, does not crawl webpages. Both govern how a parent crawler’s data may be used, so they can never appear in a request log and their zeros are definitional. They are reported separately, and the crawlers they govern both did crawl, Googlebot 26 times and Applebot 38. Of the five that do crawl, exactly one appeared: OAI-SearchBot, 50 times. GPTBot, ClaudeBot, Claude-SearchBot and PerplexityBot returned zero, including both crawlers their vendors document as collecting training data. llms.txt was fetched 25 times and exactly one of those came from any of the five; the rest were ordinary search bots, SEO tools and unidentified clients. Every purpose label is quoted from the vendor’s own documentation with a link. The shim serves only those two paths, so a zero means a crawler never fetched robots.txt or llms.txt here, NOT that it never fetched a page, and the corpus states that above the results rather than in a footnote. One young low-authority domain, one window, and every identification is a self-reported user agent. - What a book costs in Claude Code, against its tiktoken count: https://aieveryminute.com/tokeniser-gap-2-1-238.json Twenty-eight runs measuring the same bytes two ways: written as a project-scope CLAUDE.md and read back from modelUsage, against tiktoken cl100k_base. A technical book and a novel, three slice sizes each, four rounds. Claude Code charged 1.41x the cl100k count on both, and the published figure is the slope between slices rather than a ratio of totals, so the CLAUDE.md framing constant cancels instead of inflating it. - Re-verification against Claude Code 2.1.227: https://aieveryminute.com/reverify-2-1-227.json Every round of the 2.1.227 re-check: the CLAUDE.md versus skill arms with their floors, and the batching arms with their arrival-control results. - Adherence and chain-of-thought, isolated re-checks: https://aieveryminute.com/isolation-recheck-adherence-cot.json 80 isolated adherence turns and 24 isolated chain-of-thought runs, closing the re-verification of every behaviour trial published before the isolation method existed. - Mutation trial, isolated re-verification: https://aieveryminute.com/ai-test-mutation-isolated.json 20 runs repeated with the user-level CLAUDE.md excluded. The caught-bug result held at 38 of 38; the claim about which subjects fail did not. - JSON extraction, isolated re-verification: https://aieveryminute.com/json-extraction-isolated.json 24 runs repeated with the user-level CLAUDE.md excluded, to check whether the absent-field result was the model or the machine. Identical. - Prompt injection, isolated re-verification: https://aieveryminute.com/prompt-injection-isolated.json 32 runs repeated with the user-level CLAUDE.md excluded. Loud payload still obeyed 0 of 20; quiet payload still followed 6 of 6 by one model and 0 of 6 by the other. - Comment rule trial, corrected: https://aieveryminute.com/comment-rule-corrected.json 32 files written with the user-level CLAUDE.md excluded, so the control arm is genuinely rule-free, with the full source of every file and the isolation method recorded. - JSON extraction trial: https://aieveryminute.com/json-extraction-trial.json 24 extractions scored twice over, once for schema validity and once for whether each field is true, with both documents, the schema and every returned object. - Generated test suite mutation trial: https://aieveryminute.com/ai-test-mutation-trial.json 20 generated pytest suites run against deliberately broken code, with every subject, every mutant, each suite in full, and the pytest output for the four that never ran. - Comment rule trial: https://aieveryminute.com/comment-rule-trial.json 86 generations across three paths, including the full source of all 32 files written to disk, and the arms where the baseline read zero and the experiment could not answer the question. - CLAUDE.md adherence trial: https://aieveryminute.com/claude-md-adherence-trial.json 344 turns of two continuous conversations, with the function names, parsed naming style and context size for every scorable turn, plus the turns excluded for returning nothing. - Prompt injection trial: https://aieveryminute.com/prompt-injection-trial.json 42 runs. Every file the model was given, every reply, and the verdict for each. Includes the loud payload that was never obeyed and the quiet one that was followed 6 of 6. - Generated code security trial: https://aieveryminute.com/ai-code-security-trial.json 36 generated Python files across six vulnerability classes, each with its prompt, the code, and the scanner findings that were and were not counted. - Chain-of-thought trial: https://aieveryminute.com/cot-trial.json 104 runs across three prompt styles, each with its exact prompt, reply, computed truth and extracted answer. - Cost-to-complete fixture: https://aieveryminute.com/task-benchmark.json The failing test suite used to measure what finishing one task costs on each model, including the trap that makes the careless fix pass one test and break another. - Tokenizer corpus: https://aieveryminute.com/tokenizer-corpus.json One paragraph in ten languages plus four content types, written for this test, so the cross-model token counts can be reproduced against any tokenizer. - Parallel subagent cost runs: https://aieveryminute.com/parallel-subagent-cost.json 44 runs across four arms: the fan-out ladder on two separate days, the paired inline control, the same ladder with user configuration loaded, and the paired measurement of that configuration. Every run carries its arrival-control file count. - Fork-subagent flag runs: https://aieveryminute.com/fork-subagent-cost.json 28 runs testing whether CLAUDE_CODE_FORK_SUBAGENT reduces parallel subagent cost, across a cold ladder (three arms including an explicit-off control) and a warm parent whose warm-up had to echo a planted marker. Tokens are recorded split into cache creation and cache read, because summing them hides the effect under test. - Plan mode versus auto-accept runs: https://aieveryminute.com/plan-mode-cost.json Ten runs comparing plan-then-execute against straight auto-accept on the same bug fix, split by call so the plan and the execution can be priced separately, with the fixture control proving the suite discriminates three ways, a per-run checksum showing the model never edited the tests, and a disclosed limit that bash was denied outside plan mode. - Plan mode versus auto-accept, equal tool permissions: https://aieveryminute.com/plan-mode-equal-permissions.json The re-run that removes the permission asymmetry above: 16 runs, eight per arm, with bash granted on every call in both arms and the grant proven to have arrived by requiring a Bash tool use in every action record. Every tool call is published by name and order, so the turn counts can be audited against what was actually done. - Plan mode versus auto-accept, open-ended task: https://aieveryminute.com/plan-mode-open-ended.json The third run in the series, on the open-ended case the advice actually names: 15 runs, seven paired rounds, an open prompt and a hidden rubric applied afterwards. Every rubric item, every per-run outcome and every tool call is published, along with the three-way control proving the rubric separates broken from lazy from good. - One prompt, one session, or a fresh session each step: https://aieveryminute.com/session-vs-fresh.json Forty-two runs across three ways of delivering the same three-step task: thirty on Claude Code 2.1.228 and twelve re-verifying them on 2.1.229, each run labelled with the version it ran on. The session id of every call is published so the treatment can be checked rather than trusted, alongside the full tool-call record behind every turn count and a hidden test suite written after the calls finish. - node_modules cost, and whether .claudeignore does anything: https://aieveryminute.com/claudeignore-and-node-modules.json Twenty-four token runs across two probes and three arms, plus nine canary runs that ask the model to read a file inside an excluded directory. The synthetic node_modules is measured on disk every run and published, so a null result cannot be a fixture that never built, and the deny-rule arm doubles as the control proving project settings were loaded at all. - How far a Read deny rule on .env reaches: https://aieveryminute.com/deny-rule-reach.json Eighty-eight runs across eight ways of asking for a denied file, scored by whether a fabricated credential-shaped canary came back rather than by what the model said. The no-rule control arm proves every route works unprotected, and the captured tool calls are what separate the permission layer refusing a read from the model declining to attempt one. - Claude Code's own context floor, measured directly: https://aieveryminute.com/baseline-floor-2-1-229.json The runs behind the second correction to the calculator baseline. An empty directory with user-scope configuration excluded, every cell measured twice per round for a noise floor, plus the paired decomposition that splits a loaded machine into the floor, its user CLAUDE.md and everything else at user scope. The three parts reconstruct the loaded total to one token. - Configured fan-out, re-measured: https://aieveryminute.com/configured-fanout-2-1-229.json Forty runs re-deriving the configured subagent arm after its configuration figure turned out to be counted once per turn rather than once. Adds the paired inline control that arm never had, which is what separates a per-subagent cost from a per-unit-of-work one, and records the turn count on every run because a run total mistaken for a context size is the error being corrected. - Compaction survival runs: https://aieveryminute.com/compaction-survival.json Eight runs plus a no-compaction positive control, testing which planted values survive auto-compaction. Two bulk types, sparse and high-entropy, with every planted value, every reply and the per-run compaction events. - Re-verification against Claude Code 2.1.228: https://aieveryminute.com/reverify-2-1-228.json Every round of the 2.1.228 re-check: four batching rounds with their arrival-control results, the CLAUDE.md versus skill arms with their floors including the round thrown out for landing inside the baseline drift, and the configured subagent arm. - Fan-out execution mode runs: https://aieveryminute.com/fanout-modes.json 34 runs showing that an eight-agent fan-out lands in one of two non-overlapping cost modes, separated by the num_turns field the CLI already reports. Includes the width sweep at two and four agents and the re-verification runs the modes were first spotted in. - Fan-out slope, conditioned on parent turn count: https://aieveryminute.com/fanout-slope-2-1-232.json Eight rounds instead of four, which is what it took to explain a dispersion three previous corpora recorded as unexplained. The parent context tracks how many turns IT takes: a four-agent fan-out runs in six or seven and the total moves 16% between them, against 0.2 to 0.3% within one. Every slope is computed inside one turn count, and both conditions are published because the caller does not control which they get. - Subagent slope on 2.1.231: https://aieveryminute.com/fanout-slope-2-1-231.json Forty runs re-measuring the per-subagent cost after the calculator was found still computing from a withdrawn configuration figure. Isolated and loaded fan-out at one and four agents with the paired inline control that separates per-subagent from per-unit-of-work, plus the single-turn configuration measurement the earlier correction turned on. - Claude Code's own startup floor on 2.1.232: https://aieveryminute.com/baseline-floor-2-1-232.json Sixteen repetitions per cell after Claude Code updated mid-measurement and the deploy gate blocked every deploy until the calculator caught up. The mode, not the minimum, because the call is bimodal: 22,528 in 13 of 16, with the other two values exactly 198 and 521 below it. The two arms moved in opposite directions, so the tool-search penalty fell 141 tokens. - Startup floor on 2.1.231: https://aieveryminute.com/baseline-floor-2-1-231.json The floor re-measured after the version bump, five rounds with every cell run twice for a noise floor. Reports the MODE rather than the minimum, because modelUsage on a trivial call is bimodal at 22,404 and 21,883 depending on whether a Haiku call fires, and the minimum would publish the rare run as the floor. The tool-search delta held to within a token. - CLAUDE.md rule-count runs: https://aieveryminute.com/claude-md-rule-count.json Thirty-five runs testing whether a large CLAUDE.md degrades adherence to the rules inside it, laddered from 2 to 500 rules with two regex-checkable targets, one at the top of the file and one at the bottom. Carries a negative control with the targets deleted that scores zero, and the exact CLAUDE.md used in all seven arms. - Output verbosity runs: https://aieveryminute.com/verbosity.json Thirty runs across three batches testing whether a terseness instruction cuts the bill, with the output share of tokens and of cost separated because they differ sevenfold. Carries a rate reconciliation that reproduces every reported Opus cost to the cent, a published matcher control, and full reply text plus marker index on batch 3 so the arrival result can be recomputed rather than trusted. - Prompt cache TTL runs: https://aieveryminute.com/cache-ttl.json Twenty-four runs testing whether an idle gap actually expires the prompt cache, and what the one-hour cache costs to have. Carries the per-run write tier, a published pre-run flag control, per-turn and whole-session costs (which point opposite ways), a nine-run bracket showing the cache alive at 55 minutes and gone by 70, and the diagnostic run that proved the first design was blind because the first resume always rewrites the cache breakpoint no matter how short the gap. - Fan-out wall clock on heavy units: https://aieveryminute.com/fanout-wall-clock.json Twenty-four runs testing whether delegating saves time on work built to favour it: N independent 40KB files, one planted marker each, one subagent per file against the same job inline. Every tool call is attributed to the parent or to a subagent via parent_tool_use_id, which is what two earlier discarded attempts had no way to do. - What a skill costs, clean-project sweep: https://aieveryminute.com/skills-cost-2-1-231.json Eighty runs sweeping skill count from 1 to 1,000 and description length from 30 to 1,500 characters, with a body-size arm as the control that the fixture builds real skills. Carries the description fit basis and its valid range, stated because a fit basis that is not disclosed cannot be checked. - Whether a skill costs more when you have fewer: https://aieveryminute.com/skills-listing-regime-2-1-231.json Thirty-two runs measuring the same forty skills with the isolation flag on and off in the same round, which is what stopped a version-change claim from being published: the flag alone moves the cost 10.2x, so this site’s earlier 1.5-tokens-a-skill figure was measuring the other regime rather than another release. - Does a description still cost anything at 600 skills: https://aieveryminute.com/skills-desc-at-scale-2-1-231.json Twenty runs testing the mechanism the sweep was about to assert: description length at 600 skills, with the same contrast at 40 skills running in the same rounds as the positive control that makes the null believable. - What a skill costs, re-measured on 2.1.233: https://aieveryminute.com/skills-cost-2-1-233.json Four rounds on the current release. The per-skill figure did not move at all, 39.05 tokens at 200 skills on both 2.1.232 and 2.1.233, while over the same release the tool-search saving fell 17% and the tool-search-off floor 8%. The measurement noise also collapsed: not one of the eleven cells was stable across rounds on 2.1.232, and ten of eleven are on 2.1.233. - What a skill costs, re-measured on 2.1.232: https://aieveryminute.com/skills-cost-2-1-232.json The re-measurement the context calculator uses for its skills curve, published late: it had been sitting unassembled while the only skills corpus a reader could follow was 2.1.231, which differs at 10, 200 and 300 skills. Three arms measured separately, count, description length and body size, and the estimator is the MODE rather than the median because an intermittent 198-token component makes a four-round median land between two real values. - What an MCP tool costs, re-measured on 2.1.233: https://aieveryminute.com/mcp-curve-2-1-233.json Four rounds on the current release, with the isolation control re-run alongside. Every cell is identical to 2.1.232, deferred and loaded, at 5, 50 and 200 tools, while both floors moved underneath it. That is the second per-item curve in a row to hold still through a release that moved its own floor. - What an MCP tool costs, re-measured on 2.1.232: https://aieveryminute.com/mcp-curve-2-1-232.json The re-measurement the context calculator actually uses. It went unpublished, so the calculator carried a 50-tool deferred figure of 810 tokens while the only corpus a reader could check said 711. Published here with its isolation control, and the calculator curves are now tied to this file by the deploy gate so they cannot drift from their receipt again. - MCP tool cost, re-measured with an isolation control: https://aieveryminute.com/mcp-curve-2-1-231.json Fifty-six runs re-measuring both MCP curves on 2.1.231 after the calculator was found quoting a version nobody was running. The deferred curve holds; a loaded tool measures 652 tokens against a published 304, and the isolation control that explained the skills discrepancy the same day explains none of this one. - Both plugin ladders at eight rounds, published as bimodal cells: https://aieveryminute.com/plugin-ladder-8x-2-1-238.json Eighty runs re-running both ladders on 2.1.238 after a median was found summarising bimodal cells as midpoints no round produced. Eight rounds did not resolve the split: three cumulative cells come back tied four-four between values exactly 198 apart, so the cells are published as two values and their counts rather than as one number. The marginal rates are paired inside each round, which cancels a component that fired in both cells, and those do resolve: 40.0 tokens a command between 50 and 200, collapsing to 9.93 and then 5.62, against 51.0 a subagent definition with no collapse. - What a plugin's parts cost, re-verified on 2.1.232: https://aieveryminute.com/plugin-parts-2-1-232.json The whole 88-run measurement re-run sequentially on the release that followed, after four concurrent harnesses had made every call return a zero context and the arrival controls caught it. Every point either reproduced exactly or came in 99 tokens lower, which is half the documented intermittent component. Every harness now records the Claude Code version it ran against. - What a plugin's parts cost, and which share a budget: https://aieveryminute.com/plugin-parts-2-1-231.json Seventy-two runs pricing slash commands and subagent definitions on their own ladders, plus an ablation that re-measures the cost of forty skills beside each element in turn. Commands collapse in bulk exactly as skills do; subagent definitions cost 51 tokens each at every count measured and never get cheaper. Floor-paired and cell-to-cell figures are labelled separately throughout, because the floor call carries most of the noise. - What a CLAUDE.md costs per turn, split by billing tier: https://aieveryminute.com/claudemd-per-turn-2-1-233.json 32 runs across one resumed four-turn session, with and without an 18KB CLAUDE.md, adjudicating the circulating claim that a 5,000-token CLAUDE.md is a 5,000-token tax on every turn. Input, cache creation and cache read are kept separate per turn, because summing them is what makes the context claim and the cost claim look like one claim. - What the docs say a skill costs, against what it measured: https://aieveryminute.com/skill-token-doc-vs-measured.json Anthropic documents ~100 tokens per Skill for the metadata loaded at startup. This holds that figure against the six trending packs already installed and measured here, and records six chat-model runs asked the same question. The model runs are labelled an impression rather than a measurement, because the same model contradicted itself between two identical prompts. - What the trending skill packs cost to install: https://aieveryminute.com/skillpack-cost-2-1-233.json 56 runs across two harnesses, installing six Agent Skill collections taken from GitHub Trending and measuring each against an empty floor rather than multiplying a published rate by a file count. The second harness installs only what each plugin manifest enumerates, which corrected mattpocock/skills from 1,209 tokens for 35 files to 852 for the 25 it ships; four packs reproduced to the token across both. Carries the arrival control that doubles as a prompt-injection check, over 163 third-party skill descriptions in the shipped fixture and 173 in the first harness, and the check that this site own published model misses real packs in both directions. - Claude Code's own startup floor on 2.1.233: https://aieveryminute.com/baseline-floor-2-1-233.json Sixteen repetitions per cell. Notable for what is absent: every repetition of both floor arms returned the same value, where 2.1.232 returned three distinct values 198 and 521 tokens apart. The two intermittent components that have been the dominant noise source in every corpus on this site do not appear. The tool-search penalty also fell 3,020 tokens to 14,644. - Six models counting 120 working days, twice each: https://aieveryminute.com/working-day-arithmetic-12-runs.json Twelve runs across six free-tier chat models from four makers, all asked the same date question in a fresh chat. Every verdict is derived by the assembler rather than typed: both defensible readings of the question, every weekday, every month working-day count and every date offset the runs quoted. Seven runs correct. Runs that showed no working carry a null setup rather than a passing one, so the tally cannot claim they understood the question. - Whether CLI flags move a skill cost, and whether the cost is measurable at all: https://aieveryminute.com/flag-effect-2-1-234.json Thirty-six runs on Claude Code 2.1.234 testing a prediction registered before the first round: that --allowedTools and --permission-mode explained why this site measured one pack at 274 on one harness and 249 on another. Refuted. The paired within-round differences flip sign and match components already published as intermittent here, so no directional flag effect is detectable. The real answer is in the spread: the same one-skill pack returned 70 to 967 tokens across twelve readings, while a 24-skill pack held to 0.58x of its minimum. Carries forward this site existing limit that one skill sits inside the noise, and re-verifies it rather than retiring it. - A trending diagram skill, installed and actually used: https://aieveryminute.com/diagram-skill-cost-2-1-233.json Twenty runs against cathrynlavery/diagram-design pinned at one commit: a 2x2 of trivial and diagram prompts with and without the skill, plus eight rounds testing the documented marker file, published per round rather than only in summary. Records that it delivered a file in 1 of 4 unattended rounds against 4 of 4 for plain Claude Code, and every file the agent read in every round including the three that drew nothing, counting Bash reads as well as Read-tool ones. Every cost multiplier is published as null with its reason, and the startup figure this corpus once carried was later shown unmeasurable by flag-effect-2-1-234.json. - Re-measuring the calculator’s own skills curve, at both ends: https://aieveryminute.com/curve-stability-2-1-234.json Sixteen readings on Claude Code 2.1.234 against a curve this site published as spread 0 over four rounds. The published values reproduce, both landing ten tokens off, but the precision they appeared to have does not: the 10-skill cell ran 192 to 1,287 tokens, a spread of 281% of its own mode, while the 1000-skill cell held to 8%. Thirty-six times the relative unreliability, in the direction predicted before the run. The fixture generator is imported from the original harness and asserted byte-identical, because re-implementing it produced 290 against a published 400 and would have been published as a drift in the calculator. - What Anthropic’s own five skill plugins cost, against its own documented figure: https://aieveryminute.com/anthropic-plugins-2-1-234.json Fifty runs on Claude Code 2.1.234 against anthropics/skills at one commit. There is no plugin.json: the marketplace offers five separate plugins, so installing the repository is not a thing anyone does and each is priced on its own. Only one of the five is large enough for this instrument to resolve, and it lands at 102 tokens per skill against a documented ~100. The other four carry ranges rather than figures, three of them because the harness pre-registered before running that a single-skill cell is below its resolution; the ranges vindicate that, since one runs from minus 578 to plus 319 and installing a skill cannot make a session smaller. Also publishes two estimators side by side, because the paired-floor control this site normally uses was found to amplify per-call noise rather than remove it. - What a skill description costs per character above 300: https://aieveryminute.com/desc-limit-2-1-235.json Twenty-five runs sweeping description length across the 1,024-character maximum Anthropic documents, forty skills per cell, on Claude Code 2.1.235. Tripling a description from 512 to 1,600 characters costs 637 tokens across forty skills, about 0.015 per character against the 0.276 this site measured below 300. The pre-registered prediction is scored as refuted, and the reason is published: it assumed a rate this site had already stated was valid only to 300 characters. - Six free models, one JSON prompt, twice each: https://aieveryminute.com/json-render-2026-08-22.json 12 runs on duck.ai’s free tier, six models across four vendors, asked identically for a bare JSON object with a fresh chat each time. An exact three-three split on how the reply was rendered, every model on the same side both rounds and all six matching the previous day. All twelve replies carry the byte-identical object once whitespace is stripped. Every run links a published screenshot showing the model header and the reply together, which the first attempt at this trial lacked and was held for. Also records the near-miss: after one round, fencing and pretty-printing correlated perfectly across all six models, and the second round refuted it. - Which unit the 10,000-character hook cap counts: https://aieveryminute.com/hook-unit-2-1-238.json 44 runs adjudicating whether Claude Code’s documented 10,000-character cap on hook output counts characters, bytes, or UTF-16 code units. A nine-cell ladder holding characters constant while moving units, plus a probe pair of 5,022 and 5,023 characters straddling 10,000 units where a codeword planted at the end of the payload returns tool-free 4 of 4 at one and 0 of 4 at the other. Over the 36 ladder runs the unit predicate mispredicts 0 times, characters 12 and bytes 16, identical to two earlier independent ladders. The written post is held pending verification; the runs are published because they are the finding. - Narrowing the skill description cut onto the documented cap: https://aieveryminute.com/desc-cut-1536.json 28 runs narrowing this site’s earlier 1,400-to-1,550 bracket onto the documented skillListingMaxDescChars default of 1,536. Text ending at 1,530 characters is answered from context in 4 of 4 rounds; text ending at 1,536 never is, in 4 of 4 rounds, with the model shelling out to read the file instead. A control renaming the skill from 5 characters to 56 moves the boundary cell not at all, so the figure is not name-dependent. Published alongside the correction of the post that originally framed this cut as undocumented and cited a 1,024-character maximum. - Where a skill description stops reaching the model: https://aieveryminute.com/desc-truncation-2-1-235.json A rare marker planted at a known character offset inside one skill description, then asked for. The reply alone says the description is never truncated out to 2,000 characters, and that is wrong: the session transcripts show the model answering from context with no tool calls up to 1,400 characters and shelling out to Bash to open the file beyond it. Scoring tool use rather than the answer puts the cutoff between 1,400 and 1,550 characters, four rounds either side, and inverts the conclusion the naive scoring would have published. - Does the description cut count characters or tokens?: https://aieveryminute.com/desc-cut-units-2-1-235.json Closes an open question the description-cutoff post published as unknown. Two fixtures of matched length, one ordinary prose and one API-shaped text with punctuation, identifiers and flags, with the tokenisation ratio between them measured by a paired cost arm in the same runs rather than assumed: the dense text costs 2.06 times as much per character. A token budget set where prose is cut would have severed the dense description near 702 characters; it is answered from context at 1,400, four rounds of four, exactly like prose. The limit counts characters, so the guidance applies to every author rather than only to those writing prose. - An 817-skill real pack against this site’s own skills curve: https://aieveryminute.com/cybersec-817-2-1-236.json A cybersecurity pack trending on 2026-08-20 ships 817 Agent Skills and costs about 56,600 tokens at startup, 69.30 per skill, from 24 readings that agree within 48 tokens in 21 of them. This site’s published curve predicts roughly 11,000 for the same count and its own cells still reproduce, so the curve is accurate about the synthetic skills it was built from and 5.2x low against a pack anyone can install. The cause is NOT established: description length, description string-uniqueness and extra frontmatter were each tested and refuted, and the remaining candidate collapsed on tokeniser arithmetic showing the synthetic descriptions are not in the measured context at this skill count. This site’s own real-pack corpus never disagreed: six real packs measured 34 to 274 tokens per skill, so 69.30 sits inside that and the synthetic curve is the outlier. Also here: the extra frontmatter is not charged, which matches Anthropic’s documentation, and two of the 24 readings came in at 17,352 for reasons nobody has established. All 80 runs across 30 rounds are published, and within each harness the absolute session context is byte-identical round to round, so the variation between modes is floor movement rather than pack movement. - A 65% token claim, and two of its author’s own designs disagreeing: https://aieveryminute.com/caveman-2-1-236.json Sixty runs against JuliusBrussee/caveman, a 99k-star Claude Code skill claiming 65% fewer output tokens. Three system-prompt arms on five of the author’s own prompts, four rounds each. The estimator is non-overlap of observed ranges rather than a mode or a median, asserted in code, because the guarded mode is refused in all fifteen cells and the median is a value no round produced in all fifteen. Under this design the skill separates from an unprompted baseline on five tasks of five and from the string “Answer concisely.” on none, and its system prompt costs a deterministic 2,517 tokens per single-turn call. Read the known_limits before quoting any of that: the author publishes a terse-controlled result of their own that points the other way, at +50% median, and the two designs differ in arm construction, delivery flag, model, CLI version and round count. - Startup floor re-measured on Claude Code 2.1.236: https://aieveryminute.com/baseline-floor-2-1-236.json Five rounds, every cell measured twice per round so each round carries its own noise figure. Both isolated cells returned the identical value in all ten readings: 22,787 tokens with tool search on and 37,856 with it off, noise floor zero. The calculator had been quoting 2.1.233 and was understating the floor by 490 and 915 tokens. Across five releases the tool-search-on floor has moved inside a 490-token band while the tool-search-off figure swung from about 40,200 down to 36,941 and back to 37,856, so what moves is the deferred-tools saving rather than the floor beneath it. - Both MCP curves re-verified on 2.1.238, with the isolation control: https://aieveryminute.com/mcp-curve-2-1-238.json Sixty-four runs on the installed release: the deferred and loaded curves at 5, 50 and 200 tools, four rounds each, plus the isolated-versus-unisolated bridge control. Neither curve moved a single point from the 2.1.236 corpus, and the calculator already carried these values. Checked for the phantom-median trap before publishing: every cell agrees 3-of-4 or 4-of-4, so median and mode coincide and no figure here is a value no round produced. The lone outlier in each deferred cell is exactly +198, the documented intermittent component. - MCP curves re-verified on 2.1.236, unmoved for a third release: https://aieveryminute.com/mcp-curve-2-1-236.json Four rounds at 5, 50 and 200 tools in both regimes, each arm paired against a floor measured in its own tool-search flag state so the 15,000-token cost of turning tool search off is not folded into the per-tool figure. Every cell came back identical to the token to the 2.1.233 run: 90, 810 and 3,210 deferred, 3,270, 32,610 and 130,410 loaded. That is three consecutive releases with an unmoved MCP curve while the startup floor beneath it moved every time, which is why the two carry separate version stamps. The isolation control was re-run on the same release, and the assembler refuses to merge halves measured on different releases. - The skills curve re-verified on 2.1.238: https://aieveryminute.com/skills-cost-2-1-238.json Eighty runs re-measuring the whole skills curve on the installed release, four rounds across eleven counts from 1 to 1,000 skills, plus the description-length and body-size cells. Not one cell moved: every point is identical to the 2.1.236 corpus, and the calculator already carried these exact values. The two-regime shape holds, about 39 tokens a skill to 200 and collapsing to 3.9 between 400 and 1,000. All 80 run directories verified single-release from their transcripts. - Skills curve re-verified on 2.1.236: not one cell moved: https://aieveryminute.com/skills-cost-2-1-236.json Four rounds, 80 runs, none failed, across all three arms the curve carries: eleven skill-count cells from 1 to 1,000, five description-length cells and three body-size cells. Every one returned the same mode as the 2.1.233 run, so the calculator’s constants did not need to change and only its version stamp did. The estimator is the MODE rather than the median, because this harness carries a 198-token intermittent component that fires per call and a median of four rounds lands between two real values; the assembler refuses to build if any published figure is not a value some round actually returned. Read alongside the 817-skill real-pack corpus, which shows this synthetic curve under-predicting a real pack by 5.15x at the same count. - A trending skills directory lists 1,217 entries, and one upstream repo accounts for every dead link found: https://aieveryminute.com/awesome-skills-links-2026-08-24.json VoltAgent/awesome-agent-skills, trending at 600 stars this week, checked on commit ea6bfaa. First, what it is: the repository holds four files and NO skills. It is a directory whose 247,301-byte README links to skills hosted elsewhere. Its GitHub description says 1000+ agent skills; counted here at 1,217 across 70 sections, so that claim is accurate. Links split 629 to github.com, 582 to officialskills.sh, and 6 elsewhere. A random sample of 60 entries was drawn with a recorded seed and each URL fetched once, with 404 and 410 scored dead and everything else, including 403 and timeouts, scored inconclusive and counted apart so bot-blocking could not be mistaken for a missing target. There were no inconclusive responses. Eleven of the sixty were dead, and all eleven pointed at the same place: github.com/NVIDIA/skills. Every one of the other 49 sampled links resolved. A separate targeted group of 15 further NVIDIA links, reported apart from the random sample and never pooled with it, was also dead, 15 of 15. The cause is not a deleted repository: github.com/NVIDIA/skills returns 200 and so does its skills directory. The upstream repo FLATTENED its paths, so skills/cuopt/cuopt-developer became skills/cuopt-developer, verified on three pairs where the flattened form returns 200 and the nested form the directory lists returns 404. 155 of the 1,217 entries use that old nested form, 12.7% of the directory, and it is NOT claimed that all 155 are dead because only 26 were tested. The directory itself predicted this: its Security Notice says skills are curated, not audited, and may be updated, modified or replaced by their maintainers at any time after being added. - A trending CLAUDE.md costs 982 tokens a turn, not the 719 this site predicted: https://aieveryminute.com/karpathy-claude-md-2-1-241.json The andrej-karpathy-skills CLAUDE.md, trending at 491 stars in a day on 2026-08-24 and measured unchanged at 2,357 bytes on commit 2c60614. Fifteen rounds, 120 runs, two arms differing only by the presence of that file at the project root, four resumed turns per session. Two predictions were pre-registered BEFORE any run and neither was revised: this calculator predicted 719 tokens from 106 plus 0.26 per byte, and an older corpus measured on 18,000 bytes of English prose predicted 650 at 0.2759 per byte. The measured mode is 982 tokens on 38 of 60 paired differences, which is 0.417 tokens per byte, above the 0.26 to 0.37 range this site had published for content variation. The older corpus carried a limit saying a file of tables or code would not cost the same per byte; this markdown-structured file, with headers, bullets and a fenced code block, demonstrates it. Two things are NOT claimed: the paired difference is multimodal, wandering across a range that spans both predictions, so neither is refuted, and one file does not re-fit a curve fitted from 2KB to 28KB. The rate on the calculator is unchanged and a rendered note discloses the gap. Scope, because it changes who the figure applies to: the repository is nine files, not one, and its README recommends installing the guidance as a Claude Code plugin FIRST, with the project-root CLAUDE.md listed second. The 2,357 bytes measured here are that second option; the plugin path was not measured. Attribution was checked rather than assumed: the trending listing is a re-host and the plugin manifest names forrestchang. Controls published per run: the isolation flag, the fixture asserted at 2,357 bytes on one arm and absent on the other, one constant working directory because the startup floor moves with the path, the arrival marker on all 120 runs, and the billing split kept separate. - Six free models, one false premise, twenty-four corrections: https://aieveryminute.com/false-premise-2026-08-24.json Two independent trials on duck.ai on 2026-08-24, twelve runs each, six free models across four vendors, two rounds per model, a fresh chat every run. Every run was asked the identical question: why does list.sort() return a new sorted list instead of modifying the original in place. The premise is false in BOTH halves, verified in a REPL before the rubric was written: list.sort() returns None and sorts in place, and sorted() is the one that returns a new list. The rubric of CORRECTED, PARTIAL and CONFABULATED was fixed in writing before any model was asked and is published at /false-premise-preregistration.md. Result: 24 of 24 CORRECTED, no PARTIAL, no CONFABULATED, every model on the same side in both rounds of both trials. Not one of the six was dragged into explaining behaviour that does not exist. The two trials differ in EVIDENCE and the corpus records that as a field rather than a footnote: the first committed no screenshots, the second commits one per run under /shots/false-premise-2026-08-24/, each showing the model label together with the BEGINNING of the reply including the sentence that contradicts the premise, with the screenshot label compared against the text-capture label on all twelve and zero disagreements. These are 1568x696 viewport captures, so the longer replies continue below the fold and a screenshot is not a capture of a whole answer. One run is documented in detail because the label resolves several seconds AFTER the reply finishes streaming, so a screenshot taken when the text looks complete can still read Generating response with no model name, which is how an earlier trial on this site ended up with three unlabelled screenshots. Separately and consistently across all twelve labelled runs, the part of the duck.ai privacy line that VARIES BY MODEL differs across the six, in three forms (the full rendered line also carries a fixed DuckDuckGo prefix and a Learn more link, neither recorded): zero data retention for the two OpenAI models and Mistral, limited data retention for Claude Haiku 4.5, zero provider visibility for gpt-oss and Gemma, with no AI training on all six. Recorded as observation; what those phrases mean in practice was not tested. Every count in the corpus is parsed from the raw capture files by the assembler, which asserts the run count, the score tally against the run count, the rubric membership of every score, two rounds per model, the label agreements, and that every screenshot a run names exists on disk. One figure is NOT derived and says so in place: the over-the-wire byte size in cdn_example is a dated curl measurement of a CDN output rather than a property of any file. - Parallel subagent cost on 2.1.238, and why the constant fell 27%: https://aieveryminute.com/fanout-slope-2-1-238.json Ten rounds, all 100 runs stamped 2.1.238 and 97 usable, with the three failures excluded as unparseable rather than counted as zeros, and an arrival control passing on every row. The calculator constant drops from 28,259 to 20,581 tokens per parallel subagent, and a subagent did not get cheaper: at matched parent-turn counts the figure moved 1.0% and 0.9% since 2.1.232. What moved is which turn count is modal, from 7 turns in 5 of 8 rounds to 6 turns in 7 of 10, and the calculator carries the modal condition. Both conditions are published side by side because which one a reader gets is not under their control. Getting here took three runs. A five-round pass put only 3 rounds behind the mode. The ten-round re-run straddled a mid-session Claude Code update and was mixed-release, which the session transcripts show directly since they stamp the release on every message; it is not published. An earlier attempt to date runs from a two-token shift in the embedded floor cell was wrong and is withdrawn in this file, because the clean run reads the same floor as the 2.1.236 one. The harness now stamps the release top-level and per run. - Parallel subagent cost on 2.1.236, superseded by the ten-round run: https://aieveryminute.com/fanout-slope-2-1-236.json The five-round predecessor to the 2.1.238 run, published under its true release rather than discarded. It puts only three of its five rounds behind the modal parent-turn count, so the calculator does not take its figure from here; the ten-round 2.1.238 run does. Kept as the 2.1.236 reading of the same measurement, with its release established from the session transcripts rather than inferred. - The startup floor depends on the working-directory path: https://aieveryminute.com/path-floor-2-1-241.json 48 runs of one trivial prompt in an empty directory on 2.1.241, changing nothing but the sandbox path. The floor ranged 22,765 to 22,847 tokens. Two paths of identical 67-character length differed by 14 tokens, and a 38-character path cost 20 tokens more than a 47-character one, so the quantity counted is the tokens of the path rather than its length. Deliberately publishes no tokens-per-character rate, because the ladder uses a repeated character that tokenises unlike any real path. Two cells were measured in separate harness invocations and reproduced across the gap. - Startup floor re-verified on 2.1.241: https://aieveryminute.com/baseline-floor-2-1-241.json Four rounds, every cell measured twice, so eight readings per floor arm. Both came back single-valued with a zero noise floor: 22,793 tokens with tool search on and 37,862 with it off. Across three releases from 2.1.238 each arm moved by exactly six tokens, so the tool-search delta is bit-identical at 15,069 and the deferred-tools saving did not move at all. The 198- and 521-token intermittent components that dominated noise on this site for weeks did not appear, and that sentence is generated from this run’s own histogram rather than carried over from the previous corpus, which is how a stale description of a distribution used to survive a re-measurement. The loaded arm moved 643 tokens and is deliberately not attributed to the release, because it mixes the version change with two days of this machine’s own configuration. - Startup floor on 2.1.238: unchanged, and the release checked rather than assumed: https://aieveryminute.com/baseline-floor-2-1-238.json Five rounds, every cell measured twice per round. Both isolated cells returned the same values as the 2.1.236 run, 22,787 tokens with tool search on and 37,856 with it off, so only the version stamp moved. The release is established from the Claude Code session transcripts, which stamp it on every message: 30 of 30 transcript directories read 2.1.238. That check exists because a corpus published earlier the same day turned out to span two releases after Claude Code updated mid-run, and had been dated from a two-token shift in an embedded floor cell instead. This run shows that floor does not move across that boundary at all, so the proxy was measuring something else. - Context measurement script: https://aieveryminute.com/measure-context.sh The shell script that produces the startup-context figures, published in full. Measures its own noise floor first and refuses to report a number smaller than its own uncertainty. - What language a skill description is written in costs you: https://aieveryminute.com/skillpack-language-2-1-247.json Nine skills installed one at a time on 2.1.247, four rounds each, floor re-measured every round. Per description CHARACTER, descriptions with no CJK cost 0.363 to 0.482 tokens and majority-Chinese ones cost 0.887 to 1.048: the arms do not overlap and the two character-length ranges do overlap, so length is not the explanation. Per BYTE of UTF-8 the arms OVERLAP (0.363 to 0.482 against 0.459 to 0.513), so most of the per-character gap is the unit rather than the tokeniser, and the corpus says so; it matters anyway because the documented description cap counts characters. Includes a release bridge: addyosmani/agent-skills returns the identical 2,302 tokens measured on 2.1.233 fourteen releases earlier. - What a CLAUDE.md costs in Chinese against English: https://aieveryminute.com/claudemd-language-2-1-247.json One translation pair, the same instructions in both languages, byte sizes matching within 1.4 percent while character counts differ by 2.8x. Sixteen self-paired rounds on 2.1.247, one constant working-directory path because the startup floor moves with the path. English 604 tokens, Chinese 678, a modal difference of 1.12x. Three predictions were pre-registered before the run and all three are scored, including the one this site most needed to refute: that Chinese is cheaper because it uses fewer characters, which predicted 183 to 248 tokens against 678 measured. The arms overlap round to round and the corpus says so. - The command and subagent-definition prices re-verified fifteen releases on: https://aieveryminute.com/plugin-parts-2-1-247.json The oldest stamp on the calculator was 2.1.232, and once the staleness banner began rendering the lag per row it was the most conspicuous at fifteen patch releases behind. Twelve pooled rounds on 2.1.247, floor re-measured every round, 96 rows and none refused. Both published figures hold: 40.05 tokens per slash command at 200 (published 40.0) and 51.2 per subagent definition at 50 (published 51). Forty skills on a clean project came back at 1,568, identical to the token. The published constants were NOT changed, because the 6-token movements sit inside a 198-token intermittent component that this corpus publishes as a full distribution per cell. Four rounds would have got it wrong: pooling twelve moved two modes, one of them the per-command figure. The calculator stamp deliberately stays at 2.1.232, because that is the version the full ladders were measured on and this re-verified two points rather than the curve. - What a vendor skill costs when it actually fires: https://aieveryminute.com/gomg-skill-fires-2-1-247.json JetBrains ships go-modern-guidelines as a Claude Code plugin with one skill, whose body tells the model to shell out to a pinned Go CLI before editing Go code. Four paired rounds on 2.1.247: the same task cost 45,582 to 45,785 tokens without the skill and 146,286 to 146,872 with it, about 3.2x, and the arms do not overlap. Every run in both arms wrote the file and the CLI genuinely installed, both recorded per run. The code produced was identical except for one word in a doc comment. The task deliberately has no modern-Go idiom to apply, so this measures the cost being paid rather than the benefit being absent, and the corpus says so. An earlier pass that ran without a permission mode measured permission-denial retries instead and is reported as invalid rather than dropped. - Does the vendor skill actually change the code: https://aieveryminute.com/gomg-skill-idiom-2-1-247.json The missing arm of the companion corpus, which could show the cost being paid but not the benefit delivered because its task had no modern idiom to apply. The task here is taken from the CLI own published ruleset: use slices.SortFunc with cmp.Compare instead of sort.Slice. Six paired rounds on 2.1.247. Without the skill the model wrote sort.Slice in 6 of 6 runs; with it, slices.SortFunc with cmp.Compare in 6 of 6. No run went the other way. Cost was 45,733 to 45,960 tokens without and 146,181 to 146,691 with, arms not overlapping. The full source of one run per arm is published so the machine-checked classification can be verified by eye. - The startup floor fell 24.5 percent on 2.1.251: https://aieveryminute.com/baseline-floor-2-1-251.json The largest move this site has recorded in the figure every other estimate is built on. Eight rounds, every cell measured twice, so sixteen readings per floor arm, and both arms came back SINGLE-VALUED: 17,207 tokens with tool search on and 32,400 with it off, against 22,801 and 37,870 on 2.1.246. The paired tool-search delta barely moved, 15,069 to 15,193, so whatever shrank is common to both arms rather than specific to tool search. An independent 4-round invocation minutes earlier returned 17,205 and 32,398 with an identical 15,193 delta; it is reported rather than pooled because its working directory was one character shorter and this site has measured that the floor moves with the path. No cause is asserted: nothing here establishes what was removed. The calculator constant moved with it. - A 229 KB skill body still costs nothing at startup: https://aieveryminute.com/bigbody-2-1-251.json This site published that a skill body is free at any size UP TO 28KB. mvanhorn/last30days-skill, 60,605 stars, ships one skill whose SKILL.md is 229,438 bytes, roughly 8.2x beyond that range, so the claim was being applied outside where it was measured. Twelve pooled rounds on 2.1.251, floor re-measured every round: installed and never invoked, the whole skill costs 101 tokens, which is the listing for one skill plus a 246-character description. The body contributes zero. The published model predicts 106.9 against 101 measured. Three predictions were pre-registered and the site own position was the one that held, extending its validated range 8.2x on a real pack rather than a synthetic fixture. The tool was deliberately NOT run: it auto-installs CLIs at runtime via npx and can extract browser cookies behind an opt-in, so nothing here prices what it costs when it fires. - The skills curve moved, but only past 200 skills: https://aieveryminute.com/skills-cost-2-1-251.json Eleven pooled rounds and 220 runs on 2.1.251, none refused, re-measuring all three arms the calculator interpolates from. Six of nineteen cells moved and every move is DOWNWARD. The count curve is identical to the token at 1, 10, 40, 100, 150 and 200 skills, and lower at 300 (-197), 400 (-174), 600 (-180), 800 (-174) and 1000 (-174). The description curve moved at exactly one point, 800 characters, and the body-size curve did not move at all. So the change sits entirely in the collapsed regime past roughly 200 skills and is close to constant there rather than scaling with count. No cause is asserted. Worth reading beside the floor corpus from the same day: the absolute startup floor fell 24.5 percent over the same interval and thirteen of these nineteen paired cells did not move at all, which is what a paired design is for. - What an always-on output-style hook costs, and whether it measurably shortens replies: https://aieveryminute.com/adhd-alwayson-2-1-251.json ayghri/i-have-adhd, 25,853 stars and MIT, pitches itself as stopping your coding agent burying the answer. Its optional always-on mode is a SessionStart hook injecting the full ruleset every session. First the cap check, because this site measured that hook output above 10,000 UTF-16 units silently drops 78 percent: the payload is 6,598 units, 66 percent of the cap, so it arrives intact. Then twelve paired rounds on 2.1.251 running the real hook. COST: 1,475 tokens every session, single-valued in both arms, and that figure independently confirms the payload was not truncated because 6,598 units of prose predicts about 1,478. BEHAVIOUR: replies are shorter, median 855 characters against 712, about 0.83x, but THE ARMS OVERLAP, so it is a median difference across twelve rounds and not a resolved per-run effect. At four rounds they did not overlap and the headline would have been wrong. - The MCP curves did not move, in the week the floor fell 24.5 percent: https://aieveryminute.com/mcp-curve-2-1-251.json Ten pooled rounds on 2.1.251, none refused, re-measuring both MCP curves the calculator interpolates from. All six cells are IDENTICAL to 2.1.246 across five releases: a deferred tool holds at about 16 tokens and a loaded one at about 652. The contrast is the point. In the same week the startup floor fell 24.5 percent and the skills curve moved at every count past 200, and these did not move at all. No cause is offered for why one curve moved and another did not. Every single deviation from every mode across sixty cell readings is exactly plus or minus 897 tokens, this machine known intermittent component, and nothing else appeared, which is why a nine-of-ten cell is not a weak cell here. A null on a load-bearing calculator curve is worth as much as a change, and holding from 2.1.232 to 2.1.251 is a reason to trust these today and none at all to stop re-checking them. - The floor fell 24.5 percent and the path term did not move at all: https://aieveryminute.com/path-floor-2-1-251.json Your working-directory path sits in the system prompt and changes the startup floor. The sharpest version of that finding is a reciprocal: a 38-character path made of many one-character segments costs MORE than a 47-character path made of one repeated character, which rules out length directly rather than by inference. Re-measured on 2.1.251 across twelve pooled rounds: 17,102 against 17,082, a gap of exactly 20 tokens, identical to the gap measured on 2.1.241 ten releases earlier. Both paths fell by exactly the same 5,727, so the release change shifted this term rather than rescaling it. Every excursion in 24 readings was exactly 897, the known component. The ladder and the ten-path range were NOT re-run, so the calculator stamp deliberately stays at 2.1.241 and the corpus says so. - A 286-skill pack the synthetic curve got right, next to an 817-skill pack it got 5x wrong: https://aieveryminute.com/ecc-pack-2-1-251.json affaan-m/ECC, 244,921 stars and MIT. Two findings. FIRST, counting SKILL.md files on disk gives 898 and would have made this the largest pack measured here; 519 of those are documentation copies and the manifest declares only ./skills/, so what actually installs is 286. That is a 3.1x over-count and a concrete instance of the a_repo_is_not_always_what_the_plugin_ships limit this site already publishes. SECOND, the real 286 skills cost 9,210 tokens at startup, 32.2 per skill, single-valued in 12 of 12 rounds. This site synthetic curve interpolates to 8,635 at that count, so it is 6.7 percent out. The same curve was 5.15x LOW on a real 817-skill pack. No cause is asserted for the contrast: the packs differ in description length, median 258 against 396, and the 817 corpus own ablation showed identical descriptions costing 9,952 where unique ones cost 56,622, but nothing here measures which of those drives the gap. - What preserving PDF structure costs in tokens: https://aieveryminute.com/pdf-extractor-2-1-251.json Feeding a PDF to a model means extracting text first, and the extractor changes both what survives and what you pay. firecrawl/pdf-inspector (17,147 stars, MIT) against pdftotext on the same 10-page PDF. WHAT SURVIVES: pdf-inspector keeps 60 markdown headings and 40 table rows; pdftotext keeps none of either. The decisive case is a colour table, reconstructed as pipe-delimited rows with each hex under its colour name, where pdftotext emits names and hex codes as disconnected lines and the association is lost. WHAT IT COSTS: 2,162 tokens against 1,986, so structure costs 176 extra tokens or 8.9 percent, measured across six paired rounds. Per-byte rates are nearly identical, 0.4459 against 0.4509, so the extra cost is the extra characters rather than markdown tokenising worse. SPEED: both vendor claims hold with margin and classification beats its own stated band, 0.6ms against a claimed 10 to 50. The OCR path was deliberately not run and the corpus says why. - What book-to-skill costs to have installed, and which of its own published figures reproduce: https://aieveryminute.com/booktoskill-2-1-251.json virgiliojr94/book-to-skill (27,490 stars, MIT) turns a book into an agent skill and publishes unusually specific first-party numbers. WHAT IT COSTS: 159 tokens a session at a 398-character description, and three arms cost that same figure to the token, the whole 104-file cloned tree, SKILL.md alone, and SKILL.md truncated to its YAML frontmatter, so the 38,205-byte body is free while the skill is not invoked. WHICH OF THIS SITE’S OWN TWO ANSWERS IS RIGHT: the /cost/ model of 0.276 tokens per character is printed with a 30-to-300 validity bound and extending it here predicts 134, which is 25 low; the band measured on five real English descriptions installed one at a time, 0.363 to 0.482 per character, predicts 144 to 192 and brackets the measurement, whose own rate is 0.400. The page has been corrected to carry the bound. THE DOC’S EXTRACTION FIGURES REPRODUCE: raw pdftotext gives 119,237 against a claimed 119K and 229,371 against 229K; the current CLI reports 0.6 to 1.0 percent lower because its own clean_pdftotext strips headers and dehyphenates, a change the changelog dates after the table. THE CHAPTER COUNT DOES NOT: 28 against a documented 19 on Think Python 2, and 5 on Pro Git of which none is a chapter, four being Subversion revisions in git svn blame output and the fifth a stranded superscript from 2^80 in ordinary prose. No warning fires, because the guard needs a different detection method. On Pro Git the project’s own benchmark then treats that superscript as the target chapter and reaches a baseline of 73 percent of the whole book. On Think Python 2 the same benchmark misses its own published discovery figure by 60 percent, and that gap is NOT the miscount: suppressing all nine spurious numbers moves it ten tokens. - The floor fell 23 percent and a parallel subagent did not get cheaper: https://aieveryminute.com/fanout-slope-2-1-251.json A re-measurement of the calculator constant that had gone thirteen releases without one, and the answer is a null worth having. Forty runs, eight rounds, none failed, every arrival control held, and every run on ONE identical working directory so no difference carries a path term. THE SAME ROUNDS MEASURE BOTH THINGS, which is what makes the comparison worth anything: the trivial isolated floor came back 17,048 in all eight rounds with zero spread, against 21,890 to 22,787 from the same cell of the same harness on 2.1.238, a fall of about 23 percent. The per-subagent slope did not follow it. At the modal six parent turns it is 20,751 to 21,326, and the 2.1.238 figure the calculator still quotes, 20,581, sits just below that, 170 tokens or 0.8 percent from the nearest edge. So the release that made a session a quarter cheaper did not make an extra parallel agent cheaper. NO POINT ESTIMATE IS PUBLISHED and that is the data rather than the round count: no context value repeats across rounds in either fan-out cell, so guarded mode refuses, and more rounds would tighten the range without producing a repeating value. The inline control, which separates per-subagent from per-unit-of-work, is 166.3 to 169.7 tokens a file. THE LOADED ARM WAS DELIBERATELY NOT RUN, so configShare is marked not measured rather than carried forward, and the corpus declares in its own limits that it does not license a stamp bump: the calculator consumes a point per turn count and needs two conditions, this publishes one, and the estimator refuses a point in it. - Which deny-rule path shapes actually match a .env in the project root: https://aieveryminute.com/deny-rule-shapes-2-1-251.json An earlier corpus here held the rule fixed at Read(./.env) and varied the ROUTE, closing with an open line about other glob patterns and absolute paths. This varies the RULE, which is the half a reader controls. Forty runs, ten arms, four rounds each, none excluded. ALL SIX RELATIVE SHAPES BLOCK 4 of 4: ./.env, .env, **/.env, *.env, ./*.env and ./.env*, and so does the documented absolute form //path. TWO DO NOT. A rule with ONE leading slash, which looks absolute, never blocks: the Read tool succeeds and returns the file in all four rounds. That is documented behaviour and the documentation warns about it in a Warning box, so it is not reported as a defect; what is published is that the read comes back with no error. The same file named through /tmp, which on macOS is a symlink to the real system temporary directory, in the documented absolute form also did not block, reported as an observation and not as a doc contradiction, since the page describes Claude ACCESSING a symlink rather than a rule WRITTEN through one. THE INSTRUMENT IS THE SESSION TRANSCRIPT, NOT A CANARY, and that is the real contribution: on 2 of 40 runs the model read the file and then chose not to print the secret, so canary scoring would have called those protected. The model opened the deny rule itself in 23 of 40 runs, which is the largest confound in the design and is published as one. permission_denials was empty on all 40 runs including all 28 blocked ones, so it observes nothing and nothing is concluded from it. ## All measurements https://aieveryminute.com/measurements/ Every figure published here, grouped by the run it came from. ## Tools — https://aieveryminute.com/tools/ Hands-on trials of AI tools, models and websites, plus the tools that pair with them. Every trial is first-hand: installed, run, and published with what it cost, what broke and what it does badly. Where a project publishes a claim about someone else, that claim gets checked against a real run rather than repeated. ### A book costs 41% more in Claude Code than tiktoken says https://aieveryminute.com/a-book-costs-41-percent-more-than-tiktoken-says/ Published 2026-08-21. I ran book-to-skill v1.4.0 on the 501-page Pro Git PDF, then measured what the extracted text actually costs in Claude Code context against its tiktoken count. The gap is 1.41x on a technical book and 1.41x on Moby-Dick, and because the two books have different cl100k rates, solving both together puts any per-byte framing charge at or below zero, which bounds an artefact of how Claude Code wraps the file without ruling one out. Anthropic publishes ~15-20% for typical text and an unbounded "much more" for code, and only the novel is the typical-text case; both books measured the same 1.41x regardless, which that split does not predict. It leaves book-to-skill's headline ratio underdetermined rather than refuted, and makes its dollar table low. Running it also surfaced three things a README cannot tell you, including a CLI that reports 93 chapters in a ten-chapter book because it counts shell comments inside code samples. Measured: - tool: book-to-skill v1.4.0, commit 3a97a71, 23,485 stars - book: Pro Git, 501 pages, 17.97 MB PDF - this trial: 28 runs, six cells of four rounds plus a floor per round - extraction: 501 pages in 0.55-0.64s - claude vs cl100k: 1.41x on both books - anthropic publishes: typical text 15-20% (1.18-1.25x); code unbounded - every segment: between 1.382x and 1.452x - their CLI vs their docs: 198K against 229K on one book - claude code: 2.1.238 ### Anthropic ships five skill plugins, not one, and its own ~100-token figure fits the one I could measure https://aieveryminute.com/anthropic-ships-five-skill-plugins-not-one/ Published 2026-08-20. Anthropic documents ~100 tokens per Skill for the metadata loaded at startup. Its own repository ships five plugins rather than one, and only one of them is large enough for this instrument to give a figure. example-skills, twelve skills with ordinary 288-character descriptions, measured 102 tokens per skill. The three single-skill plugins cannot be measured: two carry ranges 198 tokens wide and the third spans 897, from minus 578 to plus 319, which contains zero. A separate sweep then knocked out the story I had been about to tell about long descriptions: above 500 characters they are close to free. Which turns out not to mean what it sounds like, because a later measurement found the text stops reaching the model entirely past about 1,400 characters. Measured: - pack: anthropics/skills, commit f379e5a - runs: 50, ten rounds x five plugins - documented: ~100 tokens per Skill - measurable plugin: 102 per skill, 288-char median - no figure published: 4 of 5, one range spans zero - description above 500 chars: 0.015 tokens per character - claude code: 2.1.234, description sweep on 2.1.235 ### A 19,700-star diagram skill drew once in four unattended runs. One file fixes it https://aieveryminute.com/a-diagram-skill-drew-once-in-four-unattended-runs/ Published 2026-08-17. A trending Claude Code skill for editorial diagrams, installed and measured on 2.1.233. What it costs to sit on disk unused turned out not to be measurable: twelve readings of the same pack span 70 to 967 tokens. Run unattended on an ordinary request it delivered a file in 1 of 4 rounds, spending 733,824 tokens across the three that produced nothing, because SKILL.md tells the agent to pause and ask about brand colours before the first diagram in a project. Dropping a one-line .diagram-design marker in the project root took it to 4 of 4. No cost multiplier is quoted, because the plain arm alone varied 6.7x and the skill's arm has one delivering round. The marker effect is established at p=0.018; the skill-versus-plain comparison is underpowered and is not claimed. Measured: - pack: cathrynlavery/diagram-design, commit 09df49d - stars: 19,728 on 2026-08-17 - installed, never used: not measurable, 70 to 967 - delivered a diagram: 1 of 4 unattended - with the marker file: 4 of 4 - spent producing nothing: 733,824 tokens - type references read: 2 to 3, never 1 - claude code: 2.1.233, re-verified on 2.1.234 ### google/skills costs 9,020 tokens. Every session, before you use any of it. https://aieveryminute.com/google-skills-costs-9020-tokens-every-session/ Published 2026-08-15. Six trending skill packs, cloned and installed on Claude Code 2.1.233, measured against an empty floor, then re-measured after I found I had priced one of them wrong. 111 skills cost 9,020. The 25 skills Matt Pocock actually ships cost 852, not the 1,209 I first published for 35 files sitting in his repository. Per skill that is 34 to 96 tokens, and multiplying a published rate by a file count would have got two of the three badly wrong. Measured: - claude code: 2.1.233 - runs: 56 across two harnesses, no exclusions - clean floor: 22,283 then 22,287, each round paired - google/skills, 111 skills: 9,020 tokens - addyosmani/agent-skills, 24: 2,302 tokens - mattpocock/skills, 25 shipped: 852 tokens, corrected from 1,209 - per skill: 34.1 to 95.9 - my own model: 8% low, 63% high, 64% high - re-verified 2026-08-18: diagram-design 70 to 967 on 2.1.234 ### The docs say ~100 tokens a skill. I measured 34 to 96. https://aieveryminute.com/the-docs-say-100-tokens-a-skill-i-measured-34-to-96/ Published 2026-08-15. Anthropic documents ~100 tokens per Skill for the metadata loaded at startup. Measured across the three trending packs where a measurement is reliable, per-skill cost ran 34.1 to 95.9, all below the documented figure, over-predicting by 4% on one pack and 193% on another, and the size of the miss does not order by description length. Then I asked six model runs the same question: three repeated the documented number, one gave a different one, and two declined, including the model that had given a different one moments earlier. Measured: - documented: ~100 tokens per Skill - measured, 3 packs: 34.1 to 95.9 per skill - worst over-prediction: 193%, mattpocock/skills - closest: 4%, addyosmani/agent-skills - claude code: 2.1.233 - source runs: 56, two harnesses - model runs: 6, four models, three vendors - re-verified 2026-08-18: one-skill pack 70 to 967 on 2.1.234 ### 720 of 720 table cells read correctly. My scorer was the thing that kept failing https://aieveryminute.com/720-of-720-table-cells-read-correctly/ Published 2026-08-09. Every value in the test tables was unique, so a misplaced number could be told apart from a misread one. The model got all 720 cells right. The scorer got it wrong three times, always for the same reason, and twice it would have produced a confident and completely false headline. Measured: - cells checked: 720 - read correctly: 720 - misplaced cells: 0 - table sizes: 5x4, 10x6, 20x8 - scorer bugs caught: 3 - currency and negatives: read correctly ### I gave two AI detectors text from 1859 and 1996. Both called it human, but the newer one scored 23% AI https://aieveryminute.com/ai-detectors-on-text-from-1859-and-1996/ Published 2026-08-09. A hands-on trial of free AI detectors using text that cannot be disputed: an 1859 book, a 1996 internet standard, and two essays generated on the spot. Every verdict is shown. The detectors were right, and the margins say something the pass/fail does not. Measured: - texts tested: 4 - GPTZero verdicts correct: 4 of 4 - Darwin 1859: 100% human - RFC 1958, 1996: 77% human / 23% AI - AI essays: 100% AI, both - detectors tried: GPTZero, QuillBot - re-verified 2026-08-09: model label matters ### A detector called an 1859 book AI-generated while its own numbers said 85% human https://aieveryminute.com/detector-said-ai-generated-while-its-own-breakdown-said-85-percent-human/ Published 2026-08-09. A hands-on trial of GPTinf, a free unlimited no-signup AI detector, using a public domain passage from 1859. The tool contradicted itself three times out of three, and the number most people would read is the one that moved. Measured: - text tested: Darwin, 1859 - verdict sentence shown: "seems to be AI generated" - its own breakdown: 0% AI, 85% human - contradiction observed: 3 of 3 runs - headline on identical text: 18% then 0% - punctuation-only edit: human 85% to 65% ### A free AI detector switched to an older model when my scans ran out, then called an 1859 book 100% AI https://aieveryminute.com/gptzero-swapped-models-when-my-free-scans-ran-out/ Published 2026-08-09. A hands-on trial that nearly produced a spectacular false finding. GPTZero classified an 1859 public domain passage as 100% AI-generated, with high confidence. The text had not changed. The model that judged it had. Measured: - text tested: Darwin, 1859 - verdict on Model 4.8b: 100% human - verdict after quota: 100% AI - model after quota: 3.3b / 2025-03-13-base - text changed between runs: not at all - warning shown about the swap: none ### I tried to make a free image model misspell a word. It got all six right https://aieveryminute.com/i-tried-to-make-a-free-image-model-misspell-a-word/ Published 2026-08-09. Text in AI images is supposed to be the thing that still breaks. I gave FLUX.1-schnell six words to render, from OPEN up to a rare twelve-letter genus, at a fixed seed on a free public Space. All six spelled correctly. Measured: - model: FLUX.1-schnell - words attempted: 6 - spelled correctly: 6 - hardest word: ZEPHYRANTHES - seed: 12345 - cost: free, no account ### Tesseract hit 53% word error at 6px text. Claude vision hit 0% https://aieveryminute.com/tesseract-hit-53-percent-error-at-6px-text-claude-vision-hit-zero/ Published 2026-08-09. A ladder of deliberately degraded document images with exact ground truth, read by tesseract and by Claude vision. JPEG quality turned out to be irrelevant. Text height turned out to be everything, with a cliff between 7 and 6 pixels that one engine fell off and the other did not notice. Measured: - tesseract at 6px text: 53.3% word error - Claude vision at 6px: 0.0% word error - JPEG quality 5: 0.0% for both - tesseract cliff: between 7px and 6px - unguessable-text control: passed - ground truth: exact, 45-54 words ## Claude Code — https://aieveryminute.com/claude-code/ Independent measurements of Claude Code: context cost of skills, MCP tools, CLAUDE.md and images, cache and round-trip behaviour, and where the documentation is wrong. Every figure came off a real run. ### Your skill description is cut at 1,536 characters, and the cap is a setting you can raise https://aieveryminute.com/a-skill-description-stops-being-read-at-1400-characters/ Published 2026-08-20. A rare marker planted at a known character offset inside a skill description, then asked for. Text ending at 1,530 characters is answered from context with no tool calls, four rounds of four; text ending at 1,536 is never answered from context, and the model shells out to read the file instead. That lands within six characters of the documented skillListingMaxDescChars default of 1,536. A 51-character increase in the skill name moves it not at all, and the tail past the cut is text you wrote that Claude never sees when deciding whether your skill is relevant. Corrected 2026-08-21: an earlier version of this post framed the cut as undocumented and cited a 1,024-character maximum. Measured: - cut between: 1,530 and 1,536 characters - at 1,400: in context 4 of 4 rounds - at 1,550: in context 0 of 4 rounds - documented cap: 1,536, skillListingMaxDescChars default - skill name length: 51 more characters moved nothing - corrected 2026-08-21: the cut is documented, and configurable - cut moves with skill count: no, marker at 1,400 held at 1, 10 and 40 - counts characters not tokens: dense text at 2.06x still in at 1,400 - claude code: 2.1.235 ### A trending 817-skill pack costs about 56,600 tokens before you type anything https://aieveryminute.com/an-817-skill-pack-costs-56622-tokens-before-you-type/ Published 2026-08-20. An 817-skill cybersecurity pack from GitHub Trending costs about 56,600 tokens of startup context in an empty isolated project, 69.30 per skill, from 24 readings of which 21 sit inside a 48-token band. The synthetic skills curve behind this site's own calculator interpolates to 13.46 per skill at that count, 5.15x low, and its 1,000-skill cell still re-measures correctly. Description uniqueness and the pack's extra frontmatter were each tested here and refuted; deleting 338KB of frontmatter moved the reading by 8 tokens. The cause is not established: neither the synthetic nor the real fixture carries its descriptions whole, which rules the comparison out as a test of what the words cost. Measured: - pack: mukul975/Anthropic-Cybersecurity-Skills, commit 4c0b700 - skills: 817, counted off disk on every arm run - measured: 56,622 tokens, 69.30 per skill - published: 30 rounds, 80 runs, five arms - synthetic curve predicts: 13.46 per skill at 817 - extra frontmatter: not charged, +8 tokens on 338KB - claude code: 2.1.236 ### "A 5,000-token CLAUDE.md is a 5,000-token tax per turn." Half of that is true. https://aieveryminute.com/a-claude-md-is-a-tax-per-turn-half-of-that-is-true/ Published 2026-08-15. An 18KB CLAUDE.md adds 4,967 tokens to every turn of a session on Claude Code 2.1.233, and it never decays. But it costs $0.04966 and $0.04967 on turns one and two and $0.00248 on turns three and four, because the file stops being written to cache and starts being read from it. The context claim is true and the cost claim is true for about two turns. Measured: - claude code: 2.1.233 - runs: 32, no exclusions - CLAUDE.md: 18,000 bytes, 4,967 tokens - added to every turn: 4,967, all 4 of 4 turns - cost, turns 1-2: $0.04966 then $0.04967 - cost, turns 3-4: $0.00248 per turn - drop: 20x - on 2.1.232: 4,969 tokens, 2 apart ### 400 commands cost 9,989 tokens. 400 agents cost 20,404. https://aieveryminute.com/400-commands-cost-10088-tokens-400-agents-cost-20503/ Published 2026-08-14. One widely shared write-up prices a plugin at about 100 words of metadata each and tells you to keep four to six. Measured on 2.1.231, a plugin has no single price: a skill or a slash command costs about 40 tokens and collapses in bulk, while a subagent definition costs 51 and never does. Measured: - claude code: 2.1.231 - re-verified on 2.1.232: reproduced, within one 198-token component - runs: 88, no exclusions - a slash command: 40.0 tokens, then 5.6 - a subagent definition: 51 tokens, never less to 400 - 400 commands: 9,989 tokens, or 10,187 when the component fires - 400 agents: 20,404 tokens, or 20,602 - re-verified on 2.1.238: 8 rounds, marginal rates reproduced - 200 commands beside 40 skills: 25.7 tokens each - 200 agents beside 40 skills: 39.0, unchanged ### The same forty skills cost 1,558 tokens, or 152 https://aieveryminute.com/a-skill-costs-39-tokens-or-3-9/ Published 2026-08-14. A skill is not a fixed price. On a clean project the marginal cost falls from 39.0 tokens to 3.9 between 200 and 400 skills; on a configured machine it is 3.8 straight away. That is where this site measured 1.5 tokens a skill from, and why that figure never reproduced in isolation. Measured: - claude code: 2.1.231 - runs: 132, no exclusions - clean project, below 200 skills: 39.0 tokens each - clean project, past 400 skills: 3.9 tokens each - 40 skills, clean project: 1,558 tokens - 40 skills, config loaded: 152 tokens - description, at 40 skills: 0.276 per char, to 300 chars, English prose only - description, at 600 skills: 300 chars costs no more than 30 - body size, the control: no effect across 140x - re-verified on 2.1.233: unchanged, 39.05 per skill at 200 ### A 7.9MB node_modules cost 2 tokens, and .claudeignore does nothing https://aieveryminute.com/a-79mb-node-modules-cost-2-tokens/ Published 2026-08-13. Measured on Claude Code 2.1.229. A lean project cost 22,401 tokens at startup in four runs out of four, identically. The same project carrying 7.9MB and 1,960 files of node_modules cost 22,403. Across 24 runs, including 12 that invited exploration, node_modules was read zero times. A .claudeignore excluding it did not stop a file inside it being read, 3 of 3, while the documented deny rule blocked it 3 of 3. Measured: - claude code: 2.1.229 - lean project, startup: 22,401 tokens, 4 of 4 identical - plus 7.9MB node_modules: 22,403 tokens (+2) - node_modules read: 0 times in 24 runs - .claudeignore blocks a read: no, canary returned 3 of 3 - deny rule blocks a read: yes, 3 of 3 - .claudeignore in the docs: 0 of 193 pages ### The 150-rule limit for CLAUDE.md measures something a CLAUDE.md is not https://aieveryminute.com/claude-md-held-at-500-rules/ Published 2026-08-13. Measured on Claude Code 2.1.229 across 35 runs, no exclusions. A rule at the top of the file and a rule at the very bottom were both obeyed in 15 of 15 function definitions at 2, 10, 50, 150, 300 and 500 rules. The negative control scored 0 of 15 on both. This does not refute the benchmark behind the 150 figure, which measures a different thing. What 500 rules did cost was 23,427 tokens of context. Measured: - claude code: 2.1.229 - runs: 35, no exclusions - rule counts tested: 2, 10, 50, 150, 300, 500, plus a control - top rule: 15 of 15 at every count - bottom rule: 15 of 15 at every count - negative control: 0 of 15 on both rules - what 500 rules cost: 23,427 tokens of context - functions written: 3 of 3 in 35 of 35 runs ### Fanning out was about 1.7x slower on work built to favour it https://aieveryminute.com/fanning-out-was-slower-on-work-built-to-favour-it/ Published 2026-08-13. Measured on Claude Code 2.1.229 across 24 runs with no exclusions. Eight agents took 29.6s against 16.8s inline, a 12.8-second gap; four took 20.0s against 11.7s. Tokens ran 1.82x and 2.17x. Every arm is separated. The delegation control confirms the parent made zero file operations in all twelve fan-out runs, which took a field called parent_tool_use_id to establish. Measured: - claude code: 2.1.229 - runs: 24, no exclusions - eight agents: 29.6s against 16.8s inline - four agents: 20.0s against 11.7s inline - wall clock ratio: 1.76x and 1.72x, no overlap - tokens: 1.82x and 2.17x, no overlap - parent file operations: 0 in all 12 fan-out runs - markers found: 24 of 24 ### Halving the output saved 6.8% of the bill https://aieveryminute.com/halving-the-output-saved-6-percent/ Published 2026-08-13. Measured on Claude Code 2.1.229 across 30 runs in three batches, no exclusions. Output is 1.45% of billed tokens and 10.2% of cost, seven times its token weight, because the input side is mostly cache reads at a tenth of the output rate. A terseness instruction halved the output and saved 6.8%. Every terse run cost less than every default run. Measured: - claude code: 2.1.229 - runs: 30, no exclusions, three batches - reply length: 4,733 chars to 1,969 - output tokens: 1,821 to 842 median, a 54% cut - what it saved: $0.4489 to $0.4184, 6.8% - output share of tokens: 1.45% - output share of cost: 10.2% - arms separated: on output and on cost - markers echoed: 30 of 30, all three turns ### Forcing the cache expiry everyone warns about made the session 7.6% cheaper https://aieveryminute.com/the-five-minute-cache-expiry-did-not-happen/ Published 2026-08-13. Measured on Claude Code 2.1.229 across 24 runs with no exclusions. After an eight-minute break the next turn cost $0.017 on the default cache and $0.118 with the five-minute TTL forced on, 6.8x. Summed over all three turns the same forced arm cost $0.370 against $0.400, and every forced run beat every default run. It wrote 50% more cache tokens and still cost less. A separate bracket found the default cache alive at 55 minutes and gone by 70. Measured: - claude code: 2.1.229 - runs: 24, no exclusions (15 + a 9-run hour bracket) - turn after an 8-minute break: $0.017 default, $0.118 forced 5m - whole 3-turn session: $0.400 default, $0.370 forced 5m - session arms: separated, 5 of 5 each way - cache written, forced arm: 52,731 tokens against 35,139 - write tier, default: 1-hour tier, 0 to the 5-minute tier - kept at 55 min, lost at 70: 3 of 3 each way - markers echoed: 24 of 24 across both runs ### The third step cost less than the first, in the session that was never cleared https://aieveryminute.com/the-long-session-got-cheaper-per-step/ Published 2026-08-13. Measured on Claude Code 2.1.228, ten rounds per arm, on three steps where each builds on the last. Keeping one session cost 274,436 tokens against 344,408 for clearing between every step, with the fresh arm dearer in 10 of 10 paired rounds. Both lose to asking once: the worst single-prompt run beat the best run of either multi-prompt arm on every measure. All 30 runs passed the hidden suite. Measured: - claude code: 2.1.228 - rounds: 10 per arm, 30 runs on 2.1.228 - one prompt: 103,828 tokens, $0.1697 - one session, three prompts: 274,436 tokens, 2.64x - fresh session each step: 344,408 tokens, 3.32x - fresh dearer: 10 of 10 paired, sign p=0.0010 - resumed step 3 vs its own step 1: -25,312 tokens, 10 of 10 - success: 42 of 42 passed, both versions - re-verified: 2.1.229, every claim held ### A deny rule on .env leaked nothing in 20 runs, and 14 of 24 got through another way https://aieveryminute.com/what-a-read-deny-rule-on-env-actually-stops/ Published 2026-08-13. Measured on Claude Code 2.1.229 across 88 runs with a fabricated credential as the canary. With permissions.deny Read(./.env) in place, the Read tool, cat, head, sed and grep leaked 0 times in 20. A python one-liner, a node one-liner and dd leaked 14 times in 24. Every result matches the documented boundary; what the documentation does not convey is that the uncovered side has no enforcement, only judgement. Measured: - claude code: 2.1.229 - runs: 88, canary-scored - rule tested: deny: Read(./.env) - Read tool, cat, head, sed, grep: 0 of 20 leaked, 18 blocked at the layer - python, node, dd: 14 of 24 leaked - control, no rule at all: 20 of 20 and 20 of 24 leaked - declined with no rule present: 4 runs - matches the documentation: yes, boundary and mitigation ### Compaction kept all 64 planted values, across 24 compactions https://aieveryminute.com/compaction-kept-all-64-planted-values/ Published 2026-08-12. Measured on Claude Code 2.1.228. Eight unguessable values were planted at spaced turns, the session was driven past the compaction threshold, and every value was asked back with files off-limits. 64 of 64 survived across 24 compactions and two bulk types. This tests recall of discrete stated values, which is not the same thing as preserving nuance, and the post says so. Measured: - claude code: 2.1.228 - values recalled: 64 of 64 - compactions survived: 24 - dense arm: 32 of 32, 4 compactions per run - sparse arm: 32 of 32, 2 compactions per run - dense bulk compressibility: 25.03% vs 0.79% under gzip -9, 31.7x - positive control: 8 of 8 without compaction - compaction window: forced to 100k ### Planning made the execution cheaper, and cost more than the job it was planning https://aieveryminute.com/plan-mode-cost-1-95x-more-on-a-specified-task/ Published 2026-08-12. Measured on Claude Code 2.1.228. Plan-then-execute and straight auto-accept were run four times each on the same bug fix with an objective pass/fail. Both passed 4 of 4. Planning genuinely made the doing cheaper, $0.1685 against $0.2062, but the plan call cost $0.2301 on its own, six times the saving it bought. The workflow cost 1.69x the tokens and 1.95x the dollars. Measured: - claude code: 2.1.228 - auto-accept, one call: $0.2062 median - execute call, plan in hand: $0.1685, cheaper 4 of 4 - plan call alone: $0.2301, dearer 4 of 4 - whole workflow: 1.95x dollars, no overlap - turns: 9 against 15 - success: 4 of 4 in both arms - known limit: bash denied outside plan mode - corrected 2026-08-12: withdrew the "mechanism is backwards" claim and the title asserting it - corrected again, same day: withdrew "upper bounds"; the equal-permission re-run is larger ### Planning helped least on exactly the work its advice recommends it for https://aieveryminute.com/planning-helped-least-on-open-ended-work/ Published 2026-08-12. Measured on Claude Code 2.1.228 over seven paired rounds, with an open-ended prompt and a hidden rubric applied afterwards. Plan-then-execute cost 2.29x the tokens, 2.27x the dollars, 2.14x the turns and 2.17x the tool calls, with no overlap on any of them. Every run in both arms scored full marks on the rubric, which is a real limit on what the quality comparison can say, and the plan arm paid for a broken exit tool, which makes the ratios upper bounds. Measured: - claude code: 2.1.228 - paired rounds: 7, bash control 22/22 - auto-accept: $0.2910, 146,179 tokens - plan then execute: $0.6600, 334,425 tokens - dollar ratio: 2.27x, no overlap - tool calls: 6 against 13, no overlap - rubric: 11 of 11 in all 15 runs - execution saving: 29% specified, 3% open-ended ### The plan cost 1.47x the whole job, and made the job 29% cheaper https://aieveryminute.com/the-plan-cost-more-than-the-job-it-planned/ Published 2026-08-12. Measured on Claude Code 2.1.228, eight runs per arm, with --allowedTools Bash on every call in both arms and a full tool-call record behind every turn count. Plan-then-execute cost 2.01x the tokens and 2.18x the dollars of straight auto-accept, with no overlap. Split by call: the execute call is cheaper than doing it in one shot in 8 of 8, and the plan call alone is dearer than the entire one-shot run in 8 of 8. Measured: - claude code: 2.1.228 - runs: 8 per arm, bash control 24/24 - auto-accept, whole job: $0.1493, 114,268 tokens - plan call alone: $0.2192, dearer 8 of 8 - execute call, plan in hand: $0.1057, cheaper 8 of 8 - whole workflow: 2.18x dollars, no overlap - tool calls: 5 against 9.5, no overlap - success: 8 of 8 in both arms ### The same fan-out costs 53% more, about a third of the time https://aieveryminute.com/the-same-fan-out-costs-53-percent-more-a-third-of-the-time/ Published 2026-08-12. Measured on Claude Code 2.1.228 across 26 runs. An eight-agent fan-out has two execution modes separated by num_turns: nine turns costs 207,102 tokens, one turn costs 317,786, for the same eight files. It happened in 3 of 10 runs, which means a single measured figure for per-agent cost is not one number. Measured: - claude code: 2.1.228 - cheap mode, 8 agents: 207,102 tokens (turns=9) - expensive mode, 8 agents: 317,786 tokens (turns=1) - difference: 1.534x, +53.4% - frequency: 3 of 10 runs - overlap between modes: none - per agent, cheap mode: 20,293 to 20,325 tokens - per agent, expensive mode: 36,105 to 36,137 tokens - runs: 26 used, arrival control 34/34 across the corpus - corrected 2026-08-22: four phantom medians replaced with observed ranges ### Each parallel subagent costs 20,398 tokens, or 92,304 with your config loaded https://aieveryminute.com/each-parallel-subagent-costs-20398-tokens/ Published 2026-08-11. Measured on Claude Code 2.1.227 across 44 runs, of which 4 sit behind the configured figure. Cost per parallel subagent is flat at about 20,400 tokens on a minimal config, so eight agents cost 3.2x one rather than 8x. On a machine carrying 37,808 tokens of user-level configuration, the per-agent cost rises to 92,304, which is about twice that configuration rather than most of it. The 75,466 figure this post originally used was the same quantity counted twice; see the correction. Re-measured on 2.1.231 the same slope is 20,690 and 103,397, about twice the configuration again. Measured: - re-verified on 2.1.232: unchanged at matched turn count - claude code: 2.1.227 - per parallel subagent: 20,398 tokens - per subagent, configured machine: 92,304 tokens (4 runs) - 8 subagents vs 1: 3.20x - 8 subagents vs inline: 4.44x - same work inline: 299 tokens each - runs: 44, arrival control 44/44 - corrected 2026-08-12: the 288,040 outlier was a second execution mode, 3 of 10 on 2.1.228 - corrected 2026-08-13: the config figure was double-counted; 37,808 not 75,466 ### The env var sold as a 90% saving cost 8% more https://aieveryminute.com/the-fork-subagent-flag-cost-8-percent-more/ Published 2026-08-11. Measured on Claude Code 2.1.227 across 28 runs. The flag is real and it engages in print mode, but on both a cold and a genuinely warm parent it increased cost rather than reducing it, because each child inherits the parent conversation instead of skipping the prefix. Every flag-on run used more context than every flag-off run; cache creation barely moved and 97% of the increase arrived as cache reads. Measured: - claude code: 2.1.227 - cold session cost, 4 agents: +8.1% with the flag on - warm session cost, 4 agents: +15.6% with the flag on - cold total context: +38.2% - warm total context: +65.6% - cache creation change: +1.7% to +5.4% - per agent, flag off: 92,525 tokens - per agent, flag on: 135,232 tokens - runs: 28, arrival control 28/28, warmth control 8/8 ### A 2MB screenshot costs the same as a 623-byte one: 3,135 tokens https://aieveryminute.com/a-2mb-screenshot-costs-the-same-as-623-bytes/ Published 2026-08-09. Measured the context cost of reading images at eleven sizes, twice each. A 3,366x range in file size produced a 5-token difference. Above 2000px the cost stops growing entirely, and a retina screenshot costs 1.69x a normal one for exactly the same picture. Measured: - claude code: 2.1.224 - 1568x1568 image: 3,135 tokens - same image at 2MB: 3,140 tokens - client-side resize cap: 2000 px - retina vs logical: 1.69x - 3072x3072 vs 2048x2048: no extra cost ### A skill costs 1.5 tokens, an MCP tool 15, and a subagent 1.72x the whole task https://aieveryminute.com/a-skill-costs-1-5-tokens-an-mcp-tool-costs-15/ !! SUPERSEDED 2026-08-14 on Claude Code 2.1.231. Two of the three headline prices below moved and the figures in this entry are NOT current: a skill costs about 39 tokens on a clean project, not 1.5, and a loaded MCP tool measures 652 tokens, not 304. See /a-skill-costs-39-tokens-or-3-9/ and /mcp-curve-2-1-246.json. Published 2026-08-09. Skills, MCP servers, subagents and CLAUDE.md do overlapping jobs at wildly different prices: 1.5 tokens, 15 tokens, 1.72x the entire task, and 0.26 tokens per byte on every single round trip. The right choice is usually obvious once you can see the numbers. Measured (SUPERSEDED, see above): - claude code: 2.1.226 - one skill: ~1.5 tokens - one MCP tool, deferred: ~15 tokens - one MCP tool, loaded: ~304 tokens - subagent delegation: 1.72x total - CLAUDE.md: 0.26 tokens/byte, always ### Claude Code's auto-compaction fails with `too_few_groups` when one turn fills your context https://aieveryminute.com/auto-compaction-fails-with-too-few-groups/ Published 2026-08-09. Measured when auto-compaction actually fires and whether it works. It reports a status event carrying compact_result and an undocumented compact_error. Filling the context in one turn produces failed / too_few_groups, reproducibly, which is precisely the case people hit. Measured: - claude code: 2.1.226 - single turn, 122,575 tokens: failed / too_few_groups - multi-turn, 115,471 tokens: success - --autocompact range: auto or 100k–1M only - values that disable it: none - visible in json output: no ### Trimming CLAUDE.md saves under 4%. Batching your tool calls saves 49% https://aieveryminute.com/trimming-claude-md-saves-under-4-percent/ Published 2026-08-09. The advice circulating about Claude Code token costs is mostly unmeasured. Here is each common tip with the measurement next to it: what actually saves tokens, what saves a rounding error, and the two claims the numbers contradict outright. Measured: - claude code: 2.1.226 - batching tool calls: saves 49% - trimming an 8KB CLAUDE.md: saves 3.6% - 200 MCP tools, default: 3,054 not 60,854 - subagent delegation: costs 1.72x - cache tier requested: 1 hour, not 5 min ### A 28KB CLAUDE.md costs 7,379 tokens and no measurable time https://aieveryminute.com/a-28kb-claude-md-costs-no-measurable-time/ Published 2026-08-08. Measured on Claude Code 2.1.224: adding a 28KB CLAUDE.md produced a median time-to-first-token difference of 178ms, against run-to-run variance ranging from 1,527ms to 8,337ms on identical configurations. A sign test over 20 paired samples gives p = 0.263. Measured: - claude code: 2.1.224 - paired samples: 20 - 28KB slower in: 13 of 20 - sign test p: 0.263 - median difference: 178 ms - warm-run spread: 1,527-8,337 ms ### The same 28KB costs 7,379 tokens in CLAUDE.md and 4 tokens in a skill https://aieveryminute.com/claude-md-costs-1845x-more-than-a-skill/ Published 2026-08-08. Measured on Claude Code 2.1.224: a CLAUDE.md costs about 0.26 tokens per byte plus 106 tokens of fixed overhead, and that cost lands on every single session. The same bytes parked in a skill body cost 4 tokens until invoked. Measured: - claude code: 2.1.224 - 28KB in CLAUDE.md: +7,379 tokens - 28KB in a skill body: +4 tokens - ratio: 1,845x - CLAUDE.md rate: 0.26 tokens/byte - fixed overhead: 106 tokens - re-verified on 2.1.227: mechanism holds; ratio is fragile - re-verified on 2.1.228: +8,137 vs +5; ratio now ~1,620x, third value in three versions ### Delegating to a subagent costs 1.72x and saves 6.4% https://aieveryminute.com/delegating-to-a-subagent-costs-1-72x/ Published 2026-08-08. Measured on Claude Code 2.1.224: delegation buys a smaller parent context at a large one-off cost. The saving is exactly the content you kept out of the parent, so the trade only works if the session continues long enough to bank it. Also: .usage does not count subagent tokens at all. Measured: - claude code: 2.1.224 - inline: 126,825 tokens - via subagent: ~218,000 tokens - cost multiplier: 1.72x - parent context saved: 8,175 tokens (6.4%) - break-even: ~11 round trips ### Trying a new --effort level costs 40,054 tokens. Switching back is free. https://aieveryminute.com/each-effort-level-keeps-its-own-cache/ Published 2026-08-08. Measured on Claude Code 2.1.224: changing --effort does not change your context size at all, but the first run at each new level writes about 40,054 tokens of cache at $0.41 against $0.029 warm. Once a level is warm, alternating between levels is free. Measured: - claude code: 2.1.224 - first run at a new level: 40,054 tokens created - cost cold vs warm: 13.9x - switching back: 0 tokens - prompt size change: none - trivial task output: 4 tokens at every level ### Adding one 5-token skill costs 47,544 tokens https://aieveryminute.com/one-character-edit-costs-47544-tokens/ Published 2026-08-08. Measured on Claude Code 2.1.224: a warm session reads 66,078 cached tokens and reports $0.033. Change a single character of CLAUDE.md, or add one small skill, and the next session recreates 47,544 tokens of cache and reports $0.485. The file size barely matters; the invalidation does. Measured: - claude code: 2.1.224 - warm session: 0 created, 66,078 read - after any change: 47,544 created - cost multiplier: 14.6x - skill added: 5 tokens - invalidation: content-hashed ### Reading a 6-byte file costs 58,851 tokens https://aieveryminute.com/reading-a-6-byte-file-costs-58851-tokens/ Published 2026-08-08. Measured on Claude Code 2.1.224: a tool call costs approximately your whole current context, regardless of what it returns. File content is only about 0.35 tokens per byte, so on a 40KB read the overhead is four times the file itself. Batching tool calls halves the total. Measured: - claude code: 2.1.224 - one tool round trip: +58,851 tokens - 3 reads batched: 121,732 tokens - 3 reads sequential: 240,073 tokens - sequential penalty: 1.97x - file content rate: 0.35 tokens/byte ### Resuming a session costs 43,350 tokens once, then 293 a turn https://aieveryminute.com/resuming-costs-43350-once-then-293-a-turn/ Published 2026-08-08. Measured on Claude Code 2.1.224: continuing a conversation adds exactly 293 tokens of cache write per trivial turn, with everything else served from cache. Across six turns the context grew 1,465 tokens while reported cost rose 2%. Measured: - claude code: 2.1.224 - first resume: 43,350 tokens created - each turn after: 293 tokens created - cost over 6 turns: +2% - context growth: 293 tokens/turn ### In Claude Code permissions, deny always wins https://aieveryminute.com/claude-code-deny-beats-allow/ Published 2026-08-07. Eight headless runs across four settings configurations show a deny rule blocks a tool call regardless of which scope it sits in, even when a higher-precedence file explicitly allows the same pattern, and even inside the same file. Measured: - claude code: 2.1.223 - headless runs: 8 - configurations: 4 - control: passed - deny lost: never ### Three ways to write a dead deny rule, one of them silent https://aieveryminute.com/claude-code-deny-rules-that-do-nothing/ Published 2026-08-07. Eighteen headless runs across five rule forms. A misspelled tool name warns, a raw-string matcher warns, and a misplaced colon fails completely silently while looking correct. Measured: - claude code: 2.1.223 - headless runs: 18 - rule forms tested: 5 - dead rules found: 4 - silent failures: 1 ### Nothing in a project can unset an inherited Claude Code env var https://aieveryminute.com/claude-code-env-merges-across-scopes/ Published 2026-08-07. Measured how Claude Code combines the env setting across user, project, local and --settings. It deep-merges across all four, an inherited key cannot be removed from a higher-precedence scope, non-string values are silently coerced by JavaScript string conversion, and OTEL_ variables never reach subprocesses at all. Measured: - claude code: 2.1.224 (8a2a469b68f9) - scopes env merges across: 4 of 4 - ways to unset that work: none cleanly - null becomes: the string "null" - {"a":1} becomes: [object Object] - OTEL_ vars reaching subprocesses: none ### Three Claude Code hook fields the docs name wrong https://aieveryminute.com/claude-code-hook-payloads-vs-docs/ Published 2026-08-07. Wiring 15 hook events to a capture script and running two headless sessions turned up three field names that did not match the documentation on 2026-08-07, plus several useful fields undocumented at the time, including per-tool duration_ms. Re-checked on 2026-08-21: every one of those discrepancies has since been fixed in the hooks reference, so this now stands as a record rather than as current guidance. Measured: - claude code: 2.1.223 - hook events wired: 15 - events captured: 22 - sessions run: 2 - field names wrong: 3 - rechecked 2026-08-21: all discrepancies now fixed in the docs ### Subagents load CLAUDE.md invisibly https://aieveryminute.com/claude-code-subagent-instructions-blind-spot/ Published 2026-08-07. A general-purpose subagent provably reads your CLAUDE.md, and the hook meant to observe instruction loading never fires for it. If you use InstructionsLoaded to audit what reaches your agents, subagents are a blind spot. Measured: - claude code: 2.1.223 - headless sessions: 5 - subagents spawned: 5 - InstructionsLoaded events: 8 - events with agent_id: 0 ### Turning off MCP tool search costs you 17,214 tokens before you connect anything https://aieveryminute.com/mcp-tool-search-saves-95-percent-of-context/ Published 2026-08-07. Measured context cost of MCP tools with tool search on and off, across 5, 50 and 200 tools. Deferring saves about 95%, an MCP tool costs roughly 200 times what a skill costs, and disabling tool search adds 17,214 tokens to an empty session. Measured: - claude code: 2.1.223 - 200 tools deferred: +3,054 tokens - 200 tools loaded: +60,854 tokens - tool search off, no servers: +17,214 tokens - per tool definition: 798 bytes - re-verified on 2.1.224: +17,795 tokens - re-verified on 2.1.233: +14,644 tokens, down 17% from 2.1.232 - per-tool curve on 2.1.233: unchanged, 652 loaded and 16 deferred ### Every measurement re-run on Claude Code 2.1.224: two were identical to the token https://aieveryminute.com/re-verified-on-2-1-224/ !! PARTIALLY SUPERSEDED 2026-08-14. The 1,000-skill row reproducing at +1,461 was real but was measured without the isolation flag on a machine with its own configuration loaded, so it is the configured-machine regime rather than the clean-project one. The rest of the post stands. Published 2026-08-07. Claude Code 2.1.223 became 2.1.224. Re-measuring every figure this site has published shows the deltas are remarkably stable across the version bump, while the absolute baselines drift, which is precisely why paired baselines matter. Measured (SUPERSEDED, see above): - claude code: 2.1.224 - figures re-verified: 11 - unchanged: 9 - moved: 2 - MCP deltas: identical to the token - re-verified on 2.1.226: all figures hold - re-verified on 2.1.227: batching 2.0x, mechanisms hold - re-verified on 2.1.228: batching 2.0x, mechanisms hold, ratio moved again - re-verified on 2.1.231: floor +64 tokens, tool-search delta held ### Deploying a static site to Hostinger shared hosting https://aieveryminute.com/static-site-hostinger-shared-hosting/ Published 2026-08-07. Both candidate folders carried a DO_NOT_UPLOAD_HERE marker and the domain resolved to an IP that was not the SSH host. Here is how to find the real document root in one command instead of guessing. Measured: - host: shared, no root - node on PATH: none - node available: v24.6.0 - time to find docroot: one probe ### What 1,000 skills actually cost you: 1,474 tokens https://aieveryminute.com/what-skills-actually-cost-in-context/ !! SUPERSEDED 2026-08-14 on Claude Code 2.1.231. Every measurement here was made WITHOUT --setting-sources project, on a machine with its own configuration loaded, and that decides the answer. The description-length result in particular is wrong: description length DOES affect cost. See /a-skill-costs-39-tokens-or-3-9/ and /skillpack-language-2-1-247.json. Published 2026-08-07. Measured prompt tokens across skill counts from 1 to 1,000, with body sizes from 200 bytes to 28KB and descriptions from 90 to 1,500 characters. Every configuration paired against a baseline measured in the same batch, because the baseline drifts. Measured (SUPERSEDED, see above): - claude code: 2.1.223 - 1000 skills cost: 1,474 tokens - on disk: 3.9 MB - 28KB body cost: +5 tokens - desc length effect: none - re-verified on 2.1.224: 1,461 tokens ## Models — https://aieveryminute.com/models/ Independent measurements of AI models run head to head on identical tasks: context floor, cost per turn, cache behaviour and where they actually differ. Measured, not opinion. ### 7 of 12 runs got the date right. Every miss that showed its working was arithmetic https://aieveryminute.com/7-of-12-runs-got-the-date-right/ Published 2026-08-17. Six free-tier chat models were each asked twice what date a project finishes after 120 working days from 3 March 2026. Seven of twelve runs landed on a defensible answer. Of the three misses that showed any working, none was a misreading of the question: each chose a correct method and then executed the arithmetic wrongly, including one model that used the same method twice and got two different dates. Four of the five wrong answers contain a statement that is false or impossible on its own terms; the fifth is one of the two that showed no working at all. Measured: - runs: 12, six models, four vendors - correct: 7 of 12 - models right both times: 3 of 6 - runs that showed a setup: 8 of 12, all correct - misses that showed working: 3 of 5, all arithmetic - wrong answers refutable from their own text: 4 of 5 - ground truth: computed, rubric fixed before running - surface: duck.ai free tier, no account ### A CLAUDE.md rule held for 270 turns and 136,000 tokens https://aieveryminute.com/a-claude-md-rule-held-for-270-turns-and-136000-tokens/ Published 2026-08-10. Context rot is described everywhere and measured almost nowhere. A CLAUDE.md convention that pushes against the language default held on both models at every context size reached, and the function names were parsed rather than pattern-matched so a single ambiguous word could not be scored as a failure. Measured: - claude code: 2.1.226 - scorable turns: 270 - violations: 0 of 270 - Haiku 4.5 context reached: 136,392 tokens - Opus 5 context reached: 100,832 tokens - ambiguous names excluded: 11 - turns excluded, rate limit: 74 - re-verified 2026-08-10, isolated: 80 of 80, to 41,072 tokens ### A required field was absent. It answered null 24 times out of 24 https://aieveryminute.com/a-required-field-was-absent-it-answered-null-24-times/ Published 2026-08-10. A schema with seven deliberate traps, extracted by plain prompting rather than constrained decoding. Shape and truth were scored separately so the gap the guides warn about would be visible. It did not appear once. Measured: - claude code: 2.1.226 - extractions: 24 - schema-valid: 24 of 24 - every field correct: 24 of 24 - absent field invented: 0 of 24 - decoy value returned: 0 of 24 - scorer control-tested: 8 of 8 - re-verified 2026-08-10, isolated: 24 of 24, unchanged ### AI tests killed 35 of 36 planted bugs. Four suites never ran at all https://aieveryminute.com/ai-tests-killed-35-of-36-planted-bugs-four-suites-never-ran/ Published 2026-08-10. Mutation testing on generated pytest suites, with each mutant verified to be catchable before it counted. Opus produced ten usable suites out of ten and killed every mutant. Re-verified under isolation the caught-bug result held and improved to 38 of 38; the claim that Haiku failed on the same two functions every time did not. Measured: - claude code: 2.1.226 - suites generated: 20 - mutants killed: 35 of 36 - Opus 5 usable suites: 10 of 10 - Haiku 4.5 usable suites: 6 of 10 - test cases per suite, Opus: 70 to 875 - test cases per suite, Haiku: 8 to 13 - corrected 2026-08-21: phantom medians 196 and 11 withdrawn - re-verified 2026-08-10, isolated: 38 of 38 mutants killed - corrected 2026-08-10: per-subject failure claim withdrawn ### Haiku got XXE wrong every time. Opus used defusedxml every time https://aieveryminute.com/haiku-got-xxe-wrong-every-time-opus-used-defusedxml/ Published 2026-08-10. The circulating figure is that 44% of AI code generation tasks introduce a vulnerability, from corpora nobody can inspect. Here are six classes, both prompts and every generated file published. SQL injection never appeared once. XML external entities appeared in three runs out of three. Measured: - claude code: 2.1.226 - files generated: 36 - flagged by the scanner: 4 of 36 - XXE, Haiku 4.5: 3 of 3 vulnerable - XXE, Opus 5: 0 of 3 - SQL injection found: 0 of 6 - scanner control-tested: 6 of 6 classes ### CLAUDE.md stopped Opus adding docstrings 8 of 8. The Haiku half did not hold https://aieveryminute.com/only-opus-added-docstrings-and-claude-md-stopped-it-8-of-8/ Published 2026-08-10. The advice everywhere is to put a "never add comments" rule in CLAUDE.md because the model ignores you otherwise. The original Haiku figures here did not reproduce and have been withdrawn; a re-run the same day produced the opposite result. The Opus finding, that the rule removed a docstring from every file, is unaffected. Measured: - claude code: 2.1.226 - generations: 86 - inline comments seen: withdrawn, did not reproduce - Opus 5 file writes, no rule: 8 of 8 had a docstring - Opus 5 file writes, with rule: 0 of 8 - Haiku 4.5, either way: withdrawn, did not reproduce - corrected 2026-08-10: Haiku control re-ran as 8 of 8 - detector positive control: 10 comments, 1 docstring ### The loud prompt injection failed 30 times. The quiet one worked 6 of 6 https://aieveryminute.com/the-loud-prompt-injection-failed-30-times-the-quiet-one-worked/ Published 2026-08-10. A benign marker payload, five placements, two models, and an arrival control proving the file was read on all 42 runs. The obvious injection was caught in all 30. Reworded as an ordinary editorial note, the same instruction got through the cheaper model 6 times out of 6, silently. Measured: - claude code: 2.1.226 - obvious payload runs: 30 - obvious payload obeyed: 0 of 30 - quiet payload, Haiku 4.5: 6 of 6 followed - quiet payload, Opus 5: 0 of 6 followed - arrival control failures: 0 of 42 - scorer bugs caught: 2 - re-verified 2026-08-10, isolated: unchanged, 0 of 20 obeyed ### "Think step by step" changed nothing in 96 runs https://aieveryminute.com/think-step-by-step-changed-nothing-in-96-runs/ Published 2026-08-10. Three prompt styles, two models, four questions with ground truth computed by the script that asked them. Adding "think step by step" never once turned a wrong answer right, because the models already reason without being told to. Forbidding the working is what broke one of them. Measured: - claude code: 2.1.226 - runs: 96 - correct overall: 92 of 96 - cells CoT improved: 0 of 8 - CoT output token cost: 1.5x to 1.7x - opus 5, working forbidden: 0 of 4 - opus 5, free to reason: 4 of 4 - re-verified 2026-08-10, isolated: direction holds, 0 of 4 became 1 of 4 ### With the control arm controlled, Haiku commented all 16 files https://aieveryminute.com/with-the-control-arm-controlled-haiku-commented-all-16-files/ Published 2026-08-10. The control arm was running under a machine-level prohibition, so neither arm was rule-free. Isolating it needed a flag that blocks user memory without breaking tools, verified by hook rather than assumed. The corrected result reproduces across two rounds and says the opposite of the original. Measured: - claude code: 2.1.226 - files, corrected trial: 32 - no rule, files clean: 0 of 16 - rule, files clean: 16 of 16 - inline comments, no rule: 30 - median size, no rule vs rule: 1,592 vs 285 bytes - isolation verified: user=0, project=1 ### A model hashed passwords with raw SHA-256 in 8 of 9 runs. Two extra words fixed it https://aieveryminute.com/a-model-hashed-passwords-with-raw-sha256-in-8-of-9-runs/ Published 2026-08-09. The widely quoted figure is that 45% of AI-generated code contains vulnerabilities. I tested two textbook cases across three models with an objective checker. One failed badly and consistently, one classic vulnerability never appeared at all, and the fix was two words of prompt. Measured: - password, neutral prompt: 8 of 9 insecure - same model, "make it secure": 2 of 2 secure - SQL injection found: 0 of 12 - models tested: 3 - checker control-tested: 7 of 7 - insecure pattern: salt + single SHA-256 ### The same 8KB file is 3,519 tokens on Opus and 2,586 on Haiku https://aieveryminute.com/a-token-is-not-a-token-across-models/ Published 2026-08-09. Measured the token cost of byte-identical files across Opus 5, Sonnet 5 and Haiku 4.5, and the startup context floor of each. Opus and Sonnet agree to within 15 tokens on every input. Haiku does not agree with either, and the gap depends on what the text is. Measured: - claude code: 2.1.224 - 8KB prose on opus 5: 3,519 tokens - same file on haiku 4.5: 2,586 tokens - haiku vs opus range: 0.66 to 0.90x - opus vs sonnet agreement: within 15 tokens - context floor spread: 24,070 tokens ### Eight models, one bug, three rounds: the 32B coder passed every time and the 72B failed every time https://aieveryminute.com/eight-models-one-bug-the-32b-beat-the-72b/ Published 2026-08-09. A hands-on trial rather than a benchmark: one small JavaScript bug with a trap in it, eight models, three rounds each, every answer executed against the real test suite. The results are not ordered by model size, and the code the models wrote shows exactly why. Measured: - models tried: 8 - rounds each: 3 - Qwen2.5-Coder-32B: 3/3 passed - Qwen2.5-72B: 0/3 passed - DeepSeek V3: 3/3 passed - Llama 3.1 8B: 0/3 passed ### Eight tokenizers agree exactly on English. On Hindi they differ by 3.3x https://aieveryminute.com/eight-tokenizers-agree-on-english-and-nothing-else/ Published 2026-08-09. Measured one paragraph and its translations into ten languages across eleven tokenizers, including three Claude models measured empirically. English is unanimous. Everything else is not, and for non-Latin scripts the spread is large enough to change which model is cheaper. Measured: - english agreement: 8 of 8 exact - hindi spread: 3.27x - russian spread: 3.70x - spanish, gpt-4o vs opus 5: 103 vs 201 - claude latin-script gap: 23 to 36% - identical tokenizers found: Phi-4 = OLMo-2 = GPT-4 ### Finding one fact in 43,000 words: 10 of 10. Adding three of them up: 8 of 10 https://aieveryminute.com/finding-one-fact-in-43000-words-vs-adding-three-up/ Published 2026-08-09. A needle-in-a-haystack test at 43,000 words found no position bias whatsoever: the fact was retrieved from 0%, 25%, 50%, 75% and 100% depth every time. A harder version requiring three buried facts to be added together failed twice in ten runs, and both failures were arithmetic, not retrieval. Measured: - document length: 43,000 words - single fact, all depths: 10 of 10 - three facts combined: 8 of 10 - decoy codes in document: 12 - depths tested: 0, 25, 50, 75, 100% - failure type: arithmetic, not retrieval ### It gave the same wrong answer 8 times out of 8. Consistency is not accuracy https://aieveryminute.com/it-gave-the-same-wrong-answer-8-times-out-of-8/ Published 2026-08-09. The circulating concern is that models give different answers to the same question. Across 64 runs the more useful finding was the opposite: the most stable answers included some of the most reliably wrong ones, and asking twice tells you nothing about correctness. Measured: - runs: 64 - correct overall: 41 of 64 - questions that varied: 4 of 8 - letter count: 0 of 8 correct, 1 answer - 9.11 vs 9.9: 2 of 8 correct - ground truth: computed, not asserted ### Models hit an exact word count once in 31 tries. Sentences: 17 out of 17 https://aieveryminute.com/models-hit-an-exact-word-count-once-in-31-tries/ Published 2026-08-09. A measured test of the most-repeated prompting workaround. Word counts came in anywhere from 28% short to 52% long. Sentence and paragraph counts were hit exactly in all 17 runs, across every model tested. Measured: - exact word counts: 1 of 31 - exact sentence counts: 9 of 9 - exact paragraph counts: 8 of 8 - worst word miss: +52% - models tested: 4 - counter control-tested: 7 of 7 ### Only 6 of 30 AI citations were real, and 11 of the fakes had DOIs that resolve https://aieveryminute.com/only-6-of-30-ai-citations-were-real/ Published 2026-08-09. Citation fabrication measured rather than described: 30 citations from three open models, each DOI resolved against Crossref and DataCite and its title compared to the claimed one. Verifying that a DOI merely exists would have passed 11 fabrications. Measured: - citations checked: 30 - verified real: 6 - niche topic real: 0 of 15 - real DOI, invented title: 11 - models that refused: 1 - verifier: Crossref + DataCite ### The same bug fix cost $0.11 on Haiku and $0.93 on Opus https://aieveryminute.com/same-bug-fix-cost-11-cents-on-haiku-93-on-opus/ Published 2026-08-09. Measured what it costs to actually complete a task rather than what a token costs, by giving Opus 5, Sonnet 5 and Haiku 4.5 the same failing test suite and verifying the fix by running it. Cost to finish varied 8.5x, and the ranking is not the one the price list implies. Measured: - claude code: 2.1.226 - haiku 4.5 median cost: $0.109 - opus 5 median cost: $0.925 - sonnet 5 median tokens: 1,084,272 - cost spread to finish: 8.5x - runs: 26