aieveryminute

AI every minute

First-hand trials of AI tools and models. Everything here was installed and run: what it cost, what broke, and where a published claim did not survive being checked. Every prompt, output and figure came off a real run, including the ones that contradicted what everybody repeats, and the ones that contradicted what this site said last week.

68trials and measurements published
490figures with a run behind them
2.1.251newest Claude Code version tested
2runs minimum before publishing

Start here

Everything published is listed below in full. These twenty-seven are the ones that changed what I do.

  1. 8 of 8

    Given a task that never mentioned a command line, Claude Code built one in 8 of 8 runs

    The task was to read a CSV of orders and report revenue per customer. It never mentioned a command line. Claude Code with no instructions built an argparse interface in 8 of 8 runs, wrote two files every time, and came in at 73 to 88 lines. With a standing terseness instruction installed through a SessionStart hook, the same task produced an argparse interface in 0 of 8 runs and 37 to 60 lines. The arms do not overlap, with a gap of 13 lines. Eight rounds per arm, none excluded, and an arrival control confirmed the instruction reached the model in every round. The instruction is a third party plugin and the point here is not the plugin: it is that the DEFAULT adds structure nobody asked for, reliably enough to separate cleanly across sixteen runs, and that saying what you do not want changes it. What is NOT shown is that shorter is better. Neither program was run against real data or reviewed for correctness.

  2. 78%

    Crossing the documented hook output cap silently drops 78% of the context you sent

    Anthropic caps SessionStart hook output at 10,000 and documents it twice. Forty runs on 2.1.238 measure the price of crossing: 10,016 tokens of context arrive below the cap, identical in all four rounds, against 2,172 to 2,181 above it. The arms do not overlap. The payload is not destroyed, it is written to a file the model must spend a tool call to read, so a plugin that injects standing instructions silently stops delivering most of them. obra/superpowers at 3,334 bytes is well under. The cap counts UTF-16 code units, not characters, so every emoji counts twice; this corpus RETIRES its own earlier characters claim rather than rewording it.

  3. 1 of 5

    Four of the five AI crawlers this site allows never fetched robots.txt or llms.txt in 14 days

    Over 14 days and 19 hours a logging shim on robots.txt and llms.txt recorded 278 non-curl requests. robots.txt names seven AI agents but only five of them crawl: Google-Extended and Applebot-Extended are training-use control tokens that make no requests at all, so they are counted separately rather than as absent crawlers, and the crawlers they govern both showed up, Googlebot 26 times and Applebot 38. Of the five that do crawl, only OAI-SearchBot appeared, 50 times. GPTBot, ClaudeBot, Claude-SearchBot and PerplexityBot returned zero, including both crawlers their vendors document as collecting training data. llms.txt was fetched 25 times and exactly one came from any of the five; the rest were ordinary search bots, SEO tools and unidentified clients. The shim serves only those two paths, so a zero means never fetched robots.txt or llms.txt here, not never crawled the site. One young low-authority domain, one window, self-reported user agents.

  4. 1.41x

    A book costs 41% more than tiktoken says, so a tiktoken estimate runs about 29% under

    The same bytes written as a project-scope CLAUDE.md and read back from modelUsage cost 1.41x what tiktoken cl100k_base counts, on a 501-page technical book and on Moby-Dick, across 28 runs. The published figure is the slope between slices, so the CLAUDE.md framing constant cancels rather than inflating it, and every segment measured falls between 1.382x and 1.452x. The two books have different cl100k rates, so solving both for a tokeniser factor and a per-byte framing charge puts the framing term at -0.0133 tokens per byte, indistinguishable from zero and unable to be positive, which bounds a size-proportional overhead without ruling it out. Anthropic's own guidance puts this gap at 15-20% on typical text, implying 1.18x to 1.25x, and at an unstated 'much more' on code. Only the novel is the typical-text case, and at 29.3% it sits above the published band; the code-heavy book cannot contradict a clause that names no number. What the guidance does not predict is that both books landed at the same ratio, 29.08% and 29.28%, with the prose one marginally higher. A book tiktoken prices at 227,000 tokens is nearer 320,000 of Claude Code context, and any dollar figure computed that way is low by the same margin.

  5. 56,600

    A trending 817-skill pack costs about 56,600 tokens before you type anything

    mukul975/Anthropic-Cybersecurity-Skills ships 817 Agent Skills and adds about 56,600 tokens of startup context in an empty isolated project, 69.30 per skill, from 24 readings of which 21 sit inside a 48-token band. The synthetic curve behind this site’s own calculator interpolates to 13.46 per skill at that count, 5.15x low, while its real-pack measurements never disagreed. Description uniqueness and the pack’s extra frontmatter were tested and refuted, and deleting 338KB of frontmatter moved the reading 8 tokens, which matches what Anthropic documents. Why a real pack costs so much more than the synthetic fixture is not established.

  6. 1,536

    Your skill description is cut at 1,536 characters, and the cap is a setting you can raise

    A rare marker planted at a known offset inside a skill description, then asked for. Text ending at 1,530 characters is answered from context with no tool calls, four rounds of four; text ending at 1,536 never is, and the model shells out to Bash to open the file instead. That brackets the documented skillListingMaxDescChars default of 1,536 to within six characters, and the cap is a setting you can raise. Nothing errors: the words past the cut are simply absent when Claude decides whether your skill is relevant, so trigger conditions written last silently stop working. A 51-character increase in the skill name moves the cut not at all, and it does not move as more skills are installed, tested at 1, 10 and 40.

  7. 1 of 4

    A trending diagram skill drew once in four unattended runs. One file fixes it

    cathrynlavery/diagram-design carries two pause points that a headless run has nobody to clear, so it delivered a file in 1 of 4 rounds and spent 733,824 tokens across the three that produced nothing. A one-line .diagram-design marker in the project root took it to 4 of 4 across eight further rounds, which is the comparison that reaches significance at p=0.018. What it costs to sit installed and unused is not measurable at all: twelve readings of the same pack span 70 to 967 tokens.

  8. 7 of 12

    Six models, one date question, and every visible miss was arithmetic

    Six free-tier models from four vendors, asked twice each what date falls 120 working days after 3 March 2026. Seven of twelve runs landed on a defensible answer. All eight runs that showed a route to their answer set the problem up correctly and the wrong ones then slipped on the arithmetic, including one model that used the same method twice and got two different dates. Four of the five wrong answers contradict themselves in their own text.

  9. ~100 vs 34

    The docs put a skill at ~100 tokens. Three measured packs came in at 34 to 96

    Anthropic documents ~100 tokens per Skill for the metadata loaded at startup. Across the three trending packs where a measurement is reliable, per-skill cost ran 34.1 to 95.9 and every one came in below it, over-predicting by 4% on one and 193% on another, and the size of the miss does not order by description length. It is NOT a ceiling: this site later measured the example-skills pack Anthropic itself ships, twelve skills with 288-character descriptions, at 102 per skill, above the documented figure. So ~100 over-predicts for most packs and under-predicts for at least one. Measure the pack if the number matters.

  10. 9,020

    The biggest trending skill pack costs 9,020 tokens on every session

    Six Agent Skill collections from GitHub Trending, installed and measured rather than estimated. google/skills is 111 skills and 40% of a clean startup context, paid whether or not you invoke any of them. Per skill it is 34.1 to 95.9, and it does not order by description length: the pack with the longest descriptions is not the dearest one. Check the plugin manifest before pricing a pack: the SKILL.md files in a repository are not all the ones you get.

  11. 2.04x

    Subagent definitions never get cheaper in bulk, and everything else does

    A skill or a slash command costs about 40 tokens and collapses to under 6 once you pass a few hundred. A subagent definition costs 51 at every count measured, so 400 of them cost 20,404 to 20,602 tokens against 9,989 to 10,187 for the same number of commands.

  12. 49%

    Batching your tool calls saves about half a session

    Three files read in one batch cost 121,732 tokens. The same three read one at a time cost 240,073, because every round trip re-sends the whole conversation.

  13. 20,581

    A parallel subagent costs 20,600 to 28,500 tokens before it does anything, and far more with config loaded

    The spread has a cause: a four-way fan-out takes six or seven parent turns and the total moves 16% between them, and which one you get is not under your control. On 2.1.238 six turns is the common case at 20,581 per agent, against 28,504 at seven; at matched turn counts those figures moved 1.0% and 0.9% since 2.1.232. On a configured machine every subagent re-pays about twice your CLAUDE.md and tools.

  14. 64 of 64

    Compaction kept every planted value, across 24 compactions

    Arbitrary values survived auto-compaction intact, including when the surrounding bulk was 31.7x less compressible and competing for room in the summary. Discrete stated facts survive; the loss people report is somewhere else.

  15. 7,379 vs 4

    The same 28,000 bytes costs 7,379 tokens in CLAUDE.md and 4 as a skill body

    A skill body only loads when it is invoked, so reference material belongs there. Move it, do not trim it. The gap is the durable part; the headline ratio it produces is not.

  16. 20x

    Your CLAUDE.md is re-sent every turn, and after turn two it costs 20x less

    An 18KB CLAUDE.md added 4,967 tokens to every turn of a session, identical on all four turns with zero spread, so the context cost is permanent. The money is not: the same tokens billed at $0.04966 and $0.04967 on turns one and two and $0.00248 from turn three, because the file stops being written to cache and starts being read from it. Trim it for the window you get back, not for the bill.

  17. 2.2x

    Planning first cost about 2.2x, on a specified task and on an open-ended one

    Plan mode is recommended as the cheaper way to work, specifically on open-ended prompts. It did make the execution cheaper, but only on the task with one right answer, and the plan cost about five times the saving it bought.

  18. +8%

    The env var sold as a 90% saving cost 8% more

    CLAUDE_CODE_FORK_SUBAGENT raised total context 38% cold and 66% warm across 28 runs. Cache creation, the thing it should remove, moved upward.

  19. 6 of 30

    Only 6 of 30 AI-generated citations were real, and 11 fakes had working DOIs

    Checked against Crossref and DataCite. On a niche topic, zero of fifteen were real. Verifying that a DOI merely resolves would have passed eleven fabrications.

  20. 8.5x

    The same bug fix cost $0.11 on Haiku and $0.93 on Opus

    Cost per token is the wrong unit. Sonnet used 74% more tokens than Opus to do identical work, so most of its per-token discount disappears.

  21. 8 of 8

    It gave the same wrong answer eight times out of eight

    Consistency is not accuracy. The question it was most stable on is one it never got right, while the visibly unstable answer is the one you actually catch by asking twice.

  22. 1 of 31

    Exact word counts were hit once in 31 tries. Sentences and paragraphs, 17 out of 17

    The workaround every guide repeats turns out to be right, across four models. If the number has to be exact, ask for sentences or paragraphs, never words.

  23. 2.2x per char

    A Chinese skill description costs 2.2x per character, and the documented cap counts characters

    ConardLi/garden-skills reached GitHub Trending at 11,112 stars with five skills whose descriptions are roughly half Chinese. Nine skills were installed one at a time on 2.1.247, four rounds each, every delta against a floor re-measured in the same round. Per description CHARACTER, descriptions with no CJK cost 0.363 to 0.482 tokens and majority-Chinese ones cost 0.887 to 1.048: the arms do not overlap, and the character-length ranges DO overlap, so length is not the explanation. Per BYTE of UTF-8 the arms OVERLAP (0.363 to 0.482 against 0.459 to 0.513), which is the honest qualification: a Chinese character is about 2.3 bytes, and that accounts for most of the per-character gap. It matters anyway, because the description cap Anthropic documents counts CHARACTERS and this site prices skills per character, so a Chinese description reaches that cap having spent roughly twice the tokens, and the published model, 39 tokens plus 0.276 per description character and declared valid only from 30 to 300 characters, lands 2.48x under on the Chinese arm while landing within 5 percent on the English one. What is NOT supported: that the tokeniser handles Chinese badly per unit of encoded text. Skill BODIES are unmeasured.

  24. 1.12x

    The same CLAUDE.md in Chinese costs 12 percent more than in English, not two to three times

    One translation pair, the same project instructions written in English and in Chinese, with byte sizes matching within 1.4 percent while character counts differ by 2.8x. That is what lets a single pair separate the two units. Sixteen self-paired rounds on 2.1.247, floor re-measured every round, one constant working-directory path because this site has measured the startup floor moving with the path. English is 604 tokens at the mode and Chinese 678: 1.12x, or 0.388 against 0.441 tokens per byte. Three predictions were pre-registered before anything ran and all are published. The one that matters most is refuted decisively: that Chinese is CHEAPER because it uses fewer characters predicted 183 to 248 tokens against 678 measured. What is NOT claimed: separated arms. Both arms wander in quantised 198-token steps and their ranges overlap, so 1.12x is a modal difference across 16 rounds, not a resolved per-run effect. One pair, one author, no native-speaker check.

  25. 6 of 6

    A vendor skill rewrote the old Go idiom in 6 of 6 runs, and tripled the cost of the task

    JetBrains publishes go-modern-guidelines, 1,916 stars, as a Claude Code plugin with a single skill whose body tells the model to run a pinned Go CLI before editing Go code. Two tasks were measured on 2.1.247, paired, same constant working directory. WHAT IT DOES: on a task the tool own ruleset covers, the arm without the skill wrote the pre-1.21 sort.Slice in 6 of 6 runs and the arm with it wrote slices.SortFunc with cmp.Compare in 6 of 6. No run went the other way, and the rule was taken from the CLI own published list rather than invented. WHAT IT COSTS: 45,582 to 45,960 tokens without against 146,181 to 146,872 with, roughly 3.2x, arms not overlapping. It does not inject 100,000 tokens, it adds tool round trips and each tool call re-sends the whole conversation. Sitting installed and unfired it costs only tens of tokens, so the listing cost usually quoted for skills is the wrong number here by three orders of magnitude. On a second task that had no modern idiom to apply, the same 3.2x bought code identical apart from one word of a doc comment, and the skill description fires on any Go edit at all. Neither arm was compiled or tested, and one rule of roughly two dozen was exercised.

  26. -24.5%

    Claude Code startup floor fell 5,594 tokens between 2.1.246 and 2.1.251

    The startup floor is what a session costs before you add anything, and it is the base every other figure here is measured against. Re-measured on 2.1.251: 17,207 tokens with tool search on and 32,400 with it off, against 22,801 and 37,870 on 2.1.246. That is a 24.5 percent fall and the largest move recorded on this site. Eight rounds with every cell measured twice, and both floor arms returned a SINGLE value across all sixteen repetitions, histograms 17207 x16 and 32400 x16. An independent four-round invocation run minutes earlier returned 17,205 and 32,398 with an IDENTICAL 15,193 tool-search delta; it is reported separately rather than pooled because its working directory was one character shorter, and this site has measured that the floor moves with the path, so pooling would have made an exact cell look bimodal. The paired tool-search delta moved only 15,069 to 15,193, so whatever shrank is common to both arms rather than specific to tool search. What is NOT claimed: any cause. Nothing here establishes what Anthropic removed.

  27. 0 tokens

    A 229 KB skill body costs nothing at startup, 8x beyond where this site had tested

    This site published that a skill body is free at any size up to 28KB. mvanhorn/last30days-skill, 60,605 stars and MIT, ships one skill whose SKILL.md is 229,438 bytes: about 8.2x beyond that range, so the claim was being applied outside where it had been measured. Twelve pooled rounds on 2.1.251, floor re-measured every round, skill installed and NEVER invoked. The whole thing costs 101 tokens, which is just the listing: one skill plus a 246-character description. The body contributes zero, and the published model predicts 106.9 against 101 measured. Three predictions were recorded before the run; the site own position held and its validated range now extends 8.2x on a real pack rather than a synthetic fixture. One round read 998 instead of 101 because its FLOOR was 897 low, this machine known intermittent component, and that is published rather than dropped. What is NOT measured: what the skill costs when it FIRES. The tool was deliberately not run, because doing so auto-installs third-party CLIs via npx and can extract browser cookies behind an opt-in.

Tools11

AI tools installed, run and measured first-hand, including what they cost you and what they do badly.

tried it, here is what happened 11

Claude Code39

What the tool actually costs and how it actually behaves, measured against its own documentation.

what things cost in context 28

how it actually behaves 9

keeping the numbers honest 1

running the site 1

Models18

The same method pointed at the models themselves: what each one costs to run, and where they measurably differ.

what each model costs 3

where they measurably differ 15

Every figure above is collected on the measurements page.

INDEXaieveryminute.com⏵68 notes⏵built 2026-08-31 17:47 UTC