Tools
AI tools installed, run and measured first-hand, including what they cost you and what they do badly.
tried it, here is what happened 11
A book costs 41% more in Claude Code than tiktoken says
I ran book-to-skill v1.4.0 on the 501-page Pro Git PDF, then measured what the extracted text actually costs in Claude Code context against its tiktoken count. The gap is 1.41x on a technical book and 1.41x on Moby-Dick, and because the two books have different cl100k rates, solving both together puts any per-byte framing charge at or below zero, which bounds an artefact of how Claude Code wraps the file without ruling one out. Anthropic publishes ~15-20% for typical text and an unbounded "much more" for code, and only the novel is the typical-text case; both books measured the same 1.41x regardless, which that split does not predict. It leaves book-to-skill's headline ratio underdetermined rather than refuted, and makes its dollar table low. Running it also surfaced three things a README cannot tell you, including a CLI that reports 93 chapters in a ten-chapter book because it counts shell comments inside code samples.
- tool book-to-skill v1.4.0, commit 3a97a71, 23,485 stars
- book Pro Git, 501 pages, 17.97 MB PDF
- this trial 28 runs, six cells of four rounds plus a floor per round
- extraction 501 pages in 0.55-0.64s
Anthropic ships five skill plugins, not one, and its own ~100-token figure fits the one I could measure
Anthropic documents ~100 tokens per Skill for the metadata loaded at startup. Its own repository ships five plugins rather than one, and only one of them is large enough for this instrument to give a figure. example-skills, twelve skills with ordinary 288-character descriptions, measured 102 tokens per skill. The three single-skill plugins cannot be measured: two carry ranges 198 tokens wide and the third spans 897, from minus 578 to plus 319, which contains zero. A separate sweep then knocked out the story I had been about to tell about long descriptions: above 500 characters they are close to free. Which turns out not to mean what it sounds like, because a later measurement found the text stops reaching the model entirely past about 1,400 characters.
- pack anthropics/skills, commit f379e5a
- runs 50, ten rounds x five plugins
- documented ~100 tokens per Skill
- measurable plugin 102 per skill, 288-char median
A 19,700-star diagram skill drew once in four unattended runs. One file fixes it
A trending Claude Code skill for editorial diagrams, installed and measured on 2.1.233. What it costs to sit on disk unused turned out not to be measurable: twelve readings of the same pack span 70 to 967 tokens. Run unattended on an ordinary request it delivered a file in 1 of 4 rounds, spending 733,824 tokens across the three that produced nothing, because SKILL.md tells the agent to pause and ask about brand colours before the first diagram in a project. Dropping a one-line .diagram-design marker in the project root took it to 4 of 4. No cost multiplier is quoted, because the plain arm alone varied 6.7x and the skill's arm has one delivering round. The marker effect is established at p=0.018; the skill-versus-plain comparison is underpowered and is not claimed.
- pack cathrynlavery/diagram-design, commit 09df49d
- stars 19,728 on 2026-08-17
- installed, never used not measurable, 70 to 967
- delivered a diagram 1 of 4 unattended
google/skills costs 9,020 tokens. Every session, before you use any of it.
Six trending skill packs, cloned and installed on Claude Code 2.1.233, measured against an empty floor, then re-measured after I found I had priced one of them wrong. 111 skills cost 9,020. The 25 skills Matt Pocock actually ships cost 852, not the 1,209 I first published for 35 files sitting in his repository. Per skill that is 34 to 96 tokens, and multiplying a published rate by a file count would have got two of the three badly wrong.
- claude code 2.1.233
- runs 56 across two harnesses, no exclusions
- clean floor 22,283 then 22,287, each round paired
- google/skills, 111 skills 9,020 tokens
- re-verified 2026-08-18 diagram-design 70 to 967 on 2.1.234
The docs say ~100 tokens a skill. I measured 34 to 96.
Anthropic documents ~100 tokens per Skill for the metadata loaded at startup. Measured across the three trending packs where a measurement is reliable, per-skill cost ran 34.1 to 95.9, all below the documented figure, over-predicting by 4% on one pack and 193% on another, and the size of the miss does not order by description length. Then I asked six model runs the same question: three repeated the documented number, one gave a different one, and two declined, including the model that had given a different one moments earlier.
- documented ~100 tokens per Skill
- measured, 3 packs 34.1 to 95.9 per skill
- worst over-prediction 193%, mattpocock/skills
- closest 4%, addyosmani/agent-skills
- re-verified 2026-08-18 one-skill pack 70 to 967 on 2.1.234
720 of 720 table cells read correctly. My scorer was the thing that kept failing
Every value in the test tables was unique, so a misplaced number could be told apart from a misread one. The model got all 720 cells right. The scorer got it wrong three times, always for the same reason, and twice it would have produced a confident and completely false headline.
- cells checked 720
- read correctly 720
- misplaced cells 0
- table sizes 5x4, 10x6, 20x8
I gave two AI detectors text from 1859 and 1996. Both called it human, but the newer one scored 23% AI
A hands-on trial of free AI detectors using text that cannot be disputed: an 1859 book, a 1996 internet standard, and two essays generated on the spot. Every verdict is shown. The detectors were right, and the margins say something the pass/fail does not.
- texts tested 4
- GPTZero verdicts correct 4 of 4
- Darwin 1859 100% human
- RFC 1958, 1996 77% human / 23% AI
- re-verified 2026-08-09 model label matters
A detector called an 1859 book AI-generated while its own numbers said 85% human
A hands-on trial of GPTinf, a free unlimited no-signup AI detector, using a public domain passage from 1859. The tool contradicted itself three times out of three, and the number most people would read is the one that moved.
- text tested Darwin, 1859
- verdict sentence shown "seems to be AI generated"
- its own breakdown 0% AI, 85% human
- contradiction observed 3 of 3 runs
A free AI detector switched to an older model when my scans ran out, then called an 1859 book 100% AI
A hands-on trial that nearly produced a spectacular false finding. GPTZero classified an 1859 public domain passage as 100% AI-generated, with high confidence. The text had not changed. The model that judged it had.
- text tested Darwin, 1859
- verdict on Model 4.8b 100% human
- verdict after quota 100% AI
- model after quota 3.3b / 2025-03-13-base
I tried to make a free image model misspell a word. It got all six right
Text in AI images is supposed to be the thing that still breaks. I gave FLUX.1-schnell six words to render, from OPEN up to a rare twelve-letter genus, at a fixed seed on a free public Space. All six spelled correctly.
- model FLUX.1-schnell
- words attempted 6
- spelled correctly 6
- hardest word ZEPHYRANTHES
Tesseract hit 53% word error at 6px text. Claude vision hit 0%
A ladder of deliberately degraded document images with exact ground truth, read by tesseract and by Claude vision. JPEG quality turned out to be irrelevant. Text height turned out to be everything, with a cliff between 7 and 6 pixels that one engine fell off and the other did not notice.
- tesseract at 6px text 53.3% word error
- Claude vision at 6px 0.0% word error
- JPEG quality 5 0.0% for both
- tesseract cliff between 7px and 6px
Every figure is collected on the measurements page, and the method behind them is written up here.