A book costs 41% more in Claude Code than tiktoken says

book-to-skill had 23,485 stars on 2026-08-21. It turns a book into an agent skill, and it is one of the more carefully built things I have run: the code is MIT, dependency-light, and its benchmark script labels its own assumptions out loud, which is rarer than it should be.

It also publishes a token-cost claim computed with tiktoken cl100k_base. It is a cross-host tool: its README names GitHub Copilot CLI, Amp and Claude Code. cl100k is not strictly right for any of them. tiktoken’s own table maps OpenAI’s current chat and reasoning models (GPT-4o, GPT-4.1, GPT-5, the o-series) to o200k_base, and keeps cl100k_base for the original GPT-4, GPT-3.5 and the text-embedding-3 pair. Not every model: the open-weight gpt-oss pair maps to o200k_harmony. But for the OpenAI-backed hosts the choice barely matters: on this book o200k lands 0.2% from cl100k, and on Moby-Dick 0.7% under it. For Claude Code it is out by far more, and that is what this post measures.

What I actually ran

docs/performance.md opens with “All numbers below are measured, not estimated, using tiktoken (cl100k_base) for token counts and tools/discovery_tax.py for the discovery model.” It reports four books. One of them is Pro Git at 501 pages, which is freely downloadable, so I ran their extractor on the real thing rather than on something of my own choosing.

Mode: text — using pdftotext...
  chapters: 93 (structural)
   Total Pages: 501
   Words   : 148,733
   Tokens  : ~198K

That worked first time, with no configuration, in 0.64s and 0.55s across two runs. A 501-page, 17.97 MB PDF to structured text in under a second. The extraction half of this tool is genuinely good.

Then I counted the same output with cl100k_base myself: 227,179 tokens, which reproduces their published 229K to within 1%. So their table’s numbers are real and I could not fault them.

Anthropic already publishes a number for this, and it does not settle it

This is not new ground and I should not have implied it was. Anthropic’s own skills repository says it directly, in skills/claude-api/shared/token-counting.md:

Do not use tiktoken. It’s OpenAI’s tokenizer. It undercounts Claude tokens by ~15-20% on typical text, and by much more on code or non-English input.

That is two claims, and they have to be kept apart. On typical text, 15-20% means a tiktoken count sits that far below the true one, so correcting it means multiplying by 1.18 to 1.25. On code or non-English input the doc names no number at all, only “much more”, with no upper bound.

Which clause applies depends on the book, so I measured the actual composition of the exact interval each slope covers, bytes 20,000 to 120,000:

Book	shell/comment lines	indented code lines	non-ASCII	clause
Pro Git	8.2%	43.0%	0.37%	“much more”, unbounded
Moby-Dick	0.0%	10.0%	0.74%	typical text, 15-20%

Only one of my two books is a conflict. Moby-Dick is plain English prose and measured a 29.3% undercount against a published 15-20%, which is outside the band. Pro Git is genuinely code-bearing, and for that input the doc supplies no upper bound, so nothing measured on it can contradict the guidance in either direction. My previous version claimed the conflict for both, by quoting the code carve-out and then reasoning as though it were not there.

The result the doc does not predict is that the two books came out the same. 29.08% on the code-heavy technical book against 29.28% on the novel, with the prose book marginally higher. Whatever separates code from typical text in Anthropic’s figure did not show up across these two books at this size. That is the part of this worth keeping.

One caveat on the comparison itself. Their statement is about the tokeniser; mine is what a CLAUDE.md of that text costs inside a real Claude Code context, which is the number a reader actually pays. That distinction is real but thinner than it sounds, since the doc’s own worked examples call count_tokens on a full message payload rather than on bare text. I am not going to lean on it to explain the gap away.

The denominator is a constant, and their own tool prints it

The ratio’s other half is where I would spend attention. Running their benchmark on Moby-Dick:

book-to-skill : 5,000  core [design cap (no --skill-dir)] (4,000) + compiled chapter (1,000)

design cap. Unless you pass --skill-dir, the denominator is 4,000 + 1,000 hardcoded in tools/discovery_tax.py. The published table shows ~5,000 identically for all three books, which is what that default looks like. The tilde is honest; the opening sentence promising every number is measured is in tension with it.

The same run also prints this:

vs discovery best : 0.2x fewer tokens
vs discovery loop : 0.7x fewer tokens

Both are below 1.0, which means book-to-skill cost more than simply reading the table of contents and the target chapter, by about 4.6x. The label still says “fewer tokens” in that direction. Their docs do disclose that this comparison “scales with chapter size”, and Moby-Dick’s chapters are tiny, so this is a disclosed limitation showing up rather than a hidden one. It is worth seeing before you assume the discovery-loop rows generalise.

Three things only running it surfaces

Its CLI and its docs disagree by 15% on the same book. The CLI printed Tokens: ~198K; the performance table says 229K. Neither is wrong: estimate_tokens in book_to_skill/utils.py is a deliberate words / 0.75 heuristic, kept dependency-free so the same book always yields the same number, while the docs use tiktoken. A reader comparing the two will not be told they are different instruments.

Its two halves disagree about the same file, and the CLI is the one that is wrong. The extractor printed chapters: 93 (structural) for Pro Git. tools/discovery_tax.py, on that exact full_text.txt, printed:

No chapters detected — cannot model discovery. The source may be a
technical PDF whose headings were flattened by text extraction; try
technical mode (Docling) so chapter structure is preserved.

Pro Git has ten chapters and three appendices, so neither number is right, but the benchmark’s zero is far closer. I enumerated all 93: not one is a chapter heading. They are # lines lifted out of code samples and terminal transcripts — # ignore all .a files, # Commands:, # p, pick <commit> = use commit. _ATX_HEADING in book_to_skill/utils.py treats a leading # as a Markdown heading, and the fenced-code guard that would exclude them found exactly one closed fence in the whole book, six lines, because pdftotext output preserved only two ``` markers out of a book full of code samples. Nothing warns you: the check beside it fires only when the structural count is exactly one. At zero the method is recorded as none, so that branch is unreachable as well, and at 93 it is far past it.

Their docs are the honest version here. The performance table gives Pro Git’s chapter cell as a dash, footnoted that it “heads chapters with section titles (no Chapter N), so it does not auto-segment”. The dash is right and the CLI’s 93 is not.

What remains odd is the comment in discovery_tax.py saying it reuses the extractor’s detection “so discovery_tax and the pipeline always agree on what a chapter is”. On this book they disagree, 93 against 0.

It asks before installing. dependencies.py runs pip install for missing packages, but prompts first. Worth knowing before you run it in an environment you care about.

The supply-chain notice you should read first

The project ships SECURITY-NOTICE.md, dated 2026-08-17, stating that a separate repository re-uploading this project under another account contained wallet-enumeration and data-exfiltration behaviour not present in the original. I am reporting that the official project published that notice; I did not download the re-upload to verify its contents, and I am not going to.

What I did verify is the official repository at commit 3a97a71. No eval or exec, no shell=True anywhere, and no network client imported anywhere in the package: every subprocess call is an argv list invoking a local binary (pdftotext, pdfinfo, ebook-convert, pip). The danger grep carried a positive control that fired on planted strings. That is a clean scan of the code I ran, on one commit, and it is not a warranty about any other commit or any other copy.

What to actually do

Use it for the text extraction, not for its chapter count. A 501-page PDF to structured text in about 0.6 seconds with no configuration is a real result. The chapters: N (structural) line beside it is not trustworthy on a PDF: on this book it counted shell comments in code samples and reported 93 for a ten-chapter work, with no warning.

Do not budget from any tiktoken figure if your agent is Claude. Anthropic’s guidance, converted the same way throughout, means multiplying a tiktoken count by 1.18 to 1.25 on typical text and by an unstated amount more on code. These two books both measured 1.41x. Take 1.2x as the published floor and 1.4x as what this trial actually saw. That applies to this project’s tables, and to any other estimate computed with tiktoken for work that will run on Claude Code. How common that is I have not measured and will not guess at.

Ignore the ~5,000 until you generate a skill and measure yours. Pass --skill-dir and the script will measure your real one instead of the design cap. That is the honest version of the comparison and it is one flag away.

What this does not say

It is not a refutation of 24x-51x. The measurement moves their numerator up. Whether the whole ratio holds depends on the denominator, which I did not measure, because measuring it means generating a skill and that is a separate trial.

Two books, one delivery mechanism. Every byte arrived as a CLAUDE.md. A book pasted into a chat turn, read through a tool call, or loaded as a skill body may be framed differently, and none of those are measured here.

Measured to 120,000 bytes, not to a whole book. Both books are seven to nine times larger than the largest slice. The 320,000-token figure applies the measured ratio to the book’s full cl100k count, which is an extrapolation of the ratio beyond the range it was measured in. Applying the measured per-byte rate to the book’s full byte count instead gives 296,000, and the two methods disagreeing by 8% is itself a reason to treat either as an estimate.

One machine, one release, and the model is not controlled. Claude Code 2.1.238. The harness sums modelUsage across every model that appears in a run rather than pinning one, so “Opus” describes the session’s configured model, not something the measurement enforced.

Nothing here is a quality judgement. Whether a book-derived skill answers questions well is the thing the project is actually for, and this measures none of it.

Controls

Paired. The floor was re-measured every round and every delta is against that round’s floor.

Fixture. The byte count was read back off disk on every run and asserted equal to the slice written.

Slices are valid UTF-8. Each slice backs off to a UTF-8 boundary first and to a whitespace boundary second, asserted per run. Doing those in the other order fails, loudly, which is how the first attempt at this harness died.

Arrival, 28 of 28. Every reply had to be exactly OK.

Isolation. --setting-sources project asserted onto the command line in every call.

Single release. All 28 run directories were verified as 2.1.238 by reading the version field of every message in their Claude Code transcripts, rather than by trusting the harness’s own stamp, which is a single claude --version read at process start. The verifier also matched a 29th directory, a probe left at the same path prefix by earlier work; it is 2.1.238 too and is not part of this trial.

Estimator. The guarded mode of four rounds, which refuses a cell where nothing repeated or the top count is tied. No cell here refused.

Adversarial pass, and a correction to this post. Four reviewers attacked this draft, each recomputing every figure from the raw runs and checking every statement about book-to-skill against the cloned repository, and each finding was then handed to a separate verifier told to refute the reviewer. They raised 57 defects; verifiers confirmed 20 and killed the rest as themselves wrong.

A first version went live before that pass had finished, carrying two of its findings. It said the benchmark could not see chapters the extractor could, when the truth is the reverse: the extractor’s 93 are false positives and the benchmark’s zero is nearer right. And it treated book-to-skill as a Claude Code tool, which it is not. Also fixed then: a ratio of totals quoted where a slope was meant, a claim that discovery_tax.py imports one symbol when it imports two, and an unsupported claim about how common tiktoken estimates are.

A second pass was then run against those corrections, because a correction that is itself wrong is the worse failure, and it found one. The fix had said that “for an OpenAI-backed host cl100k is the right tokeniser”. It is not: tiktoken maps OpenAI’s current chat and reasoning models to o200k_base. That is repaired at the top, along with the discovery that Anthropic already publishes a figure for this gap, which this post had claimed nobody had. Three smaller repairs came from the same pass: the structural-count warning cannot fire at zero, only at exactly one; the fenced-code guard found one real fence rather than none; and the segment bound printed as “1.383x to 1.451x” excluded both of its own extremes, 1.3829 and 1.4513, because the generator rounded them. The bound is now floored and ceiled with the containment asserted in code.

A third pass then attacked round two, and found four more. The largest: this post quoted Anthropic’s “much more on code or non-English input” clause and then reasoned as though only the “typical text” number existed, claiming a conflict for both books when only the novel is the typical-text case. That is rewritten above, and the finding that survives is better than the one it replaces. Also corrected: two incompatible conversions of the same sentence in one post (“1.18x to 1.25x” in one place, “add 15-20%” in another, which are different conventions); “every current OpenAI model maps to o200k_base”, which is false because the open-weight gpt-oss pair maps to o200k_harmony; and an overstated distinction between a tokeniser figure and a context figure, when the cited doc’s own examples count a full message payload too.

Three passes, 91 findings raised, 40 confirmed, two fatal and both in the first version. One reviewer figure was itself wrong on recomputation, an intercept given as +1,769 against a measured +1,753, and is not used.

All 28 runs are in tokeniser-gap-2-1-238.json.