Eight tokenizers agree exactly on English. On Hindi they differ by 3.3x

Everyone prices models in dollars per million tokens, which only means something if a token is the same size everywhere. For English, remarkably, it very nearly is. For anything else it is not, and the gap is wide enough to reverse which model is cheaper for your text.

I wrote one paragraph, translated it into nine more languages so the meaning is held constant, and counted the tokens each model’s tokenizer produces. The full corpus is published so every number here can be reproduced exactly.

The whole table

Tokens for the same paragraph. Lower is cheaper.

Tokenizer	English	Spanish	Russian	Arabic	Hindi	Chinese
GPT-4o (o200k)	92	103	122	113	154	105
GPT-4 (cl100k)	92	119	210	286	492	169
Qwen3	92	118	145	136	468	89
DeepSeek-V3	92	114	135	132	283	87
Mistral-Nemo	92	105	137	106	186	138
Phi-4	92	119	210	286	492	169
Falcon3	92	107	452	307	504	176
OLMo-2	92	119	210	286	492	169
Claude Opus 5	121	201	183	264	355	154
Claude Sonnet 5	120	200	186	259	—	—
Claude Haiku 4.5	93	128	186	261	355	155

Language	Cheapest	Dearest	Spread
English	92	121	1.32x
Spanish	103	201	1.95x
Chinese	87	176	2.02x
Arabic	106	307	2.90x
Hindi	154	504	3.27x
Russian	122	452	3.70x

Eight tokenizers agree on English to the token

Every one of the eight open and OpenAI tokenizers returned exactly 92 tokens. Not close: identical. I checked it was not a fluke of one paragraph by running a second, unrelated English text through all eight, and they returned exactly 87 apiece.

That unanimity is specifically an English phenomenon, not a “clean text” phenomenon. The same eight tokenizers disagree on a TypeScript snippet (110 to 150), on a JSON payload (76 to 102) and on a markdown table (66 to 98). English prose is the one input on which the industry has converged.

The exceptions are Claude Opus 5 and Sonnet 5 at 120-121, about 30% above the consensus.

Three of these are literally the same tokenizer

Phi-4, OLMo-2 and GPT-4’s cl100k_base do not merely produce equal counts. They produce identical token ID sequences on every one of five test inputs. That is shared vocabulary, not coincidence, and it is why those three rows are the same across the entire table.

Worth knowing before you treat a benchmark of “three different models” as three independent data points about tokenization.

Non-Latin scripts are where the money is

Russian is the widest gap: 122 tokens on GPT-4o against 452 on Falcon3, for the same sentence. Hindi runs 154 to 504.

Measured against each tokenizer’s own English baseline, this is the tax a non-English user pays:

	Spanish	Russian	Arabic	Hindi	Chinese
GPT-4o	1.12x	1.33x	1.23x	1.67x	1.14x
GPT-4 / Phi-4 / OLMo-2	1.29x	2.28x	3.11x	5.35x	1.84x
Falcon3	1.16x	4.91x	3.34x	5.48x	1.91x
Qwen3	1.28x	1.58x	1.48x	5.09x	0.97x
DeepSeek-V3	1.24x	1.47x	1.43x	3.08x	0.95x

Two things fall out of that.

GPT-4o’s tokenizer is the single biggest upgrade in this table. Against the older cl100k_base it cuts Hindi from 492 tokens to 154 and Arabic from 286 to 113. Same vendor, same sentence, a third of the tokens.

Chinese-developed models are the only ones where Chinese is cheaper than English. Qwen3 at 0.97x and DeepSeek-V3 at 0.95x both encode the Chinese paragraph in fewer tokens than the English one. Every other tokenizer charges a premium of 1.14x to 1.91x.

Claude’s models disagree with each other, but only on Latin script

This was the surprise, and it sharpens an earlier finding here that Haiku 4.5 counted the same file more cheaply than Opus 5. That is true, but it is not a general discount. It is confined to one script:

Script	Opus 5	Sonnet 5	Haiku 4.5	Haiku ÷ Opus
Latin (English)	121	120	93	0.77
Latin (Spanish)	201	200	128	0.64
Cyrillic (Russian)	183	186	186	1.02
Arabic	264	259	261	0.99
Devanagari (Hindi)	355	—	355	1.00
Han (Chinese)	154	—	155	1.01

On Latin-script text Haiku needs 23% to 36% fewer tokens than Opus. On every non-Latin script tested the three models agree within 2%.

That is why the earlier per-file ratios moved around between 0.66 and 0.90 depending on content: every one of those files was ASCII, and the gap tracks how compressible the Latin text is.

Spanish is the sharpest case. It is only 10% more bytes than the English paragraph, and GPT-4o charges 12% more for it. Claude Opus charges 66% more, ending at 201 tokens against GPT-4o’s 103. For Spanish text, Opus needs nearly double the tokens.

I am reporting where the divergence lies, not why. A difference confined to one script is a real pattern and the measurement is clean, but I have no visibility into the vocabularies and I am not going to invent a mechanism for it.

What this actually changes

Compare price lists only after converting to your text. The published rate is dollars per million tokens; what you care about is dollars per million bytes of the thing you actually send, and the conversion factor is different for every model and every language.

For an English-only workload the tokenizers are close enough that the price list is roughly the truth. For Hindi, Russian or Arabic it is not: a model advertised 30% cheaper per token can be more expensive per document, because it turns that document into three times as many tokens.

Method

The eight open and OpenAI tokenizers were run locally and offline, so those figures are exact counts rather than estimates: tiktoken for o200k_base and cl100k_base, and the published tokenizer.json for each open-weight model. Llama and Gemma are omitted because their tokenizers sit behind gated repositories.

Claude has no public tokenizer, so those three rows are measured empirically instead, by loading the text as a CLAUDE.md and reading reported prompt tokens. Each was measured at two sizes, twenty copies and forty, and the slope taken: subtracting cancels the fixed per-file overhead exactly, leaving pure content cost. Each Claude figure therefore includes one paragraph separator, worth about one token. Opus and Haiku figures reproduced across two rounds within one token.

All Claude measurements on Claude Code 2.1.226, headless, totals from modelUsage. The corpus is published in full; the numbers hold for that exact text, and a different translation would shift them without changing the picture.