The same piece of text, fed to two models, gets tokenized into 766 tokens by one, and 1170 by the other.
The person presenting these numbers is Tibo, head of OpenAI Codex.
His exact words: One OpenAI token is not equal to another model's token. A lower price per token does not necessarily mean a lower bill.
Everyone is comparing prices using 'dollars per million tokens,' treating tokens as if they were a standard unit like grams or kilowatt-hours, but they are not.
To make it easier to understand, he told a pizza story.
Two identical pizzas.
The first shop cuts it into 8 slices, each costing $2. The second shop cuts it into 16 slices, each costing $1.25. The second shop's sign says it's cheaper, but the whole pizza costs $20, while the first only costs $16.
He added: Your stomach doesn't care how many slices you just ate.
Each slice is cheaper, but the whole pie is more expensive. Different cutting methods render unit prices incomparable.
A token is the smallest unit of billing for a model; you can think of it as the model's 'knife technique' for slicing text.
The same sentence, sliced with different techniques, yields a different number of pieces. You are charged for the number of pieces. More pieces mean a higher bill.
This comparison covered English, technical text, multilingual content, and numbers.
GPT-5.6 Sol's tokenizer used 766 tokens, while Claude Opus 5's estimate was 1170 tokens.
For the same text, GPT-5.6 Sol's 'knife technique' produced 34.5% fewer pieces.
And both companies' input price is $5 per million tokens.
The unit price is identical, but with 30% fewer pieces, the input cost is also 30% lower.
That's where the trouble lies.
If even 'how big is a token' can't be aligned between two companies, then does that widely circulated API price comparison table everyone shares daily still count?
Same Text, Two Different Counts? Why?
This is because the unit 'token' simply lacks a unified measurement standard.
Each vendor trains its own tokenizer, deciding how finely to fragment the text.
Common words are swallowed whole by the tokenizer; rare words can be split into three or four pieces.
English provides the clearest example. Words like 'the,' 'and,' 'is' appear constantly, so the tokenizer gives each a dedicated ID—one word, one token.
For a longer word like 'unbelievable,' it gets split into 'un,' 'believ,' 'able'—one word occupying three tokens.
The principle is simple: tokenizers are derived from statistical analysis of training data. Frequent combinations get their own slot. The rest have to be pieced together from fragments.
So 'how many tokens in a passage' essentially asks 'how common are the things in this passage within this company's training corpus.'
And English prose happens to be the content category with the *least* variation. For code, JSON, long number strings, the differences between how two companies slice them will only be greater.
Even Within One Company, Old and New Models Can't Share Counts
This isn't a problem unique to one company.
Anthropic's own documentation is explicit: Token counts are estimates; the actual number of input tokens used when creating a message may vary slightly.
They even provide a specific figure.
Models from Claude 4.7 onwards use a new tokenizer; the same input text generates approximately 30% more tokens compared to earlier models, with the exact increase depending on content and workload type.
Anthropic Official Docs: Claude 4.7+ models use a new tokenizer; the same text yields ~30% more tokens; don't reuse counts measured on older models.
The same company, the same text, yields 30% more after a model generation change.
Therefore, the official advice is: to know the difference for your workload, measure the same request against both models and compare the returned `input_tokens`.
Don't use token counts measured on early models to estimate costs.
Counts can't be reused even between two generations of the same company's models. So cross-vendor price comparison using 'price per million tokens' is even less standardized.
Same $5 Rate, Bills Differ in Four Places
Same unit price, same input—where exactly do the bill differences come from?
First, the tokenization efficiency mentioned above. The same text, different number of tokens, multiplied by the same unit price, naturally leads to different costs.
Second, caching.
GPT-5.6 Sol's cached input price is $0.50 per million tokens, only one-tenth of the standard input price. For workloads with many repeated prefixes, this alone can restructure the entire bill.
Third, output.
GPT-5.6 Sol's output is $30/million tokens, while Claude Opus 5 starts at $25.
In real-world agent workflows, output tokens often carry more weight than input.
Meaning, the 34.5% saved earlier might very well be given back here.
Fourth, the most easily overlooked, is stated right on OpenAI's own model page. For GPT-5.6 Sol, when input exceeds 272K tokens, the *entire request's* input is billed at 2x the rate, and output at 1.5x.
GPT-5.6 Sol Official Model Page: Input $5, Cached Input $0.50, Output $30. The fine print below states the premium rate rules for exceeding 272K.
It's not the portion exceeding the limit that's charged more; the *entire request* is subject to the higher multiplier.
The same piece of code, if you ask about it within a 270K token context versus a 280K token context, the unit price jumps a tier.
This limit comes from the official pricing page itself. Longer context windows mean attention and GPU memory costs rise faster; long context has never been free.
The Million-Token Window Is Open, Money Flows Out Gradually
Tibo later posted a second thread, teaching how to manually max out the context window in Codex.
Open ~/.codex/config.toml, add three lines before any section headers:
model = "gpt-5.6-sol"
model_context_window = 1000000
model_auto_compact_token_limit = 900000
First line selects the model, second line raises the context budget to 1 million tokens, third line triggers auto-compaction around 900K tokens, leaving some margin.
Save, restart the client, start a new session for the config to take effect.
For those not wanting to change defaults, you can also temporarily override for a single CLI session:
codex -m gpt-5.6-sol
-c model_context_window=1000000
-c model_auto_compact_token_limit=900000
Both keys can be found in the Codex official configuration reference, with the described effects.
`model_context_window`: The number of context window tokens available for the current model.
`model_auto_compact_token_limit`: Threshold for triggering automatic history compaction.
But the documentation only defines the keys' meanings; it doesn't list the '1M/900K' values as universal recommendations.
Tibo himself added at the end of his post: The defaults are carefully tuned.
So why do so many people want to change them manually?
A user's test report on GitHub explains the reason.
This test report in the openai/codex repo: Codex dir caps window at 372K, effective 353.4K, while model specs state 1.05M
Under a specific version of the Codex client and a ChatGPT Pro account, the model directory listed the window for gpt-5.6-sol as 372K, with 95% utilization yielding 353.4K usable. The official model page states 1.05M.
Bought a million-token window, usable window shrinks to one-third.
This report has clear version and account restrictions and shouldn't be taken as the current state for all users. Tibo's config post was published later.
Also, clarification: Changing the config to 1 million does not instantly incur a 1 million token charge. Billing is always based on actual processing volume.
But pushing the compaction threshold to 900K means a long session will carry increasingly long history forward, re-processing that history in each subsequent request.
The larger the window, the later the compression, the more likely a request hits that 272K premium threshold.
In short dialogues, tokenizer differences are a matter of decimal points. When a session stretches to hundreds of thousands of tokens, with history repeatedly carried along, multiplied by a higher pricing tier, those decimal point differences move to the integer column.
Money isn't spent all at once; it accumulates round by round.
The Next Unit Is 'Per Successful Outcome'
There's another line in Tibo's thread, overshadowed by the numbers: What truly matters is the cost per successful outcome.
He also gave the method. Benchmarks can be a starting point, but to truly know which is more expensive, you need to run your own tasks.
This line shifts the anchor point for price comparison. From 'price per million tokens' to 'total cost to complete the same task.'
To find out which of two vendors is actually cheaper for you, test it yourself.
Take the same raw text, same language mix, same tool definitions, call both vendors' official counting APIs to get the real token counts, factor in cache hits, output length, reasoning length, and long-context multipliers, and finally compare who costs less to get the job done.
Tokenization efficiency is just the first link in this chain. A model with more efficient tokenization, if its reasoning is verbose or requires more retries, can still end up with a higher bill.
The question going forward shouldn't be how much per million tokens, but how much to fix this bug.
References:
https://x.com/thsottiaux/status/2089082893804896524?s=20
https://x.com/thsottiaux/status/2088866513008873560?s=20 https://github.com/openai/codex/issues/31860
This article is from the WeChat public account "New Zhiyuan," author: ASI Apocalypse
