DeepMind researcher reveals late at night: There's a critical bug in OpenAI's original Scaling Law paper! The global AI community has wasted trillions in compute power for nothing. GPT-3 was actually severely "overinflated".
OpenAI misled the entire AI field for years!
Over the past five years, the entire AI industry has been driven forward by the Scaling Law.
Sam Altman's confidence in AGI stems from this curve.
Now, someone has stepped forward to say: This curve was wrong from the very beginning.
This is not hindsight. The speaker is a researcher who was optimizing large models at OpenAI back then - Diogo Almeida.
Just now, he published a blog post with a chillingly straightforward title - "Scaling Laws, Honestly".
The opening sentence leaves no room for doubt: The original version of the scaling law is wrong because it contains a bug.
Portal: https://www.completeskeptic.com/p/scaling-laws-honestly
Sander Dieleman from DeepMind, famous for diffusion models, promptly boosted it on Twitter, calling it an interesting piece of LLM history:
The original scaling law was wrong due to a bug, likely causing the industry to waste massive compute power on a bunch of "overly large, undertrained" models.
One bug, burning two years.
When the bug is exposed, what we see is not just a black hole of compute power, but also a boundary of intelligence reshaped by language itself, far deeper than imagined.
Scaling Law Turns Out to Be the LLM Version of "Geocentric Theory"
In 2020, OpenAI concluded: Under a fixed compute budget, you should prioritize making the model larger, rather than feeding it more data.
In formula terms, the optimal parameter count is proportional to compute raised to the power of 0.73—parameters were the variable to aggressively scale.
This statement directly defined the shape of the GPT-3 generation. Pile on parameters. Pile them on relentlessly. 175 billion.
It told developers worldwide: Don't ask, just scale parameters; as long as you make the model big enough, miracles will happen.
Two years later, DeepMind threw down Chinchilla, overturning this conclusion: model and data should be scaled up with roughly equal importance, with about 20 tokens per parameter being cost-effective.
They trained a 70-billion parameter Chinchilla, fed it 1.4 trillion tokens—half the size of GPT-3 but with over four times its data.
The result? With the same compute budget, it comprehensively outperformed Gopher with 280 billion parameters but fed only 300 billion tokens.
In plain language: With the same money, one raised a "flabby" strongman, the other trained a lean boxer.
After a three-year delay, Beida alumna Weng Li delved into mainstream explanations for the subsequent differences, attributing the variance to their methods of calculating total parameters.
And that's not all. Even the "correct" Chinchilla itself isn't clean.
In 2024, Besiroglu et al. extracted and reran the data points from the Chinchilla paper, discovering another bug hidden in its own fitting:
The loss scale in the optimizer was set too high, applying the Huber loss per sample average instead of sum, causing premature termination of fitting.
The paper correcting a bug, itself carried another bug.
At this point, that phrase "first principles" repeated by countless people suddenly seems less solid.
So-called Scaling Law has never been an ironclad physical law like Newton's three laws; it's just an empirically fitted curve.
When Diogo Almeida believes the truth isn't so, it's not a matter of different methods, "the original scaling law itself had a bug."
Did OpenAI Deceive Global AI Peers with Three Tricks?
To create a lie believed by the global AI community, only three steps are needed.
Step One: Imprisoning Data.
OpenAI's paper fed all models—whether they were toddlers learning to walk (small models) or already giants—the exact same "meal size." Approximately 130B tokens of data.
Small models were thus "well-fed" or even "overfed," while the large models that truly needed massive data to fill their capacity were severely malnourished under the same token budget.
The Chinchilla paper later pinpointed the issue: They "used a fixed number of training tokens and learning rate schedule for all models."
This is like having kindergarteners and PhD students take the same exam with the same time limit, then claiming "performance depends only on talent."
Step Two: Deceiving LR Decay.
They used Cosine Learning Rate Decay, allowing the learning rate to smoothly approach zero as training neared its endpoint.
As training approached the preset endpoint, the learning rate was artificially pressed down to zero step by step, naturally causing the model's progress to "flatten."
Once the curve flattens, it looks like: This model has already learned all it can, feeding it more is useless.
The researchers thus concluded: "Adding more data is useless, the model has saturated."
This wasn't the model's limit; it was the learning rate artificially severing the model's growth path. It created a perfect illusion: performance has hit the ceiling, more data is futile.
But we now know those large models were nowhere near their limit.
Step Three: The Arrogance of Authority.
The third step, and the most insidious: The paper stated that the results were "largely independent of learning rate schedule."
Although many, including Diogo Almeida who was at OpenAI at the time, vaguely sensed something was off, technically this conclusion was correct under a fixed token limit.
But it simply didn't apply to the ideal "infinite data" world that the scaling law truly aimed to describe.
They mistook a localized truth under limited conditions for a universal cosmic law.
With these three steps combined, you get a law that is both wrong and extremely difficult to debug.
Even Diogo himself admits: Back then, he was also doing optimization at OpenAI and didn't spot this bug—that learning rate curve looked too much like it was "carefully set," who would have doubted it?
GPUs Wasted in Vain
Severe Compute Mismatch
Guided by OpenAI's erroneous formula, the AI industry entered the era of "brute force yields miracles."
This means that in recent years, the world's brightest minds and scarcest compute power were wasted on ineffective scale expansion.
This isn't just a matter of money; it's humanity collectively sprinting thousands of kilometers down the wrong track in the race against time towards AGI, all due to a learning rate setting.
If the discovery of the bug is heartbreaking, the ensuing deep reflection is chilling.
Researcher Adam Zachary Wasserman pointed out a blind spot overlooked by everyone: Even if the formula is corrected, the current Scaling Law is merely an "English Scaling Law."
He conducted a counterintuitive experiment: training models with the same architecture and compute power.
The result showed that a French model achieved a certain grammatical capability with efficiency 50 to 100 times higher than an English model.
Why? Because English is a "morphologically poor" language.
It relies too much on distributional patterns, requiring models to guess word meanings from massive data; whereas languages like French or Chinese, which are morphologically rich or structurally strict, carry a lot of explicit information in the words themselves.
This means all our current compute allocation plans are based on the most "data-hungry," least efficient language.
When you think you're exploring the physical laws of "general intelligence," you're actually just measuring "how wasteful English is with compute."
It's like trying to establish nutritional standards for all living creatures in the universe by studying a pig's appetite—this is not just bias, but a cognitive limitation.
We could have achieved stronger performance with smaller models and more high-quality data.
We could have saved tens of thousands of H100 runtime hours worth of electricity and heat.
We could have entered the "Efficient AI" era two years earlier.
References:
https://www.completeskeptic.com/p/scaling-laws-honestly
https://lilianweng.github.io/posts/2026-06-24-scaling-laws/
This article comes from the WeChat public account "New Zhiyuan," author: ASI Apocalypse, editor: David
