OpenAI's Misfire, Scaling Law's Original Paper Reveals Bug, Trillions of Compute Power Wasted in Vain
Recent revelations by a former OpenAI researcher, Diogo Almeida, and subsequent discussion highlighted by DeepMind's Sander Dieleman suggest a critical bug in OpenAI's seminal 2020 "Scaling Laws" paper. The analysis claims the original research contained a flawed experimental setup, leading to a misinterpretation of how to optimally scale large language models (LLMs).
The core issue involves two key methodological choices in the OpenAI paper: first, training all models (small and large) on the same fixed dataset size (~130 billion tokens), which underfed larger models; and second, using a cosine learning rate decay that prematurely flattened loss curves, creating the false impression that models had reached performance saturation with more data. This combination allegedly biased the conclusion that, for a fixed compute budget, scaling model parameters was vastly more important than scaling training data—a principle that drove the creation of "over-parameterized, under-trained" models like GPT-3.
This was later corrected by DeepMind's 2022 Chinchilla paper, which advocated for a more balanced scaling of parameters and data. Further scrutiny revealed that even the Chinchilla analysis itself had an optimization bug. The critique extends beyond the bug, questioning whether current scaling laws are inherently biased, as they are primarily derived from English data, a morphologically poor language that may be inefficient to learn compared to others like French.
The implication is that the AI industry may have wasted significant computational resources and years of effort following an erroneous scaling principle, potentially delaying more efficient model development.
marsbit5 saat önce
