In a recent interview, Sam Altman gave a rare assessment:
"He may be the most important yet least known researcher in AI history."
This person is Alec Radford, the true father of GPT.
You may not have heard of him, but you've probably used things he invented.
He is the first author of papers for GPT-1, GPT-2, CLIP, and Whisper.
Additionally, he participated in key works like GPT-3, DALL·E, Scaling Law, GPT-4, and GPT-4o.
Strangely, for someone whose resume spans almost every major turn in the development of large models, he has no PhD and rarely gives interviews.
When ChatGPT took the world by storm, he was almost invisible under the spotlight.
But if you look back through AI papers from the past decade, you'll find an even stranger phenomenon:
Whenever a model starts gaining a previously absent general ability, Alec Radford's name often already appears in the author list.
OpenAI's Real Take-off Began with Hiring Alec
In the latest episode of the "Invest Like The Best" podcast, the host asked Sam Altman:
"Looking back on the company's journey, who is the unsung hero you most admire?"
Altman immediately answered, it's Alec Radford.
Alec's work later truly evolved into the GPT series.
Beyond that, he continuously inspired and guided those around him, pushing research in directions that later proved extremely important.
Wired magazine also gave an equally weighty judgment:
OpenAI's rise truly began when it hired the then-unknown Alec Radford.
In 2016, Alec joined OpenAI. He was only 23 years old that year.
Back then, he was just a kid eating pineapple-onion pizza at midnight and participating in Kaggle competitions with classmates. He had founded a small AI company from his dorm in Boston: Indico.
After accepting the invitation to join OpenAI, Alec didn't think it was that important either, feeling the job was "a bit like graduate school."
There wasn't much short-term pressure, nor were there products that had to be built immediately. He was free to pursue a question no one knew the answer to at the time: What can language models actually do?
His first attempt was training a language model on 2 billion Reddit comments. The dataset was sizable, but the results were useless.
This experiment, like many early explorations at OpenAI, failed.
But OpenAI let him continue. Former CTO Greg Brockman recalled:
"We thought Alec was amazing, so we just let him do what he wanted."
So, the young man continued his experiments. But soon, he was limited by OpenAI's insufficient computing power.
Since he couldn't run larger experiments, Alec scaled down. He gathered about 100 million Amazon product reviews and made the model do something so simple it was almost boring: predict the next character based on the preceding text.
During this process, an unexpected thing happened.
Although no one taught the model what a positive or negative review was, a neuron inside the model began actively distinguishing the sentiment of the reviews.
Adjusting it in one direction made the model generate positive reviews more easily; adjusting it the other way made it start complaining about products.
OpenAI later called this the "Unsupervised Sentiment Neuron."
This experiment gave Alec the answer he was truly looking for:
When a model is tasked with a prediction task on a sufficiently large dataset, it may spontaneously learn abilities never explicitly stated in the training objective.
Ilya Sutskever also encouraged him to move beyond Amazon reviews and let the model learn from larger, more diverse text, even the entire internet.
The problem was, with the neural network architectures of the time, processing such vast data could take years.
But soon, OpenAI had a stroke of luck. Because in 2017, Google's paper "Attention Is All You Need," which rewrote the course of AI, was published.
Ilya immediately saw the potential of the Transformer architecture. His first reaction was: "This is exactly what we've been waiting for!"
Alec then plugged the Transformer into his previous line of experiments. He obtained the BooksCorpus dataset, containing over 7,000 unpublished English books spanning genres like romance, adventure, and fantasy.
Compared to many datasets where sentences are fragmented, books retain long continuous narratives, making them more suitable for models to learn long-distance relationships between contexts.
He didn't use Transformer for machine translation like Google did; instead, he made it predict the next most likely word.
The machine responded: one word, then another, then another—each new word inferred from the hidden patterns in those seven thousand books.
For Alec, "the progress made in these two weeks exceeded the combined progress of the past two years." This set of experiments ultimately became the pre-training foundation for GPT-1.
He and his colleagues began discussing a direction called "Big Transformer": don't design more intricate rules for the model; just make the model, data, and computational scale larger, and see what else it can learn.
In 2018, this approach got an official name:
Generative Pre-trained Transformer, or GPT.
The first GPT paper had four authors, and the first listed was Alec Radford.
GPT-1 didn't immediately shake the world, but it gave OpenAI something even more important: direction.
After Alec's breakthrough, OpenAI's management made a series of key decisions. They began consolidating research resources from scattered projects like robotics and focusing them on language models.
Rather than continuing to design complex new structures, the team preferred collecting more data, investing more computing power, and scaling up the methods that had already shown potential.
A year later, GPT-2 was released.
Its parameter count increased from GPT-1's 117 million to 1.5 billion, and its training data expanded from over 7,000 books to about 8 million web pages.
The GPT-2 paper had six authors; Alec was still first author, sharing the primary authorship with Jeffrey Wu.
This time, the model no longer needed retraining for each task. Just by predicting the next word, it could attempt translation, question-answering, summarization, and even achieved decent scores on multiple zero-shot tests.
In early 2020, OpenAI formalized this relationship in the paper "Scaling Laws for Neural Language Models."
Alec was also among the 10 authors. The judgments accumulated from previous experiments began to be described as predictable mathematical laws.
In May of the same year, OpenAI scaled the model further to 175 billion parameters, creating GPT-3.
Now, the paper's authors had grown from 4 for GPT-1 to 31.
The first author became Tom Brown, with Alec listed 29th, just ahead of Ilya Sutskever and Dario Amodei.
The change in authorship position also hinted at the shift in the nature of the GPT project.
It was no longer a small experiment led by Alec but a large-scale project undertaken by OpenAI mobilizing research, engineering, and computing resources.
The path Alec first championed had become the company's most important technical strategy.
But just as GPT was becoming widely known and about to have a massive impact, Alec had already turned his gaze elsewhere.
Beyond Language, He Continuously Anticipated Images and Speech
After GPT-3 was released, Alec returned to his old specialty: images.
Even before joining OpenAI, his most influential work was DCGAN.
Together with Luke Metz and Soumith Chintala, he introduced convolutional networks into GANs, solving the problem of difficulty in training such models stably.
They also found that when the model learned to generate bedroom pictures, it spontaneously formed visual representations about beds, windows, and scene structure.
The intuition that would repeatedly appear in Alec's later research was already taking shape:
As long as you find a sufficiently general training task, the model might learn additional abilities in the process of completing it.
Actually, there's a bit of gossip in between.
In 2016, Jensen Huang showcased images generated by DCGAN at an Nvidia conference but attributed the main credit to the Facebook lab led by Yann LeCun at the time.
Alec and several companions watched the livestream from their Boston office, feeling very hurt.
Not long after, Alec left Boston and joined OpenAI.
His friend Victoroff even believes this oversight was an important reason Alec decided to go to OpenAI.
Years later, he returned to images with GPT.
In 2020, OpenAI released Image GPT: flatten an image into a sequence of pixels, then predict the next pixel just like predicting the next word.
Alec was listed as the second author.
It proved that GPT is not inherently limited to language. As long as data can be represented as a sequence, the same method can be used to learn images.
But generating pixels one by one was too slow. Half a year later, DALL·E compressed images into visual tokens and placed both text and images into the same Transformer sequence.
"An avocado-shaped armchair," "a radish baby walking a dog"—these never-before-seen combinations were drawn by the model for the first time.
Released on the same day as DALL·E was CLIP.
This time, Alec returned to the position of co-first author.
OpenAI used 400 million image-text pairs collected from the internet, making the model do just one multiple-choice question: which piece of text and which image are a pair?
No fixed categories, no specialized training for each task.
In the end, CLIP, without using the ImageNet training set, achieved zero-shot recognition accuracy comparable to early supervised-trained ResNet-50.
Beyond images, he also experimented with sound.
The 2020 Jukebox compressed audio into discrete codes, then made the Transformer generate music like it generated text.
Two years later, the same line led to the more practical Whisper.
Alec was again the first author.
OpenAI collected 680,000 hours of audio and its corresponding text from the internet. The data wasn't perfect but contained different languages, accents, noises, and usage scenarios.
Whisper wasn't specifically optimized for any single benchmark, but when tested zero-shot on multiple different datasets, it performed more robustly, better handling real-world accents, background noise, and specialized vocabulary.
These works superficially span images, language, multimodal, and speech, but stem from Alec's same judgment:
Prescribe fewer rules, provide more data and computing power, and general capabilities might emerge on their own.
He always seemed to find the next simple task worth scaling one step ahead of industry consensus.
The Most Mysterious Man at OpenAI
And so, the illustrious past of this big shot ends here.
Returning to the man himself, he could almost be described as "formerly" the most mysterious man at OpenAI.
Compared to the influence of his papers, Alec almost never cultivates a personal brand.
Although he has an X account with over 70,000 followers, he rarely posts about himself, mostly just retweeting.
The pinned post at the top is still the one from 2018 announcing GPT-1.
"This is the thing I've been doing for the past year."
He first listed three sources of inspiration: CoVe, ELMo, and ULMFiT, then described the achievement in a rather simple sentence:
"A Transformer language model that can be transferred to a wide range of NLP tasks with only minor adjustments."
Who would have thought this work heralded a whole new era?
Public long-form interviews with Alec are few and far between; his personal website is also simplistic, almost blank.
Only his name, "Latest Posts," and pagination buttons, with almost no other content.
(He even uses the same photo all the time. This big shot is way too low-key.)
This creates a strange contrast with his status in the industry.
When the public talks about OpenAI and GPT, the first names that come to mind are Sam Altman and Ilya Sutskever; but when researchers discuss key papers from the past decade, it's hard to bypass Alec Radford.
Altman described in the latest interview that if you ask those who have worked with Alec, of course they'd first say:
He is a "once-in-a-generation genius," an extremely creative thinker with a profound understanding of his research.
But before they finish, everyone would add another sentence:
He's also one of the kindest, nicest people they've ever collaborated with.
One More Thing
However, this highly praised father of GPT is quite unconventional in his research...
In December 2024, Alec left OpenAI after nearly nine years to pursue independent research.
But when everyone expected him to create the next GPT—
Alec went and made an "Old-timer AI."
In April this year, together with Nick Levine and David Duvenaud, he launched Talkie.
This is a 13-billion parameter "antique language model," with the base model trained on 260 billion tokens.
Its pre-training corpus has one hard rule:
English materials published after January 1, 1931, are absolutely NOT ALLOWED in!
So, it has read about steam engines, airplanes, and World War I, but doesn't know what happened later with television, the internet, or World War II.
It certainly doesn't know Python.
But for such an old-timer, after researchers showed it a few Python program examples and asked it to solve HumanEval programming problems,
It actually managed to write some simple code.
This experiment truly tests: how quickly can a language model base that has never encountered modern computers absorb a completely unfamiliar skill after receiving a small number of new examples and post-training?
What? You're saying while the entire internet is still scaling like crazy, trying to get models to read more...
The father of GPT instead "disconnected" it from the internet for nearly a century??
If a model hasn't seen the answer but can still learn and apply on the spot, is it relying on memory, or learning?
Who knows what the big shot is betting on this time...
But following his usual pattern, by the time the outside world finally understands the question, he himself has likely already moved on to the next one~
References:
[1]https://x.com/InvestLikeBest/status/2086849947119333878?s=20
[2]https://www.wired.com/story/what-openai-really-wants/
[3]https://www.bostonglobe.com/2023/06/10/business/how-couple-olin-college-students-helped-spark-ai-chatbot-revolution/
[4]https://www.newyorker.com/books/under-review/can-sam-altman-be-trusted-with-the-future
This article is from the WeChat public account "Quantum Bit," author: Focus on Frontier Technology
