69 is the prime age to "create"!
Richard Sutton, the foundational figure of reinforcement learning and recipient of the 2024 Turing Award, announced that he and his student Khurram Javed have left Keen Technologies, founded by John Carmack, to start their own venture named Oak Lab.
Richard Sutton can be considered the pioneer of modern reinforcement learning:
He pursued his undergraduate studies in psychology at Stanford and earned his Ph.D. under the guidance of reinforcement learning pioneer Andrew Barto. Early in his career, he conducted research at GTE Labs and AT&T Labs.
He proposed the Temporal Difference algorithm, and co-authored with his advisor Andrew Barto the book "Reinforcement Learning: An Introduction," which serves as the universal textbook in the field.
From 2003 onwards, Sutton served as a full professor at the University of Alberta for a long time, establishing the university's Reinforcement Learning and Artificial Intelligence (RLAI) laboratory.
From 2017 to 2023, he worked as a Distinguished Research Scientist at DeepMind, leading the establishment of the DeepMind Edmonton research team.
Over more than forty years, Sutton has also nurtured a large number of top talents in the AI industry.
These include David Silver, the core designer of AlphaGo; Doina Precup, Head of DeepMind Montreal; Michael Bowling, an expert in game AI; and his current co-founder Khurram Javed.
Judging from his tweets, Sutton's reason for this independent venture is straightforward:
He believes current deep learning methods are weak and inefficient, requiring not minor fixes but entirely new foundational ideas and a complete overhaul.
In other words, Sutton thinks the existing AI path has difficulty advancing further toward higher levels of general intelligence.
The ultimate goal of Oak Lab is:
To build an agent with a trillion parameters, capable of real-time learning and real-time planning, with a total system power consumption of only 20 watts.
20 watts is precisely the energy consumption level of the human brain.
While the entire AI industry is still stacking GPUs and expanding data centers, the founding father of reinforcement learning is preparing to redefine what intelligence is.
Parting Ways with the "Carmack God"
To understand why Sutton is leaving, we must first clarify where he is leaving from.
The founder of Keen Technologies, John Carmack, is the creator of "Doom" and "Quake," the former CTO of Oculus, a legendary figure in the programming world known as "Carmack God."
After leaving Meta in 2022, he fully dedicated himself to an AI startup, also focused on reinforcement learning.
In September 2023, Sutton chose to join Keen, prompted by Google DeepMind's closure of his Edmonton laboratory in Canada.
At the time, their collaboration was seen as a dream team:
One was a legendary figure in low-level systems engineering, the other a foundational theorist in reinforcement learning, aiming to build a prototype system showing "signs of AGI life" by 2030.
Now, less than three years into the collaboration, Sutton has chosen to step away.
However, he specifically dedicated the first line of his tweet to Carmack:
I can't say enough good things about John Carmack and Keen Technologies.
The implication is clear: the departure isn't because Carmack wasn't good enough, but because they diverged on how to reach the end goal.
In Sutton's view, the current developmental trajectory of deep learning is untenable.
Models don't need endless iterative fine-tuning; the entire industry urgently requires a paradigm-level disruption and reconstruction.
What Does Oak Lab Aim to Do?
The core bet of Sutton's startup can be summarized in one sentence:
Intelligence arises from continuously generated experience at runtime.
The working mode of current mainstream large models essentially involves months of offline pre-training with enormous costs, relying on massive amounts of textual data.
After training concludes, the model's parameters are mostly fixed before deployment.
Even though it converses with billions of users daily, the vast majority of these interactions cannot transform into new capabilities for itself.
The model can only reuse knowledge acquired during the training phase or temporarily remember information within the conversation context. It cannot update itself through continuous perception of the outside world, like humans and animals do.
But the agent Oak Lab wants to build is different.
It will perceive its surroundings, take corresponding actions, and adjust its behavior based on the outcomes.
Whenever it has a new experience, the learning process happens synchronously, without waiting for long intervals to conduct a new centralized training round.
As Sutton himself says:
Every moment an AI runs should be a moment of learning.
Currently, Oak Lab has announced its core research roadmap, centered around an architecture named OaK.
OaK stands for Options and Knowledge.
The purpose of this architecture is to enable an agent to discover temporally extended abstract structures from its own experience and transform them into verifiable, plannable, and reusable skills.
For example, when a robot goes to the kitchen to get water for the first time, the entire process involves identifying the room, avoiding obstacles, picking up a cup, turning on the faucet, and a series of other actions.
Traditional AI would treat all these steps as a single decision-making task.
The OaK architecture, however, would allow the agent to deconstruct higher-level skills like "go to kitchen," "pick up cup," and "get water" from the practical operation.
When encountering similar goals later, the agent can directly retrieve existing skills and flexibly adjust the plan according to the current environment.
This method of compressing past experiences along the temporal dimension is called temporal abstraction. It allows AI to imitate humans, solidifying a series of scattered actions into mature skills and using combinations of these skills to accomplish more complex tasks.
In addition, the Oak architecture has another design goal fundamentally different from current deep learning:
The learning phase will neither store historical data nor replay past experiences.
Current deep reinforcement learning often places a large amount of historical experience into a replay buffer for repeated sampling and training.
Oak Lab envisions using a real-time learning approach with a batch size of 1, updating immediately after obtaining each new piece of experience.
The team believes that if such algorithms are combined with event-driven neural networks, the required computation and energy consumption of the system could drop by several orders of magnitude, making continuous, real-time learning truly feasible.
Hence the long-term goal: a trillion parameters, real-time learning, real-time planning, 20-watt power consumption.
Of course, this is still just a concept for now.
There is also a theoretical cornerstone behind Oak Lab, originating from the Big World Hypothesis jointly proposed by Sutton and Javed.
The core idea is: the real world will always be more complex than AI. Even as models become more powerful, the amount of data in the external environment will similarly explode.
Models trained on pre-organized data cannot keep up with real-world changes.
AI must learn to selectively remember useful content, appropriately forget outdated information, and continuously learn online to adapt to the real world.
From "The Bitter Lesson" to Entrepreneurship
Those familiar with Sutton's work won't be surprised by the views above.
In 2019, he wrote the widely circulated short essay in the AI field: "The Bitter Lesson."
The article reviewed the developmental histories of chess, Go, speech recognition, and computer vision, concluding:
General methods leveraging computation and search that scale with computation ultimately outperform methods reliant on human-crafted knowledge in the long run.
In the era of large models, this path seems to have received strong validation: larger models, more data, and greater computing power have driven AI capabilities to soar.
Yet Sutton remains unsatisfied with today's mainstream deep learning.
In his view, current systems still heavily rely on data that has been produced, curated, and organized by humans.
What models learn is primarily things humans have already written, photographed, or labeled in the past.
A true intelligent agent needs to generate new experiences through its own actions and use those experiences to pursue long-term goals.
This also explains why he has progressed from "The Bitter Lesson" to the "Era of Experience."
In 2025, he, together with AlphaGo core figure David Silver, proposed that AI would gradually shift from relying on human data to depending on experiences generated by agents interacting with the environment.
Oak Lab is precisely the entrepreneurial embodiment of this research proposition.
Forty years ago, when Sutton wrote about "temporal credit assignment in reinforcement learning" in his doctoral dissertation, reinforcement learning was a niche field.
Forty years later, while the whole world chases the commercialization wave of large models, he is still asking the same question—
How does intelligence truly come about?
One More Thing
The first stop for the founder after launching his startup: Shanghai WAIC (World AI Conference).
At the WAIC Thinkers Forum, Sutton will deliver a keynote speech titled "The First Principles of Reinforcement Learning: Cultivating Superintelligence from Experience."
Reference link: https://x.com/RichardSSutton/status/2076663628301058329
This article is from the WeChat public account "QbitAI," author: Focus on Frontier Technology.
