Author: Li Fei-Fei
Compiler: Jiayang
"World model" is probably the hottest and most chaotic concept in the AI field since 2025. When Sora was released, OpenAI called it a world simulator; Genie lets you walk around in generated scenes and also calls it a world model; robotics companies say they are working on world models, NVIDIA says Omniverse is the infrastructure for world models, and even game engines have been pulled into this narrative. Everyone is using the same term, but they are all talking about completely different things.
Today, Li Fei-Fei published a new article on her personal Substack, clarifying this concept. She first returns to the most classic diagram from reinforcement learning textbooks (the POMDP closed loop: agent→action→state→observation→agent), and then points out: what is now called a "world model" is actually three different projections of this closed loop. What outputs pixels (observations) is a renderer, what outputs states is a simulator, and what outputs actions is a planner. The classification standard is very simple: it depends on which part of the closed loop you output.
(Source: MIT Technology Review)
She judges that among the three, the renderer is the most commercially mature but has a ceiling (looking good does not equal being physically correct), the planner is the most exciting but farthest from real-world deployment (the gap between lab demos and practical usability is still huge), and the simulator is the critically underestimated hub. Because the simulator works at the level of geometry, physics, and dynamics, it can project upward into pixels for human consumption, and also derive action consequences for robot use. Mastering simulation gives you the foundation for both rendering and planning; the reverse is not true.
This article is of course also a product manifesto for World Labs. Their Marble already outputs both Gaussian splats and collision meshes, attempting to unify the renderer and simulator into a single model. The endgame depicted at the end of the article is a unified world foundation model, capable of freely switching between rendering, simulation, and planning based on downstream needs. Whether this vision can be realized is another matter, but as an analytical framework, the tripartite classification of renderer/simulator/planner may indeed help cut through some of the noise surrounding the current concept of "world model."
The full text is translated below.
"The world is everything that is the case." — Ludwig Wittgenstein, Tractatus Logico-Philosophicus, 1921
The world is not made of words.
In an earlier article, we proposed that spatial intelligence is the next frontier of AI, and world models are the path to it. Here, the World Labs team and I want to go one level deeper: among the many things now labeled "world models," which functional components truly constitute this capability? And what are their respective uses?
Language models have endowed machines with powerful control over concepts, vocabulary, and reasoning, but the physical world, whether virtual or real, operates on a completely different substrate. Language models learn the statistical structure of text; world models learn the statistical structure of space and time: how light falls on a surface, what a garden looks like from an angle never captured by a camera, how objects respond to forces and follow physical laws.
This makes "world model" one of the most important, and also most abused, terms in AI today. Computer vision, robotics, reinforcement learning, and generative AI all claim to be building world models, but each refers to something quite different. A video model that generates gorgeous but physically impossible flames, a language model that improvises playable games, a physics engine that faithfully simulates combustion—they are all called by the same name.
The ancient Greeks could never agree on what the world was made of, be it fire, water, or indivisible atoms, because "the world" was never a single thing. It was always a stand-in a thinker used to reason about a certain totality. AI has inherited the same problem, and it happens precisely at the moment when the field most needs precision.
The Closed Loop Behind the Taxonomy
To sort out this confusion, we can start with a diagram older than all the technologies mentioned above. All reinforcement learning textbooks, including the classic Sutton and Barto, have for decades used variants of the same diagram to describe how an agent interacts with the world. The formal name of this diagram is a partially observable Markov decision process (POMDP), and the initial definition of the term "world model" belongs to this tradition.
An agent (which can be a human, robot, or software system) performs actions. These actions change the state of the world. But the agent can never directly see the state itself; what it receives are observations: photons hitting the retina, sensor readings, pixels in a video frame. New observations guide new actions, and the cycle repeats.
The word "state" needs to be unpacked because its meaning shifts across fields. This is not the chemist's state, not the distinction between solid, liquid, and gas. This is the physicist and roboticist's state: a complete description of everything that is happening in the world at a given moment, including every object, every position, every velocity, every property. State is the underlying reality of the world, complete in principle, but forever unobservable directly by any agent within it. Observations are the agent's partial perspective on this reality. Actions are the agent's response based on this.
This closed loop (agent→action→state→observation→agent) is precisely the structure that gives the term "world model" its technical meaning. The phrase itself is older, dating back to Kenneth Craik's 1943 proposal that the mind reasons by running a "small-scale model" of reality, and was introduced into the neural network field in the late 1980s and early 1990s. This closed loop also explains what people mean when they use the term today. The various things now called world models are actually different projections of the same closed loop, each outputting a different component of the loop.
The Three Functions of World Models
The first type of world model is the renderer. The renderer outputs observations, specifically pixels for human eyes, and the most important quality metric is visual fidelity. A video model that turns text prompts into cinematic aerial shots is a renderer; interactive systems like Google's Genie 3 or World Labs' own RTFM are also renderers, generating imagery in real-time based on user input. Such models lack explicit understanding of 3D structure. They generate what a viewer would see, not what things actually are. The buildings in an aerial shot might look flawless from above, but try to navigate the city below and they fall apart.
The second is the simulator. The simulator outputs states: a representation of the world that is faithful in geometry, physics, or dynamics, on which both humans and computer programs can compute and interact. The renderer's contract is purely visual, while the simulator's contract is structural—it requires geometry that holds up to inspection, physics that follows Newton's laws, and dynamics that behave as expected by physical principles. The simulator serves two types of users. Professionals like architects, designers, filmmakers, and game developers need accuracy beyond visual plausibility. Computer programs like reinforcement learning agents, robot controllers, and autonomous vehicles use the simulator as a training ground to interact with the world at scale, testing scenarios that would be dangerous, expensive, or impossible to execute in reality.
The third is the planner. The planner outputs actions. Given an observation and a goal, the planner answers the question: what should the agent do next. In many ways, the planner is the inverse of the renderer. The renderer takes actions as input and produces observations; the planner takes observations as input and produces actions, thereby closing the perception-action loop. Vision-Language-Action models (VLA), model-based systems, and the new wave of World Action Models are all attempts at planning: enabling systems to decide what a robot should do in an unstructured world.
These three categories cover most of the work currently being implemented, and distinguishing between them is useful in practice. But these categories are not fundamentally separate. They share the same underlying knowledge about how the world works: geometry, physics, dynamics. A model that can render a cup from any angle should, in principle, also be able to simulate what happens if the cup is pushed, and plan a hand to pick it up. Increasingly, the most interesting research is intentionally blurring the boundaries between these three.
Image丨The Three Types of World Models (Source: Substack)
Why Simulation is the Critical Hub
Among the three categories, the simulator receives the least public attention but is the most important. This article aims to correct that asymmetry.
Renderers are currently the most commercialized. A multitude of image- or text-to-video products are expanding rapidly in both consumer and enterprise markets. Google's Nano Banana model delivers renderer-level image generation capabilities to possibly hundreds of millions of users. The technology is real, and the market is real. However, the target of renderer optimization is visual plausibility, not physical accuracy, and that ceiling is important. Their outputs are beautiful, but you cannot use them to design a building or train a robot.
Planners are the most exciting and least mature, closely related to the rapidly evolving field of robot learning. Over the past two years, this field has produced many robot demonstrations that look impressive in videos, but we need to be honest about what these demos actually show. Almost all are confined to highly constrained lab environments, with limited object variety and short task durations. None have been validated against the complexity, diversity, and duration required for real-world deployment. The gap between a stunning demo video and a robot that works reliably in a kitchen, warehouse, or operating room remains vast.
Despite this, the scale of commercial bets is substantial. A wave of well-funded newcomers is racing to launch general-purpose planning systems, while large infrastructure players are building planning capabilities on top of broader simulation stacks.
Simulation is the bridge connecting the two. If language is an abstraction of the world, and pixels are a projection of the world, then geometry, physics, and dynamics are the world itself. The simulator must operate at this level: it is the structural skeleton from which visual appearance (for renderers) and action consequences (for planners) can both be derived.
A model that masters simulation can project its understanding into pixels for human consumption, and into action predictions for embodied agents. A model that only masters rendering or only planning can do neither. The commercial space here is immense. NVIDIA's Omniverse alone targets a total addressable market estimated by the company at over a trillion dollars, covering factories, warehouses, supply chains, and digital twins. Robot training, autonomous driving testing, architectural visualization, engineering design, drug discovery—all rely on some form of simulation.
The most difficult open questions in the field are also concentrated here. 3D data with explicit geometry, material properties, and physics annotations is several orders of magnitude scarcer than the internet videos used to train renderers. The sim-to-real gap (the difference between how objects behave in simulation and in the real world) persists. Generative simulators add a new risk: AI-generated geometry may look correct but actually contain self-intersections or incorrect scales, causing physics simulation to produce absurd results. The computational cost of large-scale multi-physics simulation (rigid bodies, deformable objects, fluids, cloth all interacting simultaneously) is still several orders of magnitude higher than single-domain simulation.
At World Labs, Marble is our first step in this direction. It takes multimodal input (text, images, video, or spatial sketches), generates explorable 3D environments, and simultaneously outputs Gaussian splats for visual exploration and collision meshes for physics engine operation. But Marble is only the first chapter of a long arc. This story is being written across the entire field as the boundaries between rendering, simulation, and planning begin to dissolve.
Boundaries are Dissolving, and What Comes Next
The most important trend in the field right now is that the three categories are beginning to merge. The underlying consensus is: the knowledge needed to render a world, simulate it, and act within it is largely the same. Following the previous example, a model that truly understands how a cup sits on a table (its geometry, material properties, response to force, etc.) should be able to render that cup from any angle, simulate what happens if the cup is pushed, and plan a hand to pick it up. The three categories are three projections of the same underlying understanding.
For instance, a small but growing body of work from various robotics labs has recently shown a possibility that is at least conceptually viable: a pre-trained video renderer can serve as a backbone network for joint world prediction and action prediction, allowing a single model to simultaneously imagine "what will happen" and "what to do," thus bridging renderer and planner. World Labs' Marble already outputs both Gaussian splats and collision meshes from a single model, dissolving the boundary between renderer and simulator. At every level, there is a shift from passive output to interactive systems: renderers become responsive to action conditions, simulators generate worlds that are more controllable and editable, and planners begin to engage in deliberative reasoning rather than just reacting.
The logical endpoint is a unified world model: a foundation model capable of rendering photorealistic views, generating physically accurate structures, planning action sequences, and switching between different output modalities based on downstream user needs. We will still face a series of daunting challenges. The data landscape is highly uneven, with renderers sitting atop vast amounts of internet video, while simulators and planners suffer from severe shortages of 3D assets and robot demonstration data. Optimization for visual aesthetics may sacrifice the precision needed for robotics or high-fidelity simulation. Reconciling these tensions within a single architecture is the central open problem in world model research today, and is what World Labs is committed to solving as Marble continues to evolve.
(Source: Substack)
But the general direction is already clear. From the late 1980s to today, the field has always bet on the same wager: if the world model is rich enough, everything an agent needs to see the world, construct it, and act within it will be inside. This bet is now driving a generation of research. What truly adds weight to it is the already ongoing fusion: the three lines of rendering, simulation, and planning, each already supporting industries worth tens of billions of dollars, initially independent research directions, are now converging. When boundaries disappear, the confluence of the three will redefine something larger: the relationship between machine intelligence and the physical world it inhabits, which is the long-term trajectory of spatial intelligence.
Language gave machines a way to talk about the world. World models are how machines will ultimately understand, imagine, reason about, and interact with it.
Reference: 1.https://drfeifei.substack.com/p/a-functional-taxonomy-of-world-models
