On June 3, 2026, the World Labs team, in collaboration with Stanford University Professor Fei-Fei Li, released a conceptual analysis article with an almost unadorned title: "A Functional Taxonomy of World Models." The opening sentence punctured an industry unspoken agreement: "'World model' is one of the most important and most abused terms in the field of artificial intelligence today."
The context for this statement is familiar to anyone who has followed the AI industry.
In February 2024, OpenAI released the video generation model Sora, whose technical report prominently featured the title "Video generation models as world simulators." NVIDIA's Robotics Director, Jim Fan, commented on LinkedIn at the time, a statement later frequently quoted: Sora is essentially "a world model that only allows 'no-op' as the single allowed action." On the other hand, according to public reports, Tesla's AI team has repeatedly referred to the predictive component within its Full Self-Driving system as a "world model" or "world simulator" in public forums. Game engines, 3D generation tools, embodied intelligence models—various products and technologies are stuffed into the same basket, labeled with the same tag.
A video generator, an autonomous driving prediction network, a robot control model, a physics engine—what do they have in common? Almost nothing. Yet, they are all called "world models."
This conceptual confusion, persisting for over two years, has finally prompted a systematic attempt at clarification. Fei-Fei Li's team did not release a new model, announce a new benchmark, or demonstrate any product functionality. They did something more fundamental: returning to the theoretical source of partially observable Markov decision processes, they reduced all systems currently called "world models" on the market to three different functional projections of the same cognitive loop.
The three projections are: Renderer, Simulator, and Planner. Under World Labs' classification framework, Sora and similar video generation models belong to the Renderer category.
Why Can One Term Contain So Many Contradictory Meanings
To understand the root of this confusion, one must ask a more fundamental question: when a company says "we are building a world model," what exactly are they saying?
For OpenAI, Sora's goal is to "understand and depict the physical world in video." According to the technical report, by learning statistical patterns from vast amounts of video data, Sora can generate scenes that conform to visual common sense: a cup shatters when dropped, a paper airplane flies when released, a person's legs alternate when walking. These scenes appear to "understand physics."
For Tesla, the "world model" is the neural network within the FSD system that predicts the motion trajectories of road participants in the coming seconds. It needs to output precise 3D positions, velocities, and orientations for the path-planning module to compute safe driving decisions. This model does not need to output pixels; it outputs vectors and probability distributions.
For robotics companies, the "world model" is the internal simulation mechanism that allows a robotic arm to predict "if I push this cup 5 centimeters to the left, will it tip over?" It needs to understand object properties, contact mechanics, and stability, outputting feasibility assessments of actions.
The goals of the three types of companies are entirely different. Video generation companies care about pixel fidelity; autonomous driving companies care about the accuracy of physical state prediction; robotics companies care about the inferability of action consequences. They are all working on "world models," but they are fundamentally not doing the same thing.
World Labs gets to the heart of the matter in the article: the reason these systems are all given the same name is that they each embody a certain aspect of "understanding the world." However, they each only complete one part of the full cognitive loop, yet are packaged by marketing language, media coverage, and capital narratives as complete world models.
Another driver of conceptual confusion is the inherent tension of the term itself. "World model" carries grand narrative connotations, sounding more imaginative than "video generation model" or "video prediction model," and better able to support high valuations and funding stories. When technical capabilities cannot match public expectations, it becomes inevitable for concepts to devolve into promotional tools.
Going Back to the 1960s: What Should a Complete 'World Model' Be
World Labs' classification framework is built upon a seemingly ancient theoretical foundation: partially observable Markov decision processes.
This framework describes the complete loop of an intelligent agent interacting with its environment. The agent exists in some environmental state, executes an action, the action changes the environmental state, the agent receives a partial observation through sensors, the observation triggers an update of its internal state, and the updated cognition drives the next action. The cycle repeats.
Within this framework, the complete function of a "world model" should include three steps: generating observations from states (pixels, point clouds seen by human eyes or collected by sensors), inferring the next state from actions and the current state (predicting physical changes), and generating actions from observations and goals (decision planning).
Language models learn statistical patterns of text sequences, while world models learn statistical properties of space and time. How light reflects off different material surfaces, how objects move under gravity, how energy transfers after rigid body collisions—these are the patterns world models aim to capture.
World Labs points out in the article that all systems currently called "world models" on the market are essentially just projections of one functional component of the aforementioned complete loop. Some systems only perform rendering ("from state to observation"), some only perform state inference ("from action and current state to next state"), and some only perform planning ("from observation to action"). They each capture an arc of the loop but are labeled as representing the full circle.
The value of this analytical framework lies in providing a comparative coordinate system that transcends marketing rhetoric. Regardless of how a company packages its product, placing it back into the POMDP loop—examining what it inputs, what it outputs, and which component it lacks—exposes the true boundaries of its capabilities.
Renderer, Simulator, Planner: The Capability Boundaries of Three Projections
In World Labs' taxonomy, the first category is defined as "Renderer." Its core objective is to generate high-fidelity pixel outputs for human visual perception. The input is a representation of some environmental state (could be text description, 3D scene parameters, or implicit encoding), and the output is a sequence of continuous frames.
The Renderer optimizes for visual realism, not physical precision. The World Labs article explicitly states that a building generated by a Renderer might look "rickety" because it does not actually solve structural mechanics equations; the splashing liquid it generates might look realistic, but the liquid volume, flow rate, and impact force might not correspond to real physical quantities at all. Therefore, such models cannot be used for architectural design, robot training, or tasks requiring physically accurate simulation.
Google's Genie 3, various text-to-video models, and almost all AI video generation tools fall into this category. Sora, of course, is among them.
The second category is "Simulator." Its core objective is not to generate visuals for human consumption but to generate precise states usable for subsequent computation. The input is the current environmental state and external forces (or actions), and the output is the next state that faithfully adheres to real-world physical and geometric laws. The state output by a Simulator can be used for stress analysis, energy consumption calculations, collision detection, or as input for a Renderer to generate visualizations. However, its core value lies in the computability of the state itself.
NVIDIA Omniverse is a typical example of such a system. It is not an AI-native model but a digital twin platform integrating traditional physics engines with AI-accelerated computation. World Labs comments in the article that Simulators are bridges connecting rendering and planning, but the scarcity of high-quality 3D physical annotation data is a major bottleneck. According to World Labs' estimates in the article, the data used to train such models is orders of magnitude less than the video data available on the internet.
The third category is "Planner." Its input is observation data (camera images, LiDAR point clouds, tactile sensor readings, etc.) and target instructions, and its output is what action to execute next. VLA (Vision-Language-Action) models and World Action Models belong to this category.
The differences among the three categories are not minor divergences in technical approach but fundamental functional distinctions. Renderers output pixels for humans to see, Simulators output states for machines to calculate, Planners output actions for actuators to perform. A system can possess multiple capabilities, but when most systems called "world models" essentially only perform rendering, equating "rendering" with "understanding the world" constitutes a severe cognitive mismatch.
A Debate Lasting Two Years: Is Sora Actually a World Model
In February 2024, OpenAI released Sora, with its technical report title directly stating "Video generation models as world simulators." This wording immediately sparked intense debate in academia and the developer community.
Supporters argued that Sora-generated videos demonstrated 3D spatial consistency, object permanence, and an intuitive understanding of physical interactions. A bitten hamburger showing teeth marks, a dog running in snow kicking up flakes—such details seemed to indicate the model had learned some physical laws.
The core argument of opponents stemmed from the classical definition of world models in reinforcement learning: a world model must be capable of state transition prediction based on actions. That is, given the current state and an action input, the model should output the state following that action. Sora cannot do this. Users cannot tell Sora "push that cup from the left" and then observe whether it will tip over, in which direction, and where the pieces might fly.
Jim Fan's comment precisely captured this contradiction: "Sora is essentially a world model, just one that only allows 'no-op' as the single allowed action." This means Sora is indeed predicting how the environment changes over time, but this change process is not subject to any external intervention; it can only unfold along the inherent causal chains present in the video data. It is not performing interactive inference but rather passively continuing observed sequences.
On the r/MachineLearning subreddit, many reinforcement learning researchers expressed sharper criticism: a system that cannot predict state transitions based on actions cannot be called a world model; it can only be called a video prediction model.
World Labs' classification framework provides a definitive answer to this debate. In the POMDP loop, action is the key input driving state transition. Systems lacking this input are merely projections of the "observation generation" component in the complete cognitive loop. Sora belongs to the Renderer category; it is not a complete world model, and certainly not a world simulator.
This does not mean Sora lacks value. Renderers solve a different problem: how to generate images that meet human visual expectations. This problem itself is extremely difficult and holds immense commercial value. The issue lies in packaging rendering capability as "understanding the world," which misleads technical decision-makers and investors, making them mistakenly believe these models already possess physical inference or embodied interaction capabilities.
The Industrial Value of Conceptual Clarification
Clarifying the definitional boundaries of "world model" is not mere academic semantics. It directly impacts technology selection, investment judgment, and public understanding of AI capability levels.
For a manufacturing company evaluating whether to use a certain "world model" for robot training, understanding whether the model is a Renderer, Simulator, or Planner is a prerequisite to avoiding costly trial-and-error worth millions of dollars. A model that can only generate video, no matter how realistic, cannot replace precise calculations of object forces, motion trajectories, and collision consequences.
For investment institutions, distinguishing between the three projections allows for more accurate identification of a project's position in the technology stack. A startup claiming to be a "world model" company, if its product is essentially a Renderer, competes with video generation companies, not digital twin platforms or robot control models. This directly determines how market size is estimated and which companies serve as benchmarks.
For academia, clear classification is a prerequisite for establishing comparable benchmarks. If the term "world model" continues to be diluted, researchers will struggle to define what constitutes an improvement versus a breakthrough, and peer review will be based on ambiguity.
World Labs also notes in the article that conceptual clarification is not meant to create opposition. The future direction will involve the convergence of the three projections. A model that truly understands the physics of a cup should be able to simultaneously render its visual appearance, simulate its physical process when pushed over, and plan how a robotic hand can stably grasp it. However, until technology reaches that stage, recognizing respective boundaries is more meaningful than envisioning convergence.
According to World Labs' estimate in the article, Simulators and digital twin technologies, represented by NVIDIA Omniverse, target a potential market exceeding trillions of dollars in sectors like factories, warehouses, and supply chains. This figure comes from the vendors' own assessments; when the market will actually reach this scale depends on whether Simulators can break through the bottleneck of scarce high-quality 3D physical data.
For the AI industry at its current stage, perhaps the most important takeaway is simple: being able to generate realistic videos does not equate to understanding the physical world; being called a world model does not mean it is actually simulating the world. Penetrating marketing language and examining what a system truly inputs, outputs, and lacks within the POMDP loop is the most honest way to judge the boundaries of its technical capabilities.
