How high is the mountain of general-purpose robotics?
Over the past year, VLAs, robotics foundation models, and world models have taken the stage one after another.
Demos look increasingly smooth: stacking bowls, inserting tubes, tidying up, pouring water, organizing desktops—robots finally seem to understand human language, comprehend the world, and get to work.
But the question is: Which of these models is stronger? Where are they strong? Can they move from simulation to the real world? How far are we from truly general-purpose manipulative robots?
Now, a new "route map for climbing the mountain" has arrived.
The same team behind the RoboTwin series of benchmarks brings RoboDojo: a unified benchmark for robot manipulation evaluation across both simulation and the real world.
Website: https://robodojo-benchmark.com/arXiv: https://arxiv.org/abs/2607.04434Leaderboard: https://robodojo-benchmark.com/LeaderBoardBenchmark code: https://github.com/RoboDojo-Benchmark/RoboDojoXPolicyLab code: https://github.com/XPolicyLab/XPolicyLabCommunity: https://robodojo-benchmark.com/community
It's not just another benchmark; it's more like setting up a "Mount Everest" for embodied intelligence:
42 simulation tasks, 18 real-world robot tasks, 30 mainstream robot policies competing on the same stage, covering five core capabilities: generalization, memory, precision manipulation, long-horizon execution, and open semantic understanding.
The results are direct, and brutal:
The current strongest general-purpose robot policy achieves only an average success rate of 8.80% in simulation. In the real world, the best model's average success rate is only 12.8%.
What about human experts? 76.03% in simulation, and 100% in the real world.
Robotics foundation models appear to have begun their ascent of the embodied Everest, but the RoboDojo leaderboard shows: most are still at the base camp, acclimatizing to the altitude.
First, look at the task design: Why is this mountain so hard?
The difficulty of RoboDojo isn't in simply stacking up task numbers, but in breaking down robot manipulation capabilities into a set of "mountain climbing checkpoints" closer to the real world.
In the simulation environment, RoboDojo designs 42 tasks around five core capabilities:
Generalization: Can the model adapt to new backgrounds, new lighting, new objects, and complex, cluttered scenes?
Memory: Can the model remember previously seen information and use it in subsequent actions?
Precision: Can the model perform high-precision operations like insertion, alignment, and precise contact?
Long-Horizon: Can the model complete multi-step tasks with strong dependencies where errors accumulate?
Open: Can the model understand unseen open-ended semantic instructions and translate language goals into actions?
These aren't just simple pick-and-place variants.
For example, in generalization tasks, desktop clutter can be randomized up to 25 objects, with variations in background, lighting, object appearance, and layout.
In memory tasks, the robot needs to remember an object that appeared and then disappeared on a conveyor belt, and later select the matching target from subsequent candidate objects.
In precision manipulation tasks, the robot must perform high-tolerance actions like tube insertion, alignment, and peg-in-hole, where being slightly off leads to failure.
Long-horizon tasks are closer to real household chores: the robot isn't performing a single action but must complete multiple sub-steps consecutively. Pick up, move, hand over, align, place—each step can introduce error, and errors accumulate to the end.
But RoboDojo doesn't stop at simulation.
What truly makes this "embodied Everest" high is that it also moves evaluation to real robots.
RoboDojo designs 18 real-world tasks, covering three dual-arm robot platforms: ARX X5, Piper, and Piper X, with 6 tasks each.
These tasks aren't one-to-one copies of simulation tasks but are specifically designed to examine robots' deployment capabilities in the real physical world.
For instance, ARX X5 has tasks like covering blocks, making bread, preparing food, filling and emptying fruit containers, safe box storage, and tube insertion. Piper has tasks like stacking and covering blocks, filling a pen holder, placing objects in a basket, plugging in a charger, stacking bowls, and uprighting a bottle. Piper X includes tasks like object classification, disassembling Lego, hanging cups, packing items into a backpack, sweeping blocks, and capping pens.
These tasks sound very mundane, but they aren't simple for robots.
Because every step in the real world carries physical uncertainty: objects may slip, grippers may not grasp firmly, robotic arms may have slight delays, cameras may have noise, and contact may push the target off slightly.
More importantly, RoboDojo-RealEval standardizes real-robot evaluation: unified hardware configuration, workspace layout, lighting conditions, scene reset procedures, evaluation protocols, and deployment interfaces.
Before each test, evaluators reset the scene according to a preset layout; each trial is also blind-scored by three reviewers, considering both final success and completion of intermediate steps.
In other words, the real-robot part of RoboDojo isn't "filming a few demo videos" but turning real robot manipulation into a set of reproducible, comparable, remotely accessible standardized exams.
So, RoboDojo doesn't just ask robots "can you solve the problems?" in simulation; it also asks in the real world: Is it still stable when you switch to a different robot? Will it falter with real contact? Can it correct if an object is slightly off? Can it recover if it makes a mistake halfway through a task? Once you leave the simulation training ground, can you continue climbing the mountain?
This is the true meaning of the "embodied Everest": it's not about reaching the summit in a single capability, but about not dropping the ball on either the simulation diagnostic path or the real-world deployment path.
The leaderboard is out, laying the gaps bare
The core of RoboDojo is its public leaderboard.
This is also where it differs from many "our model, our test" evaluations:
RoboDojo is initiated and maintained by a consortium of purely academic institutions, with no commercial model stakeholder interests behind it. Leaderboard governance is handled by the public benefit organization AI MMLab Club Foundation.
In other words, this "embodied Everest" isn't an observation deck built by a single company for itself, but a public climbing route open to the entire community.
In the simulation leaderboard, the team integrated and evaluated 30 representative robot manipulation policies, including Hy-Embodied-0.5-VLA, Spatial Forcing, π0.5, X-VLA, GR00T-N1.7, π0, OpenVLA-OFT, and more.
Topping the leaderboard is Hy-Embodied-0.5-VLA, with an average score of 13.07 and an average success rate of 8.80%.
Closely following are models like Spatial Forcing, π0.5, X-VLA, but overall performance remains in a very low range.
Even the leading models aren't truly "all-rounders" across the five capability dimensions.
Some models are stronger in generalization, some are steadier in precision manipulation, some can advance a few more steps in long-horizon tasks. But once you look at the full leaderboard, the shortcomings become very apparent.
A key takeaway from RoboDojo is: Today's robot models aren't incapable of moving; they're not stable enough. They aren't completely unable to perform tasks; they struggle to reliably complete tasks.
Many policies can complete some steps, but final success rates are low.
For example, in long-horizon tasks, a robot might have already picked up the object and moved it near the target, only to fail in the final alignment, insertion, placement, or recovery stages.
This is also the biggest difference between embodied intelligence and pure language or vision tasks: In the physical world, being off by a little means failure.
The real-world leaderboard is even more sobering
If simulation is the "training ground," real robots are the "Everest expedition site."
In the real-world leaderboard, the best-performing model is π0.5, with an overall success rate of 12.8% and an average score of 22.9.
The top tier includes InternVLA-A1, GalaxeaVLA, Xiaomi-Robotics-0, X-VLA, etc., but overall success rates still range from single digits to just over ten percent.
This highlights a crucial issue: Being relatively higher in simulation doesn't guarantee stability in the real world.
Real robots introduce additional difficulties: camera noise, calibration errors, arm delays, contact instability, motion jitter, safety boundaries, tiny deviations in object initial positions. These things are often invisible in demo videos but become apparent in standardized evaluations.
This is also the significance of RoboDojo: It doesn't just ask "did the robot succeed?" It asks:
Can this policy pass comprehensive examination in simulation while also confronting challenges head-on in the real world?
Why this is the "embodied Everest"
Looking at the results, RoboDojo reveals a realistic assessment: The current capability growth of robot foundation models is uneven.
Some models can better identify targets, some can execute actions more smoothly, some can advance more steps in long-horizon tasks.
But a truly general-purpose robot cannot be strong in just one capability dimension.
It needs to both see and understand, and also remember; it needs to plan correctly and also be precise with its hands; it needs to handle familiar tasks and also understand open-ended instructions; it needs to run in simulation and also execute stably on a real robotic arm.
The RoboDojo experimental results show that today's models still have obvious shortcomings across these dimensions.
The most typical is the Open task. Even the strongest model only achieves about a 1.67% success rate on open semantic tasks.
This means current robot foundation models are still a significant distance away from truly "understanding human language and working reliably."
They can mimic familiar tasks, but the chain linking semantic understanding, visual localization, skill selection, and action execution remains fragile when faced with new goals, new semantics, and new combinations.
This is precisely the difficulty of the embodied Everest: It's not about reaching the summit in a single point capability, but about all capabilities holding up without fail.
Not just an evaluation, but also an infrastructure suite
RoboDojo has two other important components.
One is heterogeneous parallel simulation.
Traditional parallel simulation often replicates the same scene, only changing initial positions; RoboDojo supports running different tasks, different objects, and different layouts simultaneously, greatly improving evaluation efficiency.
The other is XPolicyLab.
It's essentially the "unified access layer" behind RoboDojo, designed to solve a very practical problem in robot policy evaluation: Different models often have different data formats, preprocessing pipelines, training scripts, action representations, and deployment environments. Achieving fair comparison on the same leaderboard carries very high engineering costs.
XPolicyLab standardizes these external processes.
It provides unified data conversion, training templates, deployment pipelines, and evaluation scripts, while preserving the original model architectures and implementations of each policy.
This way, different robot policies only need to connect to a unified observation-action interface to run on RoboDojo's simulation environment and the RoboDojo-RealEval real-robot platform.
In this paper, the team has already integrated 30 representative robot manipulation models via XPolicyLab.
For researchers, this means models can be "integrated once, evaluated in many places": first quickly iterate and diagnose capability gaps in simulation, then deploy to real robots for standardized testing.
Therefore, RoboDojo isn't just a static benchmark in a paper; it's a continuously updatable arena for embodied intelligence.
Models can keep climbing the leaderboard, tasks can be continuously expanded, and real-robot evaluation can be accessed remotely.
This is important for the field of robotics foundation models.
Because on the path to general-purpose manipulative robots, we need not only bigger models and cooler demos, but also a reliable "altimeter" to repeatedly measure progress.
Embodied intelligence finally has a higher mountain
In the past, the robotics field was often driven by demos.
A model completing a few beautiful tasks easily creates the illusion that "general-purpose robots are almost here."
But RoboDojo offers a more sober conclusion: Current models are indeed improving, but they are still far from reliable, generalizable, deployable general-purpose robot manipulation.
This isn't bad news.
On the contrary, RoboDojo clarifies the problems: Who can generalize, who forgets, whose actions are shaky, who can only do half the task, who falls behind in the real world, who can climb higher on the leaderboard.
Embodied intelligence is finally not just about competing with promotional videos, but about competing with real scores on standardized tracks.
This "embodied Everest" has been erected. Next, we'll see who reaches the summit first.
Project leads introduction
Tianhang Chen, Ph.D. student at HKU MMLab, advised by Prof. Ping Luo.
Published over ten papers at top conferences like ICML, CVPR, ICLR, RSS, and received multiple Best Paper awards at conference workshops, as well as championships and runner-up positions in multiple top conference academic competitions.
Selected as AI25 (Top 25 AI Innovators Under 25) by Sequoia China and MIT Technology Review China, recipient of Shenzhen University Special Award (highest student honor), CCF Outstanding College Student (99 nationwide).
First author of RoboTwin 2.0, founder of the leading embodied open-source community Lumina. Open-source projects have accumulated nearly 20,000 GitHub stars.
Yue Chen, Master's student at Peking University, main research interests in 3D visual representation and robot simulation.
Published over 10 high-level papers in CCF A and CAAI A categories, with multiple works presented as Oral or Spotlight and receiving Best Paper awards at international conference workshops like CVPR and IROS. Recipient of National Scholarship and Peking University Merit Student honors.
Future expansions
The RoboDojo team will continue to release benchmarks for dexterous manipulation, mobile manipulation, tactile manipulation, and full-body humanoid manipulation. Stay tuned.
*This article is authorized for publication by QbitAI. The views expressed are solely those of the original author.
This article comes from the WeChat public account "QbitAI," author: Yunzhong
