Embodied Intelligence 'Gaokao' is Insanely Hard, Humans Score 100, Best Model Only 12.8
Embodied AI Faces a Daunting "Everest": New Benchmark Reveals Huge Gap Between Models and Humans
A comprehensive new benchmark for robotic manipulation, RoboDojo, has been released, painting a stark picture of the current state of embodied AI. It serves as a unified evaluation platform covering both simulation and real-world robot tasks.
The benchmark assesses five core capabilities: Generalization (adapting to new scenes/objects), Memory, Precision manipulation, Long-Horizon multi-step tasks, and Open semantic understanding. It includes 42 simulation tasks and 18 standardized real-world tasks across three dual-arm robot platforms.
The results are sobering. In simulation, the best-performing generalist robot policy achieved an average success rate of only 8.80%. Performance in the real world was slightly higher but still low, with the top model succeeding 12.8% of the time on average. In stark contrast, human experts scored 76.03% in simulation and 100% in real-world tests.
The benchmark highlights significant, uneven gaps in current models' abilities. While some excel in specific areas like visual recognition or simple actions, they struggle with reliability, especially in long-horizon tasks where errors accumulate and in open-ended semantic instructions. The low scores, particularly in real-world deployment with physical uncertainties like camera noise and contact dynamics, underscore that today's models are far from being robust, general-purpose operational robots.
RoboDojo is more than just a ranking; it's an infrastructure designed for fair, reproducible comparison. Its companion system, XPolicyLab, standardizes the interface for different models to be evaluated. Maintained by an academic consortium without commercial ties, it aims to provide a community-wide "altitude meter" to track genuine progress toward reliable and generalizable robot manipulation.
marsbit07/08 11:49
