Robots Learn Operations by Watching Videos: Berkeley First to Bridge the Gap from Internet Videos to Real Dexterous Hand Deployment
UC Berkeley researchers introduce "Do as I Do," an end-to-end pipeline enabling robots to learn dexterous manipulation directly from everyday monocular RGB videos. The system overcomes key bottlenecks in scaling robot learning by first reconstructing 4D hand-object interactions from noisy web videos using a guided diffusion-based tracking method that outperforms existing approaches. It then robustly retargets these trajectories—which are often discontinuous or physically implausible—to a 22-DoF Sharpa Wave dexterous hand. The method incorporates a dynamics-aware optimizer with adaptive contact modeling and multi-stage refinement to handle trajectory noise, improving real-world retargeting success from 25% to 71%. Validated across 20 complex action categories (e.g., stirring, hammering, writing), the pipeline generated 500 executable trajectories and was successfully deployed at 50Hz on a dual-UR3e arm and dual-Sharpa Wave hand platform for 10 real-world tasks. The work establishes a crucial link, transforming the vast repository of human video data into actionable robot policies for anthropomorphic hands.
marsbit07/06 07:19
