【Summary】Using only monocular RGB video, 'Do as I Do' transforms everyday human operations into executable trajectories for the Sharpa Wave, completing the critical link from video to robot data for human-like dexterous manipulation.
Humans learn dexterous operations, often by starting with "seeing."
A child watches others beat eggs, pour water, hammer nails, and gradually learns these actions through imitation. Robots are different. Today, robot learning relies more on "doing," such as costly teleoperation, extensive simulation execution, or collecting real-world data in carefully arranged scenes.
In fact, data for robots to "see" already exists. Platforms like YouTube, egocentric datasets, and generated videos contain massive amounts of footage of human hand-object interaction. The real bottleneck is not the lack of data but whether the data conversion can be completed: how to turn these noisy monocular RGB videos into action trajectories that multi-fingered dexterous hands can execute?
The end-to-end pipeline proposed by the UC Berkeley team aims to solve this problem. The research team has established the first complete pipeline capable of generating real dexterous hand execution trajectories from web videos: first reconstructing the 4D hand-object interaction process from monocular RGB videos in real scenes, then retargeting these interaction trajectories to the 22-degree-of-freedom Sharpa Wave dexterous hand.
Paper link: https://arxiv.org/abs/2606.19333
Project link: https://do-as-i-do.com/
The entire pipeline generated 500 verified trajectories across 20 categories of manipulation actions and deployed 10 real tasks on a dual UR3e robotic arm + dual Sharpa Wave platform at 50Hz.
The Problem: "Seeing" ≠ "Knowing How"
To scale dexterous robot data, three structural challenges remain:
Stable reconstruction of hand-object interaction in monocular RGB video is still difficult.
Real-world videos often have issues like motion blur, occlusion, depth ambiguity, and variable object types. Tracking methods like FoundationPose can lose pose lock even under slight blur. Some joint reconstruction methods rely more on lab environments or can only handle pre-defined object categories.
Without stable 4D hand-object reconstruction, human videos are difficult to use in robot learning.
Noisy reference trajectories can cause motion retargeting to fail.
Previous dynamics-aware motion retargeting methods, such as SPIDER or RL-based tracking methods, typically assume clean MoCap ground truth data as input. But in reality, reference trajectories reconstructed from web videos may not be clean. They may have temporal discontinuities, misaligned contact relationships, or even contain physically impossible initial states.
These issues directly affect subsequent optimization. Experiments in the paper show that directly using sampling-based optimization methods on such noisy reference trajectories can have a failure rate of up to 75%.
Teleoperation itself is difficult to scale.
Teleoperation can provide real robot data but is costly. It relies on skilled operators, specialized equipment, and needs to be collected task by task. Relying solely on teleoperation, it's difficult to cover the rich operations in even one hour of human cooking videos, let alone the vast amount of human videos across the entire internet.
Therefore, 'Do as I Do' aims to answer the question: Using only monocular RGB video, without predefined grasping priors or limiting rigid object categories—can robots move from "seeing" to "doing"?
The Solution
The 'Do as I Do' pipeline consists of two stages:
Stage One: Using Guided Diffusion for Stable Object Tracking
SAM 3D can generate object meshes for a single frame. However, processing each frame independently can easily lead to drift in results and difficulty maintaining temporal consistency.
Therefore, 'Do as I Do' first selects an anchor frame and fixes the object shape in this frame. During the flow matching denoising process for subsequent frames, the system guides the pose sampling result of the current frame towards the pose of the previous frame, thus obtaining more continuous pose trajectories while maintaining consistent object shape. Simultaneously, the system adaptively adjusts the pose based on the object's rotational speed estimated from 2D point tracking. This helps avoid overly rigid tracking and reduces erroneous flips.
In a human comparative evaluation of 150 real-world videos, evaluators considered 'Do as I Do' tracking results better than FoundationPose in 67% of samples. In many samples, multiple evaluators gave consistent judgments.
Stage Two: Robust Motion Retargeting for Noisy Reference Trajectories
Building upon the sampling / MPPI optimization framework of SPIDER, 'Do as I Do' incorporates three additional designs to handle the noisy reference trajectories reconstructed from web videos:
Combining these improvements, 'Do as I Do' increased the motion retargeting success rate from 25% to 71% on noisy real-world reference trajectories.
Experimental Results
Reconstruction Capability Benchmark (SOTA)
Motion Retargeting Benchmark
Data Source for 500 Verified Trajectories
The method ultimately covers 20 categories of manipulation actions. These are not simple pick-and-place actions but more complex operations closer to daily human life, including placing, picking, wiping, smearing, squeezing, ironing, brushing, dusting, digging, erasing, pouring, writing, whipping, stirring, poking, compacting, drilling, hammering, cutting, and spreading sauce.
Real-World Deployment
These trajectories are not just in simulation. The research team selected 10 representative actions from them and deployed them on a dual UR3e robotic arm + dual Sharpa Wave dexterous hand platform, completing real-world execution at a 50Hz control frequency.
The deployed actions cover different object shapes and various grasping modes, including tripod writing grasp, power grasp, palmar grasp, and parallel extension grasp.
The Sharpa Wave has 22 degrees of freedom and a scale close to the human hand, making it more suitable as the target body for human hand motion transfer. Actions like whipping, stirring, and hammering require bimanual coordination, which is difficult to achieve with traditional parallel grippers. Wave's hand gesture switching frequency exceeding 4Hz and fingertip force of 50N can support the strength and speed requirements of these actions.
From reconstruction, simulation (MuJoCo Warp, 200Hz) to real-world deployment, the research team used Sharpa Wave as the target hand type for motion retargeting, transferring operation trajectories from human videos to this embodiment.
EgoScale also retargets human hand keypoints to this hand type, while CAIP performs evaluation and verification on the Dexmate Vega+ dual Wave platform. Because the target hand type is closer to the human hand, the morphological gap that the system needs to bridge when transferring from human actions to robot execution is smaller.
Screening Manual: Why 95% of Web Videos Cannot Be Used Directly
For teams hoping to utilize human video data at scale, including research teams in areas like EgoScale, 'Do as I Do' also provides a very practical reminder: more videos are not necessarily better; being able to filter out usable data is equally important.
The research team analyzed 2000 ten-second video clips from the 100DOH dataset (already filtered for hand-object interaction):
The result is direct: without preprocessing raw videos first and directly feeding web videos into robot learning, the truly usable data might be only about one-twentieth. Therefore, 'Do as I Do' also summarizes a set of data filtering points: check if the hand and object are always in frame, confirm if the action crosses a camera cut, exclude segments with excessive camera motion, and identify cases where SAM 3D might fail. For any team hoping to establish a pipeline from "human video to robot execution" on dexterous hands, this screening process will become an unavoidable fundamental step.
Conclusion: Human Videos Are Becoming Robot Data
For a long time, "Do as I Do" was more like an ideal in the field of Artificial Intelligence (AI): enabling robots to understand human demonstrations and transfer those actions to their own bodies. This UC Berkeley research is turning that ideal into reality: input a video link, and the system can reconstruct the hand-object interaction within it and transform it into an executable action trajectory for Sharpa Wave.
In a sense, the world's largest dataset of manipulation already exists—it lies hidden in the videos people shoot, upload, and share every day. What 'Do as I Do' aims to do is convert these videos into 22-DOF joint trajectories that dexterous hands can execute.
Watch, reconstruct, retarget, and then execute on a real robot.
Reference: https://do-as-i-do.com/
This article is from the WeChat public account "AI Era", edited by LRST
