1 Mar 2024 | Jun Wang, Yuzhe Qin, Kaiming Kuang, Yigit Korkmaz, Akhilan Gurumoorthy, Hao Su, Xiaolong Wang
CyberDemo is a novel pipeline for robotic imitation learning that leverages simulated human demonstrations to enhance real-world dexterous manipulation tasks. The approach involves collecting human demonstrations in a simulated environment, followed by extensive data augmentation within the simulator. The augmented data is then used to train an imitation learning model, which is fine-tuned on a few real-world demonstrations. This method outperforms traditional in-domain real-world demonstrations when transferred to the real world, handling diverse physical and visual conditions. CyberDemo demonstrates superior success rates across various tasks and generalizes well to unseen objects, such as rotating novel tetra-valve and penta-valve despite initial demonstrations only involving tri-valves. The system uses a low-cost motion capture device for teleoperation and minimal human effort, making it affordable and convenient for data collection. The effectiveness of CyberDemo is validated through experiments, showing a 35% higher success rate for quasi-static pick and place tasks and a 20% higher success rate for non-quasi-static rotate tasks compared to pre-trained policies. The method also performs better in generalization tests, achieving a 42.5% success rate in rotating novel tetra-valve and penta-valve, even with significant light disturbances. The paper discusses related work, including data collection, sim2real transfer, and the importance of data augmentation and curriculum learning in improving policy performance.CyberDemo is a novel pipeline for robotic imitation learning that leverages simulated human demonstrations to enhance real-world dexterous manipulation tasks. The approach involves collecting human demonstrations in a simulated environment, followed by extensive data augmentation within the simulator. The augmented data is then used to train an imitation learning model, which is fine-tuned on a few real-world demonstrations. This method outperforms traditional in-domain real-world demonstrations when transferred to the real world, handling diverse physical and visual conditions. CyberDemo demonstrates superior success rates across various tasks and generalizes well to unseen objects, such as rotating novel tetra-valve and penta-valve despite initial demonstrations only involving tri-valves. The system uses a low-cost motion capture device for teleoperation and minimal human effort, making it affordable and convenient for data collection. The effectiveness of CyberDemo is validated through experiments, showing a 35% higher success rate for quasi-static pick and place tasks and a 20% higher success rate for non-quasi-static rotate tasks compared to pre-trained policies. The method also performs better in generalization tests, achieving a 42.5% success rate in rotating novel tetra-valve and penta-valve, even with significant light disturbances. The paper discusses related work, including data collection, sim2real transfer, and the importance of data augmentation and curriculum learning in improving policy performance.