Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learning

Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learning

3 Jul 2024 | Runyu Ding, Yuzhe Qin, Jiyue Zhu, Chengzhe Jia, Shiqi Yang, Ruihan Yang, Xiaojuan Qi, Xiaolong Wang
Bunny-VisionPro is a real-time bimanual dexterous teleoperation system that enables intuitive and safe human-robot interaction. It leverages a VR headset to capture hand movements and translate them into precise robotic commands, allowing operators to control a high-degree-of-freedom (DoF) bimanual robot. The system includes three key modules: hand motion retargeting, arm motion control, and haptic feedback. The hand motion retargeting module maps human finger movements to robotic hands, while the arm motion control module ensures safe and smooth operation by avoiding collisions and singularities. The haptic feedback module provides tactile sensations using low-cost Eccentric Rotating Mass (ERM) actuators, enhancing immersion and real-time performance. Bunny-VisionPro outperforms existing systems on a standard task suite, achieving higher success rates and reduced task completion times. It enables high-quality teleoperation demonstrations that improve downstream imitation learning performance, leading to better generalization. The system is particularly effective in multi-stage, long-horizon dexterous manipulation tasks, which have been challenging for previous work. Its ability to handle bimanual manipulations while prioritizing safety and real-time performance makes it a powerful tool for advancing dexterous manipulation and imitation learning. The system was evaluated on the Telekinesis benchmark, achieving 11% higher success rates and reducing task completion time by 45% compared to prior systems. Imitation learning policies trained on demonstrations collected by the system show a 20% improvement in generalization on novel poses and unseen objects. These results demonstrate the system's superior performance in executing complex bimanual manipulation tasks and its effectiveness in collecting high-quality data. The system also includes a haptic feedback device that provides tactile sensations to the operator, enhancing control accuracy and fostering a more immersive experience. User studies show that haptic feedback improves task success rates and reduces completion times, particularly in tasks requiring precise control. The system's haptic feedback device uses FSR sensors to measure finger pressure, processes tactile signals, and drives vibration motors to simulate tactile sensations. In terms of imitation learning, the system's high-quality demonstrations improve generalization performance, with Bunny-VisionPro outperforming AnyTeleop+ by 22% in success rates across three tasks. The system's ability to handle long-horizon tasks is also notable, with a 73% success rate for sub-tasks and 38% for entire tasks using only 30 demonstrations. The system's tactile data processing and policy input also show promise, with tactile data enhancing performance in certain tasks. Overall, Bunny-VisionPro represents a significant advancement in bimanual dexterous teleoperation and imitation learning, offering a safe, efficient, and immersive solution for human-robot interaction.Bunny-VisionPro is a real-time bimanual dexterous teleoperation system that enables intuitive and safe human-robot interaction. It leverages a VR headset to capture hand movements and translate them into precise robotic commands, allowing operators to control a high-degree-of-freedom (DoF) bimanual robot. The system includes three key modules: hand motion retargeting, arm motion control, and haptic feedback. The hand motion retargeting module maps human finger movements to robotic hands, while the arm motion control module ensures safe and smooth operation by avoiding collisions and singularities. The haptic feedback module provides tactile sensations using low-cost Eccentric Rotating Mass (ERM) actuators, enhancing immersion and real-time performance. Bunny-VisionPro outperforms existing systems on a standard task suite, achieving higher success rates and reduced task completion times. It enables high-quality teleoperation demonstrations that improve downstream imitation learning performance, leading to better generalization. The system is particularly effective in multi-stage, long-horizon dexterous manipulation tasks, which have been challenging for previous work. Its ability to handle bimanual manipulations while prioritizing safety and real-time performance makes it a powerful tool for advancing dexterous manipulation and imitation learning. The system was evaluated on the Telekinesis benchmark, achieving 11% higher success rates and reducing task completion time by 45% compared to prior systems. Imitation learning policies trained on demonstrations collected by the system show a 20% improvement in generalization on novel poses and unseen objects. These results demonstrate the system's superior performance in executing complex bimanual manipulation tasks and its effectiveness in collecting high-quality data. The system also includes a haptic feedback device that provides tactile sensations to the operator, enhancing control accuracy and fostering a more immersive experience. User studies show that haptic feedback improves task success rates and reduces completion times, particularly in tasks requiring precise control. The system's haptic feedback device uses FSR sensors to measure finger pressure, processes tactile signals, and drives vibration motors to simulate tactile sensations. In terms of imitation learning, the system's high-quality demonstrations improve generalization performance, with Bunny-VisionPro outperforming AnyTeleop+ by 22% in success rates across three tasks. The system's ability to handle long-horizon tasks is also notable, with a 73% success rate for sub-tasks and 38% for entire tasks using only 30 demonstrations. The system's tactile data processing and policy input also show promise, with tactile data enhancing performance in certain tasks. Overall, Bunny-VisionPro represents a significant advancement in bimanual dexterous teleoperation and imitation learning, offering a safe, efficient, and immersive solution for human-robot interaction.
Reach us at info@study.space