DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation

DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation

4 Jul 2024 | Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Li Fei-Fei, C. Karen Liu
DEXCAP is a portable motion capture system designed to collect high-quality human hand motion data and 3D environmental observations for training dexterous robotic manipulation policies. It combines SLAM and electromagnetic field technology to track wrist and finger motions accurately, even in occluded environments. DEXCAP also includes an RGB-D LiDAR camera to capture 3D environmental data. DEXIL, a novel imitation learning algorithm, uses this data to train robotic policies that replicate human hand actions. DEXIL includes a human-in-the-loop correction mechanism to refine robot performance during policy execution. The system enables the collection of in-the-wild motion data, which is crucial for training dexterous robotic manipulation policies. DEXCAP's portability and robustness make it suitable for data collection in various real-world environments. DEXIL leverages point cloud-based imitation learning and inverse kinematics to adapt human motion data to robotic hands. The system has been evaluated on six challenging dexterous manipulation tasks, demonstrating superior performance and the effectiveness of learning from in-the-wild motion data. DEXCAP and DEXIL significantly advance robotic dexterity by enabling robots to perform complex tasks through direct learning from human motion data. The system's open-source hardware and code provide a foundation for future research in scalable dexterous manipulation data collection. Limitations include power constraints, size differences between human and robotic hands, and the lack of force sensing in current DEXCAP. Future improvements aim to address these challenges and enhance the system's capabilities.DEXCAP is a portable motion capture system designed to collect high-quality human hand motion data and 3D environmental observations for training dexterous robotic manipulation policies. It combines SLAM and electromagnetic field technology to track wrist and finger motions accurately, even in occluded environments. DEXCAP also includes an RGB-D LiDAR camera to capture 3D environmental data. DEXIL, a novel imitation learning algorithm, uses this data to train robotic policies that replicate human hand actions. DEXIL includes a human-in-the-loop correction mechanism to refine robot performance during policy execution. The system enables the collection of in-the-wild motion data, which is crucial for training dexterous robotic manipulation policies. DEXCAP's portability and robustness make it suitable for data collection in various real-world environments. DEXIL leverages point cloud-based imitation learning and inverse kinematics to adapt human motion data to robotic hands. The system has been evaluated on six challenging dexterous manipulation tasks, demonstrating superior performance and the effectiveness of learning from in-the-wild motion data. DEXCAP and DEXIL significantly advance robotic dexterity by enabling robots to perform complex tasks through direct learning from human motion data. The system's open-source hardware and code provide a foundation for future research in scalable dexterous manipulation data collection. Limitations include power constraints, size differences between human and robotic hands, and the lack of force sensing in current DEXCAP. Future improvements aim to address these challenges and enhance the system's capabilities.
Reach us at info@study.space