2024-07-04 | Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Li Fei-Fei, C. Karen Liu
DEXCAP is a portable and scalable hand motion capture system designed to facilitate the collection of high-quality human hand motion data for training dexterous robot manipulation skills. It leverages electromagnetic field (EMF) gloves and SLAM-based wrist pose tracking to provide precise, occlusion-resistant 3D hand motion data. DEXIL, a novel imitation learning algorithm, uses this data to train robot policies for complex manipulation tasks. The system includes an optional human-in-the-loop correction mechanism to refine policy performance. Extensive evaluations across six challenging tasks demonstrate the system's effectiveness in learning from in-the-wild data, showcasing its potential for advancing robotic dexterity. The hardware and software details, as well as experimental results, are provided to support the system's design and performance.DEXCAP is a portable and scalable hand motion capture system designed to facilitate the collection of high-quality human hand motion data for training dexterous robot manipulation skills. It leverages electromagnetic field (EMF) gloves and SLAM-based wrist pose tracking to provide precise, occlusion-resistant 3D hand motion data. DEXIL, a novel imitation learning algorithm, uses this data to train robot policies for complex manipulation tasks. The system includes an optional human-in-the-loop correction mechanism to refine policy performance. Extensive evaluations across six challenging tasks demonstrate the system's effectiveness in learning from in-the-wild data, showcasing its potential for advancing robotic dexterity. The hardware and software details, as well as experimental results, are provided to support the system's design and performance.