22 Mar 2024 | Jianfeng Gao, Xiaoshu Jin, Franziska Krebs, Noémie Jaquier, and Tamim Asfour
The paper introduces Bi-KVIL, a novel keypoints-based approach for visual imitation learning of bimanual manipulation tasks. Bi-KVIL extends the previous K-VIL framework to handle complex bimanual coordination strategies and object relationships. The key contributions of Bi-KVIL include:
1. **Hybrid Master-Slave Relationship (HMSR)**: Bi-KVIL extracts HMSR, which captures the relationships between objects and hands, enabling bimanual coordination strategies and sub-symbolic task representations.
2. **Bimanual Coordination Strategies**: The approach identifies different types of bimanual coordination, such as uncoordinated unimanual, loosely-coupled, and tightly-coupled symmetric coordination.
3. **Task Reproduction**: Bi-KVIL can reproduce tasks with category-level generalization in cluttered scenes using a small number of human demonstration videos (5-10).
The paper evaluates Bi-KVIL on various real-world tasks, demonstrating its ability to extract consistent HMSR, capture fine-grained styles, and reproduce tasks with out-of-distribution objects. The approach is evaluated in eight tasks, including pouring water, placing objects, and cleaning tables, showing its effectiveness in handling different styles and variations of demonstrations. Bi-KVIL's HMSR is structured similarly across task styles but differs in sub-symbolic constraints, capturing the fine-grained motion styles. The paper also discusses limitations and future work, highlighting the need for handling object occlusion and improving dual-arm synchronization.The paper introduces Bi-KVIL, a novel keypoints-based approach for visual imitation learning of bimanual manipulation tasks. Bi-KVIL extends the previous K-VIL framework to handle complex bimanual coordination strategies and object relationships. The key contributions of Bi-KVIL include:
1. **Hybrid Master-Slave Relationship (HMSR)**: Bi-KVIL extracts HMSR, which captures the relationships between objects and hands, enabling bimanual coordination strategies and sub-symbolic task representations.
2. **Bimanual Coordination Strategies**: The approach identifies different types of bimanual coordination, such as uncoordinated unimanual, loosely-coupled, and tightly-coupled symmetric coordination.
3. **Task Reproduction**: Bi-KVIL can reproduce tasks with category-level generalization in cluttered scenes using a small number of human demonstration videos (5-10).
The paper evaluates Bi-KVIL on various real-world tasks, demonstrating its ability to extract consistent HMSR, capture fine-grained styles, and reproduce tasks with out-of-distribution objects. The approach is evaluated in eight tasks, including pouring water, placing objects, and cleaning tables, showing its effectiveness in handling different styles and variations of demonstrations. Bi-KVIL's HMSR is structured similarly across task styles but differs in sub-symbolic constraints, capturing the fine-grained motion styles. The paper also discusses limitations and future work, highlighting the need for handling object occlusion and improving dual-arm synchronization.