Adaptive Mobile Manipulation for Articulated Objects In the Open World

Adaptive Mobile Manipulation for Articulated Objects In the Open World

28 Jan 2024 | Haoyu Xiong, Russell Mendonca, Kenneth Shaw, Deepak Pathak
This paper introduces an Open-World Mobile Manipulation System, a full-stack approach for operating articulated objects like doors, cabinets, drawers, and refrigerators in open-ended, unstructured environments. The system uses an adaptive learning framework that initially learns from a small set of data via behavior cloning, followed by learning from online practice on novel objects outside the training distribution. A low-cost mobile manipulation hardware platform is developed, capable of safe and autonomous online adaptation in unstructured environments at a cost of around $25,000. The system was tested on 20 articulated objects across 4 buildings on the CMU campus, achieving a success rate increase from 50% to 95% through online adaptation. The system uses a structured action space with parametric primitives and pre-trained policies via imitation learning. Adaptive learning allows the robot to continuously learn from self-practice data via online reinforcement learning. The hardware platform is designed to be versatile, agile, affordable, and easy to build with off-the-shelf components. The system was evaluated on 8 novel objects, demonstrating its ability to adapt to new, unseen articulated objects. The system also uses rewards from large vision-language models (VLMs) for autonomous learning. The results show that the system can significantly improve performance on unseen objects through online adaptation, outperforming imitation learning and other baselines. The system is able to operate in open-world environments with high success rates, demonstrating the effectiveness of the proposed approach.This paper introduces an Open-World Mobile Manipulation System, a full-stack approach for operating articulated objects like doors, cabinets, drawers, and refrigerators in open-ended, unstructured environments. The system uses an adaptive learning framework that initially learns from a small set of data via behavior cloning, followed by learning from online practice on novel objects outside the training distribution. A low-cost mobile manipulation hardware platform is developed, capable of safe and autonomous online adaptation in unstructured environments at a cost of around $25,000. The system was tested on 20 articulated objects across 4 buildings on the CMU campus, achieving a success rate increase from 50% to 95% through online adaptation. The system uses a structured action space with parametric primitives and pre-trained policies via imitation learning. Adaptive learning allows the robot to continuously learn from self-practice data via online reinforcement learning. The hardware platform is designed to be versatile, agile, affordable, and easy to build with off-the-shelf components. The system was evaluated on 8 novel objects, demonstrating its ability to adapt to new, unseen articulated objects. The system also uses rewards from large vision-language models (VLMs) for autonomous learning. The results show that the system can significantly improve performance on unseen objects through online adaptation, outperforming imitation learning and other baselines. The system is able to operate in open-world environments with high success rates, demonstrating the effectiveness of the proposed approach.
Reach us at info@study.space
[slides] Adaptive Mobile Manipulation for Articulated Objects In the Open World | StudySpace