UMI on Legs: Making Manipulation Policies Mobile with Manipulation-Centric Whole-body Controllers

UMI on Legs: Making Manipulation Policies Mobile with Manipulation-Centric Whole-body Controllers

14 Jul 2024 | Huy Ha*,1,2, Yihuai Gao*,2, Zipeng Fu2, Jie Tan3, and Shuran Song1,2
This paper introduces UMI-on-Legs, a framework that combines real-world and simulation data for quadruped manipulation systems. The framework uses a hand-held gripper (UMI) for real-world demonstrations and a whole-body controller trained in simulation for tracking end-effector trajectories. The interface between these policies is end-effector trajectories in the task frame, enabling cross-embodiment generalization. The system achieves over 70% success rate on prehensile, non-prehensile, and dynamic manipulation tasks. The framework allows the deployment of pre-trained manipulation policies on a quadruped system, providing a scalable path towards learning expressive manipulation skills on dynamic robot embodiments. The paper also discusses related work, methodology, experiments, and limitations of the approach.This paper introduces UMI-on-Legs, a framework that combines real-world and simulation data for quadruped manipulation systems. The framework uses a hand-held gripper (UMI) for real-world demonstrations and a whole-body controller trained in simulation for tracking end-effector trajectories. The interface between these policies is end-effector trajectories in the task frame, enabling cross-embodiment generalization. The system achieves over 70% success rate on prehensile, non-prehensile, and dynamic manipulation tasks. The framework allows the deployment of pre-trained manipulation policies on a quadruped system, providing a scalable path towards learning expressive manipulation skills on dynamic robot embodiments. The paper also discusses related work, methodology, experiments, and limitations of the approach.
Reach us at info@study.space
[slides] UMI on Legs%3A Making Manipulation Policies Mobile with Manipulation-Centric Whole-body Controllers | StudySpace