Mirage: Cross-Embodiment Zero-Shot Policy Transfer with Cross-Painting

Mirage: Cross-Embodiment Zero-Shot Policy Transfer with Cross-Painting

16 Jun 2024 | Lawrence Yunliang Chen, Kush Hari, Karthik Dharmarajan, Chenfeng Xu, Quan Vuong, Ken Goldberg
Mirage is a novel method for zero-shot policy transfer across different robot embodiments. It enables the transfer of policies trained on one robot to an unseen target robot without requiring any data from the target robot. Mirage addresses the challenges of visual and control differences between robots by using "cross-painting" to create an illusion that the source robot is performing the task, and by using a forward dynamics model to bridge the control gap. The method works by masking out the target robot in the image and inpainting the source robot at the same end-effector pose, using robot URDFs and a renderer. This creates a "mirage" for the policy, making it appear as if the source robot is performing the task. Mirage can be applied to both first-person and third-person camera views and policies that take in both states and images as inputs or only images as inputs. Mirage has been tested on 9 manipulation tasks in both simulation and real-world settings across 6 different robot and gripper setups. The results show that Mirage can successfully transfer policies between different robot arms and grippers with only minimal performance degradation on a variety of manipulation tasks such as picking, stacking, and assembly, significantly outperforming a generalist policy. Mirage is effective at bridging the visual gap between robots by using cross-painting and the control gap by using a forward dynamics model. It does not require any demonstration data on the target robot or paired images or trajectories between the source and unseen target robots. The method is robust to changes in the robot's coordinate frames and camera parameters, and it can handle different control gains by using a high-gain or blocking controller on the target robot. The key contributions of Mirage include a systematic simulation study analyzing the challenges and potential for policy transfer between grippers and arms, a novel zero-shot cross-embodiment policy transfer method that uses cross-painting to bridge the visual gap and forward dynamics to bridge the control gap, and physical experiments with Franka and UR5 demonstrating that Mirage successfully transfers between robots and grippers on 4 manipulation tasks, suffering only minimal performance degradation from the source policy and significantly outperforming a state-of-the-art generalist model.Mirage is a novel method for zero-shot policy transfer across different robot embodiments. It enables the transfer of policies trained on one robot to an unseen target robot without requiring any data from the target robot. Mirage addresses the challenges of visual and control differences between robots by using "cross-painting" to create an illusion that the source robot is performing the task, and by using a forward dynamics model to bridge the control gap. The method works by masking out the target robot in the image and inpainting the source robot at the same end-effector pose, using robot URDFs and a renderer. This creates a "mirage" for the policy, making it appear as if the source robot is performing the task. Mirage can be applied to both first-person and third-person camera views and policies that take in both states and images as inputs or only images as inputs. Mirage has been tested on 9 manipulation tasks in both simulation and real-world settings across 6 different robot and gripper setups. The results show that Mirage can successfully transfer policies between different robot arms and grippers with only minimal performance degradation on a variety of manipulation tasks such as picking, stacking, and assembly, significantly outperforming a generalist policy. Mirage is effective at bridging the visual gap between robots by using cross-painting and the control gap by using a forward dynamics model. It does not require any demonstration data on the target robot or paired images or trajectories between the source and unseen target robots. The method is robust to changes in the robot's coordinate frames and camera parameters, and it can handle different control gains by using a high-gain or blocking controller on the target robot. The key contributions of Mirage include a systematic simulation study analyzing the challenges and potential for policy transfer between grippers and arms, a novel zero-shot cross-embodiment policy transfer method that uses cross-painting to bridge the visual gap and forward dynamics to bridge the control gap, and physical experiments with Franka and UR5 demonstrating that Mirage successfully transfers between robots and grippers on 4 manipulation tasks, suffering only minimal performance degradation from the source policy and significantly outperforming a state-of-the-art generalist model.
Reach us at info@study.space