16 Jun 2024 | Lawrence Yunliang Chen*1, Kush Hari*1, Karthik Dharmarajan*1, Chenfeng Xu1, Quan Vuong2, Ken Goldberg1
Mirage is a novel algorithm for zero-shot transfer of manipulation policies to unseen robot embodiments. It addresses the challenge of transferring policies between different robots by performing "cross-painting" during execution, which involves masking out the target robot and inpainting the source robot at the same end-effector pose using robot URDFs and a renderer. This creates an illusion as if the source robot were performing the task, allowing the policy to query the source robot's actions. Mirage applies to both first-person and third-person camera views and policies that take in states and images as inputs or only images as inputs. Through extensive simulation and physical experiments, Mirage demonstrates successful zero-shot transfer between different robots and grippers, achieving significantly higher performance than a state-of-the-art generalist model. The key contributions of Mirage include a systematic simulation study, a novel zero-shot cross-embodiment policy transfer method, and physical experiments showing minimal performance degradation on various manipulation tasks.Mirage is a novel algorithm for zero-shot transfer of manipulation policies to unseen robot embodiments. It addresses the challenge of transferring policies between different robots by performing "cross-painting" during execution, which involves masking out the target robot and inpainting the source robot at the same end-effector pose using robot URDFs and a renderer. This creates an illusion as if the source robot were performing the task, allowing the policy to query the source robot's actions. Mirage applies to both first-person and third-person camera views and policies that take in states and images as inputs or only images as inputs. Through extensive simulation and physical experiments, Mirage demonstrates successful zero-shot transfer between different robots and grippers, achieving significantly higher performance than a state-of-the-art generalist model. The key contributions of Mirage include a systematic simulation study, a novel zero-shot cross-embodiment policy transfer method, and physical experiments showing minimal performance degradation on various manipulation tasks.