Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation

Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation

7 Mar 2024 | Tairan He, Zhengyi Luo, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, Guanyu Shi
The paper presents Human to Humanoid (#20), a reinforcement learning (RL) framework that enables real-time whole-body teleoperation of a full-sized humanoid robot using only an RGB camera. The authors address the challenge of translating human movements into actions that the humanoid can perform by proposing a scalable "sim-to-data" process to filter and refine a large-scale human motion dataset. They train a robust real-time humanoid motion imitator in simulation and transfer it to the real humanoid robot without any additional setup. The system successfully demonstrates various dynamic whole-body motions in real-world scenarios, including walking, kicking, and boxing. The key contributions include a scalable retargeting and "sim-to-data" process, a scalable training process for the real-world motion imitator, and a real-time teleoperation system with an RGB camera and 3D human pose estimation. The paper also discusses related work, experimental results, and future directions, emphasizing the importance of closing the representation, embodiment, and sim-to-real gaps for universal humanoid teleoperation.The paper presents Human to Humanoid (#20), a reinforcement learning (RL) framework that enables real-time whole-body teleoperation of a full-sized humanoid robot using only an RGB camera. The authors address the challenge of translating human movements into actions that the humanoid can perform by proposing a scalable "sim-to-data" process to filter and refine a large-scale human motion dataset. They train a robust real-time humanoid motion imitator in simulation and transfer it to the real humanoid robot without any additional setup. The system successfully demonstrates various dynamic whole-body motions in real-world scenarios, including walking, kicking, and boxing. The key contributions include a scalable retargeting and "sim-to-data" process, a scalable training process for the real-world motion imitator, and a real-time teleoperation system with an RGB camera and 3D human pose estimation. The paper also discusses related work, experimental results, and future directions, emphasizing the importance of closing the representation, embodiment, and sim-to-real gaps for universal humanoid teleoperation.
Reach us at info@study.space
[slides and audio] Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation