11 Apr 2024 | Gen Li, Kaifeng Zhao, Siwei Zhang, Xiaozhong Lyu, Mihai Dusmanu, Yan Zhang, Marc Pollefeys, Siyu Tang
EgoGen is a novel synthetic data generator designed for egocentric perception tasks, particularly for head-mounted devices (HMDs). It addresses the challenge of simulating natural human movements and behaviors that effectively capture the 3D world from the perspective of the camera wearer. The core of EgoGen is a human motion synthesis model that leverages egocentric visual inputs to sense the environment and respond with collision-avoiding motion primitives (CAMPs). This model uses a two-stage reinforcement learning approach to couple egocentric perception and movement, eliminating the need for predefined global paths and enabling dynamic environments. EgoGen also includes a scalable data generation pipeline that outfits virtual humans with clothing, automates cloth animation, and integrates 3D assets. The effectiveness of EgoGen is demonstrated through three tasks: mapping and localization for HMDs, egocentric camera tracking, and human mesh recovery from egocentric views. The synthetic data generated by EgoGen improves the performance of state-of-the-art methods in these tasks, making it a valuable tool for egocentric computer vision research.EgoGen is a novel synthetic data generator designed for egocentric perception tasks, particularly for head-mounted devices (HMDs). It addresses the challenge of simulating natural human movements and behaviors that effectively capture the 3D world from the perspective of the camera wearer. The core of EgoGen is a human motion synthesis model that leverages egocentric visual inputs to sense the environment and respond with collision-avoiding motion primitives (CAMPs). This model uses a two-stage reinforcement learning approach to couple egocentric perception and movement, eliminating the need for predefined global paths and enabling dynamic environments. EgoGen also includes a scalable data generation pipeline that outfits virtual humans with clothing, automates cloth animation, and integrates 3D assets. The effectiveness of EgoGen is demonstrated through three tasks: mapping and localization for HMDs, egocentric camera tracking, and human mesh recovery from egocentric views. The synthetic data generated by EgoGen improves the performance of state-of-the-art methods in these tasks, making it a valuable tool for egocentric computer vision research.