8 Mar 2024 | Zhijing Shao, Zhaolong Wang, Zhuang Li, Duotun Wang, Xiangru Lin, Yu Zhang, Mingming Fan, Zeyu Wang
SplattingAvatar is a novel method for generating realistic, photorealistic human avatars in real-time, combining Gaussian Splatting with mesh-based representation. The method disentangles motion and appearance, using a mesh for low-frequency motion and surface deformation, and Gaussian Splatting for high-frequency geometry and detailed appearance. Gaussians are embedded on the mesh using barycentric coordinates and displacement, allowing for explicit control of their rotation and scaling. The method employs lifted optimization to jointly optimize Gaussian parameters and mesh embeddings, achieving high-quality rendering on both modern GPUs (over 300 FPS) and mobile devices (30 FPS). SplattingAvatar is trained from monocular videos and can be adapted to various animation techniques, demonstrating superior performance in multiple datasets compared to state-of-the-art methods like PointAvatar, INSTA, and NHA. The approach is efficient, portable, and capable of handling complex geometries, making it suitable for applications in gaming, extended reality, and tele-presentation.SplattingAvatar is a novel method for generating realistic, photorealistic human avatars in real-time, combining Gaussian Splatting with mesh-based representation. The method disentangles motion and appearance, using a mesh for low-frequency motion and surface deformation, and Gaussian Splatting for high-frequency geometry and detailed appearance. Gaussians are embedded on the mesh using barycentric coordinates and displacement, allowing for explicit control of their rotation and scaling. The method employs lifted optimization to jointly optimize Gaussian parameters and mesh embeddings, achieving high-quality rendering on both modern GPUs (over 300 FPS) and mobile devices (30 FPS). SplattingAvatar is trained from monocular videos and can be adapted to various animation techniques, demonstrating superior performance in multiple datasets compared to state-of-the-art methods like PointAvatar, INSTA, and NHA. The approach is efficient, portable, and capable of handling complex geometries, making it suitable for applications in gaming, extended reality, and tele-presentation.