29 May 2024 | SIMON GIEBENHAIN, TOBIAS KIRCHSTEIN, MARTIN RÜNZ, LOURDES AGAPIITO, MATTHIAS NIESSNER
NPGA: Neural Parametric Gaussian Avatars
NPGA is a data-driven method for creating high-fidelity, controllable avatars from multi-view video recordings. The method leverages the rich expression space of neural parametric head models (NPHM) to condition avatars' dynamics, enabling fine-grained expression control. The avatars consist of a canonical Gaussian point cloud, augmented with per-primitive features that encode semantic information. A forward deformation field is distilled from the backward deformation field of NPHM to be compatible with rasterization-based rendering. The method also introduces per-Gaussian latent features to increase dynamic expressivity, and proposes Laplacian smoothness terms to regularize the latent features and predicted dynamics. NPGA outperforms previous state-of-the-art avatars on the self-reenactment task by 2.6 PSNR and 0.021 SSIM. The method is evaluated on the NeRSemble dataset, demonstrating accurate animation capabilities from real-world monocular videos. NPGA is a novel avatar representation that leverages a learned deformation representation while ensuring that the predicted facial dynamics stay close to the prior of an underlying neural parametric head model. The method is compared with other approaches, including GaussianAvatars, GaussianHeadAvatar, and MVP, and shows superior performance in terms of PSNR, SSIM, and perceptual LPIPS. The method is also evaluated on cross-reenactment tasks, where driving expressions from another person are transferred to the avatar. The results show that NPGA can create controllable and high-fidelity virtual head avatars from multi-view video data. The method is limited by the underlying 3DMM, and further research is needed to extend the 3DMM to provide a more complete description of a person's state. NPGA is a significant advancement in the field of avatar creation, offering a new approach to creating high-fidelity, controllable avatars.NPGA: Neural Parametric Gaussian Avatars
NPGA is a data-driven method for creating high-fidelity, controllable avatars from multi-view video recordings. The method leverages the rich expression space of neural parametric head models (NPHM) to condition avatars' dynamics, enabling fine-grained expression control. The avatars consist of a canonical Gaussian point cloud, augmented with per-primitive features that encode semantic information. A forward deformation field is distilled from the backward deformation field of NPHM to be compatible with rasterization-based rendering. The method also introduces per-Gaussian latent features to increase dynamic expressivity, and proposes Laplacian smoothness terms to regularize the latent features and predicted dynamics. NPGA outperforms previous state-of-the-art avatars on the self-reenactment task by 2.6 PSNR and 0.021 SSIM. The method is evaluated on the NeRSemble dataset, demonstrating accurate animation capabilities from real-world monocular videos. NPGA is a novel avatar representation that leverages a learned deformation representation while ensuring that the predicted facial dynamics stay close to the prior of an underlying neural parametric head model. The method is compared with other approaches, including GaussianAvatars, GaussianHeadAvatar, and MVP, and shows superior performance in terms of PSNR, SSIM, and perceptual LPIPS. The method is also evaluated on cross-reenactment tasks, where driving expressions from another person are transferred to the avatar. The results show that NPGA can create controllable and high-fidelity virtual head avatars from multi-view video data. The method is limited by the underlying 3DMM, and further research is needed to extend the 3DMM to provide a more complete description of a person's state. NPGA is a significant advancement in the field of avatar creation, offering a new approach to creating high-fidelity, controllable avatars.