Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Image

Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Image

14 Mar 2024 | Yiqun Mei1 Yu Zeng1† He Zhang2† Zhixin Shu2† Xuaner Zhang2 Sai Bi2 Jianming Zhang2 HyunJoon Jung2 Vishal M. Patel1
Holo-Relighting is a novel method for volumetric relighting of portraits, enabling users to control lighting, viewpoint, and head pose from a single input image. The method leverages a pre-trained 3D GAN (EG3D) to reconstruct 3D geometry and appearance features from the input portrait. A specialized relighting module processes these features to predict a lighting-aware 3D representation in the form of a tri-plane, which can be rendered at any viewpoint using volume rendering. This approach allows for the generation of complex non-Lambertian lighting effects, such as specular highlights and cast shadows, without explicit physical lighting priors. The method is trained using data captured with a light stage and includes techniques to improve data quality, such as multi-view regularization and shading transfer. Extensive experiments demonstrate that Holo-Relighting achieves state-of-the-art performance in terms of photorealism, 3D consistency, and controllability, outperforming existing methods in both qualitative and quantitative evaluations.Holo-Relighting is a novel method for volumetric relighting of portraits, enabling users to control lighting, viewpoint, and head pose from a single input image. The method leverages a pre-trained 3D GAN (EG3D) to reconstruct 3D geometry and appearance features from the input portrait. A specialized relighting module processes these features to predict a lighting-aware 3D representation in the form of a tri-plane, which can be rendered at any viewpoint using volume rendering. This approach allows for the generation of complex non-Lambertian lighting effects, such as specular highlights and cast shadows, without explicit physical lighting priors. The method is trained using data captured with a light stage and includes techniques to improve data quality, such as multi-view regularization and shading transfer. Extensive experiments demonstrate that Holo-Relighting achieves state-of-the-art performance in terms of photorealism, 3D consistency, and controllability, outperforming existing methods in both qualitative and quantitative evaluations.
Reach us at info@study.space
Understanding Holo-Relighting%3A Controllable Volumetric Portrait Relighting from a Single Image