14 Mar 2024 | Yiqun Mei¹, Yu Zeng¹†, He Zhang²†, Zhixin Shu²†, Xuaner Zhang², Sai Bi², Jianming Zhang², HyunJoon Jung², Vishal M. Patel¹
Holo-Relighting is a novel volumetric relighting method that enables controllable relighting of a single portrait image, allowing users to adjust lighting, viewpoint, and head pose. The method leverages a pretrained 3D GAN (EG3D) to reconstruct 3D geometry and appearance from the input image as 3D-aware features. A relighting module processes these features to generate a tri-plane representation that can be rendered to any viewpoint through volume rendering. The method also incorporates head pose as a condition to enable head-pose-dependent lighting effects. Holo-Relighting is trained using data captured with a light stage and two data-rendering techniques to improve training quality. The method achieves state-of-the-art relighting quality with better photorealism, 3D consistency, and controllability. It can generate complex non-Lambertian lighting effects such as specular highlights and cast shadows without explicit physical lighting priors. The method is evaluated on both in-the-wild and OLAT test sets, demonstrating superior performance compared to existing methods in terms of relighting quality, fidelity, and identity preservation. Holo-Relighting offers flexible control over lighting, viewpoint, and head pose, making it suitable for practical applications such as shadow diffusion. The method is implemented using PyTorch and trained on 8 A100 GPUs.Holo-Relighting is a novel volumetric relighting method that enables controllable relighting of a single portrait image, allowing users to adjust lighting, viewpoint, and head pose. The method leverages a pretrained 3D GAN (EG3D) to reconstruct 3D geometry and appearance from the input image as 3D-aware features. A relighting module processes these features to generate a tri-plane representation that can be rendered to any viewpoint through volume rendering. The method also incorporates head pose as a condition to enable head-pose-dependent lighting effects. Holo-Relighting is trained using data captured with a light stage and two data-rendering techniques to improve training quality. The method achieves state-of-the-art relighting quality with better photorealism, 3D consistency, and controllability. It can generate complex non-Lambertian lighting effects such as specular highlights and cast shadows without explicit physical lighting priors. The method is evaluated on both in-the-wild and OLAT test sets, demonstrating superior performance compared to existing methods in terms of relighting quality, fidelity, and identity preservation. Holo-Relighting offers flexible control over lighting, viewpoint, and head pose, making it suitable for practical applications such as shadow diffusion. The method is implemented using PyTorch and trained on 8 A100 GPUs.