A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis

A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis

2024 | Y Poirier-Ginter, A Gauthier, J Phillip, J.-F Lalonde, George Drettakis
This paper introduces a novel method for relighting radiance fields using multi-illumination synthesis and diffusion models. The method leverages a pre-trained diffusion model to generate relit versions of images under different lighting conditions, enabling the creation of relightable radiance fields from single-illumination multi-view datasets. The approach involves fine-tuning a diffusion model on a multi-illumination dataset conditioned on light direction, allowing the generation of realistic but potentially inconsistent multi-illumination data. This augmented data is then used to create a relightable radiance field represented by 3D Gaussian splats. To enable direct control of light direction for low-frequency lighting, the method represents appearance with a multi-layer perceptron parameterized on light direction. An auxiliary feature vector is optimized to enforce multi-view consistency and overcome inaccuracies. The method is evaluated on synthetic and real multi-view data under single illumination, demonstrating that it successfully exploits 2D diffusion model priors to allow realistic 3D relighting for complete scenes. The method is compared to previous work and shows superior performance in terms of realism and consistency. The results show that the method can produce realistic relighting of captured scenes while allowing interactive novel-view synthesis. The method is trained for indoor scenes and shows additional results on out-of-distribution samples, demonstrating its ability to generalize to unseen scenes and lighting conditions. The method is limited by its lack of physical accuracy, as the target light direction is noisy and the ControlNet relies mostly on its powerful Stable Diffusion prior to relight rather than performing physics-based reasoning. The method demonstrates that 2D diffusion model priors can be used for realistic relighting, but more complex relighting requires significant future research.This paper introduces a novel method for relighting radiance fields using multi-illumination synthesis and diffusion models. The method leverages a pre-trained diffusion model to generate relit versions of images under different lighting conditions, enabling the creation of relightable radiance fields from single-illumination multi-view datasets. The approach involves fine-tuning a diffusion model on a multi-illumination dataset conditioned on light direction, allowing the generation of realistic but potentially inconsistent multi-illumination data. This augmented data is then used to create a relightable radiance field represented by 3D Gaussian splats. To enable direct control of light direction for low-frequency lighting, the method represents appearance with a multi-layer perceptron parameterized on light direction. An auxiliary feature vector is optimized to enforce multi-view consistency and overcome inaccuracies. The method is evaluated on synthetic and real multi-view data under single illumination, demonstrating that it successfully exploits 2D diffusion model priors to allow realistic 3D relighting for complete scenes. The method is compared to previous work and shows superior performance in terms of realism and consistency. The results show that the method can produce realistic relighting of captured scenes while allowing interactive novel-view synthesis. The method is trained for indoor scenes and shows additional results on out-of-distribution samples, demonstrating its ability to generalize to unseen scenes and lighting conditions. The method is limited by its lack of physical accuracy, as the target light direction is noisy and the ControlNet relies mostly on its powerful Stable Diffusion prior to relight rather than performing physics-based reasoning. The method demonstrates that 2D diffusion model priors can be used for realistic relighting, but more complex relighting requires significant future research.
Reach us at info@study.space
[slides] A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis | StudySpace