**Neural Gaffer: Relighting Any Object via Diffusion** This paper introduces Neural Gaffer, an end-to-end 2D relighting diffusion model that can synthesize high-quality, accurate relit images for any object under novel environmental lighting conditions. The model takes a single image as input and conditions it on a target environment map, without requiring explicit scene decomposition. By fine-tuning a pre-trained diffusion model on a synthetic relighting dataset, Neural Gaffer learns to understand lighting and material properties, achieving superior generalization and accuracy on both synthetic and real-world images. **Key Contributions:** 1. **Category-Agnostic Relighting:** Neural Gaffer can relight any object category without explicit scene decomposition. 2. **Diffusion Model Integration:** The model leverages diffusion models, trained on diverse data, to enable accurate and realistic relighting. 3. **Synthetic Dataset:** A large-scale synthetic dataset, RelitObjaverse, is constructed using 800K synthetic 3D object models and 1,870 HDR environment maps. 4. **Two-Stage 3D Relighting:** Neural Gaffer can be used as a prior for 3D relighting, improving the quality of radiance fields. **Methods:** - **RelitObjaverse Dataset:** The dataset includes 18.4M rendered images with ground-truth lighting. - **Diffusion Model Architecture:** The model consists of an encoder, denoiser (U-Net), and decoder, conditioned on the input image and rotated lighting maps. - **Training:** The model is fine-tuned using a combination of LDR and normalized HDR environment maps to handle lighting conditioning effectively. **Applications:** - **2D Tasks:** Neural Gaffer supports text-based relighting and object insertion, achieving high-quality results. - **3D Tasks:** It can be used as a prior for 3D relighting, improving the quality of radiance fields. **Experiments:** - **Quantitative and Qualitative Evaluations:** The model outperforms existing methods in terms of fidelity, consistency, and accuracy. - **Ablation Studies:** Various designs for lighting conditioning and 3D relighting are evaluated, demonstrating the effectiveness of the proposed approach. **Limitations:** - Minor inconsistencies due to the generative nature of the model. - Limited resolution and potential domain-specific performance. **Conclusion:** Neural Gaffer addresses the challenge of single-image relighting by leveraging diffusion models, achieving high-quality and accurate relit images under diverse lighting conditions. The model enhances various 2D and 3D tasks, demonstrating its broad applicability and effectiveness.**Neural Gaffer: Relighting Any Object via Diffusion** This paper introduces Neural Gaffer, an end-to-end 2D relighting diffusion model that can synthesize high-quality, accurate relit images for any object under novel environmental lighting conditions. The model takes a single image as input and conditions it on a target environment map, without requiring explicit scene decomposition. By fine-tuning a pre-trained diffusion model on a synthetic relighting dataset, Neural Gaffer learns to understand lighting and material properties, achieving superior generalization and accuracy on both synthetic and real-world images. **Key Contributions:** 1. **Category-Agnostic Relighting:** Neural Gaffer can relight any object category without explicit scene decomposition. 2. **Diffusion Model Integration:** The model leverages diffusion models, trained on diverse data, to enable accurate and realistic relighting. 3. **Synthetic Dataset:** A large-scale synthetic dataset, RelitObjaverse, is constructed using 800K synthetic 3D object models and 1,870 HDR environment maps. 4. **Two-Stage 3D Relighting:** Neural Gaffer can be used as a prior for 3D relighting, improving the quality of radiance fields. **Methods:** - **RelitObjaverse Dataset:** The dataset includes 18.4M rendered images with ground-truth lighting. - **Diffusion Model Architecture:** The model consists of an encoder, denoiser (U-Net), and decoder, conditioned on the input image and rotated lighting maps. - **Training:** The model is fine-tuned using a combination of LDR and normalized HDR environment maps to handle lighting conditioning effectively. **Applications:** - **2D Tasks:** Neural Gaffer supports text-based relighting and object insertion, achieving high-quality results. - **3D Tasks:** It can be used as a prior for 3D relighting, improving the quality of radiance fields. **Experiments:** - **Quantitative and Qualitative Evaluations:** The model outperforms existing methods in terms of fidelity, consistency, and accuracy. - **Ablation Studies:** Various designs for lighting conditioning and 3D relighting are evaluated, demonstrating the effectiveness of the proposed approach. **Limitations:** - Minor inconsistencies due to the generative nature of the model. - Limited resolution and potential domain-specific performance. **Conclusion:** Neural Gaffer addresses the challenge of single-image relighting by leveraging diffusion models, achieving high-quality and accurate relit images under diverse lighting conditions. The model enhances various 2D and 3D tasks, demonstrating its broad applicability and effectiveness.