SWAG: Splatting in the Wild images with Appearance-conditioned Gaussians

SWAG: Splatting in the Wild images with Appearance-conditioned Gaussians

5 Apr 2024 | Hiba Dahmani, Moussab Bennehar, Nathan Piasco, and Luis Roldão and Dzmitry Tsishkou
SWAG: Splatting in the Wild images with Appearance-conditioned Gaussians This paper introduces SWAG, a novel method for 3D scene reconstruction from in-the-wild photo collections. SWAG extends 3D Gaussian Splatting (3DGS) to handle unstructured image collections by modeling appearance to capture photometric variations in rendered images. Additionally, it introduces a new mechanism to train transient Gaussians to handle the presence of scene occluders in an unsupervised manner. Experiments on diverse photo collection scenes and multi-pass acquisition of outdoor landmarks show that SWAG achieves state-of-the-art results with improved efficiency compared to prior works. The paper discusses the challenges of 3D scene reconstruction from in-the-wild data, including dynamic scenarios with moving transient objects and changing conditions such as weather, exposure, and lighting. It reviews related work in neural rendering in-the-wild, point-based rendering, and 3DGS rendering improvement. The paper presents the 3DGS method, which represents scenes as a large number of anisotropic 3D Gaussians with color features and opacities. The Gaussians are projected to 2D splats and blended during a fast differentiable α-blending process to get 2D rendered images. The paper introduces SWAG, a novel 3DGS-based method for 3D scene reconstruction from in-the-wild photo collections. It proposes to adapt the 3DGS parameters to handle variable visual appearances and the presence of occluders typically found in such unconstrained image collections. The method models appearance variations using image-dependent embeddings injected into the Gaussian's colors and handles transient occluders by learning image-dependent Gaussians' opacities variations. The overall architecture of the method is illustrated in Figure 2. The paper presents experiments on the Phototourism dataset and NeRF-OSR benchmark, showing that SWAG improves 3DGS performance in these scenarios and achieves state-of-the-art rendering quality with significantly faster training and rendering speed compared to previous works. The paper also demonstrates the ability of SWAG to generate new images with smooth visual transitions from learned appearances and to remove in an unsupervised manner transient objects from a captured scene. The paper discusses the challenges of 3D scene reconstruction from in-the-wild data, including dynamic scenarios with moving transient objects and changing conditions such as weather, exposure, and lighting. It reviews related work in neural rendering in-the-wild, point-based rendering, and 3DGS rendering improvement. The paper presents the 3DGS method, which represents scenes as a large number of anisotropic 3D Gaussians with color features and opacities. The Gaussians are projected to 2D splats and blended during a fast differentiable α-blending process to get 2D rendered images.SWAG: Splatting in the Wild images with Appearance-conditioned Gaussians This paper introduces SWAG, a novel method for 3D scene reconstruction from in-the-wild photo collections. SWAG extends 3D Gaussian Splatting (3DGS) to handle unstructured image collections by modeling appearance to capture photometric variations in rendered images. Additionally, it introduces a new mechanism to train transient Gaussians to handle the presence of scene occluders in an unsupervised manner. Experiments on diverse photo collection scenes and multi-pass acquisition of outdoor landmarks show that SWAG achieves state-of-the-art results with improved efficiency compared to prior works. The paper discusses the challenges of 3D scene reconstruction from in-the-wild data, including dynamic scenarios with moving transient objects and changing conditions such as weather, exposure, and lighting. It reviews related work in neural rendering in-the-wild, point-based rendering, and 3DGS rendering improvement. The paper presents the 3DGS method, which represents scenes as a large number of anisotropic 3D Gaussians with color features and opacities. The Gaussians are projected to 2D splats and blended during a fast differentiable α-blending process to get 2D rendered images. The paper introduces SWAG, a novel 3DGS-based method for 3D scene reconstruction from in-the-wild photo collections. It proposes to adapt the 3DGS parameters to handle variable visual appearances and the presence of occluders typically found in such unconstrained image collections. The method models appearance variations using image-dependent embeddings injected into the Gaussian's colors and handles transient occluders by learning image-dependent Gaussians' opacities variations. The overall architecture of the method is illustrated in Figure 2. The paper presents experiments on the Phototourism dataset and NeRF-OSR benchmark, showing that SWAG improves 3DGS performance in these scenarios and achieves state-of-the-art rendering quality with significantly faster training and rendering speed compared to previous works. The paper also demonstrates the ability of SWAG to generate new images with smooth visual transitions from learned appearances and to remove in an unsupervised manner transient objects from a captured scene. The paper discusses the challenges of 3D scene reconstruction from in-the-wild data, including dynamic scenarios with moving transient objects and changing conditions such as weather, exposure, and lighting. It reviews related work in neural rendering in-the-wild, point-based rendering, and 3DGS rendering improvement. The paper presents the 3DGS method, which represents scenes as a large number of anisotropic 3D Gaussians with color features and opacities. The Gaussians are projected to 2D splats and blended during a fast differentiable α-blending process to get 2D rendered images.
Reach us at info@study.space