2407.08447v1 11 Jul 2024 | Jonas Kulhanek, Songyou Peng, Zuzana Kukelova, Marc Pollefeys, Torsten Sattler
WildGaussians is a novel approach to 3D Gaussian Splatting (3DGS) that addresses the challenges of handling occlusions and appearance changes in uncontrolled, in-the-wild scenes. While 3DGS offers real-time rendering speeds and high-quality results, it struggles with dynamic environments due to its explicit representation and lack of shared parameters. WildGaussians extends 3DGS by integrating robust DINO features and an appearance modeling module, enabling it to handle complex scenes with varying illumination and occlusions. The method jointly optimizes a DINO-based uncertainty predictor to handle occlusions and uses trainable appearance embeddings to model scene appearance changes. This allows WildGaussians to achieve state-of-the-art results while maintaining the efficiency and flexibility of 3DGS. The approach is validated on two challenging datasets: the NeRF On-the-go dataset and the Photo Tourism dataset. Results show that WildGaussians outperforms existing methods in handling in-the-wild data, with faster rendering times and better performance in scenarios with high occlusion and appearance changes. The method also introduces an uncertainty modeling strategy that uses DINO features to create an uncertainty mask, effectively removing the influence of occluders during training. The contributions of WildGaussians include appearance modeling and uncertainty optimization, which enable the method to handle dynamic scenes with varying conditions. The approach is implemented using a combination of DSSIM and L1 losses, with a focus on robustness to appearance changes. The method is also efficient in terms of training and rendering, making it suitable for real-time applications. The results demonstrate that WildGaussians is a significant advancement in 3D scene reconstruction, particularly for uncontrolled environments.WildGaussians is a novel approach to 3D Gaussian Splatting (3DGS) that addresses the challenges of handling occlusions and appearance changes in uncontrolled, in-the-wild scenes. While 3DGS offers real-time rendering speeds and high-quality results, it struggles with dynamic environments due to its explicit representation and lack of shared parameters. WildGaussians extends 3DGS by integrating robust DINO features and an appearance modeling module, enabling it to handle complex scenes with varying illumination and occlusions. The method jointly optimizes a DINO-based uncertainty predictor to handle occlusions and uses trainable appearance embeddings to model scene appearance changes. This allows WildGaussians to achieve state-of-the-art results while maintaining the efficiency and flexibility of 3DGS. The approach is validated on two challenging datasets: the NeRF On-the-go dataset and the Photo Tourism dataset. Results show that WildGaussians outperforms existing methods in handling in-the-wild data, with faster rendering times and better performance in scenarios with high occlusion and appearance changes. The method also introduces an uncertainty modeling strategy that uses DINO features to create an uncertainty mask, effectively removing the influence of occluders during training. The contributions of WildGaussians include appearance modeling and uncertainty optimization, which enable the method to handle dynamic scenes with varying conditions. The approach is implemented using a combination of DSSIM and L1 losses, with a focus on robustness to appearance changes. The method is also efficient in terms of training and rendering, making it suitable for real-time applications. The results demonstrate that WildGaussians is a significant advancement in 3D scene reconstruction, particularly for uncontrolled environments.