WildGaussians: 3D Gaussian Splatting in the Wild

WildGaussians: 3D Gaussian Splatting in the Wild

11 Jul 2024 | Jonas Kulhanek, Songyou Peng, Zuzana Kukelova, Marc Pollefeys, Torsten Sattler
WildGaussians is a novel approach that extends 3D Gaussian Splatting (3DGS) to handle occlusions and appearance changes in uncontrolled real-world scenes. The method leverages robust DINO features and integrates an appearance modeling module within 3DGS to achieve state-of-the-art results. Key contributions include: 1. **Appearance Modeling**: Extending 3DGS with per-Gaussian and per-image embeddings to model varying appearances and illuminations, enabling the rendered image to be conditioned on specific input images. 2. **Uncertainty Optimization**: Introducing an uncertainty optimization scheme robust to appearance changes, using DINO features to create an uncertainty mask that effectively removes occluders during training. The method is evaluated on two datasets: the NeRF On-the-go dataset and the Photo Tourism dataset, showing superior performance in handling occlusions and appearance changes while maintaining real-time rendering speeds. WildGaussians outperforms both 3DGS and NeRF baselines, demonstrating its effectiveness in handling complex and dynamic scenes. The source code, model checkpoints, and video comparisons are available at: \url{https://wild-gaussians.github.io/}WildGaussians is a novel approach that extends 3D Gaussian Splatting (3DGS) to handle occlusions and appearance changes in uncontrolled real-world scenes. The method leverages robust DINO features and integrates an appearance modeling module within 3DGS to achieve state-of-the-art results. Key contributions include: 1. **Appearance Modeling**: Extending 3DGS with per-Gaussian and per-image embeddings to model varying appearances and illuminations, enabling the rendered image to be conditioned on specific input images. 2. **Uncertainty Optimization**: Introducing an uncertainty optimization scheme robust to appearance changes, using DINO features to create an uncertainty mask that effectively removes occluders during training. The method is evaluated on two datasets: the NeRF On-the-go dataset and the Photo Tourism dataset, showing superior performance in handling occlusions and appearance changes while maintaining real-time rendering speeds. WildGaussians outperforms both 3DGS and NeRF baselines, demonstrating its effectiveness in handling complex and dynamic scenes. The source code, model checkpoints, and video comparisons are available at: \url{https://wild-gaussians.github.io/}
Reach us at info@study.space