SpotLessSPlats: Ignoring Distractors in 3D Gaussian Splatting

SpotLessSPlats: Ignoring Distractors in 3D Gaussian Splatting

29 Jul 2024 | SARA SABOUR* and LILY GOLI*, GEORGE KOPANAS, MARK MATHEWS, DMITRY LAGUN, LEONIDAS GUIBAS, ALEC JACOBSON, DAVID FLEET, ANDREA TAGLIASACCHI
SpotLessSplats is a method for suppressing transient distractors in 3D Gaussian Splatting (3DGS). The method leverages pre-trained and general-purpose features, along with robust optimization, to effectively ignore transient distractors. It achieves state-of-the-art reconstruction quality on casual captures. The method uses semantic features from text-to-image models to detect outliers, rather than relying on RGB space. It introduces two approaches: spatial clustering and spatio-temporal clustering. The spatial clustering method uses non-parametric clustering of local feature embeddings to find image regions of structured outliers. The spatio-temporal clustering method uses an MLP trained in an unsupervised fashion to predict the portion of the feature space likely to be associated with distractors. The method also introduces a sparsification strategy that significantly reduces the number of Gaussians, saving compute and memory without loss of fidelity. The method is evaluated on standard benchmarks, demonstrating SOTA robust reconstruction, outperforming existing methods by a substantial margin. The method is compatible with robust optimization and is effective in removing transient distractors. The method is also effective in reducing the number of Gaussians, achieving similar reconstruction quality with two to four times fewer splats. The method is tested on challenging benchmarks of casually captured scenes, showing consistent outperformance of competing methods in reconstruction accuracy. The method is also effective in handling transient occlusions, avoiding leakage of transients or under-reconstruction. The method is also effective in handling low-resolution features, missing thin structures such as the balloon string of Figure 8. The method is also effective in handling clustering, upsampling the features to image resolution results in imprecise edges. The method is based on epistemic uncertainty computation per primitive, which is effective in removing lesser utilized Gaussians. However, if the uncertainty is thresholded too aggressively, it can remove parts of the scene that are rarely viewed in the training data. The method is also effective in handling the use of clustering, upsampling the features to image resolution results in imprecise edges. The method is also effective in handling the use of clustering, upsampling the features to image resolution results in imprecise edges. The method is also effective in handling the use of clustering, upsampling the features to image resolution results in imprecise edges.SpotLessSplats is a method for suppressing transient distractors in 3D Gaussian Splatting (3DGS). The method leverages pre-trained and general-purpose features, along with robust optimization, to effectively ignore transient distractors. It achieves state-of-the-art reconstruction quality on casual captures. The method uses semantic features from text-to-image models to detect outliers, rather than relying on RGB space. It introduces two approaches: spatial clustering and spatio-temporal clustering. The spatial clustering method uses non-parametric clustering of local feature embeddings to find image regions of structured outliers. The spatio-temporal clustering method uses an MLP trained in an unsupervised fashion to predict the portion of the feature space likely to be associated with distractors. The method also introduces a sparsification strategy that significantly reduces the number of Gaussians, saving compute and memory without loss of fidelity. The method is evaluated on standard benchmarks, demonstrating SOTA robust reconstruction, outperforming existing methods by a substantial margin. The method is compatible with robust optimization and is effective in removing transient distractors. The method is also effective in reducing the number of Gaussians, achieving similar reconstruction quality with two to four times fewer splats. The method is tested on challenging benchmarks of casually captured scenes, showing consistent outperformance of competing methods in reconstruction accuracy. The method is also effective in handling transient occlusions, avoiding leakage of transients or under-reconstruction. The method is also effective in handling low-resolution features, missing thin structures such as the balloon string of Figure 8. The method is also effective in handling clustering, upsampling the features to image resolution results in imprecise edges. The method is based on epistemic uncertainty computation per primitive, which is effective in removing lesser utilized Gaussians. However, if the uncertainty is thresholded too aggressively, it can remove parts of the scene that are rarely viewed in the training data. The method is also effective in handling the use of clustering, upsampling the features to image resolution results in imprecise edges. The method is also effective in handling the use of clustering, upsampling the features to image resolution results in imprecise edges. The method is also effective in handling the use of clustering, upsampling the features to image resolution results in imprecise edges.
Reach us at info@study.space
[slides and audio] SpotlessSplats%3A Ignoring Distractors in 3D Gaussian Splatting