Learning 3D Geometry and Feature Consistent Gaussian Splatting for Object Removal

Learning 3D Geometry and Feature Consistent Gaussian Splatting for Object Removal

21 Apr 2024 | Yuxin Wang, Qianyi Wu, Guofeng Zhang, Dan Xu
This paper presents a novel framework called GScream for object removal using 3D Gaussian Splatting (3DGS), which addresses the challenges of maintaining geometric consistency and texture coherence in 3D scenes. The method enhances information exchange between visible and invisible areas to improve both geometry and texture restoration. It begins by optimizing Gaussian primitive positions using online registration guided by monocular depth estimation, ensuring geometric consistency. A novel feature propagation mechanism with cross-attention design bridges sampling from uncertain and certain areas, improving texture coherence. The method also introduces a lightweight Gaussian Splatting architecture, Scaffold-GS, to enhance efficiency. Extensive experiments show that GScream achieves high-quality novel view synthesis and improves training and rendering speeds compared to traditional NeRF-based methods. The framework maintains geometry and feature consistency while efficiently removing objects from pre-captured scenes. GScream incorporates monocular depth guidance and cross-attention feature regularization to refine geometry and texture. The method outperforms existing approaches in terms of PSNR, SSIM, and LPIPS metrics, and demonstrates superior performance in both quantitative and qualitative evaluations. Ablation studies confirm the effectiveness of depth supervision and cross-attention feature regularization in enhancing the results. The framework is efficient and effective for 3D object removal, providing realistic and coherent results in novel views. The project page is available at https://w-ted.github.io/publications/gscream.This paper presents a novel framework called GScream for object removal using 3D Gaussian Splatting (3DGS), which addresses the challenges of maintaining geometric consistency and texture coherence in 3D scenes. The method enhances information exchange between visible and invisible areas to improve both geometry and texture restoration. It begins by optimizing Gaussian primitive positions using online registration guided by monocular depth estimation, ensuring geometric consistency. A novel feature propagation mechanism with cross-attention design bridges sampling from uncertain and certain areas, improving texture coherence. The method also introduces a lightweight Gaussian Splatting architecture, Scaffold-GS, to enhance efficiency. Extensive experiments show that GScream achieves high-quality novel view synthesis and improves training and rendering speeds compared to traditional NeRF-based methods. The framework maintains geometry and feature consistency while efficiently removing objects from pre-captured scenes. GScream incorporates monocular depth guidance and cross-attention feature regularization to refine geometry and texture. The method outperforms existing approaches in terms of PSNR, SSIM, and LPIPS metrics, and demonstrates superior performance in both quantitative and qualitative evaluations. Ablation studies confirm the effectiveness of depth supervision and cross-attention feature regularization in enhancing the results. The framework is efficient and effective for 3D object removal, providing realistic and coherent results in novel views. The project page is available at https://w-ted.github.io/publications/gscream.
Reach us at info@study.space