InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior

InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior

17 Apr 2024 | Zhiheng Liu, Hao Ouyang, Qiyu Wang, Ka Leong Cheng, Jie Xiao, Kai Zhu, Nan Xue, Yu Liu, Yujun Shen, and Yang Cao
InFusion is a method for 3D Gaussian inpainting that leverages diffusion models for depth completion. The approach addresses the challenge of accurately placing initial points in 3D Gaussians for inpainting by using a depth completion model trained with diffusion priors. This model helps generate accurate depth maps, which are then used to unproject points into 3D space for initialization. The method is efficient and effective, achieving significantly better fidelity and speed compared to existing alternatives. InFusion is demonstrated through various applications, including user-specific texture inpainting and novel object insertion. The model is trained on a large-scale diffusion prior, enabling strong generalization and accurate depth completion. The method is evaluated on multiple datasets, showing superior performance in terms of visual quality and inpainting speed, with results up to 20 times faster than baseline methods. The approach also supports interactive texture editing and object insertion, demonstrating its versatility in 3D scene editing. The method's effectiveness is further validated through ablation studies and comparisons with other baselines, highlighting its advantages in handling complex scenarios and occlusions. Overall, InFusion provides an efficient and effective solution for 3D Gaussian inpainting, with potential applications in various 3D scene editing tasks.InFusion is a method for 3D Gaussian inpainting that leverages diffusion models for depth completion. The approach addresses the challenge of accurately placing initial points in 3D Gaussians for inpainting by using a depth completion model trained with diffusion priors. This model helps generate accurate depth maps, which are then used to unproject points into 3D space for initialization. The method is efficient and effective, achieving significantly better fidelity and speed compared to existing alternatives. InFusion is demonstrated through various applications, including user-specific texture inpainting and novel object insertion. The model is trained on a large-scale diffusion prior, enabling strong generalization and accurate depth completion. The method is evaluated on multiple datasets, showing superior performance in terms of visual quality and inpainting speed, with results up to 20 times faster than baseline methods. The approach also supports interactive texture editing and object insertion, demonstrating its versatility in 3D scene editing. The method's effectiveness is further validated through ablation studies and comparisons with other baselines, highlighting its advantages in handling complex scenarios and occlusions. Overall, InFusion provides an efficient and effective solution for 3D Gaussian inpainting, with potential applications in various 3D scene editing tasks.
Reach us at info@study.space