MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior

MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior

5 May 2024 | Honghua Chen Chen Change Loy Xingang Pan
The paper "MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior" introduces a novel approach called MVIP-NeRF for inpainting in Neural Radiance Fields (NeRF) scenes. Traditional methods often rely on explicit RGB and depth 2D inpainting, which can lead to inconsistencies, inaccuracies, and misalignments. MVIP-NeRF addresses these issues by leveraging diffusion priors to perform joint inpainting across multiple views, ensuring both appearance and geometry consistency. The method uses Score Distillation Sampling (SDS) to iteratively optimize the inpainting process, recovering both rendered RGB images and normal maps. This approach ensures high-quality geometry completion and alignment with the inpainted RGB images. Additionally, a multi-view SDS score function is introduced to enhance consistency and sharpness in large view variations. Experimental results demonstrate that MVIP-NeRF outperforms existing NeRF inpainting methods in terms of appearance and geometry recovery. The paper also includes a detailed analysis of the method's effectiveness, ablation studies, and visual comparisons with state-of-the-art approaches.The paper "MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior" introduces a novel approach called MVIP-NeRF for inpainting in Neural Radiance Fields (NeRF) scenes. Traditional methods often rely on explicit RGB and depth 2D inpainting, which can lead to inconsistencies, inaccuracies, and misalignments. MVIP-NeRF addresses these issues by leveraging diffusion priors to perform joint inpainting across multiple views, ensuring both appearance and geometry consistency. The method uses Score Distillation Sampling (SDS) to iteratively optimize the inpainting process, recovering both rendered RGB images and normal maps. This approach ensures high-quality geometry completion and alignment with the inpainted RGB images. Additionally, a multi-view SDS score function is introduced to enhance consistency and sharpness in large view variations. Experimental results demonstrate that MVIP-NeRF outperforms existing NeRF inpainting methods in terms of appearance and geometry recovery. The paper also includes a detailed analysis of the method's effectiveness, ablation studies, and visual comparisons with state-of-the-art approaches.
Reach us at info@study.space
[slides and audio] MVIP-NeRF%3A Multi-View 3D Inpainting on NeRF Scenes via Diffusion Prior