2021-09-10 | Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, Ricardo Martin-Brualla
The paper introduces a method to photorealistically reconstruct deformable scenes using casual mobile phone captures, extending the Neural Radiance Fields (NeRF) framework. The authors address the challenge of non-rigid deformations by augmenting NeRF with a continuous volumetric deformation field, which warps each observed point into a canonical 5D NeRF. They propose a coarse-to-fine optimization method to robustly optimize the deformation field, incorporating elastic regularization to improve robustness. The method is evaluated using a rig with two synchronized mobile phones, capturing time-synchronized data. The results show that the method can faithfully reconstruct non-rigidly deforming scenes and produce high-fidelity renderings from arbitrary viewpoints, referred to as "nerfies." The paper also includes a detailed analysis of the contributions, ablation studies, and comparisons with other methods, demonstrating the effectiveness of their approach.The paper introduces a method to photorealistically reconstruct deformable scenes using casual mobile phone captures, extending the Neural Radiance Fields (NeRF) framework. The authors address the challenge of non-rigid deformations by augmenting NeRF with a continuous volumetric deformation field, which warps each observed point into a canonical 5D NeRF. They propose a coarse-to-fine optimization method to robustly optimize the deformation field, incorporating elastic regularization to improve robustness. The method is evaluated using a rig with two synchronized mobile phones, capturing time-synchronized data. The results show that the method can faithfully reconstruct non-rigidly deforming scenes and produce high-fidelity renderings from arbitrary viewpoints, referred to as "nerfies." The paper also includes a detailed analysis of the contributions, ablation studies, and comparisons with other methods, demonstrating the effectiveness of their approach.