Dynamic Gaussian Marbles for Novel View Synthesis of Casual Monocular Videos

Dynamic Gaussian Marbles for Novel View Synthesis of Casual Monocular Videos

26 Jun 2024 | COLTON STEARNS, ADAM HARLEY, MIKAELA UY, FLORIAN DUBOST, FEDERICO TOMBARI, GORDON WETZSTEIN, LEONIDAS GUIBAS
This paper introduces Dynamic Gaussian Marbles (DGMarbles), a method for novel-view synthesis from casual monocular videos. DGMarbles improves upon existing Gaussian-based methods by using isotropic Gaussian "marbles" that reduce degrees of freedom and focus on motion and appearance. It employs a divide-and-conquer learning strategy to efficiently guide optimization towards globally coherent motion and incorporates image-level and geometry-level priors, including a tracking loss that leverages recent advances in point tracking. DGMarbles outperforms other Gaussian baselines in quality and is on-par with non-Gaussian representations, while maintaining the efficiency, compositionality, editability, and tracking benefits of Gaussians. The method is evaluated on the Nvidia Dynamic Scenes and DyCheck iPhone datasets, showing significant improvements in novel-view synthesis. DGMarbles is well-suited for tracking and editing, and significantly outperforms previous reconstruction methods in tracking accuracy. However, it has limitations in handling scenes with rapid and non-rigid motion, where further progress in 3D priors and visual tracking is needed. The paper also compares DGMarbles with NeRF baselines, showing that it is on-par with them in novel-view synthesis, while offering faster rendering, better tracking, and more editability. The method is a significant step forward in bringing Gaussian representations to the challenging setting of general monocular novel-view synthesis.This paper introduces Dynamic Gaussian Marbles (DGMarbles), a method for novel-view synthesis from casual monocular videos. DGMarbles improves upon existing Gaussian-based methods by using isotropic Gaussian "marbles" that reduce degrees of freedom and focus on motion and appearance. It employs a divide-and-conquer learning strategy to efficiently guide optimization towards globally coherent motion and incorporates image-level and geometry-level priors, including a tracking loss that leverages recent advances in point tracking. DGMarbles outperforms other Gaussian baselines in quality and is on-par with non-Gaussian representations, while maintaining the efficiency, compositionality, editability, and tracking benefits of Gaussians. The method is evaluated on the Nvidia Dynamic Scenes and DyCheck iPhone datasets, showing significant improvements in novel-view synthesis. DGMarbles is well-suited for tracking and editing, and significantly outperforms previous reconstruction methods in tracking accuracy. However, it has limitations in handling scenes with rapid and non-rigid motion, where further progress in 3D priors and visual tracking is needed. The paper also compares DGMarbles with NeRF baselines, showing that it is on-par with them in novel-view synthesis, while offering faster rendering, better tracking, and more editability. The method is a significant step forward in bringing Gaussian representations to the challenging setting of general monocular novel-view synthesis.
Reach us at info@study.space