MVS Gaussian: Fast Generalizable Gaussian Splatting Reconstruction from Multi-View Stereo

MVS Gaussian: Fast Generalizable Gaussian Splatting Reconstruction from Multi-View Stereo

15 Jul 2024 | Tianqi Liu, Guangcong Wang, Shoukang Hu, Liao Shen, Xinyi Ye, Yuhang Zang, Zhiguo Cao, Wei Li, Ziwei Liu
MVSGaussian is a fast and generalizable Gaussian Splatting method for multi-view stereo (MVS) that enables efficient reconstruction of unseen scenes. The method leverages MVS to encode geometry-aware Gaussian representations and decode them into Gaussian parameters. It introduces a hybrid Gaussian rendering approach that integrates an efficient volume rendering design for novel view synthesis. Additionally, a multi-view geometric consistent aggregation strategy is proposed to aggregate point clouds generated by the generalizable model, serving as initialization for per-scene optimization. Compared to previous generalizable NeRF-based methods, MVSGaussian achieves real-time rendering with better synthesis quality for each scene. It also outperforms the vanilla 3D-GS in view synthesis with less training computational cost. Extensive experiments on DTU, Real Forward-facing, NeRF Synthetic, and Tanks and Temples datasets validate that MVSGaussian attains state-of-the-art performance with convincing generalizability, real-time rendering speed, and fast per-scene optimization. The main contributions include a generalizable Gaussian Splatting method derived from MVS and a pixel-aligned Gaussian representation, an efficient hybrid Gaussian rendering approach to boost generalization learning, and a consistent aggregation strategy to provide high-quality initialization for fast per-scene optimization.MVSGaussian is a fast and generalizable Gaussian Splatting method for multi-view stereo (MVS) that enables efficient reconstruction of unseen scenes. The method leverages MVS to encode geometry-aware Gaussian representations and decode them into Gaussian parameters. It introduces a hybrid Gaussian rendering approach that integrates an efficient volume rendering design for novel view synthesis. Additionally, a multi-view geometric consistent aggregation strategy is proposed to aggregate point clouds generated by the generalizable model, serving as initialization for per-scene optimization. Compared to previous generalizable NeRF-based methods, MVSGaussian achieves real-time rendering with better synthesis quality for each scene. It also outperforms the vanilla 3D-GS in view synthesis with less training computational cost. Extensive experiments on DTU, Real Forward-facing, NeRF Synthetic, and Tanks and Temples datasets validate that MVSGaussian attains state-of-the-art performance with convincing generalizability, real-time rendering speed, and fast per-scene optimization. The main contributions include a generalizable Gaussian Splatting method derived from MVS and a pixel-aligned Gaussian representation, an efficient hybrid Gaussian rendering approach to boost generalization learning, and a consistent aggregation strategy to provide high-quality initialization for fast per-scene optimization.
Reach us at info@study.space
[slides] MVSGaussian%3A Fast Generalizable Gaussian Splatting Reconstruction from Multi-View Stereo | StudySpace