Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting

Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting

2 Apr 2024 | Yiming Huang1 *, Beilei Cui1 *, Long Bai1 *, Ziqi Guo1, Mengya Xu1, Mobarakol Islam3, and Hongliang Ren1,2,4 **
Endo-4DGS is a real-time endoscopic dynamic reconstruction method that utilizes 3D Gaussian Splatting (GS) for 3D scene representation. The method addresses the challenges of slow inference speed, prolonged training, and inconsistent depth estimation in neural radiance field (NeRF)-based methods. By incorporating lightweight MLPs to capture temporal dynamics with Gaussian deformation fields, Endo-4DGS effectively reconstructs deformable surgical scenes. The depth estimation is enhanced using Depth-Anything, a powerful foundation model, to generate pseudo-depth maps as a geometry prior. Confidence-guided learning is introduced to tackle the ill-posed problems in monocular depth estimation, and surface normal constraints and depth regularization are implemented to improve the accuracy of the pseudo-depth. Extensive validation on two surgical datasets demonstrates that Endo-4DGS achieves high-quality reconstruction, real-time performance, and efficient training, making it suitable for robot-assisted surgery. The method's effectiveness is further validated through ablation studies, which show that each component significantly contributes to the overall performance.Endo-4DGS is a real-time endoscopic dynamic reconstruction method that utilizes 3D Gaussian Splatting (GS) for 3D scene representation. The method addresses the challenges of slow inference speed, prolonged training, and inconsistent depth estimation in neural radiance field (NeRF)-based methods. By incorporating lightweight MLPs to capture temporal dynamics with Gaussian deformation fields, Endo-4DGS effectively reconstructs deformable surgical scenes. The depth estimation is enhanced using Depth-Anything, a powerful foundation model, to generate pseudo-depth maps as a geometry prior. Confidence-guided learning is introduced to tackle the ill-posed problems in monocular depth estimation, and surface normal constraints and depth regularization are implemented to improve the accuracy of the pseudo-depth. Extensive validation on two surgical datasets demonstrates that Endo-4DGS achieves high-quality reconstruction, real-time performance, and efficient training, making it suitable for robot-assisted surgery. The method's effectiveness is further validated through ablation studies, which show that each component significantly contributes to the overall performance.
Reach us at info@study.space