VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality

VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality

4 May 2024 | YING JIANG, CHANG YU, TIANYI XIE, XUAN LI, YUTAO FENG, HUAMIN WANG, MINCHEN LI, HENRY LAU, FENG GAO, YIN YANG, CHENFANFU JIANG
VR-GS is a physical dynamics-aware interactive Gaussian Splatting (GS) system designed for real-time interaction in virtual reality (VR). It enables intuitive and physics-based manipulation of 3D content represented with GS, offering a seamless and realistic user experience. The system integrates a two-level embedding strategy with deformable body simulations to ensure real-time execution and realistic dynamic responses. Key components include real-time deformation embedding, dynamic shadow casting, and a unified framework for rendering and simulation. VR-GS leverages eXtended Position-based Dynamics (XPBD) for real-time deformation simulation and a tetrahedral cage mesh for embedding Gaussian kernels. The two-level embedding approach allows each Gaussian kernel to adapt to a smoothed average deformation of the surrounding tetrahedra, reducing spiky artifacts and enhancing deformation realism. The system also incorporates a shadow map to enhance the immersive experience by generating dynamic shadows. VR-GS is tested on VR devices with segmentation, inpainting, and shadow map functionalities, providing users with a rich platform for 3D content manipulation. The system is evaluated for performance, visual quality, and user interaction, demonstrating its effectiveness in real-time physics-based editing and immersive VR experiences. The results show that VR-GS significantly enhances user immersion and realism compared to transform-based interaction, with high usability and satisfaction ratings from users. Future work includes incorporating a broader range of materials and exploring the use of large multimodal models for assessing generated dynamics.VR-GS is a physical dynamics-aware interactive Gaussian Splatting (GS) system designed for real-time interaction in virtual reality (VR). It enables intuitive and physics-based manipulation of 3D content represented with GS, offering a seamless and realistic user experience. The system integrates a two-level embedding strategy with deformable body simulations to ensure real-time execution and realistic dynamic responses. Key components include real-time deformation embedding, dynamic shadow casting, and a unified framework for rendering and simulation. VR-GS leverages eXtended Position-based Dynamics (XPBD) for real-time deformation simulation and a tetrahedral cage mesh for embedding Gaussian kernels. The two-level embedding approach allows each Gaussian kernel to adapt to a smoothed average deformation of the surrounding tetrahedra, reducing spiky artifacts and enhancing deformation realism. The system also incorporates a shadow map to enhance the immersive experience by generating dynamic shadows. VR-GS is tested on VR devices with segmentation, inpainting, and shadow map functionalities, providing users with a rich platform for 3D content manipulation. The system is evaluated for performance, visual quality, and user interaction, demonstrating its effectiveness in real-time physics-based editing and immersive VR experiences. The results show that VR-GS significantly enhances user immersion and realism compared to transform-based interaction, with high usability and satisfaction ratings from users. Future work includes incorporating a broader range of materials and exploring the use of large multimodal models for assessing generated dynamics.
Reach us at info@study.space