VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality

VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality

4 May 2024 | YING JIANG*, UCLA, USA and HKU, Hong Kong CHANG YU*, UCLA, USA TIANYI XIE*, UCLA, USA XUAN LI*, UCLA, USA YUTAO FENG, University of Utah, USA and Zhejiang University, China HUAMIN WANG, Style3D Research, China MINCHEN LI, CMU, USA HENRY LAU, HKU, Hong Kong FENG GAO*, Amazon, USA YIN YANG, University of Utah, USA CHENFANFU JIANG, UCLA, USA
**VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality** **Authors:** Ying Jiang, Chang Yu, Tianyi Xie, Xu Li, Yutao Feng, Huamin Wang, Minchen Li, Henry Lau, Feng Gao, Yin Yang, Chenfanfu Jiang **Abstract:** The paper introduces VR-GS, a system that integrates 3D Gaussian Splatting (GS) with eXtended Position-based Dynamics (XPBD) to enable real-time, physics-aware interactions in virtual reality (VR). VR-GS addresses the challenges of traditional 3D content creation and manipulation by leveraging GS for efficient and explicit rendering, and XPBD for realistic dynamics. The system offers a seamless and intuitive user experience, allowing users to interact with deformable virtual objects and realistic environments. Key contributions include a two-level embedding strategy to smooth deformation, real-time physics-based simulation, and the integration of shadow mapping for enhanced immersion. The system is designed to be efficient and effective, from scene reconstruction and object segmentation to multi-view image inpainting and interactive physics-based editing. VR-GS is evaluated through various demonstrations and a user study, showing its ability to create high-fidelity, realistic VR experiences. **Keywords:** Gaussian Splatting, Neural Radiance Fields, Real-Time Interactions **Contributions:** - **Physics-Aware Interactive System:** Development of a system that enables interactive, physics-aware manipulation of 3D content represented with GS. - **Two-Level Deformation Embedding:** Introduction of a novel two-level embedding approach to allow Gaussians to adapt smoothly to the mesh, enhancing deformation realism and preventing spiky artifacts. **System Design:** - **Immersive and Realistic Generative Dynamics:** The system aims to provide an immersive and realistic experience with dynamic motions, 3D virtual shapes, and realistic lighting. - **Real-Time Interaction:** Low latency is crucial for preventing disorientation or motion sickness. The system supports basic transformations and real-time physics-based interactions. - **Unified Framework:** VR-GS integrates rendering and simulation within a unified pipeline, ensuring that what is seen is what is simulated. - **Additional Components:** The system includes object segmentation, scene inpainting, and shadow mapping to enhance the user experience. **Method:** - **Gaussian Splatting:** GS represents 3D scene information using an array of 3D anisotropic Gaussian kernels, providing explicit and efficient rendering. - **VR-GS Assets Preparation:** Each 3D asset consists of a high-fidelity GS reconstruction and a simulatable tetrahedral mesh to enable real-time physics-aware dynamics. - **Segmentation, Inpainting, and Mesh Generation:** These processes are conducted offline to prepare the 3D GS assets for interactive editing within the VR environment. **Evaluation:** - **Two-Level Embedding:** Ablation study shows that the two-level embedding approach effectively reduces spiky**VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality** **Authors:** Ying Jiang, Chang Yu, Tianyi Xie, Xu Li, Yutao Feng, Huamin Wang, Minchen Li, Henry Lau, Feng Gao, Yin Yang, Chenfanfu Jiang **Abstract:** The paper introduces VR-GS, a system that integrates 3D Gaussian Splatting (GS) with eXtended Position-based Dynamics (XPBD) to enable real-time, physics-aware interactions in virtual reality (VR). VR-GS addresses the challenges of traditional 3D content creation and manipulation by leveraging GS for efficient and explicit rendering, and XPBD for realistic dynamics. The system offers a seamless and intuitive user experience, allowing users to interact with deformable virtual objects and realistic environments. Key contributions include a two-level embedding strategy to smooth deformation, real-time physics-based simulation, and the integration of shadow mapping for enhanced immersion. The system is designed to be efficient and effective, from scene reconstruction and object segmentation to multi-view image inpainting and interactive physics-based editing. VR-GS is evaluated through various demonstrations and a user study, showing its ability to create high-fidelity, realistic VR experiences. **Keywords:** Gaussian Splatting, Neural Radiance Fields, Real-Time Interactions **Contributions:** - **Physics-Aware Interactive System:** Development of a system that enables interactive, physics-aware manipulation of 3D content represented with GS. - **Two-Level Deformation Embedding:** Introduction of a novel two-level embedding approach to allow Gaussians to adapt smoothly to the mesh, enhancing deformation realism and preventing spiky artifacts. **System Design:** - **Immersive and Realistic Generative Dynamics:** The system aims to provide an immersive and realistic experience with dynamic motions, 3D virtual shapes, and realistic lighting. - **Real-Time Interaction:** Low latency is crucial for preventing disorientation or motion sickness. The system supports basic transformations and real-time physics-based interactions. - **Unified Framework:** VR-GS integrates rendering and simulation within a unified pipeline, ensuring that what is seen is what is simulated. - **Additional Components:** The system includes object segmentation, scene inpainting, and shadow mapping to enhance the user experience. **Method:** - **Gaussian Splatting:** GS represents 3D scene information using an array of 3D anisotropic Gaussian kernels, providing explicit and efficient rendering. - **VR-GS Assets Preparation:** Each 3D asset consists of a high-fidelity GS reconstruction and a simulatable tetrahedral mesh to enable real-time physics-aware dynamics. - **Segmentation, Inpainting, and Mesh Generation:** These processes are conducted offline to prepare the 3D GS assets for interactive editing within the VR environment. **Evaluation:** - **Two-Level Embedding:** Ablation study shows that the two-level embedding approach effectively reduces spiky
Reach us at info@study.space