GaMeS: Mesh-Based Adapting and Modification of Gaussian Splatting

GaMeS: Mesh-Based Adapting and Modification of Gaussian Splatting

15 Feb 2024 | Joanna Waczyńska, Piotr Borycki, Sławomir Tadeja, Jacek Tabor, Przemysław Spurek
The paper introduces GaMeS (Gaussian Mesh Splatting), a novel method that combines the benefits of Gaussian Splatting (GS) and mesh representations to enable real-time editing and adaptation of 3D scenes. GS, known for its fast training and real-time rendering capabilities, uses Gaussian distributions to represent 3D objects, but lacks a well-defined conditioning approach due to the large number of Gaussian components. GaMeS addresses this by parameterizing each Gaussian component using the vertices of a mesh face, allowing for modifications similar to those made on classical meshes. The model can be trained on existing meshes or estimated during training, and it automatically adjusts Gaussian components in response to changes in the mesh, enabling real-time animation and editing. The paper discusses the method's implementation, including the parameterization of Gaussian distributions on mesh faces and the initialization of the pseudo-mesh. Experimental results on synthetic, Mip-NeRF360, and human face datasets demonstrate the effectiveness of GaMeS in both static and animated scenes, showing comparable performance to state-of-the-art methods while offering the advantage of real-time editing. The contributions of GaMeS include a hybrid representation for 3D objects, efficient training on pseudo-meshes, and the ability to render dynamic scenes at similar speeds to static ones.The paper introduces GaMeS (Gaussian Mesh Splatting), a novel method that combines the benefits of Gaussian Splatting (GS) and mesh representations to enable real-time editing and adaptation of 3D scenes. GS, known for its fast training and real-time rendering capabilities, uses Gaussian distributions to represent 3D objects, but lacks a well-defined conditioning approach due to the large number of Gaussian components. GaMeS addresses this by parameterizing each Gaussian component using the vertices of a mesh face, allowing for modifications similar to those made on classical meshes. The model can be trained on existing meshes or estimated during training, and it automatically adjusts Gaussian components in response to changes in the mesh, enabling real-time animation and editing. The paper discusses the method's implementation, including the parameterization of Gaussian distributions on mesh faces and the initialization of the pseudo-mesh. Experimental results on synthetic, Mip-NeRF360, and human face datasets demonstrate the effectiveness of GaMeS in both static and animated scenes, showing comparable performance to state-of-the-art methods while offering the advantage of real-time editing. The contributions of GaMeS include a hybrid representation for 3D objects, efficient training on pseudo-meshes, and the ability to render dynamic scenes at similar speeds to static ones.
Reach us at info@study.space
Understanding GaMeS%3A Mesh-Based Adapting and Modification of Gaussian Splatting