PivotMesh: Generic 3D Mesh Generation via Pivot Vertices Guidance

PivotMesh: Generic 3D Mesh Generation via Pivot Vertices Guidance

27 May 2024 | Haohan Weng, Yikai Wang, Tong Zhang, C. L. Philip Chen, Jun Zhu
PivotMesh is a generic and scalable framework for generating 3D meshes by using pivot vertices as a coarse representation to guide the mesh generation process. The framework employs a transformer-based auto-encoder to encode meshes into discrete tokens and decode them from face level to vertex level hierarchically. It first learns to generate pivot vertices as a coarse mesh representation and then generates the complete mesh tokens using an auto-regressive Transformer. This approach reduces the difficulty of directly modeling the mesh distribution and improves the model's controllability. PivotMesh demonstrates its versatility by effectively learning from both small datasets like ShapeNet and large-scale datasets like Objaverse and Objaverse-xl. Extensive experiments show that PivotMesh can generate compact and sharp 3D meshes across various categories, highlighting its great potential for native mesh modeling. The framework is designed to be scalable and extensible, and it outperforms existing methods like PolyGen and MeshGPT in terms of mesh quality and performance. The model can generate meshes from scratch, starting with the generation of pivot vertices followed by the complete mesh token sequence. It also supports conditional generation given the pivot vertices from the reference mesh and downstream applications like mesh variation and refinement. The effectiveness of the framework is validated through various experiments, including shape novelty analysis and comparison with baselines. The results show that PivotMesh can generate novel and realistic shapes, demonstrating its potential for creating new 3D meshes and supporting downstream applications.PivotMesh is a generic and scalable framework for generating 3D meshes by using pivot vertices as a coarse representation to guide the mesh generation process. The framework employs a transformer-based auto-encoder to encode meshes into discrete tokens and decode them from face level to vertex level hierarchically. It first learns to generate pivot vertices as a coarse mesh representation and then generates the complete mesh tokens using an auto-regressive Transformer. This approach reduces the difficulty of directly modeling the mesh distribution and improves the model's controllability. PivotMesh demonstrates its versatility by effectively learning from both small datasets like ShapeNet and large-scale datasets like Objaverse and Objaverse-xl. Extensive experiments show that PivotMesh can generate compact and sharp 3D meshes across various categories, highlighting its great potential for native mesh modeling. The framework is designed to be scalable and extensible, and it outperforms existing methods like PolyGen and MeshGPT in terms of mesh quality and performance. The model can generate meshes from scratch, starting with the generation of pivot vertices followed by the complete mesh token sequence. It also supports conditional generation given the pivot vertices from the reference mesh and downstream applications like mesh variation and refinement. The effectiveness of the framework is validated through various experiments, including shape novelty analysis and comparison with baselines. The results show that PivotMesh can generate novel and realistic shapes, demonstrating its potential for creating new 3D meshes and supporting downstream applications.
Reach us at info@study.space