Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials

Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials

2 Jul 2024 | Yawar Siddiqui, Tom Monnier, Filippos Kokkinos, Mahendra Kariya, Yanir Kleiman, Emilien Garreau, Oran Gafni, Natalia Neverova, Andrea Vedaldi, Roman Shapovalov, David Novotny
Meta 3D AssetGen is a novel text- or image-conditioned generator of 3D meshes with physically-based rendering (PBR) materials. It produces detailed geometry and high-fidelity textures, and decomposes materials into albedo, metalness, and roughness, enabling realistic relighting in new environments. The method consists of two stages: a text-to-image stage that generates 4-view grids of images with material information, and an image-to-3D stage that reconstructs the 3D shape and appearance from these views. The image-to-3D stage uses a deferred shading loss for efficient supervision and introduces a signed-distance function (SDF) to represent 3D shape more reliably. A texture refinement transformer further improves the sharpness and details of the generated textures. AssetGen achieves significant improvements over existing methods in Chamfer Distance and LPIPS, and outperforms industry competitors in visual quality and text alignment. The project page provides generated assets: https://assetgen.github.io.Meta 3D AssetGen is a novel text- or image-conditioned generator of 3D meshes with physically-based rendering (PBR) materials. It produces detailed geometry and high-fidelity textures, and decomposes materials into albedo, metalness, and roughness, enabling realistic relighting in new environments. The method consists of two stages: a text-to-image stage that generates 4-view grids of images with material information, and an image-to-3D stage that reconstructs the 3D shape and appearance from these views. The image-to-3D stage uses a deferred shading loss for efficient supervision and introduces a signed-distance function (SDF) to represent 3D shape more reliably. A texture refinement transformer further improves the sharpness and details of the generated textures. AssetGen achieves significant improvements over existing methods in Chamfer Distance and LPIPS, and outperforms industry competitors in visual quality and text alignment. The project page provides generated assets: https://assetgen.github.io.
Reach us at info@study.space