FlashTex: Fast Relightable Mesh Texturing with LightControlNet

FlashTex: Fast Relightable Mesh Texturing with LightControlNet

22 Apr 2024 | Kangle Deng, Timothy Omerick, Alexander Weiss, Deva Ramanan, Jun-Yan Zhu, Tinghui Zhou, Maneesh Agrawala
FlashTex is a fast and efficient method for generating relightable textures for 3D meshes based on user-provided text prompts. The method disentangles lighting from surface material/reflectance, enabling the mesh to be properly relit in any lighting environment. The key innovation is LightControlNet, a text-to-image model that allows the specification of desired lighting as a conditioning image. The text-to-texture pipeline consists of two stages: first, generating a sparse set of visually consistent reference views using LightControlNet; second, applying texture optimization based on Score Distillation Sampling (SDS) to enhance texture quality while disentangling surface material from lighting. The method is significantly faster than previous text-to-texture methods, producing high-quality and relightable textures. The method is evaluated on the Objaverse dataset and 3D game assets, demonstrating superior performance in terms of quality and runtime compared to existing baselines. The method also shows strong results in user studies, with participants consistently preferring the results over baselines. The method is also compatible with off-the-shelf depth ControlNet and Stable Diffusion, offering additional speed improvements when relightability is not required. The method is effective in generating highly detailed and relightable textures for a wide variety of meshes, as demonstrated in the experiments.FlashTex is a fast and efficient method for generating relightable textures for 3D meshes based on user-provided text prompts. The method disentangles lighting from surface material/reflectance, enabling the mesh to be properly relit in any lighting environment. The key innovation is LightControlNet, a text-to-image model that allows the specification of desired lighting as a conditioning image. The text-to-texture pipeline consists of two stages: first, generating a sparse set of visually consistent reference views using LightControlNet; second, applying texture optimization based on Score Distillation Sampling (SDS) to enhance texture quality while disentangling surface material from lighting. The method is significantly faster than previous text-to-texture methods, producing high-quality and relightable textures. The method is evaluated on the Objaverse dataset and 3D game assets, demonstrating superior performance in terms of quality and runtime compared to existing baselines. The method also shows strong results in user studies, with participants consistently preferring the results over baselines. The method is also compatible with off-the-shelf depth ControlNet and Stable Diffusion, offering additional speed improvements when relightability is not required. The method is effective in generating highly detailed and relightable textures for a wide variety of meshes, as demonstrated in the experiments.
Reach us at info@study.space
Understanding FlashTex%3A Fast Relightable Mesh Texturing with LightControlNet