**DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models** **Authors:** Yuqing Zhang, Yuan Liu, Zhiyu Xie, Lei Yang, Zhongyuan Liu, Mengzhou Yang, Runze Zhang, Qilong Kou, Cheng Lin, Wenping Wang, Xiaogang Jin **Abstract:** DreamMat is an innovative method for generating high-quality Physically Based Rendering (PBR) materials from text descriptions. It addresses the limitations of existing methods that often produce unrealistic rendering effects due to incorrect material decomposition and baked-in shading effects. The key contributions of DreamMat include: 1. **Geometry- and Light-aware Diffusion Model:** This model is trained to generate images consistent with the given geometry and lighting conditions, ensuring that the generated materials are consistent with the input mesh and environment light. 2. **Text-guided Material Generation:** The method uses a text-to-image diffusion model to distill material parameters, focusing on generating high-quality albedo, roughness, and metallic properties. 3. **CSD Loss:** A Classifier Score Distillation (CSD) loss is applied to optimize the material representation, improving the quality and fidelity of the generated materials. **Methods:** - **Material Representation:** The simplified Disney BRDF is used to represent the SVBRDF parameters, including albedo, roughness, and metallic. - **Distillation Loss:** The CSD loss is used to distill the material representation from noisy rendered images, ensuring that the generated materials align with the text prompts. - **Geometry- and Light-aware Diffusion Model:** This model is finetuned to generate images consistent with the given geometry and lighting conditions, using depth and normal maps as geometry conditions and predefined materials as lighting conditions. **Experiments:** - **Qualitative Results:** comparisons with baselines show that DreamMat generates more realistic and diverse materials, with better disentanglement of albedo from shading effects. - **User Study:** A user study with 42 respondents rated the generated materials, showing high satisfaction with the quality and realism of the results. - **Quantitative Analysis:** DreamMat outperforms baseline methods in CLIP score and FID metrics, indicating superior text fidelity and visual quality. - **Ablative Study:** The effectiveness of each component is validated through ablation studies, demonstrating the importance of the geometry- and light-aware diffusion model. **Limitations and Conclusions:** - **Limitations:** The method still struggles with materials that exhibit transparency, high reflection, or subsurface scattering due to the limitations of the BRDF model. - **Conclusions:** DreamMat effectively generates high-quality PBR materials for untextured 3D meshes, offering enhanced realism for applications in gaming and simulation.**DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models** **Authors:** Yuqing Zhang, Yuan Liu, Zhiyu Xie, Lei Yang, Zhongyuan Liu, Mengzhou Yang, Runze Zhang, Qilong Kou, Cheng Lin, Wenping Wang, Xiaogang Jin **Abstract:** DreamMat is an innovative method for generating high-quality Physically Based Rendering (PBR) materials from text descriptions. It addresses the limitations of existing methods that often produce unrealistic rendering effects due to incorrect material decomposition and baked-in shading effects. The key contributions of DreamMat include: 1. **Geometry- and Light-aware Diffusion Model:** This model is trained to generate images consistent with the given geometry and lighting conditions, ensuring that the generated materials are consistent with the input mesh and environment light. 2. **Text-guided Material Generation:** The method uses a text-to-image diffusion model to distill material parameters, focusing on generating high-quality albedo, roughness, and metallic properties. 3. **CSD Loss:** A Classifier Score Distillation (CSD) loss is applied to optimize the material representation, improving the quality and fidelity of the generated materials. **Methods:** - **Material Representation:** The simplified Disney BRDF is used to represent the SVBRDF parameters, including albedo, roughness, and metallic. - **Distillation Loss:** The CSD loss is used to distill the material representation from noisy rendered images, ensuring that the generated materials align with the text prompts. - **Geometry- and Light-aware Diffusion Model:** This model is finetuned to generate images consistent with the given geometry and lighting conditions, using depth and normal maps as geometry conditions and predefined materials as lighting conditions. **Experiments:** - **Qualitative Results:** comparisons with baselines show that DreamMat generates more realistic and diverse materials, with better disentanglement of albedo from shading effects. - **User Study:** A user study with 42 respondents rated the generated materials, showing high satisfaction with the quality and realism of the results. - **Quantitative Analysis:** DreamMat outperforms baseline methods in CLIP score and FID metrics, indicating superior text fidelity and visual quality. - **Ablative Study:** The effectiveness of each component is validated through ablation studies, demonstrating the importance of the geometry- and light-aware diffusion model. **Limitations and Conclusions:** - **Limitations:** The method still struggles with materials that exhibit transparency, high reflection, or subsurface scattering due to the limitations of the BRDF model. - **Conclusions:** DreamMat effectively generates high-quality PBR materials for untextured 3D meshes, offering enhanced realism for applications in gaming and simulation.