Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation

Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation

2 Apr 2024 | Xiyi Chen, Marko Mihajlovic, Shaofei Wang, Sergey Prokudin, Siyu Tang
**Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation** This paper introduces a novel method called Morphable Diffusion, which enables the creation of 3D-consistent, photorealistic human avatars from a single input image. The method integrates a 3D morphable model into a multi-view consistent diffusion framework, enhancing the quality and functionality of avatar creation. Key contributions include: 1. **Enhanced Reconstruction Quality**: By conditioning the generative process on a deformable 3D model, the method improves the reconstruction quality of novel views and animations. 2. **Controlled Novel View Synthesis**: The method allows for explicit control over facial expressions and body poses, achieving high-fidelity and multi-view consistent results. 3. **Animatable Avatars**: The generated avatars are not only photorealistic but also animatable, allowing for realistic facial expressions and body movements. The paper evaluates the method on datasets such as FaceScape and THuman 2.0, demonstrating superior performance in novel view synthesis and novel expression synthesis compared to state-of-the-art methods. The results show that Morphable Diffusion produces more realistic and detailed images, with better preservation of identity and facial expressions. **Limitations and Future Work**: The method has limitations in generalizing to diverse ethnicities and hair types, and it struggles with out-of-distribution camera parameters. Future work could focus on improving generalizability and integrating a more self-contained 3D reconstruction process. **Conclusion**: Morphable Diffusion represents a significant advancement in the field of photorealistic avatar creation, offering a powerful tool for generating high-quality, controllable avatars from a single image.**Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation** This paper introduces a novel method called Morphable Diffusion, which enables the creation of 3D-consistent, photorealistic human avatars from a single input image. The method integrates a 3D morphable model into a multi-view consistent diffusion framework, enhancing the quality and functionality of avatar creation. Key contributions include: 1. **Enhanced Reconstruction Quality**: By conditioning the generative process on a deformable 3D model, the method improves the reconstruction quality of novel views and animations. 2. **Controlled Novel View Synthesis**: The method allows for explicit control over facial expressions and body poses, achieving high-fidelity and multi-view consistent results. 3. **Animatable Avatars**: The generated avatars are not only photorealistic but also animatable, allowing for realistic facial expressions and body movements. The paper evaluates the method on datasets such as FaceScape and THuman 2.0, demonstrating superior performance in novel view synthesis and novel expression synthesis compared to state-of-the-art methods. The results show that Morphable Diffusion produces more realistic and detailed images, with better preservation of identity and facial expressions. **Limitations and Future Work**: The method has limitations in generalizing to diverse ethnicities and hair types, and it struggles with out-of-distribution camera parameters. Future work could focus on improving generalizability and integrating a more self-contained 3D reconstruction process. **Conclusion**: Morphable Diffusion represents a significant advancement in the field of photorealistic avatar creation, offering a powerful tool for generating high-quality, controllable avatars from a single image.
Reach us at info@study.space