DREAMFLOW: HIGH-QUALITY TEXT-TO-3D GENERATION BY APPROXIMATING PROBABILITY FLOW

DREAMFLOW: HIGH-QUALITY TEXT-TO-3D GENERATION BY APPROXIMATING PROBABILITY FLOW

2024 | Kyungmin Lee, Kihyuk Sohn, Jinwoo Shin
DreamFlow is a text-to-3D generation method that improves upon existing approaches by approximating the probability flow of diffusion models. The method leverages the pre-trained text-to-image diffusion prior to optimize a 3D representation in a multi-view image-to-image translation framework. By using a predetermined timestep schedule, DreamFlow reduces the variance of the gradient and accelerates the optimization process. The method is designed as a three-stage coarse-to-fine optimization framework, where it first optimizes a Neural Radiance Field (NeRF), then extracts and fine-tunes a 3D mesh, and finally refines the mesh using a high-resolution diffusion prior. This approach enables the generation of high-quality and high-resolution 3D content, with DreamFlow being 5 times faster than existing state-of-the-art methods while producing more photorealistic results. The method also demonstrates superior performance in user preference studies and quantitative comparisons, showing better results in terms of photorealism, 3D consistency, and prompt fidelity. DreamFlow's optimization strategy is based on approximating the probability flow ODE, which allows for more stable and efficient 3D optimization compared to score distillation methods. The method is evaluated on various benchmarks and shows significant improvements in both qualitative and quantitative metrics.DreamFlow is a text-to-3D generation method that improves upon existing approaches by approximating the probability flow of diffusion models. The method leverages the pre-trained text-to-image diffusion prior to optimize a 3D representation in a multi-view image-to-image translation framework. By using a predetermined timestep schedule, DreamFlow reduces the variance of the gradient and accelerates the optimization process. The method is designed as a three-stage coarse-to-fine optimization framework, where it first optimizes a Neural Radiance Field (NeRF), then extracts and fine-tunes a 3D mesh, and finally refines the mesh using a high-resolution diffusion prior. This approach enables the generation of high-quality and high-resolution 3D content, with DreamFlow being 5 times faster than existing state-of-the-art methods while producing more photorealistic results. The method also demonstrates superior performance in user preference studies and quantitative comparisons, showing better results in terms of photorealism, 3D consistency, and prompt fidelity. DreamFlow's optimization strategy is based on approximating the probability flow ODE, which allows for more stable and efficient 3D optimization compared to score distillation methods. The method is evaluated on various benchmarks and shows significant improvements in both qualitative and quantitative metrics.
Reach us at info@study.space
[slides] DreamFlow%3A High-Quality Text-to-3D Generation by Approximating Probability Flow | StudySpace