D-Flow: Differentiating through Flows for Controlled Generation

D-Flow: Differentiating through Flows for Controlled Generation

2024 | Heli Ben-Hamu, Omri Puny, Itai Gat, Brian Karrer, Uriel Singer, Yaron Lipman
The paper introduces *D-Flow*, a framework for controlled generation using differentiable flow models. D-Flow differentiates through the flow process, optimizing the source (noise) point to control the generation outcome. The key observation is that for Diffusion/FM models trained with Gaussian probability paths, differentiating through the generation process projects the gradient onto the data manifold, implicitly injecting a prior into the optimization. This framework is validated on various controlled generation problems, including image and audio inverse problems, and conditional molecule generation, achieving state-of-the-art performance. The method is simple and effective, but it has relatively long runtimes due to the need to backpropagate through multiple compositions of the velocity field. The paper also provides theoretical support for the implicit regularization claim and discusses related work and future directions.The paper introduces *D-Flow*, a framework for controlled generation using differentiable flow models. D-Flow differentiates through the flow process, optimizing the source (noise) point to control the generation outcome. The key observation is that for Diffusion/FM models trained with Gaussian probability paths, differentiating through the generation process projects the gradient onto the data manifold, implicitly injecting a prior into the optimization. This framework is validated on various controlled generation problems, including image and audio inverse problems, and conditional molecule generation, achieving state-of-the-art performance. The method is simple and effective, but it has relatively long runtimes due to the need to backpropagate through multiple compositions of the velocity field. The paper also provides theoretical support for the implicit regularization claim and discusses related work and future directions.
Reach us at info@study.space