Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators

Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators

31 Jan 2024 | Daniel Geng, Andrew Owens
Diffusion models are powerful tools for generating and editing images, but they struggle with precise manipulation of object positions, shapes, and deformations. To address this, the authors propose *motion guidance*, a zero-shot technique that allows users to specify dense and complex motion fields to edit images. The method leverages off-the-shelf optical flow networks to guide the diffusion sampling process, ensuring that the edited image adheres to the desired motion while maintaining visual similarity to the original image. The authors demonstrate that their technique works effectively on both real and synthetic images, handling a wide range of complex motions, including translations, rotations, stretches, deformations, and even video-based flows. The method does not require training or specific diffusion network architectures, making it a versatile and user-friendly approach for image editing.Diffusion models are powerful tools for generating and editing images, but they struggle with precise manipulation of object positions, shapes, and deformations. To address this, the authors propose *motion guidance*, a zero-shot technique that allows users to specify dense and complex motion fields to edit images. The method leverages off-the-shelf optical flow networks to guide the diffusion sampling process, ensuring that the edited image adheres to the desired motion while maintaining visual similarity to the original image. The authors demonstrate that their technique works effectively on both real and synthetic images, handling a wide range of complex motions, including translations, rotations, stretches, deformations, and even video-based flows. The method does not require training or specific diffusion network architectures, making it a versatile and user-friendly approach for image editing.
Reach us at info@study.space