FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models

FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models

24 Jun 2024 | Haonan Qiu1, Zhaoxi Chen1, Zhouxia Wang1, Yingqing He2, Menghan Xia*3, Ziwei Liu*1
**FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models** Diffusion models have shown remarkable capabilities in video generation, but existing works often rely on training-based methods for trajectory control. This study introduces FreeTraj, a tuning-free framework that enables trajectory-controllable video generation by guiding noise construction and attention computation. Specifically: 1. **Noise Guidance**: FreeTraj injects target trajectories into the low-frequency components of initial noises, which influence the motion trajectory of generated content. 2. **Attention Guidance**: It modifies attention mechanisms to ensure objects follow specified trajectories, addressing issues like attention isolation. 3. **Longer and Larger Video Generation**: FreeTraj extends to generate high-fidelity, long videos with controllable trajectories and suppresses duplication in larger videos. Experiments validate the effectiveness of FreeTraj in enhancing trajectory controllability, with competitive performance in video quality and superior trajectory control compared to baseline methods. User studies further confirm the superiority of FreeTraj in trajectory alignment, video-text alignment, and video quality.**FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models** Diffusion models have shown remarkable capabilities in video generation, but existing works often rely on training-based methods for trajectory control. This study introduces FreeTraj, a tuning-free framework that enables trajectory-controllable video generation by guiding noise construction and attention computation. Specifically: 1. **Noise Guidance**: FreeTraj injects target trajectories into the low-frequency components of initial noises, which influence the motion trajectory of generated content. 2. **Attention Guidance**: It modifies attention mechanisms to ensure objects follow specified trajectories, addressing issues like attention isolation. 3. **Longer and Larger Video Generation**: FreeTraj extends to generate high-fidelity, long videos with controllable trajectories and suppresses duplication in larger videos. Experiments validate the effectiveness of FreeTraj in enhancing trajectory controllability, with competitive performance in video quality and superior trajectory control compared to baseline methods. User studies further confirm the superiority of FreeTraj in trajectory alignment, video-text alignment, and video quality.
Reach us at info@study.space