Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control

Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control

27 May 2024 | Zhengfei Kuang*1 Hongsheng Li2 Shengqu Cai*1 Leonidas Guibas1 Hao He2 Yinghao Xu1 Gordon Wetzstein1
The paper introduces Collaborative Video Diffusion (CVD), a novel framework for generating consistent multi-video content with camera control. CVD addresses the challenge of generating videos from different camera trajectories while maintaining consistent content and dynamics. The key innovation is the *Cross-Video Synchronization Module*, which uses epipolar attention to align corresponding frames from different videos, ensuring geometric and semantic consistency. CVD is trained using a hybrid approach, combining monocular static data from RealEstate10k and dynamic data from WebVid10M. Extensive experiments demonstrate that CVD outperforms existing methods in generating multi-view videos with consistent content and dynamics, making it a significant advancement in video synthesis for applications such as 3D scene generation.The paper introduces Collaborative Video Diffusion (CVD), a novel framework for generating consistent multi-video content with camera control. CVD addresses the challenge of generating videos from different camera trajectories while maintaining consistent content and dynamics. The key innovation is the *Cross-Video Synchronization Module*, which uses epipolar attention to align corresponding frames from different videos, ensuring geometric and semantic consistency. CVD is trained using a hybrid approach, combining monocular static data from RealEstate10k and dynamic data from WebVid10M. Extensive experiments demonstrate that CVD outperforms existing methods in generating multi-view videos with consistent content and dynamics, making it a significant advancement in video synthesis for applications such as 3D scene generation.
Reach us at info@study.space
[slides] Collaborative Video Diffusion%3A Consistent Multi-video Generation with Camera Control | StudySpace