CameraCtrl: Enabling Camera Control for Text-to-Video Generation

CameraCtrl: Enabling Camera Control for Text-to-Video Generation

2 Apr 2024 | Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang
CameraCtrl is a novel method that enables precise camera control in text-to-video (T2V) generation, enhancing the controllability and realism of generated videos. The method introduces a plug-and-play camera module that can be seamlessly integrated into existing T2V models without affecting their performance. By parameterizing the camera trajectory using plücker embeddings, which provide geometric interpretations for each pixel, CameraCtrl accurately controls camera viewpoints. The module is trained on a dataset with diverse camera poses and similar appearance to the base T2V model, ensuring both generalizability and controllability. Experimental results demonstrate that CameraCtrl effectively controls camera movements, producing videos with smooth and visually appealing camera trajectories. The method is versatile and can be applied to various video domains, including natural scenes, stylized objects, and cartoon character videos. Additionally, CameraCtrl can be integrated with other video control methods, such as SparseCtrl, to enhance the generation process. The project website is available at <https://hehaol3.github.io/projects-CameraCtrl/>.CameraCtrl is a novel method that enables precise camera control in text-to-video (T2V) generation, enhancing the controllability and realism of generated videos. The method introduces a plug-and-play camera module that can be seamlessly integrated into existing T2V models without affecting their performance. By parameterizing the camera trajectory using plücker embeddings, which provide geometric interpretations for each pixel, CameraCtrl accurately controls camera viewpoints. The module is trained on a dataset with diverse camera poses and similar appearance to the base T2V model, ensuring both generalizability and controllability. Experimental results demonstrate that CameraCtrl effectively controls camera movements, producing videos with smooth and visually appealing camera trajectories. The method is versatile and can be applied to various video domains, including natural scenes, stylized objects, and cartoon character videos. Additionally, CameraCtrl can be integrated with other video control methods, such as SparseCtrl, to enhance the generation process. The project website is available at <https://hehaol3.github.io/projects-CameraCtrl/>.
Reach us at info@study.space
[slides and audio] CameraCtrl%3A Enabling Camera Control for Text-to-Video Generation