360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model

360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model

10 May 2024 | Qian Wang1,2, Weiqi Li1, Chong Mou1,2, Xinhua Cheng1,2, Jian Zhang1,2
This paper introduces 360-Degree Video Diffusion Model (360DVD), a pipeline for generating 360° panoramic videos based on text prompts and motion conditions. The main challenges in generating 360° panoramic videos include content distribution, motion patterns, and continuity at the left and right ends of the video. To address these challenges, the authors propose a lightweight 360-Adapter and several 360 Enhancement Techniques. The 360-Adapter transforms pre-trained text-to-video (T2V) models to generate panoramic videos, while the enhancement techniques improve the quality and coherence of the generated videos. The authors also introduce a new dataset, WEB360, consisting of panoramic video-text pairs for training. Extensive experiments demonstrate the effectiveness of 360DVD in generating high-quality, coherent, and diverse 360° panoramic videos. The method is evaluated through user preference studies, showing superior performance in various criteria such as left-right continuity, content distribution, and motion patterns.This paper introduces 360-Degree Video Diffusion Model (360DVD), a pipeline for generating 360° panoramic videos based on text prompts and motion conditions. The main challenges in generating 360° panoramic videos include content distribution, motion patterns, and continuity at the left and right ends of the video. To address these challenges, the authors propose a lightweight 360-Adapter and several 360 Enhancement Techniques. The 360-Adapter transforms pre-trained text-to-video (T2V) models to generate panoramic videos, while the enhancement techniques improve the quality and coherence of the generated videos. The authors also introduce a new dataset, WEB360, consisting of panoramic video-text pairs for training. Extensive experiments demonstrate the effectiveness of 360DVD in generating high-quality, coherent, and diverse 360° panoramic videos. The method is evaluated through user preference studies, showing superior performance in various criteria such as left-right continuity, content distribution, and motion patterns.
Reach us at info@study.space
Understanding 360DVD%3A Controllable Panorama Video Generation with 360-Degree Video Diffusion Model