StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text

StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text

21 Mar 2024 | Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tadevosyan, Zhangyang Wang, Shant Navasardyan, Humphrey Shi
StreamingT2V is an advanced autoregressive technique designed to generate long videos (up to 1200 frames) with rich motion dynamics, temporal consistency, and high frame-level image quality. The method addresses the limitations of existing approaches, which often suffer from video stagnation and inconsistent scene transitions when extended to long video synthesis. Key components include: 1. **Conditional Attention Module (CAM)**: Conditions the current generation on features extracted from the previous chunk via attentional mechanisms, ensuring smooth transitions between chunks. 2. **Appearance Preservation Module (APM)**: Extracts high-level scene and object features from the first chunk to prevent the model from forgetting initial scene details. 3. **Randomized Blending Approach**: Enhances the quality and resolution of generated videos by applying a video enhancer autoregressively, using a randomized blending technique to ensure seamless transitions between overlapping chunks. Experiments demonstrate that StreamingT2V outperforms competing methods in terms of temporal consistency, motion amount, and per-frame quality. The method is effective across various text-to-video models and can be extended for even longer video durations.StreamingT2V is an advanced autoregressive technique designed to generate long videos (up to 1200 frames) with rich motion dynamics, temporal consistency, and high frame-level image quality. The method addresses the limitations of existing approaches, which often suffer from video stagnation and inconsistent scene transitions when extended to long video synthesis. Key components include: 1. **Conditional Attention Module (CAM)**: Conditions the current generation on features extracted from the previous chunk via attentional mechanisms, ensuring smooth transitions between chunks. 2. **Appearance Preservation Module (APM)**: Extracts high-level scene and object features from the first chunk to prevent the model from forgetting initial scene details. 3. **Randomized Blending Approach**: Enhances the quality and resolution of generated videos by applying a video enhancer autoregressively, using a randomized blending technique to ensure seamless transitions between overlapping chunks. Experiments demonstrate that StreamingT2V outperforms competing methods in terms of temporal consistency, motion amount, and per-frame quality. The method is effective across various text-to-video models and can be extended for even longer video durations.
Reach us at info@study.space