AnimateLCM: Accelerating the Animation of Personalized Diffusion Models and Adapters with Decoupled Consistency Learning

AnimateLCM: Accelerating the Animation of Personalized Diffusion Models and Adapters with Decoupled Consistency Learning

1 Feb 2024 | Fu-Yun Wang, Zhaoyang Huang, Xiaoyu Shi, Weikang Bian, Guanglu Song, Yu Liu, Hongsheng Li
AnimateLCM is a novel approach designed to accelerate the generation of high-fidelity videos using personalized diffusion models and adapters. Inspired by the Latent Consistency Model (LCM), which accelerates image generation with minimal steps, AnimateLCM decouples the distillation of image generation priors and motion generation priors to improve training efficiency and visual quality. The method first adapts a stable diffusion model into an image consistency model using high-quality image-text datasets, then inflates these models to accommodate 3D video features, and finally distills the video data to obtain a video consistency model. An efficient initialization strategy is proposed to mitigate feature corruption during the inflation process. Additionally, AnimateLCM introduces a teacher-free adaptation strategy to integrate existing adapters or train new ones without compromising sampling speed. Experimental results demonstrate that AnimateLCM achieves top-performing results in image-conditioned and layout-conditioned video generation, validating its effectiveness and efficiency.AnimateLCM is a novel approach designed to accelerate the generation of high-fidelity videos using personalized diffusion models and adapters. Inspired by the Latent Consistency Model (LCM), which accelerates image generation with minimal steps, AnimateLCM decouples the distillation of image generation priors and motion generation priors to improve training efficiency and visual quality. The method first adapts a stable diffusion model into an image consistency model using high-quality image-text datasets, then inflates these models to accommodate 3D video features, and finally distills the video data to obtain a video consistency model. An efficient initialization strategy is proposed to mitigate feature corruption during the inflation process. Additionally, AnimateLCM introduces a teacher-free adaptation strategy to integrate existing adapters or train new ones without compromising sampling speed. Experimental results demonstrate that AnimateLCM achieves top-performing results in image-conditioned and layout-conditioned video generation, validating its effectiveness and efficiency.
Reach us at info@study.space
Understanding AnimateLCM%3A Computation-Efficient Personalized Style Video Generation without Personalized Video Data