LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control

LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control

3 Jul 2024 | Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, Di Zhang
This paper introduces LivePortrait, an efficient framework for animating static portraits across various styles and sizes. The model uses implicit-keypoint-based techniques to balance computational efficiency and controllability. It enhances the original face vid2vid framework by scaling training data to 69 million high-quality frames, adopting a mixed image-video training strategy, upgrading the network architecture, and designing better motion transformation and optimization objectives. The model also introduces stitching and retargeting modules that use small MLPs with negligible computational overhead to improve controllability. Experimental results show that LivePortrait achieves high inference efficiency, with a generation speed of 12.8ms on an RTX 4090 GPU. The model outperforms diffusion-based methods in terms of generation quality and generalization ability. It can animate static portraits with precise control over eyes and lip movements, ensuring seamless stitching and realistic results. The model is evaluated on self-reenactment and cross-reenactment tasks, demonstrating its effectiveness in generating realistic and expressive animations. The framework is also compared with other methods, showing its advantages in both generation quality and motion accuracy. The model's stitching and retargeting modules are shown to be effective in handling large poses and cross-identity reenactment. The paper also discusses the limitations of the current model, including its performance in cross-reenactment scenarios with large pose variations and potential jitter in shoulder movements. Ethical considerations are also addressed, highlighting the social risks of portrait animation technologies, including their potential misuse for deepfakes. The paper concludes that LivePortrait provides a promising solution for real-time portrait animation applications in various scenarios.This paper introduces LivePortrait, an efficient framework for animating static portraits across various styles and sizes. The model uses implicit-keypoint-based techniques to balance computational efficiency and controllability. It enhances the original face vid2vid framework by scaling training data to 69 million high-quality frames, adopting a mixed image-video training strategy, upgrading the network architecture, and designing better motion transformation and optimization objectives. The model also introduces stitching and retargeting modules that use small MLPs with negligible computational overhead to improve controllability. Experimental results show that LivePortrait achieves high inference efficiency, with a generation speed of 12.8ms on an RTX 4090 GPU. The model outperforms diffusion-based methods in terms of generation quality and generalization ability. It can animate static portraits with precise control over eyes and lip movements, ensuring seamless stitching and realistic results. The model is evaluated on self-reenactment and cross-reenactment tasks, demonstrating its effectiveness in generating realistic and expressive animations. The framework is also compared with other methods, showing its advantages in both generation quality and motion accuracy. The model's stitching and retargeting modules are shown to be effective in handling large poses and cross-identity reenactment. The paper also discusses the limitations of the current model, including its performance in cross-reenactment scenarios with large pose variations and potential jitter in shoulder movements. Ethical considerations are also addressed, highlighting the social risks of portrait animation technologies, including their potential misuse for deepfakes. The paper concludes that LivePortrait provides a promising solution for real-time portrait animation applications in various scenarios.
Reach us at info@study.space
[slides and audio] LivePortrait%3A Efficient Portrait Animation with Stitching and Retargeting Control