ToonCrafter: Generative Cartoon Interpolation

ToonCrafter: Generative Cartoon Interpolation

28 May 2024 | Jinbo Xing, Hanyuan Liu, Menghan Xia, Yong Zhang, Xintao Wang, Ying Shan, Tien-Tsin Wong
ToonCrafter is a novel approach to generative cartoon video interpolation, addressing the limitations of traditional correspondence-based methods that struggle with non-linear and large motions, dis-occlusion, and textureless regions. The method leverages live-action video priors to adapt to the cartoon domain, using a toon rectification learning strategy to resolve domain gaps and content leakage. A dual-reference-based 3D decoder compensates for detail loss in compressed latent spaces, ensuring fine details are preserved. A flexible sketch encoder allows users to control the interpolation results. Experimental results demonstrate that ToonCrafter produces visually convincing and natural dynamics, effectively handling dis-occlusion, and outperforms existing methods in various metrics. The code will be available upon publication.ToonCrafter is a novel approach to generative cartoon video interpolation, addressing the limitations of traditional correspondence-based methods that struggle with non-linear and large motions, dis-occlusion, and textureless regions. The method leverages live-action video priors to adapt to the cartoon domain, using a toon rectification learning strategy to resolve domain gaps and content leakage. A dual-reference-based 3D decoder compensates for detail loss in compressed latent spaces, ensuring fine details are preserved. A flexible sketch encoder allows users to control the interpolation results. Experimental results demonstrate that ToonCrafter produces visually convincing and natural dynamics, effectively handling dis-occlusion, and outperforms existing methods in various metrics. The code will be available upon publication.
Reach us at info@study.space