28 May 2024 | Jinbo Xing, Hanyuan Liu, Menghan Xia, Yong Zhang, Xintao Wang, Ying Shan, Tien-Tsin Wong
ToonCrafter is a novel generative framework for cartoon video interpolation that addresses the limitations of traditional correspondence-based methods. Traditional methods assume linear motion and lack complex phenomena like disocclusion, leading to implausible results in cartoons with non-linear and large motions. ToonCrafter adapts live-action video priors to cartoon interpolation within a generative framework, overcoming these challenges. It includes a toon rectification learning strategy to adapt live-action video priors to the cartoon domain, a dual-reference-based 3D decoder to compensate for lost details in compressed latent spaces, and a flexible sketch encoder for interactive control over interpolation results. Experimental results show that ToonCrafter produces visually convincing and natural dynamics, effectively handles disocclusion, and outperforms existing competitors. The framework is built on the DynamiCrafter model, with three key improvements: toon rectification learning, detail injection and propagation in decoding, and sketch-based controllable generation. The method enables users to interactively create or modify interpolation results with sketch guidance. The framework is evaluated on a large cartoon video dataset, demonstrating superior performance in quantitative and qualitative comparisons. ToonCrafter is effective in generating intermediate frames for cartoon interpolation, even in challenging cases with large non-linear motions and disocclusions. It also allows users to control the generated motion via sparse sketch input. The framework is versatile and can be applied to various applications, including cartoon sketch interpolation and reference-based sketch colorization.ToonCrafter is a novel generative framework for cartoon video interpolation that addresses the limitations of traditional correspondence-based methods. Traditional methods assume linear motion and lack complex phenomena like disocclusion, leading to implausible results in cartoons with non-linear and large motions. ToonCrafter adapts live-action video priors to cartoon interpolation within a generative framework, overcoming these challenges. It includes a toon rectification learning strategy to adapt live-action video priors to the cartoon domain, a dual-reference-based 3D decoder to compensate for lost details in compressed latent spaces, and a flexible sketch encoder for interactive control over interpolation results. Experimental results show that ToonCrafter produces visually convincing and natural dynamics, effectively handles disocclusion, and outperforms existing competitors. The framework is built on the DynamiCrafter model, with three key improvements: toon rectification learning, detail injection and propagation in decoding, and sketch-based controllable generation. The method enables users to interactively create or modify interpolation results with sketch guidance. The framework is evaluated on a large cartoon video dataset, demonstrating superior performance in quantitative and qualitative comparisons. ToonCrafter is effective in generating intermediate frames for cartoon interpolation, even in challenging cases with large non-linear motions and disocclusions. It also allows users to control the generated motion via sparse sketch input. The framework is versatile and can be applied to various applications, including cartoon sketch interpolation and reference-based sketch colorization.