T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitching

T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitching

21 Feb 2024 | Zizheng Pan, Bohan Zhuang, De-An Huang, Weili Nie, Zhiqing Yu, Chaowei Xiao, Jianfei Cai, Anima Anandkumar
T-Stitch is a technique that improves the sampling efficiency of pre-trained diffusion models (DPMs) by using a smaller model in the early denoising steps and a larger model in the later steps. This approach allows for faster sampling without significant degradation in generation quality. The key insight is that smaller models can effectively generate global structures in the early steps, while larger models are better suited for generating high-frequency details in later steps. T-Stitch is training-free and can be applied to various architectures and diffusion models, complementing existing fast sampling techniques with flexible speed and quality trade-offs. For example, on DiT-XL, 40% of the early timesteps can be replaced with a 10x faster DiT-S without performance drop. T-Stitch also improves prompt alignment for stylized SD models. The method is applicable to different DPM architectures and can be used to accelerate popular pretrained models like Stable Diffusion. The approach is complementary to existing fast sampling methods and can be combined with other techniques like model compression and step reduction. T-Stitch achieves better speed and quality trade-offs compared to model stitching methods and is compatible with various diffusion samplers and architectures. The method is also applicable to different pretrained model families and can be used to accelerate large models while maintaining generation quality. Overall, T-Stitch provides a flexible and efficient solution for improving the sampling efficiency of DPMs without requiring retraining.T-Stitch is a technique that improves the sampling efficiency of pre-trained diffusion models (DPMs) by using a smaller model in the early denoising steps and a larger model in the later steps. This approach allows for faster sampling without significant degradation in generation quality. The key insight is that smaller models can effectively generate global structures in the early steps, while larger models are better suited for generating high-frequency details in later steps. T-Stitch is training-free and can be applied to various architectures and diffusion models, complementing existing fast sampling techniques with flexible speed and quality trade-offs. For example, on DiT-XL, 40% of the early timesteps can be replaced with a 10x faster DiT-S without performance drop. T-Stitch also improves prompt alignment for stylized SD models. The method is applicable to different DPM architectures and can be used to accelerate popular pretrained models like Stable Diffusion. The approach is complementary to existing fast sampling methods and can be combined with other techniques like model compression and step reduction. T-Stitch achieves better speed and quality trade-offs compared to model stitching methods and is compatible with various diffusion samplers and architectures. The method is also applicable to different pretrained model families and can be used to accelerate large models while maintaining generation quality. Overall, T-Stitch provides a flexible and efficient solution for improving the sampling efficiency of DPMs without requiring retraining.
Reach us at info@study.space
Understanding T-Stitch%3A Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitching