30 May 2024 | Sangyun Lee, Zinan Lin, Giulia Fanti
This paper presents improved training techniques for rectified flows, allowing them to compete with knowledge distillation methods even in the low number of function evaluations (NFE) setting. The main insight is that under realistic settings, a single iteration of the Reflow algorithm for training rectified flows is sufficient to learn nearly straight trajectories, making multiple Reflow iterations unnecessary. The authors propose techniques to improve one-round training of rectified flows, including a U-shaped timestep distribution and LPIPS-Huber premetric. These techniques improve the FID of the previous 2-rectified flow by up to 72% in the 1 NFE setting on CIFAR-10. On ImageNet 64×64, the improved rectified flow outperforms state-of-the-art distillation methods such as consistency distillation and progressive distillation in both one-step and two-step settings and rivals the performance of improved consistency training (iCT) in FID. The authors also show that rectified flows can be initialized with pre-trained diffusion models such as EDM, which improves performance without requiring additional Reflow or distillation stages. The improved rectified flow achieves lower FID scores on CIFAR-10 and ImageNet 64×64 compared to previous methods, and the training techniques reduce the FID of the previous 2-rectified flow by about 72%. The results show that rectified flows can be competitive with distillation methods in the low NFE setting, and the proposed techniques significantly improve performance without harming performance at higher NFE settings. The code is available at https://github.com/sangyun884/rfpp.This paper presents improved training techniques for rectified flows, allowing them to compete with knowledge distillation methods even in the low number of function evaluations (NFE) setting. The main insight is that under realistic settings, a single iteration of the Reflow algorithm for training rectified flows is sufficient to learn nearly straight trajectories, making multiple Reflow iterations unnecessary. The authors propose techniques to improve one-round training of rectified flows, including a U-shaped timestep distribution and LPIPS-Huber premetric. These techniques improve the FID of the previous 2-rectified flow by up to 72% in the 1 NFE setting on CIFAR-10. On ImageNet 64×64, the improved rectified flow outperforms state-of-the-art distillation methods such as consistency distillation and progressive distillation in both one-step and two-step settings and rivals the performance of improved consistency training (iCT) in FID. The authors also show that rectified flows can be initialized with pre-trained diffusion models such as EDM, which improves performance without requiring additional Reflow or distillation stages. The improved rectified flow achieves lower FID scores on CIFAR-10 and ImageNet 64×64 compared to previous methods, and the training techniques reduce the FID of the previous 2-rectified flow by about 72%. The results show that rectified flows can be competitive with distillation methods in the low NFE setting, and the proposed techniques significantly improve performance without harming performance at higher NFE settings. The code is available at https://github.com/sangyun884/rfpp.