Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance

Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance

26 Mar 2024 | Donghoon Ahn*1, Hyoungwon Cho*1, Jaewon Min1, Wooseok Jang1, Jungwoo Kim1, SeonHwa Kim1, Hyun Hee Park2, Kyong Hwan Jin†1, Seungryong Kim†1
The paper introduces a novel sampling guidance method called Perturbed-Attention Guidance (PAG) for improving the quality of diffusion samples. PAG enhances the structure of samples during the denoising process by generating intermediate samples with degraded structure using an identity matrix in the self-attention maps of the diffusion U-Net. This approach effectively steers the denoising trajectory away from structural collapse, improving both conditional and unconditional generation. PAG does not require additional training or external modules and shows significant improvements in sample quality, as demonstrated through experiments on the ADM and Stable Diffusion models. Additionally, PAG enhances the performance of downstream tasks such as image restoration and ControlNet with empty prompts. The effectiveness of PAG is further validated through quantitative and qualitative evaluations, including comparisons with other guidance methods like classifier-free guidance (CFG).The paper introduces a novel sampling guidance method called Perturbed-Attention Guidance (PAG) for improving the quality of diffusion samples. PAG enhances the structure of samples during the denoising process by generating intermediate samples with degraded structure using an identity matrix in the self-attention maps of the diffusion U-Net. This approach effectively steers the denoising trajectory away from structural collapse, improving both conditional and unconditional generation. PAG does not require additional training or external modules and shows significant improvements in sample quality, as demonstrated through experiments on the ADM and Stable Diffusion models. Additionally, PAG enhances the performance of downstream tasks such as image restoration and ControlNet with empty prompts. The effectiveness of PAG is further validated through quantitative and qualitative evaluations, including comparisons with other guidance methods like classifier-free guidance (CFG).
Reach us at info@study.space
Understanding Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance