**Drag Your Noise: Interactive Point-based Editing via Diffusion Semantic Propagation** **Authors:** Haofeng Liu, Chenshu Xu, Yifei Yang, Lihua Zeng, Shengfeng He **Institution:** Singapore Management University, South China Normal University **Abstract:** Point-based interactive editing is crucial for enhancing the controllability of generative models. While DragDiffusion updates the diffusion latent map in response to user inputs, it often leads to global latent map alterations, resulting in imprecise content preservation and unsuccessful editing due to gradient vanishing. In contrast, DragNoise offers robust and accelerated editing without retracing the latent map. The core idea of DragNoise is to use the predicted noise output of each U-Net as a semantic editor. This approach leverages two key observations: (1) the bottleneck features of U-Nets inherently possess rich semantic information suitable for interactive editing, and (2) high-level semantics established early in the denoising process show minimal variation in subsequent stages. By editing diffusion semantics in a single denoising step and efficiently propagating these changes, DragNoise ensures stability and efficiency in diffusion editing. Comparative experiments show that DragNoise achieves superior control and semantic retention, reducing optimization time by over 50% compared to DragDiffusion. **Introduction:** The limited controllability of diffusion models highlights the need for interactive editing in image manipulation. Recent advancements include text-guided editing, stroke-based editing, and exemplar-based methods. DragGAN and DragDiffusion are significant milestones in point-based image editing, but they face challenges such as gradient vanishing and inversion fidelity. DragNoise addresses these issues by leveraging diffusion semantics within the editing mechanism, treating predicted noises as sequential semantic editors. This method focuses on manipulating the predicted noise, encompassing diffusion semantic optimization and propagation, to achieve stable and efficient editing. **Methodology:** 1. **Diffusion Semantic Optimization:** The method optimizes the bottleneck feature of the U-Net to reflect user edits, using a semantic alignment loss and a semantic masking loss. 2. **Diffusion Semantic Propagation:** Optimized bottleneck features are copied and substituted in subsequent timesteps to propagate the editing effect, ensuring stability and efficiency. **Experiments:** - **Qualitative Evaluations:** DragNoise outperforms existing methods in terms of semantic control and image fidelity. - **Optimization Efficiency:** DragNoise reduces optimization time by over 50% compared to DragDiffusion. - **Ablation Study:** Experiments validate the effectiveness of different initial timesteps, feature layers, and editing extents. **Conclusion:** DragNoise is a novel interactive point-based image editing method that leverages diffusion semantic propagation. It offers superior editing accuracy and fidelity, making it a promising approach for real-world applications. Future work should focus on improving handling real images and expanding editing capabilities for tasks requiring a global perspective.**Drag Your Noise: Interactive Point-based Editing via Diffusion Semantic Propagation** **Authors:** Haofeng Liu, Chenshu Xu, Yifei Yang, Lihua Zeng, Shengfeng He **Institution:** Singapore Management University, South China Normal University **Abstract:** Point-based interactive editing is crucial for enhancing the controllability of generative models. While DragDiffusion updates the diffusion latent map in response to user inputs, it often leads to global latent map alterations, resulting in imprecise content preservation and unsuccessful editing due to gradient vanishing. In contrast, DragNoise offers robust and accelerated editing without retracing the latent map. The core idea of DragNoise is to use the predicted noise output of each U-Net as a semantic editor. This approach leverages two key observations: (1) the bottleneck features of U-Nets inherently possess rich semantic information suitable for interactive editing, and (2) high-level semantics established early in the denoising process show minimal variation in subsequent stages. By editing diffusion semantics in a single denoising step and efficiently propagating these changes, DragNoise ensures stability and efficiency in diffusion editing. Comparative experiments show that DragNoise achieves superior control and semantic retention, reducing optimization time by over 50% compared to DragDiffusion. **Introduction:** The limited controllability of diffusion models highlights the need for interactive editing in image manipulation. Recent advancements include text-guided editing, stroke-based editing, and exemplar-based methods. DragGAN and DragDiffusion are significant milestones in point-based image editing, but they face challenges such as gradient vanishing and inversion fidelity. DragNoise addresses these issues by leveraging diffusion semantics within the editing mechanism, treating predicted noises as sequential semantic editors. This method focuses on manipulating the predicted noise, encompassing diffusion semantic optimization and propagation, to achieve stable and efficient editing. **Methodology:** 1. **Diffusion Semantic Optimization:** The method optimizes the bottleneck feature of the U-Net to reflect user edits, using a semantic alignment loss and a semantic masking loss. 2. **Diffusion Semantic Propagation:** Optimized bottleneck features are copied and substituted in subsequent timesteps to propagate the editing effect, ensuring stability and efficiency. **Experiments:** - **Qualitative Evaluations:** DragNoise outperforms existing methods in terms of semantic control and image fidelity. - **Optimization Efficiency:** DragNoise reduces optimization time by over 50% compared to DragDiffusion. - **Ablation Study:** Experiments validate the effectiveness of different initial timesteps, feature layers, and editing extents. **Conclusion:** DragNoise is a novel interactive point-based image editing method that leverages diffusion semantic propagation. It offers superior editing accuracy and fidelity, making it a promising approach for real-world applications. Future work should focus on improving handling real images and expanding editing capabilities for tasks requiring a global perspective.