ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback **Project Page:** liming-ai.github.io/ControlNet_Plus_Plus **Authors:** Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, Chen Chen **Institutions:** Center for Research in Computer Vision, University of Central Florida; ByteDance To enhance the controllability of text-to-image diffusion models, existing methods like ControlNet face significant challenges in generating images that align with image-based conditional controls. This paper introduces ControlNet++, a novel approach that improves controllable generation by explicitly optimizing pixel-level cycle consistency between generated images and conditional controls. Specifically, pre-trained discriminative reward models extract conditions from generated images, and the consistency loss between the input condition and extracted condition is optimized. To address efficiency issues, an efficient reward strategy is introduced, where noise is added to input images, and single-step denoised images are used for reward fine-tuning, avoiding extensive sampling costs. **Contributions:** - New Insight: Reveals that existing methods still struggle with controllability, with generated images deviating significantly from input conditions. - Consistency Reward Feedback: Shows that pre-trained discriminative models can improve controllability through cycle-consistency optimization. - Efficient Reward Fine-tuning: Introduces a method to disrupt consistency between input images and conditions, enabling single-step denoising for efficient reward fine-tuning. - Evaluation and Promising Results: Provides a comprehensive evaluation under various conditional controls, demonstrating significant improvements over existing methods. **Methods:** - **Preliminary:** Describes the diffusion model's training and inference processes. - **Reward Controllability with Consistency Feedback:** Formulates the consistency loss between input and extracted conditions. - **Efficient Reward Fine-tuning:** Proposes a method to calculate consistency loss using single-step denoising, avoiding sampling costs. **Experiments:** - **Experimental Setup:** Details the datasets, evaluation metrics, and baselines used. - **Controllability Comparison:** Shows significant improvements over existing methods in various conditional controls. - **Image Quality Comparison:** Demonstrates that ControlNet++ maintains or improves image quality while enhancing controllability. - **Qualitative Comparison:** Provides visual examples of generated images, showing better consistency with input conditions. **Ablation Study:** - **Loss Settings:** Analyzes the impact of different loss settings on controllability and image quality. - **Generalizability of Efficient Reward Fine-tuning:** Shows that efficient reward fine-tuning generalizes well to larger timesteps. - **Choice of Different Reward Models:** Evaluates the effectiveness of different reward models. **Discussion:** - **Conditions Expansion:** Plans to incorporate additional control conditions. - **Beyond Controllability:** Aims to enhance both controllability and aesthetic appeal using human feedback. - **Joint Optimization:** IntendsControlNet++: Improving Conditional Controls with Efficient Consistency Feedback **Project Page:** liming-ai.github.io/ControlNet_Plus_Plus **Authors:** Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, Chen Chen **Institutions:** Center for Research in Computer Vision, University of Central Florida; ByteDance To enhance the controllability of text-to-image diffusion models, existing methods like ControlNet face significant challenges in generating images that align with image-based conditional controls. This paper introduces ControlNet++, a novel approach that improves controllable generation by explicitly optimizing pixel-level cycle consistency between generated images and conditional controls. Specifically, pre-trained discriminative reward models extract conditions from generated images, and the consistency loss between the input condition and extracted condition is optimized. To address efficiency issues, an efficient reward strategy is introduced, where noise is added to input images, and single-step denoised images are used for reward fine-tuning, avoiding extensive sampling costs. **Contributions:** - New Insight: Reveals that existing methods still struggle with controllability, with generated images deviating significantly from input conditions. - Consistency Reward Feedback: Shows that pre-trained discriminative models can improve controllability through cycle-consistency optimization. - Efficient Reward Fine-tuning: Introduces a method to disrupt consistency between input images and conditions, enabling single-step denoising for efficient reward fine-tuning. - Evaluation and Promising Results: Provides a comprehensive evaluation under various conditional controls, demonstrating significant improvements over existing methods. **Methods:** - **Preliminary:** Describes the diffusion model's training and inference processes. - **Reward Controllability with Consistency Feedback:** Formulates the consistency loss between input and extracted conditions. - **Efficient Reward Fine-tuning:** Proposes a method to calculate consistency loss using single-step denoising, avoiding sampling costs. **Experiments:** - **Experimental Setup:** Details the datasets, evaluation metrics, and baselines used. - **Controllability Comparison:** Shows significant improvements over existing methods in various conditional controls. - **Image Quality Comparison:** Demonstrates that ControlNet++ maintains or improves image quality while enhancing controllability. - **Qualitative Comparison:** Provides visual examples of generated images, showing better consistency with input conditions. **Ablation Study:** - **Loss Settings:** Analyzes the impact of different loss settings on controllability and image quality. - **Generalizability of Efficient Reward Fine-tuning:** Shows that efficient reward fine-tuning generalizes well to larger timesteps. - **Choice of Different Reward Models:** Evaluates the effectiveness of different reward models. **Discussion:** - **Conditions Expansion:** Plans to incorporate additional control conditions. - **Beyond Controllability:** Aims to enhance both controllability and aesthetic appeal using human feedback. - **Joint Optimization:** Intends