10 Mar 2024 | Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, Adams Wai-Kin Kong
MACE: Mass Concept Erasure in Diffusion Models
MACE is a framework for erasing multiple concepts from text-to-image (T2I) diffusion models. It addresses the challenge of removing unwanted concepts while maintaining the ability to generate images of unrelated concepts. MACE achieves this by refining cross-attention mechanisms and using LoRA (Low-Rank Adaptation) to remove the intrinsic information of target concepts. The framework is designed to handle up to 100 concepts simultaneously, outperforming previous methods in terms of generality and specificity.
The core of MACE lies in its closed-form cross-attention refinement, which prevents the model from embedding residual information of the target phrase into other words. This is followed by the use of distinct LoRA modules for each concept to eliminate their intrinsic information. The framework also integrates multiple LoRA modules without interference, ensuring that the model forgets a wide array of concepts.
MACE was evaluated on four tasks: object erasure, celebrity erasure, explicit content erasure, and artistic style erasure. Results show that MACE outperforms existing methods in all tasks, demonstrating superior performance in erasing mass concepts and striking an effective balance between specificity and generality.
The framework is capable of erasing different types of concepts simultaneously, such as celebrity likenesses and artistic styles. It is also compatible with distilled diffusion models, such as the Latent Consistency Model (LCM).
MACE offers an effective solution for erasing mass concepts from T2I diffusion models, achieving a remarkable balance between specificity and generality. However, a discernible decline in performance is observed as the number of erased concepts increases. Future research could focus on further scaling up the erasure scope. MACE is a pivotal tool for generative model service providers, enabling them to efficiently eliminate a variety of unwanted concepts. This is a vital step in releasing the next wave of advanced models, contributing to the creation of a safer AI community.MACE: Mass Concept Erasure in Diffusion Models
MACE is a framework for erasing multiple concepts from text-to-image (T2I) diffusion models. It addresses the challenge of removing unwanted concepts while maintaining the ability to generate images of unrelated concepts. MACE achieves this by refining cross-attention mechanisms and using LoRA (Low-Rank Adaptation) to remove the intrinsic information of target concepts. The framework is designed to handle up to 100 concepts simultaneously, outperforming previous methods in terms of generality and specificity.
The core of MACE lies in its closed-form cross-attention refinement, which prevents the model from embedding residual information of the target phrase into other words. This is followed by the use of distinct LoRA modules for each concept to eliminate their intrinsic information. The framework also integrates multiple LoRA modules without interference, ensuring that the model forgets a wide array of concepts.
MACE was evaluated on four tasks: object erasure, celebrity erasure, explicit content erasure, and artistic style erasure. Results show that MACE outperforms existing methods in all tasks, demonstrating superior performance in erasing mass concepts and striking an effective balance between specificity and generality.
The framework is capable of erasing different types of concepts simultaneously, such as celebrity likenesses and artistic styles. It is also compatible with distilled diffusion models, such as the Latent Consistency Model (LCM).
MACE offers an effective solution for erasing mass concepts from T2I diffusion models, achieving a remarkable balance between specificity and generality. However, a discernible decline in performance is observed as the number of erased concepts increases. Future research could focus on further scaling up the erasure scope. MACE is a pivotal tool for generative model service providers, enabling them to efficiently eliminate a variety of unwanted concepts. This is a vital step in releasing the next wave of advanced models, contributing to the creation of a safer AI community.