FairRAG: Fair Human Generation via Fair Retrieval Augmentation

FairRAG: Fair Human Generation via Fair Retrieval Augmentation

5 Apr 2024 | Robik Shrestha, Yang Zou, Qiuyu Chen, Zhiheng Li, Yusheng Xie, Siqi Deng
FairRAG is a novel framework that improves fairness in human image generation by conditioning pre-trained generative models on reference images retrieved from an external database. The framework uses a lightweight linear module to project reference images into the textual space, enabling the model to generate images with improved demographic diversity. FairRAG applies simple yet effective debiasing strategies during the generative process, ensuring that images are drawn from diverse demographic groups. The framework enhances fairness by using a fair retrieval system that samples from diverse demographic groups, making it more steerable, explainable, and transparent in controlling demographic distributions. Extensive experiments show that FairRAG outperforms existing methods in terms of demographic diversity, image-text alignment, and image fidelity while incurring minimal computational overhead during inference. The framework is compared against multiple methods, including SDv2.1, Ethical Interventions, Fair Diffusion, Text Augmentation, and Baseline RAG. Results show that FairRAG improves diversity metrics for all three demographic attributes: age, gender, and skin tone. The framework also generates more realistic images due to the conditioning from real images. FairRAG is efficient and has minimal latency, making it suitable for practical applications. The framework is limited to human image generation and cannot tackle non-human prompts. Future work includes extending the framework to other domains and improving the transfer of attributes. The paper concludes that FairRAG is an effective framework for improving fairness in human image generation.FairRAG is a novel framework that improves fairness in human image generation by conditioning pre-trained generative models on reference images retrieved from an external database. The framework uses a lightweight linear module to project reference images into the textual space, enabling the model to generate images with improved demographic diversity. FairRAG applies simple yet effective debiasing strategies during the generative process, ensuring that images are drawn from diverse demographic groups. The framework enhances fairness by using a fair retrieval system that samples from diverse demographic groups, making it more steerable, explainable, and transparent in controlling demographic distributions. Extensive experiments show that FairRAG outperforms existing methods in terms of demographic diversity, image-text alignment, and image fidelity while incurring minimal computational overhead during inference. The framework is compared against multiple methods, including SDv2.1, Ethical Interventions, Fair Diffusion, Text Augmentation, and Baseline RAG. Results show that FairRAG improves diversity metrics for all three demographic attributes: age, gender, and skin tone. The framework also generates more realistic images due to the conditioning from real images. FairRAG is efficient and has minimal latency, making it suitable for practical applications. The framework is limited to human image generation and cannot tackle non-human prompts. Future work includes extending the framework to other domains and improving the transfer of attributes. The paper concludes that FairRAG is an effective framework for improving fairness in human image generation.
Reach us at info@study.space
Understanding FairRAG%3A Fair Human Generation via Fair Retrieval Augmentation