**FairRAG: Fair Human Generation via Fair Retrieval Augmentation** This paper introduces FairRAG, a novel framework designed to improve demographic diversity in human image generation by conditioning pre-trained generative models on reference images from an external database. FairRAG addresses the issue of societal biases in text-to-image models, which often reflect or amplify biases present in their training data, particularly against specific demographic groups. The framework uses a lightweight linear module to project reference images into the textual space, enabling conditioning on diverse demographic groups. To enhance fairness, FairRAG employs simple yet effective debiasing strategies, such as balanced sampling and query modification, during the generative process. **Key Contributions:** - **FairRAG Framework:** A novel framework that improves demographic diversity in human image generation by conditioning on external reference images. - **Lightweight Conditioning:** Utilizes a linear projector to condition the frozen backbone model, avoiding computational overhead. - **Fair Retrieval System:** Implementes post-hoc debiasing techniques to ensure demographic diversity in retrieved images. - **Experimental Results:** Significantly outperforms existing methods in terms of demographic diversity, image-text alignment, and image fidelity while maintaining minimal computational overhead. **Related Work:** - **Societal Biases in Diffusion Models:** Discusses how diffusion-based models can inherit biases from their training data. - **Conditioning Text-to-Image Diffusion Models:** Reviews existing approaches to conditioning on visual references, including tuning-free methods and heavy adaptor modules. - **Retrieval Augmented Generation (RAG):** Explains how RAG methods retrieve relevant items from external sources to condition the generative process. **FairRAG Architecture:** - **Linear Conditioning Mechanism:** Trains a linear projector to condition the frozen backbone model using reference images. - **Fair Retrieval System:** Uses debiased queries and balanced sampling to ensure demographic diversity in retrieved images. - **Image Generation:** Enhances attribute transfer through a text instruction during the generative process. **Experiments:** - **Setup:** Evaluates FairRAG against baselines using neutral text prompts that exhibit bias. - **Results:** Demonstrates significant improvements in demographic diversity, alignment, and fidelity compared to other methods. - **Ablation Study:** Validates the effectiveness of each component of FairRAG through ablation experiments. **Limitations and Future Directions:** - **One-to-One Image Mapping:** Suggests the use of multiple images for better conditioning. - **Disfigurements:** Highlights the need for incorporating knowledge of human anatomy to improve image quality. - **Generalization:** Discusses the potential for extending FairRAG to other domains and handling non-human prompts. **Conclusion:** FairRAG effectively addresses the issue of societal biases in text-to-image models by leveraging external reference images. The lightweight and efficient approach enhances demographic diversity, alignment, and fidelity, making it a promising solution for fair human image generation.**FairRAG: Fair Human Generation via Fair Retrieval Augmentation** This paper introduces FairRAG, a novel framework designed to improve demographic diversity in human image generation by conditioning pre-trained generative models on reference images from an external database. FairRAG addresses the issue of societal biases in text-to-image models, which often reflect or amplify biases present in their training data, particularly against specific demographic groups. The framework uses a lightweight linear module to project reference images into the textual space, enabling conditioning on diverse demographic groups. To enhance fairness, FairRAG employs simple yet effective debiasing strategies, such as balanced sampling and query modification, during the generative process. **Key Contributions:** - **FairRAG Framework:** A novel framework that improves demographic diversity in human image generation by conditioning on external reference images. - **Lightweight Conditioning:** Utilizes a linear projector to condition the frozen backbone model, avoiding computational overhead. - **Fair Retrieval System:** Implementes post-hoc debiasing techniques to ensure demographic diversity in retrieved images. - **Experimental Results:** Significantly outperforms existing methods in terms of demographic diversity, image-text alignment, and image fidelity while maintaining minimal computational overhead. **Related Work:** - **Societal Biases in Diffusion Models:** Discusses how diffusion-based models can inherit biases from their training data. - **Conditioning Text-to-Image Diffusion Models:** Reviews existing approaches to conditioning on visual references, including tuning-free methods and heavy adaptor modules. - **Retrieval Augmented Generation (RAG):** Explains how RAG methods retrieve relevant items from external sources to condition the generative process. **FairRAG Architecture:** - **Linear Conditioning Mechanism:** Trains a linear projector to condition the frozen backbone model using reference images. - **Fair Retrieval System:** Uses debiased queries and balanced sampling to ensure demographic diversity in retrieved images. - **Image Generation:** Enhances attribute transfer through a text instruction during the generative process. **Experiments:** - **Setup:** Evaluates FairRAG against baselines using neutral text prompts that exhibit bias. - **Results:** Demonstrates significant improvements in demographic diversity, alignment, and fidelity compared to other methods. - **Ablation Study:** Validates the effectiveness of each component of FairRAG through ablation experiments. **Limitations and Future Directions:** - **One-to-One Image Mapping:** Suggests the use of multiple images for better conditioning. - **Disfigurements:** Highlights the need for incorporating knowledge of human anatomy to improve image quality. - **Generalization:** Discusses the potential for extending FairRAG to other domains and handling non-human prompts. **Conclusion:** FairRAG effectively addresses the issue of societal biases in text-to-image models by leveraging external reference images. The lightweight and efficient approach enhances demographic diversity, alignment, and fidelity, making it a promising solution for fair human image generation.