This paper introduces BiasPainter, a novel evaluation framework for measuring social bias in image generation models. Image generation models, such as DALL-E and Midjourney, can generate or edit images based on text prompts. However, these models are often trained on large datasets that contain social stereotypes and biases, leading to the generation of biased content. Previous methods for evaluating bias in image generation models have limitations, including low accuracy, reliance on human labor, and lack of comprehensive analysis. BiasPainter is designed to automatically and comprehensively detect social bias in image generation models. It uses a diverse set of seed images and prompts to test how models edit these images. The prompts are neutral in terms of gender, race, and age, and cover various professions, activities, objects, and personality traits. The framework then compares the edited images to the original seed images to identify significant changes related to gender, race, and age. The key insight is that these characteristics should not change under neutral prompts. BiasPainter evaluates six widely-used image generation models, including Stable Diffusion and Midjourney. The results show that BiasPainter can effectively trigger social bias in these models. According to human evaluation, BiasPainter achieves 90.8% accuracy in detecting bias, which is significantly higher than previous methods. The framework provides valuable insights into the nature and extent of biases in image generation models and can be used to evaluate bias mitigation strategies. The paper also discusses the social implications of biased image generation, including reinforcing stereotypes, damaging brand reputation, and affecting individual well-being. The authors emphasize the importance of measuring and mitigating bias in image generation models to ensure fairness and reduce the impact of social stereotypes. The framework is implemented with a comprehensive evaluation of bias across different demographic groups and provides a dataset, code, and experimental results for further research.This paper introduces BiasPainter, a novel evaluation framework for measuring social bias in image generation models. Image generation models, such as DALL-E and Midjourney, can generate or edit images based on text prompts. However, these models are often trained on large datasets that contain social stereotypes and biases, leading to the generation of biased content. Previous methods for evaluating bias in image generation models have limitations, including low accuracy, reliance on human labor, and lack of comprehensive analysis. BiasPainter is designed to automatically and comprehensively detect social bias in image generation models. It uses a diverse set of seed images and prompts to test how models edit these images. The prompts are neutral in terms of gender, race, and age, and cover various professions, activities, objects, and personality traits. The framework then compares the edited images to the original seed images to identify significant changes related to gender, race, and age. The key insight is that these characteristics should not change under neutral prompts. BiasPainter evaluates six widely-used image generation models, including Stable Diffusion and Midjourney. The results show that BiasPainter can effectively trigger social bias in these models. According to human evaluation, BiasPainter achieves 90.8% accuracy in detecting bias, which is significantly higher than previous methods. The framework provides valuable insights into the nature and extent of biases in image generation models and can be used to evaluate bias mitigation strategies. The paper also discusses the social implications of biased image generation, including reinforcing stereotypes, damaging brand reputation, and affecting individual well-being. The authors emphasize the importance of measuring and mitigating bias in image generation models to ensure fairness and reduce the impact of social stereotypes. The framework is implemented with a comprehensive evaluation of bias across different demographic groups and provides a dataset, code, and experimental results for further research.