Beyond Aesthetics: Cultural Competence in Text-to-Image Models

Beyond Aesthetics: Cultural Competence in Text-to-Image Models

2025-01-20 | Nithish Kannen, Arif Ahmad, Marco Andreetto, Vinodkumar Prabhakaran, Utsav Prabhu, Adji Bousso Dieng, Pushpak Bhattacharyya, Shachi Dave
This paper introduces CUBE, a new benchmark for evaluating the cultural competence of text-to-image (T2I) models. The benchmark assesses two key dimensions: cultural awareness and cultural diversity. Cultural awareness refers to the model's ability to accurately represent cultural concepts, while cultural diversity refers to the model's ability to generate diverse cultural artifacts across different regions and concepts. CUBE is built using a combination of structured knowledge bases and large language models to create a large dataset of cultural artifacts. The benchmark includes CUBE-1K, a set of 1000 prompts for evaluating cultural awareness, and CUBE-CSpace, a larger dataset of cultural artifacts for evaluating cultural diversity. The paper also introduces a new evaluation component for T2I models, cultural diversity, which leverages the quality-weighted Vendi score. The results show significant gaps in cultural awareness across different countries and concepts, and the need for more diverse and inclusive T2I models. The paper also discusses the ethical considerations of using T2I models and the importance of cultural competence in AI systems.This paper introduces CUBE, a new benchmark for evaluating the cultural competence of text-to-image (T2I) models. The benchmark assesses two key dimensions: cultural awareness and cultural diversity. Cultural awareness refers to the model's ability to accurately represent cultural concepts, while cultural diversity refers to the model's ability to generate diverse cultural artifacts across different regions and concepts. CUBE is built using a combination of structured knowledge bases and large language models to create a large dataset of cultural artifacts. The benchmark includes CUBE-1K, a set of 1000 prompts for evaluating cultural awareness, and CUBE-CSpace, a larger dataset of cultural artifacts for evaluating cultural diversity. The paper also introduces a new evaluation component for T2I models, cultural diversity, which leverages the quality-weighted Vendi score. The results show significant gaps in cultural awareness across different countries and concepts, and the need for more diverse and inclusive T2I models. The paper also discusses the ethical considerations of using T2I models and the importance of cultural competence in AI systems.
Reach us at info@futurestudyspace.com
[slides and audio] Beyond Aesthetics%3A Cultural Competence in Text-to-Image Models