This paper introduces CULTURALVQA, a benchmark for evaluating the cultural understanding of Vision Language Models (VLMs). The benchmark consists of 2,378 image-question pairs from 11 countries across five continents, with 1-5 answers per question. The questions cover various aspects of culture, including clothing, food, drinks, rituals, and traditions. The benchmark aims to assess VLMs' ability to understand diverse cultural contexts, which is often overlooked in existing benchmarks focused on general scene understanding. The study evaluates several state-of-the-art VLMs, including GPT-4V and Gemini, on CULTURALVQA. Results show significant disparities in cultural understanding across regions, with stronger performance in North America and lower performance in Africa. VLMs also show varying proficiency across cultural facets, with higher performance on rituals and traditions compared to food and drink. These disparities highlight areas where VLMs lack cultural understanding and demonstrate the potential of CULTURALVQA as a comprehensive evaluation set for gauging VLM progress in understanding diverse cultures. The dataset was created by curating images and questions from culturally knowledgeable annotators, ensuring a broad representation of cultural concepts. The questions were designed to be easily answerable by someone from their own culture but challenging for outsiders. The answers were collected to reflect common agreement within their culture and were written in local languages where appropriate. The benchmarking results reveal a significant performance gap between proprietary and open-source models, with open-source models significantly underperforming. The results also show that VLMs' performance varies across countries, with some models performing better in certain regions than others. The study also highlights the challenges of evaluating cultural understanding, as it requires not only visual understanding but also cultural knowledge. The paper concludes that CULTURALVQA is a valuable benchmark for evaluating VLMs' cultural understanding and highlights the need for further research to improve VLMs' ability to understand diverse cultures. The study also acknowledges the limitations of the dataset, including the potential oversimplification of cultural identities and the use of English-only data, which may miss key cultural nuances available only in native languages.This paper introduces CULTURALVQA, a benchmark for evaluating the cultural understanding of Vision Language Models (VLMs). The benchmark consists of 2,378 image-question pairs from 11 countries across five continents, with 1-5 answers per question. The questions cover various aspects of culture, including clothing, food, drinks, rituals, and traditions. The benchmark aims to assess VLMs' ability to understand diverse cultural contexts, which is often overlooked in existing benchmarks focused on general scene understanding. The study evaluates several state-of-the-art VLMs, including GPT-4V and Gemini, on CULTURALVQA. Results show significant disparities in cultural understanding across regions, with stronger performance in North America and lower performance in Africa. VLMs also show varying proficiency across cultural facets, with higher performance on rituals and traditions compared to food and drink. These disparities highlight areas where VLMs lack cultural understanding and demonstrate the potential of CULTURALVQA as a comprehensive evaluation set for gauging VLM progress in understanding diverse cultures. The dataset was created by curating images and questions from culturally knowledgeable annotators, ensuring a broad representation of cultural concepts. The questions were designed to be easily answerable by someone from their own culture but challenging for outsiders. The answers were collected to reflect common agreement within their culture and were written in local languages where appropriate. The benchmarking results reveal a significant performance gap between proprietary and open-source models, with open-source models significantly underperforming. The results also show that VLMs' performance varies across countries, with some models performing better in certain regions than others. The study also highlights the challenges of evaluating cultural understanding, as it requires not only visual understanding but also cultural knowledge. The paper concludes that CULTURALVQA is a valuable benchmark for evaluating VLMs' cultural understanding and highlights the need for further research to improve VLMs' ability to understand diverse cultures. The study also acknowledges the limitations of the dataset, including the potential oversimplification of cultural identities and the use of English-only data, which may miss key cultural nuances available only in native languages.