PUZZLEVQA is a dataset of 2000 puzzle instances based on abstract patterns, designed to evaluate and diagnose the reasoning challenges of large multimodal models. The dataset includes puzzles that focus on fundamental concepts such as numbers, colors, shapes, and size. Each puzzle is formulated with components including objects, layout, pattern, demonstrations, and a query. The dataset is automatically constructed using multimodal templates and includes reasoning explanations for interpretability. The puzzles are presented in a multiple-choice format, with four options for most questions and three for size-related questions. The dataset was evaluated on state-of-the-art large multimodal models, including GPT-4V, which achieved a score of 46.4% on single-concept puzzles. The analysis revealed that the main bottlenecks for GPT-4V are weaker visual perception and inductive reasoning abilities. The dataset also includes dual-concept puzzles, which require models to relate two concepts to solve the puzzle. The results show that GPT-4V performs well on single-concept puzzles but struggles with dual-concept puzzles, indicating that models have difficulty with reasoning about multiple abstract concepts. The dataset was compared to human performance, with a study involving 23 university students achieving an average score of 91.6%. GPT-4V scored 47.5% on the same set of puzzles, highlighting the specific bottlenecks causing models to fall short of human cognition: primarily in visual perception and inductive reasoning. The dataset also explores the effect of few-shot demonstrations on model performance, showing that models generally achieve their best performance with the most number of demonstrations. This suggests that models are capable of analogical reasoning and that in-context learning may be a promising direction to enhance the abstract reasoning abilities of multimodal models in the future. The dataset provides a systematic analysis of multimodal reasoning through abstract patterns, including perceptual, inductive, and deductive reasoning. It includes ground truth answers, image captions, and pattern explanations that enable more detailed and systematic diagnosis of the reasoning bottlenecks for large multimodal models.PUZZLEVQA is a dataset of 2000 puzzle instances based on abstract patterns, designed to evaluate and diagnose the reasoning challenges of large multimodal models. The dataset includes puzzles that focus on fundamental concepts such as numbers, colors, shapes, and size. Each puzzle is formulated with components including objects, layout, pattern, demonstrations, and a query. The dataset is automatically constructed using multimodal templates and includes reasoning explanations for interpretability. The puzzles are presented in a multiple-choice format, with four options for most questions and three for size-related questions. The dataset was evaluated on state-of-the-art large multimodal models, including GPT-4V, which achieved a score of 46.4% on single-concept puzzles. The analysis revealed that the main bottlenecks for GPT-4V are weaker visual perception and inductive reasoning abilities. The dataset also includes dual-concept puzzles, which require models to relate two concepts to solve the puzzle. The results show that GPT-4V performs well on single-concept puzzles but struggles with dual-concept puzzles, indicating that models have difficulty with reasoning about multiple abstract concepts. The dataset was compared to human performance, with a study involving 23 university students achieving an average score of 91.6%. GPT-4V scored 47.5% on the same set of puzzles, highlighting the specific bottlenecks causing models to fall short of human cognition: primarily in visual perception and inductive reasoning. The dataset also explores the effect of few-shot demonstrations on model performance, showing that models generally achieve their best performance with the most number of demonstrations. This suggests that models are capable of analogical reasoning and that in-context learning may be a promising direction to enhance the abstract reasoning abilities of multimodal models in the future. The dataset provides a systematic analysis of multimodal reasoning through abstract patterns, including perceptual, inductive, and deductive reasoning. It includes ground truth answers, image captions, and pattern explanations that enable more detailed and systematic diagnosis of the reasoning bottlenecks for large multimodal models.