How Far Are We from Intelligent Visual Deductive Reasoning? Yizhe Zhang, He Bai, Ruixiang Zhang, Jiatao Gu, Shuangfei Zhai, Josh Susskind, Navdeep Jaitly from Apple investigate the capabilities of Vision-Language Models (VLMs) in visual deductive reasoning, particularly through Raven's Progressive Matrices (RPMs). They evaluate several popular VLMs on three datasets: Mensa IQ test, IntelligenceTest, and RAVEN. The results show that while LLMs excel in text-based reasoning, VLMs lag significantly in visual reasoning tasks. VLMs struggle to perceive and understand complex abstract patterns in RPM examples, and standard strategies effective for LLMs do not translate well to visual reasoning. The study highlights that VLMs have a perceptual limitation, and their performance is hindered by overconfidence, sensitivity to prompt design, and inability to leverage in-context examples effectively. The analysis reveals that VLMs can benefit from more structured prompts and that visual cues are crucial for complex spatial reasoning. The study also shows that VLMs can improve performance when provided with oracle descriptions, but they still fall short of human-level performance. The research underscores the need for further development in visual deductive reasoning capabilities for VLMs.How Far Are We from Intelligent Visual Deductive Reasoning? Yizhe Zhang, He Bai, Ruixiang Zhang, Jiatao Gu, Shuangfei Zhai, Josh Susskind, Navdeep Jaitly from Apple investigate the capabilities of Vision-Language Models (VLMs) in visual deductive reasoning, particularly through Raven's Progressive Matrices (RPMs). They evaluate several popular VLMs on three datasets: Mensa IQ test, IntelligenceTest, and RAVEN. The results show that while LLMs excel in text-based reasoning, VLMs lag significantly in visual reasoning tasks. VLMs struggle to perceive and understand complex abstract patterns in RPM examples, and standard strategies effective for LLMs do not translate well to visual reasoning. The study highlights that VLMs have a perceptual limitation, and their performance is hindered by overconfidence, sensitivity to prompt design, and inability to leverage in-context examples effectively. The analysis reveals that VLMs can benefit from more structured prompts and that visual cues are crucial for complex spatial reasoning. The study also shows that VLMs can improve performance when provided with oracle descriptions, but they still fall short of human-level performance. The research underscores the need for further development in visual deductive reasoning capabilities for VLMs.