The paper "MATHVERSE: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?" by Renrui Zhang et al. addresses the limitations of current benchmarks in evaluating the visual math problem-solving capabilities of Multi-modal Large Language Models (MLLMs). The authors identify three primary issues with existing benchmarks: excessive textual redundancy, lack of equitable evaluation beyond final answers, and insufficient specialization in mathematical reasoning. To address these issues, they introduce MATHVERSE, a comprehensive visual math benchmark that includes 2,612 high-quality math problems with diagrams from various subjects and subfields. Each problem is transformed into six distinct versions with varying degrees of information content, allowing for a detailed assessment of MLLMs' visual comprehension and reasoning skills. The paper proposes a Chain-of-Thought (CoT) evaluation strategy to assess the intermediate reasoning steps of MLLMs, providing a fine-grained analysis of their performance. The evaluation reveals that most existing MLLMs struggle to understand math diagrams, relying heavily on textual questions. Surprisingly, some models achieve higher accuracy without visual input, while others, like GPT-4V and ShareGPT4V, demonstrate better comprehension of visual content. The study highlights the need for more advanced math-specific vision encoders to enhance multi-modal mathematical reasoning in MLLMs. The contributions of the paper include the introduction of MATHVERSE, a holistic and specialized benchmark for evaluating MLLMs' visual mathematical reasoning, and the proposed CoT evaluation strategy for a detailed assessment of intermediate reasoning steps. The findings suggest that inadequate visual interpretation capabilities are a significant barrier for MLLMs in solving multi-modal math problems, indicating substantial potential for improvement.The paper "MATHVERSE: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?" by Renrui Zhang et al. addresses the limitations of current benchmarks in evaluating the visual math problem-solving capabilities of Multi-modal Large Language Models (MLLMs). The authors identify three primary issues with existing benchmarks: excessive textual redundancy, lack of equitable evaluation beyond final answers, and insufficient specialization in mathematical reasoning. To address these issues, they introduce MATHVERSE, a comprehensive visual math benchmark that includes 2,612 high-quality math problems with diagrams from various subjects and subfields. Each problem is transformed into six distinct versions with varying degrees of information content, allowing for a detailed assessment of MLLMs' visual comprehension and reasoning skills. The paper proposes a Chain-of-Thought (CoT) evaluation strategy to assess the intermediate reasoning steps of MLLMs, providing a fine-grained analysis of their performance. The evaluation reveals that most existing MLLMs struggle to understand math diagrams, relying heavily on textual questions. Surprisingly, some models achieve higher accuracy without visual input, while others, like GPT-4V and ShareGPT4V, demonstrate better comprehension of visual content. The study highlights the need for more advanced math-specific vision encoders to enhance multi-modal mathematical reasoning in MLLMs. The contributions of the paper include the introduction of MATHVERSE, a holistic and specialized benchmark for evaluating MLLMs' visual mathematical reasoning, and the proposed CoT evaluation strategy for a detailed assessment of intermediate reasoning steps. The findings suggest that inadequate visual interpretation capabilities are a significant barrier for MLLMs in solving multi-modal math problems, indicating substantial potential for improvement.