The paper introduces the MathOdyssey dataset, a new benchmark for evaluating the mathematical problem-solving abilities of large language models (LLMs). The dataset includes a diverse range of mathematical problems at different levels, from Olympiad-level to university-level, created by experts from various institutions. It aims to provide a comprehensive and rigorous test for LLMs, covering a wide range of mathematical subjects and problem types. The dataset is publicly available, and its results, along with the code, are also open for use. The study evaluates several LLMs, including both open-source and closed-source models, such as GPT-4, Gemini, Llama-3, and DBRX-Instruct. The results show that while LLMs perform well on routine and moderately difficult tasks, they face significant challenges with Olympiad-level problems and complex university-level questions. The performance gap between open-source and closed-source models is narrowing, but substantial challenges remain, particularly with the most demanding problems. The MathOdyssey dataset includes three types of problems: Olympiad-level, High School, and University-level. Each problem is accompanied by an answer and a detailed reasoning process. The dataset is designed to test the reasoning abilities of LLMs, providing a unique tool for assessing AI performance in complex mathematical reasoning. The dataset includes a variety of answer types, such as True-False, Multiple-Choice, and Open-Answer, ensuring a well-rounded evaluation of LLMs' mathematical capabilities. The study highlights the ongoing need for research to enhance the mathematical reasoning of LLMs. The dataset, results, and code are publicly available, allowing researchers to replicate the study, compare methods, and explore new approaches. The findings indicate that while closed-source models currently lead, open-source models are rapidly catching up, highlighting the competitive landscape of LLM capabilities in mathematical problem-solving. The research underscores the importance of continued efforts to improve the mathematical reasoning of AI systems, with the MathOdyssey dataset serving as a benchmark for future developments.The paper introduces the MathOdyssey dataset, a new benchmark for evaluating the mathematical problem-solving abilities of large language models (LLMs). The dataset includes a diverse range of mathematical problems at different levels, from Olympiad-level to university-level, created by experts from various institutions. It aims to provide a comprehensive and rigorous test for LLMs, covering a wide range of mathematical subjects and problem types. The dataset is publicly available, and its results, along with the code, are also open for use. The study evaluates several LLMs, including both open-source and closed-source models, such as GPT-4, Gemini, Llama-3, and DBRX-Instruct. The results show that while LLMs perform well on routine and moderately difficult tasks, they face significant challenges with Olympiad-level problems and complex university-level questions. The performance gap between open-source and closed-source models is narrowing, but substantial challenges remain, particularly with the most demanding problems. The MathOdyssey dataset includes three types of problems: Olympiad-level, High School, and University-level. Each problem is accompanied by an answer and a detailed reasoning process. The dataset is designed to test the reasoning abilities of LLMs, providing a unique tool for assessing AI performance in complex mathematical reasoning. The dataset includes a variety of answer types, such as True-False, Multiple-Choice, and Open-Answer, ensuring a well-rounded evaluation of LLMs' mathematical capabilities. The study highlights the ongoing need for research to enhance the mathematical reasoning of LLMs. The dataset, results, and code are publicly available, allowing researchers to replicate the study, compare methods, and explore new approaches. The findings indicate that while closed-source models currently lead, open-source models are rapidly catching up, highlighting the competitive landscape of LLM capabilities in mathematical problem-solving. The research underscores the importance of continued efforts to improve the mathematical reasoning of AI systems, with the MathOdyssey dataset serving as a benchmark for future developments.