AlphaMath Almost Zero: Process Supervision Without Process This paper introduces AlphaMath, a novel approach to enhance mathematical reasoning in large language models (LLMs) without requiring process annotations. The method leverages the Monte Carlo Tree Search (MCTS) framework to automatically generate both process supervision and step-level evaluation signals. By iteratively training policy and value models, the approach enables the LLM to progressively improve its mathematical reasoning skills. The value model assists the policy model (LLM) in navigating more effective reasoning paths, rather than relying solely on prior probabilities. Experimental results on both in-domain and out-of-domain datasets show that AlphaMath achieves comparable or superior results to previous state-of-the-art methods, even without GPT-4 or human-annotated process supervision. The code for the method is available at https://github.com/MARIO-Math-Reasoning/Super_MARIO. The paper discusses the challenges of mathematical reasoning in LLMs, including the difficulty of identifying logical errors in intermediate steps and the high cost of manually annotating these steps. It proposes an efficient inference strategy—step-level beam search—where the value model helps the policy model (LLM) navigate more effective reasoning paths. The method is evaluated on various datasets, including GSM8K, MATH, GaoKao2023, and OCWCourses, demonstrating its effectiveness in both in-domain and out-of-domain settings. The results show that AlphaMath outperforms existing methods in challenging problems and achieves competitive results in grade school math problems. The paper also discusses the performance of different inference strategies, including greedy decoding, step-level beam search, and MCTS. It highlights the advantages of step-level beam search in terms of computational efficiency and the ability to provide streaming outputs. The value model plays a crucial role in facilitating mathematical reasoning by providing feedback on intermediate steps. The paper further explores the potential of AlphaMath in enhancing the mathematical reasoning capabilities of other LLMs, including general-purpose and SFT models. The paper concludes that AlphaMath demonstrates the potential of LLMs to autonomously enhance their mathematical reasoning capabilities without relying on process annotations. The method is effective in both in-domain and out-of-domain settings and shows promise for broader applications beyond mathematical reasoning. The results indicate that AlphaMath can achieve competitive or superior performance to state-of-the-art methods, even without GPT-4 or human-annotated process supervision. The code for the method is available at https://github.com/MARIO-Math-Reasoning/Super_MARIO.AlphaMath Almost Zero: Process Supervision Without Process This paper introduces AlphaMath, a novel approach to enhance mathematical reasoning in large language models (LLMs) without requiring process annotations. The method leverages the Monte Carlo Tree Search (MCTS) framework to automatically generate both process supervision and step-level evaluation signals. By iteratively training policy and value models, the approach enables the LLM to progressively improve its mathematical reasoning skills. The value model assists the policy model (LLM) in navigating more effective reasoning paths, rather than relying solely on prior probabilities. Experimental results on both in-domain and out-of-domain datasets show that AlphaMath achieves comparable or superior results to previous state-of-the-art methods, even without GPT-4 or human-annotated process supervision. The code for the method is available at https://github.com/MARIO-Math-Reasoning/Super_MARIO. The paper discusses the challenges of mathematical reasoning in LLMs, including the difficulty of identifying logical errors in intermediate steps and the high cost of manually annotating these steps. It proposes an efficient inference strategy—step-level beam search—where the value model helps the policy model (LLM) navigate more effective reasoning paths. The method is evaluated on various datasets, including GSM8K, MATH, GaoKao2023, and OCWCourses, demonstrating its effectiveness in both in-domain and out-of-domain settings. The results show that AlphaMath outperforms existing methods in challenging problems and achieves competitive results in grade school math problems. The paper also discusses the performance of different inference strategies, including greedy decoding, step-level beam search, and MCTS. It highlights the advantages of step-level beam search in terms of computational efficiency and the ability to provide streaming outputs. The value model plays a crucial role in facilitating mathematical reasoning by providing feedback on intermediate steps. The paper further explores the potential of AlphaMath in enhancing the mathematical reasoning capabilities of other LLMs, including general-purpose and SFT models. The paper concludes that AlphaMath demonstrates the potential of LLMs to autonomously enhance their mathematical reasoning capabilities without relying on process annotations. The method is effective in both in-domain and out-of-domain settings and shows promise for broader applications beyond mathematical reasoning. The results indicate that AlphaMath can achieve competitive or superior performance to state-of-the-art methods, even without GPT-4 or human-annotated process supervision. The code for the method is available at https://github.com/MARIO-Math-Reasoning/Super_MARIO.