The paper introduces AlphaMath, an innovative approach to enhance the mathematical reasoning capabilities of large language models (LLMs) without relying on process annotations from humans or GPT-4. The method leverages Monte Carlo Tree Search (MCTS) to automatically generate process supervision and step-level evaluation signals, iteratively training the policy and value models. The value model assists the policy model in navigating more effective reasoning paths, enhancing the LLM's ability to solve complex mathematical problems. Experimental results on both in-domain and out-of-domain datasets demonstrate that AlphaMath achieves comparable or superior performance to state-of-the-art methods, even without high-quality annotated solutions. The approach is efficient and scalable, making it suitable for practical applications.The paper introduces AlphaMath, an innovative approach to enhance the mathematical reasoning capabilities of large language models (LLMs) without relying on process annotations from humans or GPT-4. The method leverages Monte Carlo Tree Search (MCTS) to automatically generate process supervision and step-level evaluation signals, iteratively training the policy and value models. The value model assists the policy model in navigating more effective reasoning paths, enhancing the LLM's ability to solve complex mathematical problems. Experimental results on both in-domain and out-of-domain datasets demonstrate that AlphaMath achieves comparable or superior performance to state-of-the-art methods, even without high-quality annotated solutions. The approach is efficient and scalable, making it suitable for practical applications.