This paper introduces a novel approach to scalable AI alignment called "Easy-to-Hard Generalization," which enables AI systems to solve complex tasks without direct human supervision. The method leverages reward models trained on simpler tasks to evaluate and improve the performance of policy models on harder tasks. By training evaluators on easier tasks, the system can effectively score solutions to harder tasks, facilitating the transfer of knowledge from easier to harder problems. This approach is demonstrated through experiments on mathematical problem-solving tasks, where evaluators trained on easier tasks significantly improved the performance of generators on harder tasks. The study shows that using process-supervised and outcome-supervised reward models can enhance the generalization ability of AI systems, allowing them to solve complex problems beyond the scope of human supervision. The results indicate that this method outperforms traditional training methods and offers a promising path toward AI systems that can surpass human capabilities in problem-solving. The paper also explores the effectiveness of reinforcement learning and re-ranking strategies in improving the performance of AI systems on complex tasks. Overall, the research highlights the potential of easy-to-hard generalization as a scalable and effective approach to AI alignment.This paper introduces a novel approach to scalable AI alignment called "Easy-to-Hard Generalization," which enables AI systems to solve complex tasks without direct human supervision. The method leverages reward models trained on simpler tasks to evaluate and improve the performance of policy models on harder tasks. By training evaluators on easier tasks, the system can effectively score solutions to harder tasks, facilitating the transfer of knowledge from easier to harder problems. This approach is demonstrated through experiments on mathematical problem-solving tasks, where evaluators trained on easier tasks significantly improved the performance of generators on harder tasks. The study shows that using process-supervised and outcome-supervised reward models can enhance the generalization ability of AI systems, allowing them to solve complex problems beyond the scope of human supervision. The results indicate that this method outperforms traditional training methods and offers a promising path toward AI systems that can surpass human capabilities in problem-solving. The paper also explores the effectiveness of reinforcement learning and re-ranking strategies in improving the performance of AI systems on complex tasks. Overall, the research highlights the potential of easy-to-hard generalization as a scalable and effective approach to AI alignment.