**REBEL: Reinforcement Learning via Regressing Relative Rewards** **Authors:** Zhaolin Gao **Abstract:** Proximal Policy Optimization (PPO) has become a dominant algorithm in reinforcement learning (RL), particularly for fine-tuning generative models. However, PPO requires multiple heuristics for stable convergence and is sensitive to implementation details. To address these issues, this paper introduces REBEL, a minimalist RL algorithm that simplifies policy optimization by regressing the relative reward between two completions of a prompt. REBEL eliminates the need for value networks and clipping, making it more lightweight and computationally efficient. Theoretical analysis shows that fundamental RL algorithms like Natural Policy Gradient (NPG) can be seen as variants of REBEL, providing strong guarantees in terms of convergence and sample complexity. Empirical results demonstrate that REBEL outperforms PPO and other baselines in language modeling and image generation tasks, achieving competitive performance on benchmarks such as AlpacaEval 2.0, MT-Bench, and Open LLM Leaderboard. **Key Contributions:** 1. **REBEL Algorithm:** A simple and scalable RL algorithm that reduces policy optimization to solving a sequence of squared loss regression problems. 2. **Theoretical Analysis:** Proves that REBEL is a generalization of NPG and provides strong convergence and regret guarantees. 3. **Empirical Evaluation:** Demonstrates superior performance in language modeling and image generation tasks compared to PPO and other baselines. **Related Work:** - **Policy Gradients:** Explains the limitations of PPO and the benefits of REBEL's approach. - **Reward Regression:** Introduces the novel idea of using regression to fit relative rewards. - **Preference Fine-Tuning (PFT):** Discusses the role of RL in aligning language models with human preferences. **Conclusion:** REBEL offers a simpler and more efficient approach to RL, particularly for fine-tuning generative models, by leveraging regression to relative rewards. Theoretical and empirical results validate its effectiveness and scalability.**REBEL: Reinforcement Learning via Regressing Relative Rewards** **Authors:** Zhaolin Gao **Abstract:** Proximal Policy Optimization (PPO) has become a dominant algorithm in reinforcement learning (RL), particularly for fine-tuning generative models. However, PPO requires multiple heuristics for stable convergence and is sensitive to implementation details. To address these issues, this paper introduces REBEL, a minimalist RL algorithm that simplifies policy optimization by regressing the relative reward between two completions of a prompt. REBEL eliminates the need for value networks and clipping, making it more lightweight and computationally efficient. Theoretical analysis shows that fundamental RL algorithms like Natural Policy Gradient (NPG) can be seen as variants of REBEL, providing strong guarantees in terms of convergence and sample complexity. Empirical results demonstrate that REBEL outperforms PPO and other baselines in language modeling and image generation tasks, achieving competitive performance on benchmarks such as AlpacaEval 2.0, MT-Bench, and Open LLM Leaderboard. **Key Contributions:** 1. **REBEL Algorithm:** A simple and scalable RL algorithm that reduces policy optimization to solving a sequence of squared loss regression problems. 2. **Theoretical Analysis:** Proves that REBEL is a generalization of NPG and provides strong convergence and regret guarantees. 3. **Empirical Evaluation:** Demonstrates superior performance in language modeling and image generation tasks compared to PPO and other baselines. **Related Work:** - **Policy Gradients:** Explains the limitations of PPO and the benefits of REBEL's approach. - **Reward Regression:** Introduces the novel idea of using regression to fit relative rewards. - **Preference Fine-Tuning (PFT):** Discusses the role of RL in aligning language models with human preferences. **Conclusion:** REBEL offers a simpler and more efficient approach to RL, particularly for fine-tuning generative models, by leveraging regression to relative rewards. Theoretical and empirical results validate its effectiveness and scalability.