This paper introduces a novel approach for multi-turn reinforcement learning from preference feedback, addressing the limitations of existing single-turn methods in scenarios requiring long-term planning and interaction. The authors propose a new policy optimization algorithm, MTPO, which is based on mirror descent and self-play, and is proven to converge to a Nash equilibrium. The algorithm is designed to handle preference feedback between entire multi-turn conversations, allowing for the capture of long-term effects of individual actions. The paper also presents a multi-turn RLHF algorithm that leverages the same framework and is proven to converge to an optimal policy. The authors evaluate their approach in two environments: Education Dialogue and Car Dealer. In the Education Dialogue environment, a teacher agent guides a student in learning a random topic, and the algorithm outperforms single-turn baselines and multi-turn RLHF. In the Car Dealer environment, the algorithm achieves comparable performance to reward-based methods despite relying on a weaker preference signal. The paper also introduces a new preference-based Q-function that accounts for the long-term consequences of individual actions, and demonstrates that even in a reward-based environment, the preference-based algorithm achieves performance comparable to reward-based methods. The authors also provide theoretical guarantees for their algorithms, including convergence to Nash equilibrium and bounds on the KL divergence between the optimal policy and the current policy. The paper highlights the importance of considering conversation-level feedback in multi-turn settings, as it allows for a more accurate representation of the long-term effects of actions. The authors argue that single-turn methods are limited in their ability to capture these effects, and that multi-turn methods are necessary for effective alignment with human preferences in complex, long-term tasks. The paper also discusses the challenges of multi-turn reinforcement learning, including the need for planning ahead and the difficulty of evaluating the quality of intermediate actions. The authors conclude that their approach provides a promising solution to these challenges and offers a new framework for multi-turn reinforcement learning from preference feedback.This paper introduces a novel approach for multi-turn reinforcement learning from preference feedback, addressing the limitations of existing single-turn methods in scenarios requiring long-term planning and interaction. The authors propose a new policy optimization algorithm, MTPO, which is based on mirror descent and self-play, and is proven to converge to a Nash equilibrium. The algorithm is designed to handle preference feedback between entire multi-turn conversations, allowing for the capture of long-term effects of individual actions. The paper also presents a multi-turn RLHF algorithm that leverages the same framework and is proven to converge to an optimal policy. The authors evaluate their approach in two environments: Education Dialogue and Car Dealer. In the Education Dialogue environment, a teacher agent guides a student in learning a random topic, and the algorithm outperforms single-turn baselines and multi-turn RLHF. In the Car Dealer environment, the algorithm achieves comparable performance to reward-based methods despite relying on a weaker preference signal. The paper also introduces a new preference-based Q-function that accounts for the long-term consequences of individual actions, and demonstrates that even in a reward-based environment, the preference-based algorithm achieves performance comparable to reward-based methods. The authors also provide theoretical guarantees for their algorithms, including convergence to Nash equilibrium and bounds on the KL divergence between the optimal policy and the current policy. The paper highlights the importance of considering conversation-level feedback in multi-turn settings, as it allows for a more accurate representation of the long-term effects of actions. The authors argue that single-turn methods are limited in their ability to capture these effects, and that multi-turn methods are necessary for effective alignment with human preferences in complex, long-term tasks. The paper also discusses the challenges of multi-turn reinforcement learning, including the need for planning ahead and the difficulty of evaluating the quality of intermediate actions. The authors conclude that their approach provides a promising solution to these challenges and offers a new framework for multi-turn reinforcement learning from preference feedback.