This paper investigates the issue of length exploitation in Direct Preference Optimization (DPO), a direct alignment algorithm used in training large language models. Unlike classical Reinforcement Learning from Human Feedback (RLHF), DPO does not require a separate reward model or reinforcement learning, making it more efficient. However, DPO is susceptible to length exploitation, where models generate longer responses without improving quality, often due to biases in human preferences towards longer answers. The authors show that this issue is linked to out-of-distribution bootstrapping, where the model generates responses that are not representative of the training data. To address this, the authors propose a regularization strategy that prevents length exploitation while maintaining model quality. This approach is tested on datasets for summarization and dialogue, achieving up to 20% improvement in win rates when controlling for length, despite the GPT4 judge's known verbosity bias. The regularization method is simple and effective, preventing the model from generating excessively long responses without compromising performance. The study also explores the relationship between length and quality in DPO, finding that while length-regularized models generate responses closer to the SFT model in terms of length, they still achieve higher win rates. The authors also investigate the relationship between length and KL divergence, finding that length is only a partial factor in human preference. They hypothesize that the length exploitation issue is driven by the evaluator's bias, and that length-regularized models can disentangle verbosity from quality, leading to better performance. The paper concludes that DPO is a promising approach for direct alignment, but it is important to address length exploitation to ensure that models are not biased towards longer responses. The proposed regularization strategy is a simple and effective way to mitigate this issue, allowing for better alignment with human preferences. The results suggest that open-source models may also suffer from similar issues, and that length-regularized models could perform as well as proprietary models on automated evaluations.This paper investigates the issue of length exploitation in Direct Preference Optimization (DPO), a direct alignment algorithm used in training large language models. Unlike classical Reinforcement Learning from Human Feedback (RLHF), DPO does not require a separate reward model or reinforcement learning, making it more efficient. However, DPO is susceptible to length exploitation, where models generate longer responses without improving quality, often due to biases in human preferences towards longer answers. The authors show that this issue is linked to out-of-distribution bootstrapping, where the model generates responses that are not representative of the training data. To address this, the authors propose a regularization strategy that prevents length exploitation while maintaining model quality. This approach is tested on datasets for summarization and dialogue, achieving up to 20% improvement in win rates when controlling for length, despite the GPT4 judge's known verbosity bias. The regularization method is simple and effective, preventing the model from generating excessively long responses without compromising performance. The study also explores the relationship between length and quality in DPO, finding that while length-regularized models generate responses closer to the SFT model in terms of length, they still achieve higher win rates. The authors also investigate the relationship between length and KL divergence, finding that length is only a partial factor in human preference. They hypothesize that the length exploitation issue is driven by the evaluator's bias, and that length-regularized models can disentangle verbosity from quality, leading to better performance. The paper concludes that DPO is a promising approach for direct alignment, but it is important to address length exploitation to ensure that models are not biased towards longer responses. The proposed regularization strategy is a simple and effective way to mitigate this issue, allowing for better alignment with human preferences. The results suggest that open-source models may also suffer from similar issues, and that length-regularized models could perform as well as proprietary models on automated evaluations.