The paper investigates the performance gap between online and offline alignment algorithms in the context of reinforcement learning from human feedback (RLHF). It begins by demonstrating that online methods generally outperform offline methods in terms of performance and optimization efficiency. Through a series of controlled experiments, the authors explore various hypotheses to explain this discrepancy, including data coverage, data quality, and the interplay between discriminative and generative capabilities. They find that while offline algorithms are better at classification, they are worse at generating high-quality responses. In contrast, online algorithms excel at generating high-quality responses but perform poorly at classification. This suggests that the sampling process plays a crucial role in the performance of these algorithms. The study also shows that the performance gap persists across different loss functions and does not improve with larger policy networks. Overall, the findings highlight the importance of on-policy sampling in AI alignment and point to fundamental challenges in offline alignment algorithms.The paper investigates the performance gap between online and offline alignment algorithms in the context of reinforcement learning from human feedback (RLHF). It begins by demonstrating that online methods generally outperform offline methods in terms of performance and optimization efficiency. Through a series of controlled experiments, the authors explore various hypotheses to explain this discrepancy, including data coverage, data quality, and the interplay between discriminative and generative capabilities. They find that while offline algorithms are better at classification, they are worse at generating high-quality responses. In contrast, online algorithms excel at generating high-quality responses but perform poorly at classification. This suggests that the sampling process plays a crucial role in the performance of these algorithms. The study also shows that the performance gap persists across different loss functions and does not improve with larger policy networks. Overall, the findings highlight the importance of on-policy sampling in AI alignment and point to fundamental challenges in offline alignment algorithms.