Conservative Q-learning (CQL) is a method for offline reinforcement learning (RL) that addresses the challenge of learning effective policies from static datasets without further interaction. Traditional off-policy RL methods often fail in offline settings due to distributional shifts between the dataset and the learned policy, leading to overestimation of values. CQL introduces a conservative Q-function that lower-bounds the true value of a policy, preventing overestimation. The method augments the standard Bellman error objective with a Q-value regularizer, making it straightforward to implement on top of existing deep Q-learning and actor-critic frameworks. Theoretical analysis shows that CQL produces a lower bound on the value of the current policy and can be integrated into a policy learning procedure with theoretical guarantees. Empirically, CQL outperforms existing offline RL methods on both discrete and continuous control domains, often achieving 2-5 times higher final returns, especially when learning from complex and multi-modal data distributions. CQL is particularly effective in domains with high-dimensional visual inputs and complex dataset compositions, where prior methods struggle. CQL is implemented with less than 20 lines of code on top of standard online RL algorithms, and it provides robustness to Q-function estimation errors. The method ensures that the expected value of the policy under the Q-function lower-bounds its true value, preventing overestimation. Theoretical results show that CQL is gap-expanding, meaning it increases the difference in Q-values between in-distribution and out-of-distribution actions, which helps mitigate the effects of distributional shifts. CQL also provides safe policy improvement guarantees, ensuring that the learned policy does not significantly underperform the behavior policy. The method has been evaluated on a range of domains, including continuous control, high-dimensional image inputs, and Atari games, demonstrating its effectiveness in various offline RL scenarios. Overall, CQL offers a promising approach for offline RL, balancing simplicity, efficacy, and theoretical guarantees.Conservative Q-learning (CQL) is a method for offline reinforcement learning (RL) that addresses the challenge of learning effective policies from static datasets without further interaction. Traditional off-policy RL methods often fail in offline settings due to distributional shifts between the dataset and the learned policy, leading to overestimation of values. CQL introduces a conservative Q-function that lower-bounds the true value of a policy, preventing overestimation. The method augments the standard Bellman error objective with a Q-value regularizer, making it straightforward to implement on top of existing deep Q-learning and actor-critic frameworks. Theoretical analysis shows that CQL produces a lower bound on the value of the current policy and can be integrated into a policy learning procedure with theoretical guarantees. Empirically, CQL outperforms existing offline RL methods on both discrete and continuous control domains, often achieving 2-5 times higher final returns, especially when learning from complex and multi-modal data distributions. CQL is particularly effective in domains with high-dimensional visual inputs and complex dataset compositions, where prior methods struggle. CQL is implemented with less than 20 lines of code on top of standard online RL algorithms, and it provides robustness to Q-function estimation errors. The method ensures that the expected value of the policy under the Q-function lower-bounds its true value, preventing overestimation. Theoretical results show that CQL is gap-expanding, meaning it increases the difference in Q-values between in-distribution and out-of-distribution actions, which helps mitigate the effects of distributional shifts. CQL also provides safe policy improvement guarantees, ensuring that the learned policy does not significantly underperform the behavior policy. The method has been evaluated on a range of domains, including continuous control, high-dimensional image inputs, and Atari games, demonstrating its effectiveness in various offline RL scenarios. Overall, CQL offers a promising approach for offline RL, balancing simplicity, efficacy, and theoretical guarantees.