The paper introduces Conservative Q-Learning (CQL), a novel approach to offline reinforcement learning (RL) that addresses the challenge of leveraging large, previously collected datasets without further interaction with the environment. CQL aims to learn a conservative Q-function, ensuring that the expected value of a policy under this Q-function lower-bounds its true value. The key idea is to minimize Q-values under a distribution over state-action pairs and incorporate a maximization term to tighten the bound. Theoretical analysis shows that CQL produces a lower bound on the policy value and can be integrated into policy learning procedures with theoretical guarantees. Empirical results demonstrate that CQL outperforms existing offline RL methods, achieving 2-5 times higher final returns, especially in complex and multi-modal data distributions. CQL is implemented with minimal additional code and is shown to be robust to Q-function estimation errors. The paper also discusses variants of CQL and provides safe policy improvement guarantees, making it a promising solution for real-world offline RL problems.The paper introduces Conservative Q-Learning (CQL), a novel approach to offline reinforcement learning (RL) that addresses the challenge of leveraging large, previously collected datasets without further interaction with the environment. CQL aims to learn a conservative Q-function, ensuring that the expected value of a policy under this Q-function lower-bounds its true value. The key idea is to minimize Q-values under a distribution over state-action pairs and incorporate a maximization term to tighten the bound. Theoretical analysis shows that CQL produces a lower bound on the policy value and can be integrated into policy learning procedures with theoretical guarantees. Empirical results demonstrate that CQL outperforms existing offline RL methods, achieving 2-5 times higher final returns, especially in complex and multi-modal data distributions. CQL is implemented with minimal additional code and is shown to be robust to Q-function estimation errors. The paper also discusses variants of CQL and provides safe policy improvement guarantees, making it a promising solution for real-world offline RL problems.