Constrained Policy Optimization (CPO) is a policy search algorithm for constrained reinforcement learning (CRL) that guarantees near-constraint satisfaction at each iteration. The method allows training of neural network policies for high-dimensional control while ensuring policy behavior adheres to constraints throughout training. CPO is based on a new theoretical result that relates the expected returns of two policies to an average divergence between them. This result tightens known bounds for policy search using trust regions and provides a tighter connection between theory and practice in deep reinforcement learning. CPO is designed to address the challenge of safely exploring environments where agents must avoid harmful behaviors. It is the first policy search algorithm for constrained Markov Decision Processes (CMDPs) that guarantees constraint satisfaction throughout training and works for arbitrary policy classes, including neural networks. The algorithm uses a trust region approach to ensure that policy updates do not violate constraints while improving reward. In experiments, CPO successfully trains neural network policies on high-dimensional simulated robot locomotion tasks, maximizing rewards while enforcing constraints. It outperforms baseline methods like primal-dual optimization in enforcing constraints without compromising performance. CPO also benefits from cost shaping, which helps in minimizing constraint violations by adjusting the upper bounds of auxiliary costs. The method is implemented with a practical approximation that allows efficient computation, even for policies with thousands of parameters. It uses a trust region approach to enable larger step sizes and ensures constraint satisfaction through a combination of theoretical guarantees and practical approximations. CPO is shown to be effective in various tasks, including robotic manipulation and gathering, where safety constraints are critical. The algorithm provides a principled approach to policy search in CMDPs, making it a valuable tool for applying reinforcement learning in real-world scenarios where safety is essential.Constrained Policy Optimization (CPO) is a policy search algorithm for constrained reinforcement learning (CRL) that guarantees near-constraint satisfaction at each iteration. The method allows training of neural network policies for high-dimensional control while ensuring policy behavior adheres to constraints throughout training. CPO is based on a new theoretical result that relates the expected returns of two policies to an average divergence between them. This result tightens known bounds for policy search using trust regions and provides a tighter connection between theory and practice in deep reinforcement learning. CPO is designed to address the challenge of safely exploring environments where agents must avoid harmful behaviors. It is the first policy search algorithm for constrained Markov Decision Processes (CMDPs) that guarantees constraint satisfaction throughout training and works for arbitrary policy classes, including neural networks. The algorithm uses a trust region approach to ensure that policy updates do not violate constraints while improving reward. In experiments, CPO successfully trains neural network policies on high-dimensional simulated robot locomotion tasks, maximizing rewards while enforcing constraints. It outperforms baseline methods like primal-dual optimization in enforcing constraints without compromising performance. CPO also benefits from cost shaping, which helps in minimizing constraint violations by adjusting the upper bounds of auxiliary costs. The method is implemented with a practical approximation that allows efficient computation, even for policies with thousands of parameters. It uses a trust region approach to enable larger step sizes and ensures constraint satisfaction through a combination of theoretical guarantees and practical approximations. CPO is shown to be effective in various tasks, including robotic manipulation and gathering, where safety constraints are critical. The algorithm provides a principled approach to policy search in CMDPs, making it a valuable tool for applying reinforcement learning in real-world scenarios where safety is essential.