This paper introduces Proximal Policy Optimization (PPO), a new family of policy gradient methods for reinforcement learning. PPO alternates between sampling data through interaction with the environment and optimizing a "surrogate" objective function using stochastic gradient ascent. Unlike standard policy gradient methods, which perform one gradient update per data sample, PPO uses a novel objective function that enables multiple epochs of minibatch updates. PPO combines some of the benefits of Trust Region Policy Optimization (TRPO), such as data efficiency and reliable performance, but is simpler to implement, more general, and has better sample complexity. PPO is tested on a variety of benchmark tasks, including simulated robotic locomotion and Atari game playing. It outperforms other online policy gradient methods, striking a favorable balance between sample complexity, simplicity, and wall-time. The key idea of PPO is the use of a clipped surrogate objective, which forms a pessimistic estimate of the policy's performance. This objective is designed to prevent large policy updates that could destabilize learning. The paper also introduces an adaptive KL penalty coefficient, which adjusts the penalty coefficient based on the target KL divergence. This method is used as an alternative to the clipped surrogate objective. The PPO algorithm is described in detail, including its implementation with actor-critic style and the use of a neural network architecture that shares parameters between the policy and value function. The algorithm is tested on a variety of continuous control tasks and the Atari domain, where it outperforms other methods in terms of sample efficiency and performance. The experiments show that PPO performs well on a range of tasks, including high-dimensional continuous control problems such as humanoid running and steering. It is also compared to other algorithms on the Atari domain, where it performs significantly better in terms of sample complexity. The results demonstrate that PPO is a promising method for reinforcement learning, offering a balance between sample efficiency, simplicity, and performance.This paper introduces Proximal Policy Optimization (PPO), a new family of policy gradient methods for reinforcement learning. PPO alternates between sampling data through interaction with the environment and optimizing a "surrogate" objective function using stochastic gradient ascent. Unlike standard policy gradient methods, which perform one gradient update per data sample, PPO uses a novel objective function that enables multiple epochs of minibatch updates. PPO combines some of the benefits of Trust Region Policy Optimization (TRPO), such as data efficiency and reliable performance, but is simpler to implement, more general, and has better sample complexity. PPO is tested on a variety of benchmark tasks, including simulated robotic locomotion and Atari game playing. It outperforms other online policy gradient methods, striking a favorable balance between sample complexity, simplicity, and wall-time. The key idea of PPO is the use of a clipped surrogate objective, which forms a pessimistic estimate of the policy's performance. This objective is designed to prevent large policy updates that could destabilize learning. The paper also introduces an adaptive KL penalty coefficient, which adjusts the penalty coefficient based on the target KL divergence. This method is used as an alternative to the clipped surrogate objective. The PPO algorithm is described in detail, including its implementation with actor-critic style and the use of a neural network architecture that shares parameters between the policy and value function. The algorithm is tested on a variety of continuous control tasks and the Atari domain, where it outperforms other methods in terms of sample efficiency and performance. The experiments show that PPO performs well on a range of tasks, including high-dimensional continuous control problems such as humanoid running and steering. It is also compared to other algorithms on the Atari domain, where it performs significantly better in terms of sample complexity. The results demonstrate that PPO is a promising method for reinforcement learning, offering a balance between sample efficiency, simplicity, and performance.