RL²: Fast Reinforcement Learning via Slow Reinforcement Learning This paper proposes RL², a method that learns a reinforcement learning (RL) algorithm by encoding it as a recurrent neural network (RNN) and training it using a general-purpose "slow" RL algorithm. The RNN is trained on data from multiple Markov Decision Processes (MDPs), and its weights are learned slowly. The RNN receives all information typical of an RL algorithm, including observations, actions, rewards, and termination flags, and retains its state across episodes. The activations of the RNN store the state of the "fast" RL algorithm on the current MDP. The method is evaluated on both small-scale and large-scale problems, showing that it performs close to human-designed algorithms with optimality guarantees on small-scale tasks and scales to high-dimensional problems on a vision-based navigation task. The paper introduces RL² as a way to learn an RL algorithm by treating the learning process of the agent itself as an objective, which can be optimized using standard RL algorithms. The objective is averaged across all possible MDPs according to a specific distribution, reflecting the prior knowledge we want to distill into the agent. The agent is structured as a recurrent neural network, which receives past rewards, actions, and termination flags as inputs in addition to the normally received observations. Its internal state is preserved across episodes, allowing it to perform learning in its own hidden activations. The learned agent thus also acts as the learning algorithm, adapting to the task at hand when deployed. The method is evaluated on two classical problems: multi-armed bandits and tabular MDPs. These problems have been extensively studied, and there exist algorithms that achieve asymptotically optimal performance. The paper demonstrates that RL² can achieve performance comparable with these theoretically justified algorithms. It is also evaluated on a vision-based navigation task using the ViZDoom environment, showing that it can scale to high-dimensional problems. The paper discusses related work in reinforcement learning, including methods that use prior experience to speed up learning, hierarchical reinforcement learning, and model-based approaches. It also discusses the challenges of applying RL² to high-dimensional tasks and the potential for improving performance through better architectures and algorithms. The paper concludes that RL² is a promising approach for learning better reinforcement learning algorithms by learning the algorithm end-to-end using standard reinforcement learning techniques.RL²: Fast Reinforcement Learning via Slow Reinforcement Learning This paper proposes RL², a method that learns a reinforcement learning (RL) algorithm by encoding it as a recurrent neural network (RNN) and training it using a general-purpose "slow" RL algorithm. The RNN is trained on data from multiple Markov Decision Processes (MDPs), and its weights are learned slowly. The RNN receives all information typical of an RL algorithm, including observations, actions, rewards, and termination flags, and retains its state across episodes. The activations of the RNN store the state of the "fast" RL algorithm on the current MDP. The method is evaluated on both small-scale and large-scale problems, showing that it performs close to human-designed algorithms with optimality guarantees on small-scale tasks and scales to high-dimensional problems on a vision-based navigation task. The paper introduces RL² as a way to learn an RL algorithm by treating the learning process of the agent itself as an objective, which can be optimized using standard RL algorithms. The objective is averaged across all possible MDPs according to a specific distribution, reflecting the prior knowledge we want to distill into the agent. The agent is structured as a recurrent neural network, which receives past rewards, actions, and termination flags as inputs in addition to the normally received observations. Its internal state is preserved across episodes, allowing it to perform learning in its own hidden activations. The learned agent thus also acts as the learning algorithm, adapting to the task at hand when deployed. The method is evaluated on two classical problems: multi-armed bandits and tabular MDPs. These problems have been extensively studied, and there exist algorithms that achieve asymptotically optimal performance. The paper demonstrates that RL² can achieve performance comparable with these theoretically justified algorithms. It is also evaluated on a vision-based navigation task using the ViZDoom environment, showing that it can scale to high-dimensional problems. The paper discusses related work in reinforcement learning, including methods that use prior experience to speed up learning, hierarchical reinforcement learning, and model-based approaches. It also discusses the challenges of applying RL² to high-dimensional tasks and the potential for improving performance through better architectures and algorithms. The paper concludes that RL² is a promising approach for learning better reinforcement learning algorithms by learning the algorithm end-to-end using standard reinforcement learning techniques.