This paper introduces temporally-aware reinforcement learning (RL) algorithms that adapt to the amount of learning time remaining. The authors propose two new methods, Temporally-Adaptive Learned Policy Gradient (TA-LPG) and Temporally-Adaptive Learned Policy Optimization (TA-LPO), which incorporate information about the agent's lifetime into their learning processes. These methods allow the learned objective functions to dynamically update throughout the agent's training, resulting in more expressive schedules and improved generalization across different training horizons. The authors compare meta-gradient approaches with evolutionary strategies for discovering non-myopic RL objective functions. They find that meta-gradient approaches fail to learn dynamic updates, while evolution strategies successfully discover highly dynamic learning rules. The proposed methods are evaluated on a wide range of tasks and environments, showing significant improvements over non-temporally-aware counterparts. The learned algorithms effectively balance exploration and exploitation by modifying their learning rules throughout the agent's lifetime. The paper highlights the importance of lifetime conditioning in meta-learning, demonstrating that it enables the discovery of more general and effective RL algorithms. The results show that temporally-aware objective functions outperform their non-temporally-aware counterparts in both in-distribution and out-of-distribution environments. The authors also analyze the dynamic schedules extracted by the discovered objective functions, finding that they implement dynamic policy importance ratio clipping and update and entropy annealing schedules that adapt to the training horizon. The study shows that evolutionary strategies are more effective than meta-gradient approaches for discovering temporally-aware RL algorithms. The results suggest that lifetime conditioning is a critical component in successfully taking advantage of temporal information, leading to better generalization and performance across different training horizons. The paper concludes that evolutionary optimization is a key factor in discovering RL algorithms capable of effective lifetime conditioning.This paper introduces temporally-aware reinforcement learning (RL) algorithms that adapt to the amount of learning time remaining. The authors propose two new methods, Temporally-Adaptive Learned Policy Gradient (TA-LPG) and Temporally-Adaptive Learned Policy Optimization (TA-LPO), which incorporate information about the agent's lifetime into their learning processes. These methods allow the learned objective functions to dynamically update throughout the agent's training, resulting in more expressive schedules and improved generalization across different training horizons. The authors compare meta-gradient approaches with evolutionary strategies for discovering non-myopic RL objective functions. They find that meta-gradient approaches fail to learn dynamic updates, while evolution strategies successfully discover highly dynamic learning rules. The proposed methods are evaluated on a wide range of tasks and environments, showing significant improvements over non-temporally-aware counterparts. The learned algorithms effectively balance exploration and exploitation by modifying their learning rules throughout the agent's lifetime. The paper highlights the importance of lifetime conditioning in meta-learning, demonstrating that it enables the discovery of more general and effective RL algorithms. The results show that temporally-aware objective functions outperform their non-temporally-aware counterparts in both in-distribution and out-of-distribution environments. The authors also analyze the dynamic schedules extracted by the discovered objective functions, finding that they implement dynamic policy importance ratio clipping and update and entropy annealing schedules that adapt to the training horizon. The study shows that evolutionary strategies are more effective than meta-gradient approaches for discovering temporally-aware RL algorithms. The results suggest that lifetime conditioning is a critical component in successfully taking advantage of temporal information, leading to better generalization and performance across different training horizons. The paper concludes that evolutionary optimization is a key factor in discovering RL algorithms capable of effective lifetime conditioning.