This paper introduces a principled penalty-based algorithmic framework for solving bilevel reinforcement learning (RL) problems. Bilevel optimization has been widely used in machine learning, but its application to RL has been limited due to the dynamic and complex nature of RL objectives. The authors propose a new approach that reformulates bilevel RL problems using penalty functions, enabling the use of existing optimization techniques in RL settings. The paper first defines a general bilevel RL formulation, where the upper-level problem involves optimizing a function subject to the lower-level problem, which is itself an RL problem. The lower-level problem involves finding an optimal policy for a Markov decision process (MDP) determined by the upper-level decision variable. The authors then introduce two penalty functions: value penalty and Bellman penalty, which are designed to capture the optimality conditions of the lower-level RL problem. The value penalty measures the lower-level optimality gap, while the Bellman penalty is based on the Bellman error. The authors show that an approximate solution to the reformulated problem is also an effective solution to the original bilevel problem. They further establish the differentiability of the reformulated problem and propose a first-order policy-gradient-based algorithm that provably converges. The paper also extends the bilevel RL framework to scenarios involving two RL agents in the lower-level problems with the goal of solving a zero-sum Markov game. Here, the authors introduce a value-based penalty function derived from the Nikaido-Isoda function for two-player games. The resulting algorithm is the first provably convergent algorithm for bilevel RL with a game constraint. The authors demonstrate the effectiveness of their algorithms through simulations in the Stackelberg Markov game, RL from human feedback, and incentive design. They also show that their algorithm can be applied to a variety of RL tasks, including reward shaping and RL from human feedback. The paper concludes that the proposed penalty-based approach provides a promising avenue for future research on bilevel RL with more complex lower-level problems. The authors establish the first provably convergent first-order algorithm for bilevel RL and show that their algorithm can be applied to a wide range of RL tasks.This paper introduces a principled penalty-based algorithmic framework for solving bilevel reinforcement learning (RL) problems. Bilevel optimization has been widely used in machine learning, but its application to RL has been limited due to the dynamic and complex nature of RL objectives. The authors propose a new approach that reformulates bilevel RL problems using penalty functions, enabling the use of existing optimization techniques in RL settings. The paper first defines a general bilevel RL formulation, where the upper-level problem involves optimizing a function subject to the lower-level problem, which is itself an RL problem. The lower-level problem involves finding an optimal policy for a Markov decision process (MDP) determined by the upper-level decision variable. The authors then introduce two penalty functions: value penalty and Bellman penalty, which are designed to capture the optimality conditions of the lower-level RL problem. The value penalty measures the lower-level optimality gap, while the Bellman penalty is based on the Bellman error. The authors show that an approximate solution to the reformulated problem is also an effective solution to the original bilevel problem. They further establish the differentiability of the reformulated problem and propose a first-order policy-gradient-based algorithm that provably converges. The paper also extends the bilevel RL framework to scenarios involving two RL agents in the lower-level problems with the goal of solving a zero-sum Markov game. Here, the authors introduce a value-based penalty function derived from the Nikaido-Isoda function for two-player games. The resulting algorithm is the first provably convergent algorithm for bilevel RL with a game constraint. The authors demonstrate the effectiveness of their algorithms through simulations in the Stackelberg Markov game, RL from human feedback, and incentive design. They also show that their algorithm can be applied to a variety of RL tasks, including reward shaping and RL from human feedback. The paper concludes that the proposed penalty-based approach provides a promising avenue for future research on bilevel RL with more complex lower-level problems. The authors establish the first provably convergent first-order algorithm for bilevel RL and show that their algorithm can be applied to a wide range of RL tasks.