The paper introduces a principled algorithmic framework for solving bilevel reinforcement learning (RL) problems using penalty formulation. Bilevel optimization has been widely applied in machine learning, but its applications in RL are limited due to the non-convex nature of the discounted return objective in the lower-level problem. The authors propose two tailored penalty functions: value penalty and Bellman penalty, which capture the optimality conditions of the lower-level RL problem. They prove that solving the reformulated single-level problem with these penalties recovers the solutions of the original bilevel RL problem. The paper also establishes the differentiability of the reformulated problem and proposes a first-order policy-gradient-based algorithm that provably converges. Additionally, the authors extend the framework to bilevel RL with two-player zero-sum games, introducing a value-based penalty function derived from the Nikaido-Isoda function. The resulting algorithm is the first provably convergent algorithm for bilevel RL with game constraints. The paper includes theoretical studies and simulations demonstrating the effectiveness of the proposed algorithms in various applications, such as Stackelberg Markov games, RL from human feedback, and incentive design.The paper introduces a principled algorithmic framework for solving bilevel reinforcement learning (RL) problems using penalty formulation. Bilevel optimization has been widely applied in machine learning, but its applications in RL are limited due to the non-convex nature of the discounted return objective in the lower-level problem. The authors propose two tailored penalty functions: value penalty and Bellman penalty, which capture the optimality conditions of the lower-level RL problem. They prove that solving the reformulated single-level problem with these penalties recovers the solutions of the original bilevel RL problem. The paper also establishes the differentiability of the reformulated problem and proposes a first-order policy-gradient-based algorithm that provably converges. Additionally, the authors extend the framework to bilevel RL with two-player zero-sum games, introducing a value-based penalty function derived from the Nikaido-Isoda function. The resulting algorithm is the first provably convergent algorithm for bilevel RL with game constraints. The paper includes theoretical studies and simulations demonstrating the effectiveness of the proposed algorithms in various applications, such as Stackelberg Markov games, RL from human feedback, and incentive design.