This paper presents a new approach and algorithm for solving constrained Bi-Level Optimization (BLO) problems where the lower-level (LL) problem involves constraints coupling both upper-level (UL) and lower-level variables. The proposed method introduces a smooth proximal Lagrangian value function to handle the constrained LL problem, enabling a single-level reformulation of BLOs into equivalent optimization problems with smooth constraints. This reformulation allows the development of a Hessian-free gradient-based algorithm, termed the Proximal Lagrangian Value Function-based Hessian-free Bi-level Algorithm (LV-HBA), which is straightforward to implement in a single loop. LV-HBA is particularly suitable for machine learning applications and provides non-asymptotic convergence analysis without requiring strong convexity assumptions for the LL problem, making it applicable to non-singleton scenarios. Empirical results demonstrate the superior practical performance of LV-HBA compared to existing methods. The algorithm is evaluated on synthetic problems, hyperparameter optimization for SVM, and federated bilevel learning, showing its effectiveness in practical scenarios. The key contributions include the introduction of a novel proximal Lagrangian value function, the development of LV-HBA, and the non-asymptotic convergence analysis of the algorithm.This paper presents a new approach and algorithm for solving constrained Bi-Level Optimization (BLO) problems where the lower-level (LL) problem involves constraints coupling both upper-level (UL) and lower-level variables. The proposed method introduces a smooth proximal Lagrangian value function to handle the constrained LL problem, enabling a single-level reformulation of BLOs into equivalent optimization problems with smooth constraints. This reformulation allows the development of a Hessian-free gradient-based algorithm, termed the Proximal Lagrangian Value Function-based Hessian-free Bi-level Algorithm (LV-HBA), which is straightforward to implement in a single loop. LV-HBA is particularly suitable for machine learning applications and provides non-asymptotic convergence analysis without requiring strong convexity assumptions for the LL problem, making it applicable to non-singleton scenarios. Empirical results demonstrate the superior practical performance of LV-HBA compared to existing methods. The algorithm is evaluated on synthetic problems, hyperparameter optimization for SVM, and federated bilevel learning, showing its effectiveness in practical scenarios. The key contributions include the introduction of a novel proximal Lagrangian value function, the development of LV-HBA, and the non-asymptotic convergence analysis of the algorithm.