This paper introduces a novel method for Bayesian optimization (BO) of Variational Quantum Eigensolvers (VQE), a hybrid quantum-classical protocol used to approximate the ground state of a quantum Hamiltonian. The authors propose a VQE-kernel that incorporates prior information about quantum circuits, significantly reducing posterior uncertainty. They also introduce a new acquisition function called Expected Maximum Improvement over Confident Regions (EMICoRe), which leverages the inductive bias of the VQE-kernel by treating regions with low predictive uncertainty as indirectly "observed." This approach combines the strengths of the Nakanishi-Fuji-Todo (NFT) method and BO, enhancing the efficiency and scalability of VQE optimization. Numerical experiments demonstrate the effectiveness of the proposed method, showing improved performance over state-of-the-art baselines in various VQE problems. The paper also proves the equivalence between two important properties of VQE: the parameter shift rule and the sinusoidal function-form, highlighting the synergy between physics and machine learning in optimizing quantum circuits.This paper introduces a novel method for Bayesian optimization (BO) of Variational Quantum Eigensolvers (VQE), a hybrid quantum-classical protocol used to approximate the ground state of a quantum Hamiltonian. The authors propose a VQE-kernel that incorporates prior information about quantum circuits, significantly reducing posterior uncertainty. They also introduce a new acquisition function called Expected Maximum Improvement over Confident Regions (EMICoRe), which leverages the inductive bias of the VQE-kernel by treating regions with low predictive uncertainty as indirectly "observed." This approach combines the strengths of the Nakanishi-Fuji-Todo (NFT) method and BO, enhancing the efficiency and scalability of VQE optimization. Numerical experiments demonstrate the effectiveness of the proposed method, showing improved performance over state-of-the-art baselines in various VQE problems. The paper also proves the equivalence between two important properties of VQE: the parameter shift rule and the sinusoidal function-form, highlighting the synergy between physics and machine learning in optimizing quantum circuits.