Vanilla Bayesian Optimization Performs Great in High Dimensions

Vanilla Bayesian Optimization Performs Great in High Dimensions

2024 | Carl Hvarfner, Erik O. Hellsten, Luigi Nardi
The paper addresses the challenges of high-dimensional Bayesian optimization (BO), which has traditionally been hindered by the curse of dimensionality. The authors identify that the primary issue is not the dimensionality itself but the assumed complexity of the objective function. They propose a modification to the vanilla BO algorithm by scaling the lengthscales of the Gaussian process (GP) kernel with the dimensionality, which effectively reduces model complexity without imposing structural restrictions on the objective. This modification allows vanilla BO to perform well in high-dimensional tasks, outperforming existing state-of-the-art algorithms on various real-world benchmarks. The paper also discusses the theoretical underpinnings of the boundary issue in high-dimensional BO and provides empirical evidence that vanilla BO does not suffer from excessive exploration around the boundaries, contrary to common belief. The results demonstrate that vanilla BO, with its modified prior assumptions, can achieve significant improvements in performance, making it a viable and efficient approach for high-dimensional optimization problems.The paper addresses the challenges of high-dimensional Bayesian optimization (BO), which has traditionally been hindered by the curse of dimensionality. The authors identify that the primary issue is not the dimensionality itself but the assumed complexity of the objective function. They propose a modification to the vanilla BO algorithm by scaling the lengthscales of the Gaussian process (GP) kernel with the dimensionality, which effectively reduces model complexity without imposing structural restrictions on the objective. This modification allows vanilla BO to perform well in high-dimensional tasks, outperforming existing state-of-the-art algorithms on various real-world benchmarks. The paper also discusses the theoretical underpinnings of the boundary issue in high-dimensional BO and provides empirical evidence that vanilla BO does not suffer from excessive exploration around the boundaries, contrary to common belief. The results demonstrate that vanilla BO, with its modified prior assumptions, can achieve significant improvements in performance, making it a viable and efficient approach for high-dimensional optimization problems.
Reach us at info@study.space
Understanding Vanilla Bayesian Optimization Performs Great in High Dimensions