The paper "Prompt Optimization in Large Language Models" by Antonio Sabbatella, Andrea Ponti, Ilaria Giordani, Antonio Candelieri, and Francesco Archetti explores the task of optimizing prompts for large language models (LLMs) to improve their performance on downstream tasks. The authors propose a Bayesian Optimization (BO) method to efficiently search for the optimal prompt in a combinatorial space, which is impractical to explore exhaustively due to the exponential growth of possible prompts. The method involves relaxing the combinatorial search space into a continuous space, allowing BO to sample efficiently. The paper focuses on Hard Prompt Tuning (HPT), where the goal is to directly find an optimal prompt without access to the LLM's gradients, making it suitable for black-box LLMs. The authors use BoTorch, a library for Bayesian Optimization, and evaluate their approach on six benchmark datasets, showing good performance compared to other state-of-the-art methods. The main contributions include the feasibility of using BO for black-box prompt optimization, significant reduction in wall-clock time, and the effectiveness of a "vanilla" BO algorithm in high-dimensional settings. The paper also discusses related work, methodological details, and computational results, highlighting the advantages of their approach in terms of sample efficiency and runtime.The paper "Prompt Optimization in Large Language Models" by Antonio Sabbatella, Andrea Ponti, Ilaria Giordani, Antonio Candelieri, and Francesco Archetti explores the task of optimizing prompts for large language models (LLMs) to improve their performance on downstream tasks. The authors propose a Bayesian Optimization (BO) method to efficiently search for the optimal prompt in a combinatorial space, which is impractical to explore exhaustively due to the exponential growth of possible prompts. The method involves relaxing the combinatorial search space into a continuous space, allowing BO to sample efficiently. The paper focuses on Hard Prompt Tuning (HPT), where the goal is to directly find an optimal prompt without access to the LLM's gradients, making it suitable for black-box LLMs. The authors use BoTorch, a library for Bayesian Optimization, and evaluate their approach on six benchmark datasets, showing good performance compared to other state-of-the-art methods. The main contributions include the feasibility of using BO for black-box prompt optimization, significant reduction in wall-clock time, and the effectiveness of a "vanilla" BO algorithm in high-dimensional settings. The paper also discusses related work, methodological details, and computational results, highlighting the advantages of their approach in terms of sample efficiency and runtime.