This paper investigates the effectiveness of Large Language Models (LLMs) as prompt optimizers in LLM-based Automatic Prompt Optimization. The authors conduct a comprehensive study to understand the underlying mechanisms of LLM-based prompt optimization and find that LLMs struggle to identify the true causes of errors during reflection, often biased by their prior knowledge. They also observe that even when the reflection is semantically valid, LLMs often fail to generate appropriate prompts for target models due to the unpredictable behaviors of these models. Based on these findings, the authors introduce a new paradigm called "Automatic Behavior Optimization" (ABO), which directly optimizes the target model's behavior in a more controllable manner. The study highlights the gap between LLMs and target models in prompt optimization and suggests that ABO can effectively improve the performance of less powerful target models. The authors hope their findings will inspire new directions in automatic prompt optimization.This paper investigates the effectiveness of Large Language Models (LLMs) as prompt optimizers in LLM-based Automatic Prompt Optimization. The authors conduct a comprehensive study to understand the underlying mechanisms of LLM-based prompt optimization and find that LLMs struggle to identify the true causes of errors during reflection, often biased by their prior knowledge. They also observe that even when the reflection is semantically valid, LLMs often fail to generate appropriate prompts for target models due to the unpredictable behaviors of these models. Based on these findings, the authors introduce a new paradigm called "Automatic Behavior Optimization" (ABO), which directly optimizes the target model's behavior in a more controllable manner. The study highlights the gap between LLMs and target models in prompt optimization and suggests that ABO can effectively improve the performance of less powerful target models. The authors hope their findings will inspire new directions in automatic prompt optimization.