This paper investigates when Large Language Models (LLMs) need Retrieval Augmentation (RA) to mitigate their overconfidence and improve performance. LLMs often overestimate their knowledge and provide incorrect answers when they lack the necessary information. Retrieval Augmentation helps by providing external information, but it is not always optimal due to the overhead and potential quality issues of retrieved documents. The paper proposes methods to enhance LLMs' ability to perceive their knowledge boundaries, reducing overconfidence and improving RA performance. The study quantifies LLMs' ability to recognize their knowledge boundaries and finds that overconfidence is the main reason for poor perception. It also explores the relationship between LLMs' certainty about their answers and their reliance on external information. The results show that more uncertain LLMs rely more on external documents. To reduce overconfidence, the paper proposes three prompting methods: Punish, Challenge, and Think-Step-by-Step, as well as two methods: Explain and Generate. These methods are shown to effectively enhance LLMs' perception of knowledge boundaries and improve alignment and accuracy. The paper also investigates adaptive retrieval augmentation, where RA is only used when LLMs are uncertain. Experiments show that with enhanced LLMs, RA can achieve comparable or better performance with fewer retrieval calls. The results demonstrate that adaptive retrieval augmentation is more efficient and robust, especially when the quality of retrieved documents is low. The study concludes that enhancing LLMs' perception of knowledge boundaries is crucial for effective and efficient retrieval augmentation. The methods proposed can be applied to various LLMs to improve their performance in QA tasks.This paper investigates when Large Language Models (LLMs) need Retrieval Augmentation (RA) to mitigate their overconfidence and improve performance. LLMs often overestimate their knowledge and provide incorrect answers when they lack the necessary information. Retrieval Augmentation helps by providing external information, but it is not always optimal due to the overhead and potential quality issues of retrieved documents. The paper proposes methods to enhance LLMs' ability to perceive their knowledge boundaries, reducing overconfidence and improving RA performance. The study quantifies LLMs' ability to recognize their knowledge boundaries and finds that overconfidence is the main reason for poor perception. It also explores the relationship between LLMs' certainty about their answers and their reliance on external information. The results show that more uncertain LLMs rely more on external documents. To reduce overconfidence, the paper proposes three prompting methods: Punish, Challenge, and Think-Step-by-Step, as well as two methods: Explain and Generate. These methods are shown to effectively enhance LLMs' perception of knowledge boundaries and improve alignment and accuracy. The paper also investigates adaptive retrieval augmentation, where RA is only used when LLMs are uncertain. Experiments show that with enhanced LLMs, RA can achieve comparable or better performance with fewer retrieval calls. The results demonstrate that adaptive retrieval augmentation is more efficient and robust, especially when the quality of retrieved documents is low. The study concludes that enhancing LLMs' perception of knowledge boundaries is crucial for effective and efficient retrieval augmentation. The methods proposed can be applied to various LLMs to improve their performance in QA tasks.