This survey paper provides a comprehensive overview of Retrieval-Augmented Language Models (RALMs), including Retrieval-Augmented Generation (RAG) and Retrieval-Augmented Understanding (RAU). It explores the paradigm, evolution, taxonomy, and applications of RALMs, emphasizing their components such as retrievers, language models, and augmentations. The paper discusses how these components interact to create diverse model structures and applications across various NLP tasks, including translation, dialogue systems, and knowledge-intensive applications. It also covers evaluation methods for RALMs, highlighting the importance of robustness, accuracy, and relevance in their assessment. The paper acknowledges the limitations of RALMs, particularly in retrieval quality and computational efficiency, and offers directions for future research. The survey includes a GitHub repository with the surveyed works and resources for further study. The paper is structured into nine sections, covering the definition of RALMs, retrieval methods, language models, RALM enhancements, and data sources. It discusses different retrieval techniques, including sparse retrieval, dense retrieval, internet retrieval, and hybrid retrieval, as well as various language models such as AutoEncoder, AutoRegressive, and Encoder-Decoder models. The paper also explores methods to enhance RALMs, including retriever enhancement, language model enhancement, and overall enhancement. Finally, it categorizes data sources into structured and unstructured data. The survey aims to provide a structured insight into RALMs, their potential, and future development in NLP.This survey paper provides a comprehensive overview of Retrieval-Augmented Language Models (RALMs), including Retrieval-Augmented Generation (RAG) and Retrieval-Augmented Understanding (RAU). It explores the paradigm, evolution, taxonomy, and applications of RALMs, emphasizing their components such as retrievers, language models, and augmentations. The paper discusses how these components interact to create diverse model structures and applications across various NLP tasks, including translation, dialogue systems, and knowledge-intensive applications. It also covers evaluation methods for RALMs, highlighting the importance of robustness, accuracy, and relevance in their assessment. The paper acknowledges the limitations of RALMs, particularly in retrieval quality and computational efficiency, and offers directions for future research. The survey includes a GitHub repository with the surveyed works and resources for further study. The paper is structured into nine sections, covering the definition of RALMs, retrieval methods, language models, RALM enhancements, and data sources. It discusses different retrieval techniques, including sparse retrieval, dense retrieval, internet retrieval, and hybrid retrieval, as well as various language models such as AutoEncoder, AutoRegressive, and Encoder-Decoder models. The paper also explores methods to enhance RALMs, including retriever enhancement, language model enhancement, and overall enhancement. Finally, it categorizes data sources into structured and unstructured data. The survey aims to provide a structured insight into RALMs, their potential, and future development in NLP.