The paper "In-context Learning with Retrieved Demonstrations for Language Models: A Survey" by Man Luo, Xin Xu, Yue Liu, Panupong Pasupat, and Mehran Kazemi reviews the advancements in in-context learning (ICL) using retrieved demonstrations for large language models (LLMs). ICL allows LLMs to perform new tasks with minimal fine-tuning, leveraging a few input-output examples. The paper highlights the importance of adaptive demonstration selection, where a specialized retriever curates tailored demonstrations for each task input, improving efficiency and reducing biases compared to static or random demonstrations. The authors discuss various design choices for retrieval models, training procedures, and inference algorithms. They explore different retrieval objectives, such as similarity and diversity, and the impact of demonstration formatting, order, and diversity on model performance. The paper also reviews off-the-shelf and fine-tuned demonstration retrievers, including term-based similarity methods (e.g., BM25), sentence embedding similarity methods (e.g., SBERT), and pre-trained dual encoder models. The effectiveness of retrieval-based ICL is demonstrated across four categories of tasks: natural language understanding, reasoning, knowledge-based QA, and text generation. The paper provides a comprehensive analysis of the strengths and limitations of different retrieval methods and their applications in various NLP tasks. Overall, the survey serves as a critical resource for researchers and practitioners interested in advancing ICL with retrieved demonstrations.The paper "In-context Learning with Retrieved Demonstrations for Language Models: A Survey" by Man Luo, Xin Xu, Yue Liu, Panupong Pasupat, and Mehran Kazemi reviews the advancements in in-context learning (ICL) using retrieved demonstrations for large language models (LLMs). ICL allows LLMs to perform new tasks with minimal fine-tuning, leveraging a few input-output examples. The paper highlights the importance of adaptive demonstration selection, where a specialized retriever curates tailored demonstrations for each task input, improving efficiency and reducing biases compared to static or random demonstrations. The authors discuss various design choices for retrieval models, training procedures, and inference algorithms. They explore different retrieval objectives, such as similarity and diversity, and the impact of demonstration formatting, order, and diversity on model performance. The paper also reviews off-the-shelf and fine-tuned demonstration retrievers, including term-based similarity methods (e.g., BM25), sentence embedding similarity methods (e.g., SBERT), and pre-trained dual encoder models. The effectiveness of retrieval-based ICL is demonstrated across four categories of tasks: natural language understanding, reasoning, knowledge-based QA, and text generation. The paper provides a comprehensive analysis of the strengths and limitations of different retrieval methods and their applications in various NLP tasks. Overall, the survey serves as a critical resource for researchers and practitioners interested in advancing ICL with retrieved demonstrations.