This paper explores the challenges and improvements in Retrieval Augmented Generation (RAG) for question answering on financial documents. The effectiveness of Large Language Models (LLMs) in generating accurate responses is heavily dependent on the quality of input, particularly the retrieval of relevant text chunks. Despite advancements, RAGs still face issues with suboptimal text chunk retrieval, leading to inaccuracies or irrelevant answers. The paper introduces several methodologies to enhance text retrieval, including sophisticated chunking techniques, query expansion, metadata annotations, re-ranking algorithms, and fine-tuning of embedding algorithms. These approaches aim to improve the retrieval quality, thereby enhancing the overall performance and reliability of LLMs in processing and responding to queries. The introduction highlights the limitations of standard LLMs, such as their tendency to hallucinate information and lack of domain-specific knowledge. Fine-tuning and RAG are discussed as primary techniques for improving LLMs on domain-specific tasks. Fine-tuning involves updating model parameters with domain-specific data, while RAG allows LLMs to access new knowledge sources through in-context learning. The limitations of current RAG pipelines are detailed, including uniform chunking, semantic search challenges, and the inability to handle complex document structures. The paper then explores various techniques to improve retrieval, such as recursive chunking, query expansion using Hypothetical Document Embeddings (HyDE), metadata annotations, re-ranking algorithms, and fine-tuning embedding algorithms. Evaluation methods, including structured and unstructured evaluation, are described to assess the model's ability to retrieve context and answer questions accurately. The results from the FinanceBench benchmark show that providing correct context significantly improves accuracy, but even with enhanced retrieval methods, the LLMs still struggle with highly complex and domain-specific questions. The paper concludes by emphasizing the importance of robust retrieval algorithms and suggests future directions, such as implementing knowledge graphs and fine-tuning embedding algorithms based on user-labeled data, to further enhance RAG systems.This paper explores the challenges and improvements in Retrieval Augmented Generation (RAG) for question answering on financial documents. The effectiveness of Large Language Models (LLMs) in generating accurate responses is heavily dependent on the quality of input, particularly the retrieval of relevant text chunks. Despite advancements, RAGs still face issues with suboptimal text chunk retrieval, leading to inaccuracies or irrelevant answers. The paper introduces several methodologies to enhance text retrieval, including sophisticated chunking techniques, query expansion, metadata annotations, re-ranking algorithms, and fine-tuning of embedding algorithms. These approaches aim to improve the retrieval quality, thereby enhancing the overall performance and reliability of LLMs in processing and responding to queries. The introduction highlights the limitations of standard LLMs, such as their tendency to hallucinate information and lack of domain-specific knowledge. Fine-tuning and RAG are discussed as primary techniques for improving LLMs on domain-specific tasks. Fine-tuning involves updating model parameters with domain-specific data, while RAG allows LLMs to access new knowledge sources through in-context learning. The limitations of current RAG pipelines are detailed, including uniform chunking, semantic search challenges, and the inability to handle complex document structures. The paper then explores various techniques to improve retrieval, such as recursive chunking, query expansion using Hypothetical Document Embeddings (HyDE), metadata annotations, re-ranking algorithms, and fine-tuning embedding algorithms. Evaluation methods, including structured and unstructured evaluation, are described to assess the model's ability to retrieve context and answer questions accurately. The results from the FinanceBench benchmark show that providing correct context significantly improves accuracy, but even with enhanced retrieval methods, the LLMs still struggle with highly complex and domain-specific questions. The paper concludes by emphasizing the importance of robust retrieval algorithms and suggests future directions, such as implementing knowledge graphs and fine-tuning embedding algorithms based on user-labeled data, to further enhance RAG systems.