This paper explores the limitations of Retrieval Augmented Generation (RAG) pipelines in financial document question-answering and proposes techniques to enhance retrieval quality. RAG improves Large Language Models (LLMs) by retrieving relevant text chunks to inform queries. However, suboptimal retrieval can lead to inaccurate answers, even with advanced LLMs. The paper examines current RAG constraints and introduces methods to improve text retrieval, including sophisticated chunking, query expansion, metadata annotations, re-ranking algorithms, and fine-tuning embedding algorithms. The paper discusses the limitations of current RAG pipelines, such as uniform chunking without considering document structure, reliance on cosine similarity leading to irrelevant results, and lack of domain-specific knowledge in embeddings. These issues can result in information loss and incorrect answers, especially for complex financial documents. To address these challenges, the paper proposes various techniques. Chunking strategies are refined to better preserve document context, such as recursive chunking and element-based chunking. Query expansion uses hypothetical document embeddings (HyDE) to generate more relevant chunks. Metadata annotations help distinguish between different documents, while re-ranking algorithms prioritize relevance over similarity. Fine-tuning embedding algorithms can improve domain-specific retrieval. The paper evaluates these techniques using the FinanceBench dataset, which contains financial questions and answers. Results show that providing correct context significantly improves accuracy, but even with advanced models like GPT-4o, there are limitations in answering complex questions. Zero-shot methods like query expansion and re-ranking show some improvement over base RAG, but more advanced techniques are needed for robust performance. The study highlights the importance of accurate retrieval in financial question-answering and suggests future research directions, including knowledge graphs and user-labeled data for embedding fine-tuning. The findings emphasize the need for improved retrieval algorithms to enhance the reliability and accuracy of LLMs in domain-specific tasks.This paper explores the limitations of Retrieval Augmented Generation (RAG) pipelines in financial document question-answering and proposes techniques to enhance retrieval quality. RAG improves Large Language Models (LLMs) by retrieving relevant text chunks to inform queries. However, suboptimal retrieval can lead to inaccurate answers, even with advanced LLMs. The paper examines current RAG constraints and introduces methods to improve text retrieval, including sophisticated chunking, query expansion, metadata annotations, re-ranking algorithms, and fine-tuning embedding algorithms. The paper discusses the limitations of current RAG pipelines, such as uniform chunking without considering document structure, reliance on cosine similarity leading to irrelevant results, and lack of domain-specific knowledge in embeddings. These issues can result in information loss and incorrect answers, especially for complex financial documents. To address these challenges, the paper proposes various techniques. Chunking strategies are refined to better preserve document context, such as recursive chunking and element-based chunking. Query expansion uses hypothetical document embeddings (HyDE) to generate more relevant chunks. Metadata annotations help distinguish between different documents, while re-ranking algorithms prioritize relevance over similarity. Fine-tuning embedding algorithms can improve domain-specific retrieval. The paper evaluates these techniques using the FinanceBench dataset, which contains financial questions and answers. Results show that providing correct context significantly improves accuracy, but even with advanced models like GPT-4o, there are limitations in answering complex questions. Zero-shot methods like query expansion and re-ranking show some improvement over base RAG, but more advanced techniques are needed for robust performance. The study highlights the importance of accurate retrieval in financial question-answering and suggests future research directions, including knowledge graphs and user-labeled data for embedding fine-tuning. The findings emphasize the need for improved retrieval algorithms to enhance the reliability and accuracy of LLMs in domain-specific tasks.