The paper "T-RAG: Lessons from the LLM Trenches" by Masoomali Fatehkia, Ji Kim Lucas, and Sanjay Chawla from the Qatar Computing Research Institute at Hamad Bin Khalifa University explores the application of Large Language Models (LLMs) in question answering over private enterprise documents. The authors focus on the challenges of data security, limited computational resources, and the need for robust and reliable applications. They introduce Tree-RAG (T-RAG), a system that combines Retrieval-Augmented Generation (RAG) with a finetuned open-source LLM to enhance context and response accuracy. T-RAG uses a tree structure to represent entity hierarchies within an organization, providing additional context for queries about entities within this hierarchy. The paper includes a detailed evaluation, including a Needle in a Haystack test, which shows that T-RAG outperforms simple RAG or finetuning implementations. The authors also share lessons learned from building and deploying an LLM application for real-world use, emphasizing the importance of domain expertise, customization, and user feedback. The paper concludes with a discussion on future work, suggesting the expansion of the system to a wider corpus of documents and the development of a chat-based application.The paper "T-RAG: Lessons from the LLM Trenches" by Masoomali Fatehkia, Ji Kim Lucas, and Sanjay Chawla from the Qatar Computing Research Institute at Hamad Bin Khalifa University explores the application of Large Language Models (LLMs) in question answering over private enterprise documents. The authors focus on the challenges of data security, limited computational resources, and the need for robust and reliable applications. They introduce Tree-RAG (T-RAG), a system that combines Retrieval-Augmented Generation (RAG) with a finetuned open-source LLM to enhance context and response accuracy. T-RAG uses a tree structure to represent entity hierarchies within an organization, providing additional context for queries about entities within this hierarchy. The paper includes a detailed evaluation, including a Needle in a Haystack test, which shows that T-RAG outperforms simple RAG or finetuning implementations. The authors also share lessons learned from building and deploying an LLM application for real-world use, emphasizing the importance of domain expertise, customization, and user feedback. The paper concludes with a discussion on future work, suggesting the expansion of the system to a wider corpus of documents and the development of a chat-based application.