T-RAG: Lessons from the LLM Trenches Masoomali Fatehkia, Ji Kim Lucas, Sanjay Chawla Qatar Computing Research Institute Hamad Bin Khalifa University Doha {mfatehkia, jlucas, schawla}@hbku.edu.qa Abstract: Large Language Models (LLMs) have shown remarkable language capabilities, fueling attempts to integrate them into applications across various domains. An important application area is question answering over private enterprise documents, where data security is a key concern. Retrieval-Augmented Generation (RAG) has emerged as the most prominent framework for building LLM-based applications. While building a RAG is relatively straightforward, making it robust and reliable requires extensive customization and domain knowledge. This paper shares experiences building and deploying an LLM application for question answering over private organizational documents. The application combines RAG with a finetuned open-source LLM and uses a tree structure to represent entity hierarchies. This tree-based context enhances the response to queries about entities within the organization's hierarchy. Evaluations show that this combination performs better than simple RAG or finetuning. Lessons learned from building an LLM application for real-world use are also shared. Introduction: Large Language Models (LLMs) represent recent advances in Natural Language Processing (NLP), demonstrating a wide range of language processing capabilities. They have gained prominence after the release of ChatGPT, an application by OpenAI. LLMs are being used for various applications, including creative writing, programming, legal, and medical domains, which require factual accuracy. A promising application area for LLMs is question answering over proprietary organizational documents such as governance/policy manuals. These documents are frequently referenced as they guide day-to-day operations and decision-making within an organization. This results in frequent references to such documents or experts within the organization who respond to queries about such information. Hence, there is potential for increased efficiency from having an application that can respond to a diverse range of user queries based on organizational documents. Deploying an LLM application in such settings requires several considerations. One major concern is the security risks given the confidential nature of such documents. As a result, it is not possible to use proprietary LLM models over an API due to data leakage risks. This necessitates the use of open-source models that can be deployed on-premise. A second concern is limited computational resources and relatively smaller training datasets that can be generated based on the available documents. Finally, any such application must be able to reliably and correctly respond to user queries. Use Case: Our use case is question answering based on an organization’s governance manual. The main features of such a document are (i) descriptions of the organization’s governing principles, duties and responsibilities of various governing bodies and (ii) details about the full hierarchy of entities under the organization and their categorizations. An LLM application responding to questions based on the document should beT-RAG: Lessons from the LLM Trenches Masoomali Fatehkia, Ji Kim Lucas, Sanjay Chawla Qatar Computing Research Institute Hamad Bin Khalifa University Doha {mfatehkia, jlucas, schawla}@hbku.edu.qa Abstract: Large Language Models (LLMs) have shown remarkable language capabilities, fueling attempts to integrate them into applications across various domains. An important application area is question answering over private enterprise documents, where data security is a key concern. Retrieval-Augmented Generation (RAG) has emerged as the most prominent framework for building LLM-based applications. While building a RAG is relatively straightforward, making it robust and reliable requires extensive customization and domain knowledge. This paper shares experiences building and deploying an LLM application for question answering over private organizational documents. The application combines RAG with a finetuned open-source LLM and uses a tree structure to represent entity hierarchies. This tree-based context enhances the response to queries about entities within the organization's hierarchy. Evaluations show that this combination performs better than simple RAG or finetuning. Lessons learned from building an LLM application for real-world use are also shared. Introduction: Large Language Models (LLMs) represent recent advances in Natural Language Processing (NLP), demonstrating a wide range of language processing capabilities. They have gained prominence after the release of ChatGPT, an application by OpenAI. LLMs are being used for various applications, including creative writing, programming, legal, and medical domains, which require factual accuracy. A promising application area for LLMs is question answering over proprietary organizational documents such as governance/policy manuals. These documents are frequently referenced as they guide day-to-day operations and decision-making within an organization. This results in frequent references to such documents or experts within the organization who respond to queries about such information. Hence, there is potential for increased efficiency from having an application that can respond to a diverse range of user queries based on organizational documents. Deploying an LLM application in such settings requires several considerations. One major concern is the security risks given the confidential nature of such documents. As a result, it is not possible to use proprietary LLM models over an API due to data leakage risks. This necessitates the use of open-source models that can be deployed on-premise. A second concern is limited computational resources and relatively smaller training datasets that can be generated based on the available documents. Finally, any such application must be able to reliably and correctly respond to user queries. Use Case: Our use case is question answering based on an organization’s governance manual. The main features of such a document are (i) descriptions of the organization’s governing principles, duties and responsibilities of various governing bodies and (ii) details about the full hierarchy of entities under the organization and their categorizations. An LLM application responding to questions based on the document should be