2024-01-30 | Angels Balaguer, Vinamra Benara, Renato Cunha, Roberto Estevão, Todd Hendry, Daniel Holstein, Jennifer Marsman, Nick Mecklenburg, Sara Malvar, Leonardo O. Nunes, Rafael Padilha, Morris Sharp, Bruno Silva, Swati Sharma, Vijay Aski, Ranveer Chandra
This paper compares Retrieval-Augmented Generation (RAG) and fine-tuning as methods for incorporating domain-specific data into Large Language Models (LLMs). The study evaluates these approaches on multiple LLMs, including Llama2-13B, GPT-3.5, and GPT-4, using a dataset from the agriculture industry. The goal is to determine which method is more effective for generating accurate, context-specific knowledge for agricultural applications.
The paper introduces a comprehensive pipeline for generating high-quality, industry-specific questions and answers. This pipeline includes data acquisition, information extraction from PDFs, question and answer generation, and fine-tuning of models. The study also evaluates the effectiveness of RAG and fine-tuning in capturing geographic-specific knowledge and improving the accuracy of answers.
The results show that fine-tuning the model increases accuracy by over 6 percentage points, and combining it with RAG further increases accuracy by 5 percentage points. The fine-tuned model also demonstrates the ability to leverage information from across geographies to answer specific questions, increasing answer similarity from 47% to 72%. The study highlights the importance of domain-specific knowledge in agricultural applications and the potential of LLMs to be adapted to respond to industry-specific needs.
The paper also discusses the trade-offs between RAG and fine-tuning, including the cost and complexity of fine-tuning, and the effectiveness of RAG in providing contextually relevant answers. The study concludes that RAG and fine-tuning can be used together to improve the performance of LLMs in agricultural applications, paving the way for further applications of LLMs in other industrial domains.This paper compares Retrieval-Augmented Generation (RAG) and fine-tuning as methods for incorporating domain-specific data into Large Language Models (LLMs). The study evaluates these approaches on multiple LLMs, including Llama2-13B, GPT-3.5, and GPT-4, using a dataset from the agriculture industry. The goal is to determine which method is more effective for generating accurate, context-specific knowledge for agricultural applications.
The paper introduces a comprehensive pipeline for generating high-quality, industry-specific questions and answers. This pipeline includes data acquisition, information extraction from PDFs, question and answer generation, and fine-tuning of models. The study also evaluates the effectiveness of RAG and fine-tuning in capturing geographic-specific knowledge and improving the accuracy of answers.
The results show that fine-tuning the model increases accuracy by over 6 percentage points, and combining it with RAG further increases accuracy by 5 percentage points. The fine-tuned model also demonstrates the ability to leverage information from across geographies to answer specific questions, increasing answer similarity from 47% to 72%. The study highlights the importance of domain-specific knowledge in agricultural applications and the potential of LLMs to be adapted to respond to industry-specific needs.
The paper also discusses the trade-offs between RAG and fine-tuning, including the cost and complexity of fine-tuning, and the effectiveness of RAG in providing contextually relevant answers. The study concludes that RAG and fine-tuning can be used together to improve the performance of LLMs in agricultural applications, paving the way for further applications of LLMs in other industrial domains.