22 May 2024 | Qingqing Zhou, Can Liu, Yuchen Duan, Kaijie Sun, Yu Li, Hongxing Kan, Zongyun Gu, Jianhua Shu and Jili Hu
This study presents GastroBot, a Chinese gastrointestinal disease chatbot based on retrieval-augmented generation (RAG). The chatbot was developed using 25 clinical guidelines and 40 recent gastrointestinal literature sources as external knowledge bases. The embedding model was fine-tuned for gastrointestinal diseases to enhance retrieval performance. GastroBot provides precise diagnosis and treatment recommendations for gastrointestinal patients, improving treatment efficacy. The chatbot was evaluated using the RAGAS framework, which showed a context recall rate of 95%, faithfulness of 93.73%, and answer relevancy of 92.28%. Human assessments indicated GastroBot's excellent performance in safety, usability, and smoothness, with scores of 2.87, 2.72, and 2.88, respectively. The study highlights the effectiveness of RAG technology in clinical gastroenterology, demonstrating its potential to enhance the accuracy and reliability of large language models in clinical applications. The research also identifies potential limitations, such as the quality and accuracy of external knowledge sources, and suggests future directions for improving the model's performance. The study contributes to the field of clinical information processing and decision support in gastroenterology.This study presents GastroBot, a Chinese gastrointestinal disease chatbot based on retrieval-augmented generation (RAG). The chatbot was developed using 25 clinical guidelines and 40 recent gastrointestinal literature sources as external knowledge bases. The embedding model was fine-tuned for gastrointestinal diseases to enhance retrieval performance. GastroBot provides precise diagnosis and treatment recommendations for gastrointestinal patients, improving treatment efficacy. The chatbot was evaluated using the RAGAS framework, which showed a context recall rate of 95%, faithfulness of 93.73%, and answer relevancy of 92.28%. Human assessments indicated GastroBot's excellent performance in safety, usability, and smoothness, with scores of 2.87, 2.72, and 2.88, respectively. The study highlights the effectiveness of RAG technology in clinical gastroenterology, demonstrating its potential to enhance the accuracy and reliability of large language models in clinical applications. The research also identifies potential limitations, such as the quality and accuracy of external knowledge sources, and suggests future directions for improving the model's performance. The study contributes to the field of clinical information processing and decision support in gastroenterology.