GastroBot: a Chinese gastrointestinal disease chatbot based on the retrieval-augmented generation

GastroBot: a Chinese gastrointestinal disease chatbot based on the retrieval-augmented generation

22 May 2024 | Qingqing Zhou, Can Liu, Yuchen Duan, Kaijie Sun, Yu Li, Hongxing Kan, Zongyun Gu, Jianhua Shu, Jili Hu
GastroBot is a Chinese gastrointestinal disease chatbot developed using Retrieval-Augmented Generation (RAG) technology. The study aims to enhance the accuracy and relevance of answers by integrating external knowledge sources, addressing the limitations of large language models (LLMs) in clinical applications. The RAG model was fine-tuned using 25 guidelines and 40 recent literature sources on gastrointestinal diseases, resulting in an 18% improvement in hit rate compared to the base model and a 20% improvement over OpenAI's embedding model. GastroBot was evaluated using the RAGAS framework, achieving a context recall rate of 95%, faithfulness of 93.73%, and answer relevancy of 92.28%. Human assessments using the SUS (Safety, Usability, and Smoothness) method also demonstrated high scores for GastroBot, indicating its effectiveness in providing precise and contextually relevant information. The research highlights the potential of RAG technology in improving clinical decision support and addresses the challenges of gastrointestinal disease management in China.GastroBot is a Chinese gastrointestinal disease chatbot developed using Retrieval-Augmented Generation (RAG) technology. The study aims to enhance the accuracy and relevance of answers by integrating external knowledge sources, addressing the limitations of large language models (LLMs) in clinical applications. The RAG model was fine-tuned using 25 guidelines and 40 recent literature sources on gastrointestinal diseases, resulting in an 18% improvement in hit rate compared to the base model and a 20% improvement over OpenAI's embedding model. GastroBot was evaluated using the RAGAS framework, achieving a context recall rate of 95%, faithfulness of 93.73%, and answer relevancy of 92.28%. Human assessments using the SUS (Safety, Usability, and Smoothness) method also demonstrated high scores for GastroBot, indicating its effectiveness in providing precise and contextually relevant information. The research highlights the potential of RAG technology in improving clinical decision support and addresses the challenges of gastrointestinal disease management in China.
Reach us at info@study.space
[slides and audio] GastroBot%3A a Chinese gastrointestinal disease chatbot based on the retrieval-augmented generation