September 2024 | Matthew Maksimoski, MD; Anisha Rhea Noble, MD; David F. Smith, MD, PhD
The study investigates the accuracy of ChatGPT in answering medical questions related to otolaryngology. A total of 93 questions were asked, covering seven major domains and 21 clinical practice guidelines (CPGs). Answers were graded as correct, partially correct, incorrect, or non-answers. The results showed that ChatGPT provided correct answers at a rate of 45.5%, partially correct answers at 31.8%, incorrect answers at 21.6%, and non-answers at 1.1%. Patient-level questions had a higher rate of correct answers (71.4%) compared to CPG-based questions (37.3%). The study found no significant difference in the rate of correct answers between CPGs published before and after the data collection period cited by ChatGPT. The authors conclude that while ChatGPT can provide some accurate information, it is not a reliable source for medical questions, especially in otolaryngology. They recommend that physicians and patients use established medical reference sites and expert content for more accurate and reliable information.The study investigates the accuracy of ChatGPT in answering medical questions related to otolaryngology. A total of 93 questions were asked, covering seven major domains and 21 clinical practice guidelines (CPGs). Answers were graded as correct, partially correct, incorrect, or non-answers. The results showed that ChatGPT provided correct answers at a rate of 45.5%, partially correct answers at 31.8%, incorrect answers at 21.6%, and non-answers at 1.1%. Patient-level questions had a higher rate of correct answers (71.4%) compared to CPG-based questions (37.3%). The study found no significant difference in the rate of correct answers between CPGs published before and after the data collection period cited by ChatGPT. The authors conclude that while ChatGPT can provide some accurate information, it is not a reliable source for medical questions, especially in otolaryngology. They recommend that physicians and patients use established medical reference sites and expert content for more accurate and reliable information.