Artificial Intelligence-Generated Draft Replies to Patient Inbox Messages

Artificial Intelligence-Generated Draft Replies to Patient Inbox Messages

2024-03-20 | Patricia Garcia, MD; Stephen P. Ma, MD, PhD; Shreya Shah, MD; Margaret Smith, MBA; Jejin Jeong, BA; Anna Devon-Sand, MPH; Ming Tai-Seale, PhD, MPH; Kevin Takazawa, BBA; Danyelle Clutter, MBA; Kyle Vogt, BA; Carlene Lugtu, MGIM; Matthew Rojo, MS; Steven Lin, MD; Tait Shanafelt, MD; Michael A. Pfeffer, MD; Christopher Sharp, MD
This study evaluates the implementation and impact of a large language model (LLM) used to draft responses to patient messages in electronic inboxes at Stanford Health Care. The study was conducted over 5 weeks with 162 clinicians from primary care and gastroenterology and hepatology divisions. The primary outcome was the utilization rate of AI-generated draft replies, which was 20%. Secondary outcomes included changes in time measures and clinician experience assessed through surveys. There were no significant changes in reply action time, read time, or write time. However, there were statistically significant reductions in the 4-item physician task load score derivative and work exhaustion scores. Clinicians reported high expectations for the utility and time-saving benefits of the LLM, which were largely met or exceeded. Qualitative feedback highlighted the need for improvements in tone, brevity, and personalization. The study suggests that LLMs can be spontaneously adopted and used to improve clinician well-being, but further research is needed to optimize their use and address time-related concerns.This study evaluates the implementation and impact of a large language model (LLM) used to draft responses to patient messages in electronic inboxes at Stanford Health Care. The study was conducted over 5 weeks with 162 clinicians from primary care and gastroenterology and hepatology divisions. The primary outcome was the utilization rate of AI-generated draft replies, which was 20%. Secondary outcomes included changes in time measures and clinician experience assessed through surveys. There were no significant changes in reply action time, read time, or write time. However, there were statistically significant reductions in the 4-item physician task load score derivative and work exhaustion scores. Clinicians reported high expectations for the utility and time-saving benefits of the LLM, which were largely met or exceeded. Qualitative feedback highlighted the need for improvements in tone, brevity, and personalization. The study suggests that LLMs can be spontaneously adopted and used to improve clinician well-being, but further research is needed to optimize their use and address time-related concerns.
Reach us at info@study.space
[slides and audio] Artificial Intelligence%E2%80%93Generated Draft Replies to Patient Inbox Messages