19 March 2024 | Feiyang Yu, Alex Moehring, Oishi Banerjee, Tobias Salz, Nikhil Agarwal & Pranav Rajpurkar
This study investigates the heterogeneous effects of AI assistance on radiologists' performance in chest X-ray diagnosis, involving 140 radiologists and 15 diagnostic tasks. The findings highlight that conventional experience-based factors, such as years of experience, subspecialty, and familiarity with AI tools, do not reliably predict the impact of AI assistance. Lower-performing radiologists do not consistently benefit more from AI assistance, challenging previous assumptions. Instead, the occurrence of AI errors significantly influences treatment outcomes, with inaccurate AI predictions adversely affecting radiologist performance on all pathologies and half of the individual pathologies. The study emphasizes the importance of personalized approaches to clinician-AI collaboration and the need for accurate AI models. By understanding the factors shaping the effectiveness of AI assistance, targeted implementation can maximize benefits for individual clinicians in clinical practice. The results also suggest that higher-quality AI assistance leads to better treatment effects, and AI predictions that underestimate probabilities can lead to better treatment outcomes compared to those that overestimate probabilities. These findings underscore the need for high-quality AI models and comprehensive assessments of multiple factors to optimize the implementation of AI assistance in clinical medicine.This study investigates the heterogeneous effects of AI assistance on radiologists' performance in chest X-ray diagnosis, involving 140 radiologists and 15 diagnostic tasks. The findings highlight that conventional experience-based factors, such as years of experience, subspecialty, and familiarity with AI tools, do not reliably predict the impact of AI assistance. Lower-performing radiologists do not consistently benefit more from AI assistance, challenging previous assumptions. Instead, the occurrence of AI errors significantly influences treatment outcomes, with inaccurate AI predictions adversely affecting radiologist performance on all pathologies and half of the individual pathologies. The study emphasizes the importance of personalized approaches to clinician-AI collaboration and the need for accurate AI models. By understanding the factors shaping the effectiveness of AI assistance, targeted implementation can maximize benefits for individual clinicians in clinical practice. The results also suggest that higher-quality AI assistance leads to better treatment effects, and AI predictions that underestimate probabilities can lead to better treatment outcomes compared to those that overestimate probabilities. These findings underscore the need for high-quality AI models and comprehensive assessments of multiple factors to optimize the implementation of AI assistance in clinical medicine.