This paper introduces ChatCAD, an interactive computer-aided diagnosis (CAD) system that integrates large language models (LLMs) with CAD networks to enhance medical image analysis. The system uses LLMs to process and summarize information from CAD outputs, such as diagnosis, lesion segmentation, and report generation, to produce high-quality, patient-friendly reports. The LLMs can interpret medical images and provide explanations, making the system accessible to patients rather than just professionals. The system was tested on chest X-rays, where it improved diagnosis performance by 16.42 percentage points compared to state-of-the-art models. The results show that ChatCAD can generate accurate reports and improve the performance of vision-based CAD models. The system also allows for interactive explanations and medical advice, helping patients understand their conditions and treatment options. The paper discusses the performance of different LLMs, including GPT-3, GPT-3.5 Turbo, and GPT-4, and highlights the benefits of using LLMs in medical diagnosis. The study also evaluates the quality of generated reports and their conciseness and appropriateness. The results show that ChatCAD-generated reports are of high quality and can be used to improve the accuracy and reliability of CAD systems. The paper also discusses the limitations of the current approach, including the need for better datasets and benchmarks, and the importance of data privacy in clinical settings. The study concludes that ChatCAD has the potential to revolutionize clinical decision-making and patient communication by integrating LLMs with CAD networks.This paper introduces ChatCAD, an interactive computer-aided diagnosis (CAD) system that integrates large language models (LLMs) with CAD networks to enhance medical image analysis. The system uses LLMs to process and summarize information from CAD outputs, such as diagnosis, lesion segmentation, and report generation, to produce high-quality, patient-friendly reports. The LLMs can interpret medical images and provide explanations, making the system accessible to patients rather than just professionals. The system was tested on chest X-rays, where it improved diagnosis performance by 16.42 percentage points compared to state-of-the-art models. The results show that ChatCAD can generate accurate reports and improve the performance of vision-based CAD models. The system also allows for interactive explanations and medical advice, helping patients understand their conditions and treatment options. The paper discusses the performance of different LLMs, including GPT-3, GPT-3.5 Turbo, and GPT-4, and highlights the benefits of using LLMs in medical diagnosis. The study also evaluates the quality of generated reports and their conciseness and appropriateness. The results show that ChatCAD-generated reports are of high quality and can be used to improve the accuracy and reliability of CAD systems. The paper also discusses the limitations of the current approach, including the need for better datasets and benchmarks, and the importance of data privacy in clinical settings. The study concludes that ChatCAD has the potential to revolutionize clinical decision-making and patient communication by integrating LLMs with CAD networks.