The chapter discusses the application of multimodal large language models (LLMs) in medicine, focusing on OpenAI's release of GPT-4V in September 2023. GPT-4V is a multimodal foundation model that integrates large language models with vision input, enabling ChatGPT to process images, speech, and text simultaneously. This advancement opens new possibilities for clinical work processes and applications, particularly in areas such as transcription, image interpretation, optical character recognition, and video understanding. Key benefits include: 1. **Seamless Transcription and Summarization**: Directly generating clinical records or letters from doctor-patient consultations. 2. **Enhanced Image Interpretation**: Integrating patient history, imaging indications, and comparisons with previous images to provide recommendations. 3. **Optical Character Recognition**: Detecting numerical and textual information from images. 4. **Video Understanding**: Automating procedural documentation and scene understanding for improved efficiency and accuracy. However, the adoption of multimodal LLMs in healthcare also poses challenges: 1. **Hallucinations**: Incorrect or nonsensical outputs, such as an incorrect ECG interpretation. 2. **Privacy Concerns**: Increased risk of patient data exposure due to the growing size of foundation models. 3. **Regulatory Challenges**: Need for novel regulatory approaches to test and mitigate AI failures, especially with the emergent intelligence of foundation models. Despite these challenges, the potential of multimodal AI in medicine is significant, and the release of GPT-4V will drive future efforts in responsible development, use, and regulation of medical AI.The chapter discusses the application of multimodal large language models (LLMs) in medicine, focusing on OpenAI's release of GPT-4V in September 2023. GPT-4V is a multimodal foundation model that integrates large language models with vision input, enabling ChatGPT to process images, speech, and text simultaneously. This advancement opens new possibilities for clinical work processes and applications, particularly in areas such as transcription, image interpretation, optical character recognition, and video understanding. Key benefits include: 1. **Seamless Transcription and Summarization**: Directly generating clinical records or letters from doctor-patient consultations. 2. **Enhanced Image Interpretation**: Integrating patient history, imaging indications, and comparisons with previous images to provide recommendations. 3. **Optical Character Recognition**: Detecting numerical and textual information from images. 4. **Video Understanding**: Automating procedural documentation and scene understanding for improved efficiency and accuracy. However, the adoption of multimodal LLMs in healthcare also poses challenges: 1. **Hallucinations**: Incorrect or nonsensical outputs, such as an incorrect ECG interpretation. 2. **Privacy Concerns**: Increased risk of patient data exposure due to the growing size of foundation models. 3. **Regulatory Challenges**: Need for novel regulatory approaches to test and mitigate AI failures, especially with the emergent intelligence of foundation models. Despite these challenges, the potential of multimodal AI in medicine is significant, and the release of GPT-4V will drive future efforts in responsible development, use, and regulation of medical AI.