This review discusses the importance of explainable artificial intelligence (XAI) in handling multimodal and longitudinal data in medical imaging. XAI aims to provide insights into AI models to enhance physician efficacy and patient safety. The study highlights the challenges of XAI systems that can handle multimodal and longitudinal data, which are essential in many clinical workflows. The authors propose the XAI orchestrator, an instance that helps clinicians synthesize multimodal and longitudinal data, AI predictions, and explainability outputs. The XAI orchestrator is designed to be adaptive, hierarchical, interactive, and uncertainty-aware. XAI systems offer advantages over "black box" models by providing better quality assurance, auditability, and user trust. However, challenges remain, including the lack of studies that integrate other clinical data types (multimodal XAI) or use longitudinal datasets. Merging these data types and deriving meaningful explanations is challenging and has received little attention. The authors argue that further developments of multimodal and longitudinal XAI are essential in many clinical workflows. The study also discusses previous work on XAI for multimodal data, highlighting the benefits of multimodal fusion over single modality data. It notes that multimodal data can improve model robustness and accuracy, enable the discovery of new biomarkers, and improve model performance. However, challenges such as the choice of XAI method, domain knowledge, and the curse of dimensionality need to be addressed. For longitudinal data, the study emphasizes the importance of temporal evolution of biological processes in healthcare. It discusses the challenges of integrating time series data into XAI models, including continuous vs. intermittent data recording, data sparsity, and the representation of spatio-temporal relationships. The authors propose the XAI orchestrator as a virtual assistant that can coordinate explanations of specific AI models and provide a user-centered mechanism for further inquiry. The XAI orchestrator is designed to be adaptive, hierarchical, interactive, and uncertainty-aware. It is also expected to be time-effective, causality- and co-dependency-aware, modular, privacy-preserving, resilient to data drift, and up-to-date. The study also discusses the potential of large language models (LLMs) in XAI, highlighting their ability to process and respond to text input and logical reasoning. The authors suggest that LLM-based orchestrators could be beneficial in clinical settings as they could provide verbal explanations adapted to the current user and situation. They also emphasize the importance of involving clinical domain experts in the design, development, implementation, and maintenance of (X)AI systems to ensure their reliability, data security, and trustworthiness.This review discusses the importance of explainable artificial intelligence (XAI) in handling multimodal and longitudinal data in medical imaging. XAI aims to provide insights into AI models to enhance physician efficacy and patient safety. The study highlights the challenges of XAI systems that can handle multimodal and longitudinal data, which are essential in many clinical workflows. The authors propose the XAI orchestrator, an instance that helps clinicians synthesize multimodal and longitudinal data, AI predictions, and explainability outputs. The XAI orchestrator is designed to be adaptive, hierarchical, interactive, and uncertainty-aware. XAI systems offer advantages over "black box" models by providing better quality assurance, auditability, and user trust. However, challenges remain, including the lack of studies that integrate other clinical data types (multimodal XAI) or use longitudinal datasets. Merging these data types and deriving meaningful explanations is challenging and has received little attention. The authors argue that further developments of multimodal and longitudinal XAI are essential in many clinical workflows. The study also discusses previous work on XAI for multimodal data, highlighting the benefits of multimodal fusion over single modality data. It notes that multimodal data can improve model robustness and accuracy, enable the discovery of new biomarkers, and improve model performance. However, challenges such as the choice of XAI method, domain knowledge, and the curse of dimensionality need to be addressed. For longitudinal data, the study emphasizes the importance of temporal evolution of biological processes in healthcare. It discusses the challenges of integrating time series data into XAI models, including continuous vs. intermittent data recording, data sparsity, and the representation of spatio-temporal relationships. The authors propose the XAI orchestrator as a virtual assistant that can coordinate explanations of specific AI models and provide a user-centered mechanism for further inquiry. The XAI orchestrator is designed to be adaptive, hierarchical, interactive, and uncertainty-aware. It is also expected to be time-effective, causality- and co-dependency-aware, modular, privacy-preserving, resilient to data drift, and up-to-date. The study also discusses the potential of large language models (LLMs) in XAI, highlighting their ability to process and respond to text input and logical reasoning. The authors suggest that LLM-based orchestrators could be beneficial in clinical settings as they could provide verbal explanations adapted to the current user and situation. They also emphasize the importance of involving clinical domain experts in the design, development, implementation, and maintenance of (X)AI systems to ensure their reliability, data security, and trustworthiness.