This paper explores the role of explainability in medical AI, particularly in the context of clinical decision support systems (CDSS). It adopts a multidisciplinary approach, examining the relevance of explainability from technological, legal, medical, and patient perspectives. The authors argue that explainability is not just a technological issue but also involves ethical, legal, and societal considerations. They conclude that omitting explainability in CDSS can threaten core ethical values and have detrimental effects on individual and public health. The paper emphasizes the need for multidisciplinary collaboration to address the challenges and limitations of opaque algorithms in medical AI, ensuring that AI-driven tools are developed and used in a way that respects patient autonomy, promotes beneficence and non-maleficence, and ensures justice.This paper explores the role of explainability in medical AI, particularly in the context of clinical decision support systems (CDSS). It adopts a multidisciplinary approach, examining the relevance of explainability from technological, legal, medical, and patient perspectives. The authors argue that explainability is not just a technological issue but also involves ethical, legal, and societal considerations. They conclude that omitting explainability in CDSS can threaten core ethical values and have detrimental effects on individual and public health. The paper emphasizes the need for multidisciplinary collaboration to address the challenges and limitations of opaque algorithms in medical AI, ensuring that AI-driven tools are developed and used in a way that respects patient autonomy, promotes beneficence and non-maleficence, and ensures justice.