Position: Explain to Question not to Justify

Position: Explain to Question not to Justify

2024 | Przemyslaw Biecek, Wojciech Samek
This paper argues that the field of Explainable Artificial Intelligence (XAI) is currently experiencing a crisis due to divergent and incompatible goals. The authors propose separating XAI into two complementary cultures: BLUE XAI, focused on human/value-oriented explanations, and RED XAI, focused on model/validation-oriented explanations. While BLUE XAI has received significant attention in recent surveys, RED XAI is underexplored and holds great potential for important research necessary to ensure the safety of AI systems. The paper highlights the need for new methods to question models, such as extracting knowledge from well-performing models and spotting and fixing bugs in faulty models. The paper identifies five fallacies that are often wrongly imputed to the entire field of XAI: (1) interpretability is a binary concept, (2) a single XAI silver bullet exists, (3) there is a "true explanation," (4) user-study is the ultimate validation of XAI methods, and (5) explanations need to be aimed at users to increase their trust in explained models. The authors argue that these fallacies are harmful and that XAI techniques should be viewed as a magnification lens that can provide helpful insights for both "transparent" and "black-box" models. The paper also identifies two cultures of thinking about explainability: BLUE XAI, which is focused on human values such as fairness, ethics, and trust, and RED XAI, which is focused on model validation, exploration, and research on model and data. The authors argue that RED XAI is more engineering-focused and that it can benefit more from automating and streamlining the way models are explored. The paper presents several challenges for RED XAI, including the need for benchmarks, tools, and standards, the need for an explorer mindset, and the need for multiple models and Rashomon explanations. The paper also discusses the application of XAI in science, where explanations may be used to generate and validate research hypotheses. The authors conclude that the field of XAI needs to move beyond the current focus on BLUE XAI and explore the potential of RED XAI to ensure the safety of AI systems.This paper argues that the field of Explainable Artificial Intelligence (XAI) is currently experiencing a crisis due to divergent and incompatible goals. The authors propose separating XAI into two complementary cultures: BLUE XAI, focused on human/value-oriented explanations, and RED XAI, focused on model/validation-oriented explanations. While BLUE XAI has received significant attention in recent surveys, RED XAI is underexplored and holds great potential for important research necessary to ensure the safety of AI systems. The paper highlights the need for new methods to question models, such as extracting knowledge from well-performing models and spotting and fixing bugs in faulty models. The paper identifies five fallacies that are often wrongly imputed to the entire field of XAI: (1) interpretability is a binary concept, (2) a single XAI silver bullet exists, (3) there is a "true explanation," (4) user-study is the ultimate validation of XAI methods, and (5) explanations need to be aimed at users to increase their trust in explained models. The authors argue that these fallacies are harmful and that XAI techniques should be viewed as a magnification lens that can provide helpful insights for both "transparent" and "black-box" models. The paper also identifies two cultures of thinking about explainability: BLUE XAI, which is focused on human values such as fairness, ethics, and trust, and RED XAI, which is focused on model validation, exploration, and research on model and data. The authors argue that RED XAI is more engineering-focused and that it can benefit more from automating and streamlining the way models are explored. The paper presents several challenges for RED XAI, including the need for benchmarks, tools, and standards, the need for an explorer mindset, and the need for multiple models and Rashomon explanations. The paper also discusses the application of XAI in science, where explanations may be used to generate and validate research hypotheses. The authors conclude that the field of XAI needs to move beyond the current focus on BLUE XAI and explore the potential of RED XAI to ensure the safety of AI systems.
Reach us at info@study.space
Understanding Explain to Question not to Justify