This article discusses the importance of understanding how machine learning models make decisions, particularly in complex tasks like computer vision and arcade games. The authors propose a method called Spectral Relevance Analysis (SpRAy) to characterize and validate the behavior of nonlinear learning machines. They argue that current performance metrics are insufficient for assessing the validity of a model's decision-making process, as they may not distinguish between different types of problem-solving behaviors, such as naive, short-sighted, or strategic approaches. The study highlights the "Clever Hans" phenomenon, where models may rely on spurious correlations in the training data, leading to incorrect decisions in real-world scenarios. The authors analyze several examples, including a model that misclassifies images based on a dataset artifact (a source tag) and a model that exploits a game loophole in Atari Pinball. They also demonstrate how SpRAy can identify these behaviors by analyzing heatmaps of model predictions. The method is applied to large datasets, enabling the detection of unexpected or undesirable decision strategies. The study emphasizes the need for more nuanced evaluation of machine learning models, beyond traditional performance metrics, to ensure their reliability and generalizability. The results show that SpRAy can effectively identify and characterize different decision strategies, providing insights into the behavior of learning machines. The authors conclude that understanding how models make decisions is crucial for assessing their validity and ensuring they perform reliably in real-world applications.This article discusses the importance of understanding how machine learning models make decisions, particularly in complex tasks like computer vision and arcade games. The authors propose a method called Spectral Relevance Analysis (SpRAy) to characterize and validate the behavior of nonlinear learning machines. They argue that current performance metrics are insufficient for assessing the validity of a model's decision-making process, as they may not distinguish between different types of problem-solving behaviors, such as naive, short-sighted, or strategic approaches. The study highlights the "Clever Hans" phenomenon, where models may rely on spurious correlations in the training data, leading to incorrect decisions in real-world scenarios. The authors analyze several examples, including a model that misclassifies images based on a dataset artifact (a source tag) and a model that exploits a game loophole in Atari Pinball. They also demonstrate how SpRAy can identify these behaviors by analyzing heatmaps of model predictions. The method is applied to large datasets, enabling the detection of unexpected or undesirable decision strategies. The study emphasizes the need for more nuanced evaluation of machine learning models, beyond traditional performance metrics, to ensure their reliability and generalizability. The results show that SpRAy can effectively identify and characterize different decision strategies, providing insights into the behavior of learning machines. The authors conclude that understanding how models make decisions is crucial for assessing their validity and ensuring they perform reliably in real-world applications.