This paper presents a methodology for evaluating the adequacy of saliency methods, which are used to highlight relevant features in an input for a learned model. The authors propose two randomization tests: one for model parameters and one for data labels. These tests assess whether a saliency method is sensitive to the model's parameters or the relationship between inputs and labels. The results show that some existing saliency methods are independent of both the model and the data generating process, making them inadequate for tasks that depend on the model or the data. The authors also compare their findings with edge detection, a technique that does not require training data or a model. They find that methods similar to edge detection, such as Guided Backprop and its variants, are insensitive to model parameters and fail the proposed tests. The paper concludes that saliency methods that fail these tests are incapable of supporting tasks that require explanations faithful to the model or the data generating process. The authors also provide an analysis of linear models and a simple 1-layer convolutional network to support their findings.This paper presents a methodology for evaluating the adequacy of saliency methods, which are used to highlight relevant features in an input for a learned model. The authors propose two randomization tests: one for model parameters and one for data labels. These tests assess whether a saliency method is sensitive to the model's parameters or the relationship between inputs and labels. The results show that some existing saliency methods are independent of both the model and the data generating process, making them inadequate for tasks that depend on the model or the data. The authors also compare their findings with edge detection, a technique that does not require training data or a model. They find that methods similar to edge detection, such as Guided Backprop and its variants, are insensitive to model parameters and fail the proposed tests. The paper concludes that saliency methods that fail these tests are incapable of supporting tasks that require explanations faithful to the model or the data generating process. The authors also provide an analysis of linear models and a simple 1-layer convolutional network to support their findings.