This paper addresses the disagreement problem in pixel flipping (PF) benchmarks for explainable AI (XAI). The disagreement arises due to varying occlusion strategies and the choice of feature removal order (most influential first, MIF vs. least influential first, LIF). The study proposes a solution by introducing the R-Reference-out-of-model-scope (R-OMS) score to assess the reliability of occluded samples. This score enables a systematic comparison of occlusion strategies and resolves the disagreement problem by grouping consistent PF rankings. Additionally, the study shows that the insightfulness of MIF and LIF is inversely related to the R-OMS score. To address this, the symmetric relevance gain (SRG) measure is introduced, which combines MIF and LIF into a single metric. This measure breaks the inherent connection to the underlying occlusion strategy and leads to consistent rankings across all strategies. The SRG measure is shown to be quantitatively stable and independent of the random baseline, making it a reliable benchmark for XAI methods. The study also demonstrates that occlusion-based attributions are more faithful to the model than pixel-wise attributions, and that the SRG measure provides a consistent evaluation of XAI methods across all design choices. Overall, the study resolves the disagreement problem in PF benchmarks by decoupling the occlusion strategy from the PF benchmark.This paper addresses the disagreement problem in pixel flipping (PF) benchmarks for explainable AI (XAI). The disagreement arises due to varying occlusion strategies and the choice of feature removal order (most influential first, MIF vs. least influential first, LIF). The study proposes a solution by introducing the R-Reference-out-of-model-scope (R-OMS) score to assess the reliability of occluded samples. This score enables a systematic comparison of occlusion strategies and resolves the disagreement problem by grouping consistent PF rankings. Additionally, the study shows that the insightfulness of MIF and LIF is inversely related to the R-OMS score. To address this, the symmetric relevance gain (SRG) measure is introduced, which combines MIF and LIF into a single metric. This measure breaks the inherent connection to the underlying occlusion strategy and leads to consistent rankings across all strategies. The SRG measure is shown to be quantitatively stable and independent of the random baseline, making it a reliable benchmark for XAI methods. The study also demonstrates that occlusion-based attributions are more faithful to the model than pixel-wise attributions, and that the SRG measure provides a consistent evaluation of XAI methods across all design choices. Overall, the study resolves the disagreement problem in PF benchmarks by decoupling the occlusion strategy from the PF benchmark.