A comprehensive study on fidelity metrics for XAI explores the challenges of evaluating the reliability of explainable AI (XAI) methods. The study highlights the lack of consensus and reliability in existing fidelity metrics, which are used to assess how well XAI methods explain model predictions. The authors propose a novel methodology to verify these metrics using a transparent model, the decision tree, which allows for perfect fidelity explanations. This approach serves as an objective benchmark for evaluating existing fidelity metrics. The study applies this benchmark to two experiments using synthetic datasets with 52,000 images of 128x128 pixels. The results show that current fidelity metrics often fail to accurately reflect the true fidelity of explanations, with some metrics showing significant deviations from expected values. The study identifies issues such as inconsistency among metrics and sensitivity to out-of-distribution (OOD) samples, which can affect the reliability of fidelity assessments. The authors conclude that existing fidelity metrics are not reliable enough for real-world applications and recommend the development of new metrics to address these limitations. The proposed methodology provides a framework for objective evaluation of fidelity metrics, helping to identify and resolve inconsistencies in current approaches. The study emphasizes the need for more robust and consistent fidelity metrics to ensure the reliability of XAI systems in practical scenarios.A comprehensive study on fidelity metrics for XAI explores the challenges of evaluating the reliability of explainable AI (XAI) methods. The study highlights the lack of consensus and reliability in existing fidelity metrics, which are used to assess how well XAI methods explain model predictions. The authors propose a novel methodology to verify these metrics using a transparent model, the decision tree, which allows for perfect fidelity explanations. This approach serves as an objective benchmark for evaluating existing fidelity metrics. The study applies this benchmark to two experiments using synthetic datasets with 52,000 images of 128x128 pixels. The results show that current fidelity metrics often fail to accurately reflect the true fidelity of explanations, with some metrics showing significant deviations from expected values. The study identifies issues such as inconsistency among metrics and sensitivity to out-of-distribution (OOD) samples, which can affect the reliability of fidelity assessments. The authors conclude that existing fidelity metrics are not reliable enough for real-world applications and recommend the development of new metrics to address these limitations. The proposed methodology provides a framework for objective evaluation of fidelity metrics, helping to identify and resolve inconsistencies in current approaches. The study emphasizes the need for more robust and consistent fidelity metrics to ensure the reliability of XAI systems in practical scenarios.