The paper introduces a novel approach called "Explainable Benchmarking" to enhance the understanding of heuristic optimization algorithms. It presents the IOHxplainer software framework, which uses explainable AI techniques to analyze and interpret the performance of various optimization algorithms and their components and hyperparameters. The framework is demonstrated through two modular optimization frameworks: modCMA and modDE. By collecting performance data on a large number of algorithm configurations, IOHxplainer provides insights into the impact of different algorithmic components and hyperparameters across diverse scenarios. The framework allows for systematic evaluation and interpretation of iterative optimization heuristics, improving benchmarking and algorithm design. The paper also discusses the limitations of traditional benchmarking methods and highlights the importance of explainable AI in understanding the behavior of complex algorithms.The paper introduces a novel approach called "Explainable Benchmarking" to enhance the understanding of heuristic optimization algorithms. It presents the IOHxplainer software framework, which uses explainable AI techniques to analyze and interpret the performance of various optimization algorithms and their components and hyperparameters. The framework is demonstrated through two modular optimization frameworks: modCMA and modDE. By collecting performance data on a large number of algorithm configurations, IOHxplainer provides insights into the impact of different algorithmic components and hyperparameters across diverse scenarios. The framework allows for systematic evaluation and interpretation of iterative optimization heuristics, improving benchmarking and algorithm design. The paper also discusses the limitations of traditional benchmarking methods and highlights the importance of explainable AI in understanding the behavior of complex algorithms.