This paper introduces a novel approach called explainable benchmarking for iterative optimization heuristics, using the IOHxplainer software framework. The framework enables the analysis and understanding of the performance of various optimization algorithms and the impact of their components and hyperparameters. The paper demonstrates the framework in the context of two modular optimization frameworks, modCMA and modDE, and shows how it can be used to examine the impact of different algorithmic components and configurations. The framework provides a systematic method for evaluating and interpreting the behavior and efficiency of iterative optimization heuristics in a more transparent and comprehensible manner, allowing for better benchmarking and algorithm design. Explainable AI (XAI) is used to enhance and better understand EC algorithms, and vice versa. XAI techniques are applied to decipher the mechanisms behind the algorithms' search processes and behaviors. These techniques can illuminate the internal workings of algorithm components, such as selection, crossover, and mutation in genetic algorithms, or the update rules in swarm intelligence algorithms. Understanding these components through the lens of XAI can lead to the development of more efficient and effective algorithms. The paper discusses the challenges of benchmarking heuristic algorithms, which are often evaluated in isolation, leading to limited insights into their comparative performance and practical applicability. The proposed framework addresses these challenges by employing modular optimization approaches and explainable AI techniques to derive insights into the algorithmic behavior of a large set of algorithm components and their hyperparameters. Modular optimization frameworks allow for the comparison of various modifications on a core algorithm, facilitating a deeper understanding of each component's influence on the algorithm's performance in different scenarios. The paper also discusses the importance of explainable benchmarking in understanding the impact of algorithmic components and hyperparameters on the performance of optimization algorithms. The framework is used to analyze the performance of modCMA and modDE on a wide variety of benchmark functions, and to identify the best configurations for each function. The results show that certain hyperparameters and configurations significantly impact the performance of the algorithms, and that the best configurations vary depending on the function and dimensionality. The paper also discusses the structural bias analysis, which is part of the IOHxplainer framework. This analysis helps to identify and correct structural biases in the algorithm configurations, which can affect the performance of the algorithms on certain functions. The results show that many single-best solutions exhibit structurally biased behavior, which can impact the performance of the algorithms on certain functions. Overall, the paper presents a comprehensive framework for explainable benchmarking in iterative optimization heuristics, which allows for a more transparent and comprehensible evaluation of the performance of optimization algorithms and their components. The framework provides insights into the impact of different algorithmic components and configurations on the performance of the algorithms, and enables better benchmarking and algorithm design.This paper introduces a novel approach called explainable benchmarking for iterative optimization heuristics, using the IOHxplainer software framework. The framework enables the analysis and understanding of the performance of various optimization algorithms and the impact of their components and hyperparameters. The paper demonstrates the framework in the context of two modular optimization frameworks, modCMA and modDE, and shows how it can be used to examine the impact of different algorithmic components and configurations. The framework provides a systematic method for evaluating and interpreting the behavior and efficiency of iterative optimization heuristics in a more transparent and comprehensible manner, allowing for better benchmarking and algorithm design. Explainable AI (XAI) is used to enhance and better understand EC algorithms, and vice versa. XAI techniques are applied to decipher the mechanisms behind the algorithms' search processes and behaviors. These techniques can illuminate the internal workings of algorithm components, such as selection, crossover, and mutation in genetic algorithms, or the update rules in swarm intelligence algorithms. Understanding these components through the lens of XAI can lead to the development of more efficient and effective algorithms. The paper discusses the challenges of benchmarking heuristic algorithms, which are often evaluated in isolation, leading to limited insights into their comparative performance and practical applicability. The proposed framework addresses these challenges by employing modular optimization approaches and explainable AI techniques to derive insights into the algorithmic behavior of a large set of algorithm components and their hyperparameters. Modular optimization frameworks allow for the comparison of various modifications on a core algorithm, facilitating a deeper understanding of each component's influence on the algorithm's performance in different scenarios. The paper also discusses the importance of explainable benchmarking in understanding the impact of algorithmic components and hyperparameters on the performance of optimization algorithms. The framework is used to analyze the performance of modCMA and modDE on a wide variety of benchmark functions, and to identify the best configurations for each function. The results show that certain hyperparameters and configurations significantly impact the performance of the algorithms, and that the best configurations vary depending on the function and dimensionality. The paper also discusses the structural bias analysis, which is part of the IOHxplainer framework. This analysis helps to identify and correct structural biases in the algorithm configurations, which can affect the performance of the algorithms on certain functions. The results show that many single-best solutions exhibit structurally biased behavior, which can impact the performance of the algorithms on certain functions. Overall, the paper presents a comprehensive framework for explainable benchmarking in iterative optimization heuristics, which allows for a more transparent and comprehensible evaluation of the performance of optimization algorithms and their components. The framework provides insights into the impact of different algorithmic components and configurations on the performance of the algorithms, and enables better benchmarking and algorithm design.