This study presents a data-driven approach using explainable artificial intelligence (XAI) to generate suggestions for improving clinical decision support (CDS) alert criteria. The research aimed to develop a method that can identify improvements in alert criteria that might be overlooked or delayed in manual reviews. The study analyzed data from Vanderbilt University Medical Center (VUMC) covering alerts from January 1, 2019, to December 31, 2020. A total of 2,991,823 alert firings with 2,689 features were examined. Machine learning models, particularly LightGBM, were used to predict user responses to alerts, achieving an AUC of 0.919. XAI techniques were applied to generate global and local explanations, which were then evaluated against historical change logs and stakeholder interviews. The study identified 96 helpful suggestions, which could eliminate 278,807 alert firings (9.3%). These suggestions also revealed workflow and education issues. The findings suggest that XAI can improve CDS by identifying scenarios where alerts are not accepted due to workflow, education, or staffing issues. The study highlights the potential of XAI to enhance the quality of CDS by providing transparent and actionable insights. The results demonstrate that XAI can generate suggestions that are more comprehensive and accurate compared to traditional rule-based models. The study also emphasizes the importance of integrating automated techniques into CDS tools to improve healthcare quality and efficiency. The research underscores the value of data-driven approaches in optimizing CDS alerts and improving clinical outcomes.This study presents a data-driven approach using explainable artificial intelligence (XAI) to generate suggestions for improving clinical decision support (CDS) alert criteria. The research aimed to develop a method that can identify improvements in alert criteria that might be overlooked or delayed in manual reviews. The study analyzed data from Vanderbilt University Medical Center (VUMC) covering alerts from January 1, 2019, to December 31, 2020. A total of 2,991,823 alert firings with 2,689 features were examined. Machine learning models, particularly LightGBM, were used to predict user responses to alerts, achieving an AUC of 0.919. XAI techniques were applied to generate global and local explanations, which were then evaluated against historical change logs and stakeholder interviews. The study identified 96 helpful suggestions, which could eliminate 278,807 alert firings (9.3%). These suggestions also revealed workflow and education issues. The findings suggest that XAI can improve CDS by identifying scenarios where alerts are not accepted due to workflow, education, or staffing issues. The study highlights the potential of XAI to enhance the quality of CDS by providing transparent and actionable insights. The results demonstrate that XAI can generate suggestions that are more comprehensive and accurate compared to traditional rule-based models. The study also emphasizes the importance of integrating automated techniques into CDS tools to improve healthcare quality and efficiency. The research underscores the value of data-driven approaches in optimizing CDS alerts and improving clinical outcomes.