Using simulation studies to evaluate statistical methods

Using simulation studies to evaluate statistical methods

Received 29 November 2017; Revised 23 August 2018; Accepted 02 October 2018 | Tim P Morris, Ian R White, Michael J Crowther
This tutorial provides a structured approach for planning and reporting simulation studies, emphasizing the importance of defining aims, data-generating mechanisms, estimands, methods, and performance measures (ADEMP). It outlines coherent terminology for simulation studies, guidance on coding, and critical discussion of key performance measures and their estimation. The tutorial also offers suggestions for structuring tabular and graphical presentations of results and introduces new graphical presentations. It reviews 100 articles from Statistics in Medicine Volume 34 that included at least one simulation study, identifying areas for improvement. The review highlights that many simulation studies lack proper design, analysis, and reporting. The tutorial provides a systematic approach to planning simulation studies, including the ADEMP framework, and discusses the purposes of various performance measures and their estimation, stressing the importance of Monte Carlo standard error (SE) as a measure of uncertainty. It also outlines how to report simulation studies, using the ADEMP structure, and offers guidance on tabular and graphical presentation of results. The tutorial includes a worked example to illustrate the approaches advocated. It emphasizes the importance of understanding the rationale, design, execution, analysis, and reporting of simulation studies to improve understanding and interpretation of findings. The tutorial also discusses computational and programming issues in simulation studies, including the use of random numbers, setting seeds, and storing states. It provides practical advice on starting small and building up code, and on using different software packages for different methods. The tutorial concludes with a discussion on the analysis of estimates data, including checking for missing values and plots of the estimates data. It emphasizes the importance of exploring the distribution of estimates and their standard errors, and identifying outliers and bivariate outliers. The tutorial also discusses the estimation of performance and Monte Carlo standard errors for some common performance measures, highlighting the importance of accurate estimation and reporting.This tutorial provides a structured approach for planning and reporting simulation studies, emphasizing the importance of defining aims, data-generating mechanisms, estimands, methods, and performance measures (ADEMP). It outlines coherent terminology for simulation studies, guidance on coding, and critical discussion of key performance measures and their estimation. The tutorial also offers suggestions for structuring tabular and graphical presentations of results and introduces new graphical presentations. It reviews 100 articles from Statistics in Medicine Volume 34 that included at least one simulation study, identifying areas for improvement. The review highlights that many simulation studies lack proper design, analysis, and reporting. The tutorial provides a systematic approach to planning simulation studies, including the ADEMP framework, and discusses the purposes of various performance measures and their estimation, stressing the importance of Monte Carlo standard error (SE) as a measure of uncertainty. It also outlines how to report simulation studies, using the ADEMP structure, and offers guidance on tabular and graphical presentation of results. The tutorial includes a worked example to illustrate the approaches advocated. It emphasizes the importance of understanding the rationale, design, execution, analysis, and reporting of simulation studies to improve understanding and interpretation of findings. The tutorial also discusses computational and programming issues in simulation studies, including the use of random numbers, setting seeds, and storing states. It provides practical advice on starting small and building up code, and on using different software packages for different methods. The tutorial concludes with a discussion on the analysis of estimates data, including checking for missing values and plots of the estimates data. It emphasizes the importance of exploring the distribution of estimates and their standard errors, and identifying outliers and bivariate outliers. The tutorial also discusses the estimation of performance and Monte Carlo standard errors for some common performance measures, highlighting the importance of accurate estimation and reporting.
Reach us at info@study.space
Understanding Using simulation studies to evaluate statistical methods