Why we (usually) don’t have to worry about multiple comparisons*

Why we (usually) don’t have to worry about multiple comparisons*

July 13, 2009 | Andrew Gelman†, Jennifer Hill‡, Masanao Yajima§
The paper by Gelman, Hill, and Yajima challenges the traditional approach to multiple comparisons in statistical inference, particularly in the context of applied research. They argue that the classical paradigm, which focuses on Type I errors, is not the most appropriate framework for addressing multiple comparisons. Instead, they propose a Bayesian hierarchical modeling approach, which can effectively handle multiple comparisons and yield more reliable estimates. The authors highlight two key differences between the classical and Bayesian perspectives. First, they argue that Type I errors are not a primary concern in many research contexts because the null hypothesis is rarely strictly true. Second, they believe that the issue is not multiple testing but rather insufficient modeling of the relationships between parameters. By using multilevel models, which perform partial pooling, the Bayesian approach addresses the multiple comparisons problem more effectively than classical methods, which typically adjust intervals by making them wider. The paper includes several examples to illustrate the advantages of the Bayesian approach. For instance, in a study comparing average test scores across U.S. states, the Bayesian model provides more informative and reliable comparisons compared to classical multiple comparisons adjustments. Similarly, in an analysis of SAT coaching in eight schools, the Bayesian model shows strong pooling of estimates, reducing the need for multiple comparisons adjustments. The authors also discuss the practical implementation of multilevel models, emphasizing the ease of fitting these models using software like R, Stata, and SAS. They conclude by addressing challenges such as multiple outcomes and other complexities, suggesting that multilevel models can be extended to handle these additional dimensions. Overall, the paper advocates for a Bayesian hierarchical modeling approach as a more robust and flexible solution to the multiple comparisons problem.The paper by Gelman, Hill, and Yajima challenges the traditional approach to multiple comparisons in statistical inference, particularly in the context of applied research. They argue that the classical paradigm, which focuses on Type I errors, is not the most appropriate framework for addressing multiple comparisons. Instead, they propose a Bayesian hierarchical modeling approach, which can effectively handle multiple comparisons and yield more reliable estimates. The authors highlight two key differences between the classical and Bayesian perspectives. First, they argue that Type I errors are not a primary concern in many research contexts because the null hypothesis is rarely strictly true. Second, they believe that the issue is not multiple testing but rather insufficient modeling of the relationships between parameters. By using multilevel models, which perform partial pooling, the Bayesian approach addresses the multiple comparisons problem more effectively than classical methods, which typically adjust intervals by making them wider. The paper includes several examples to illustrate the advantages of the Bayesian approach. For instance, in a study comparing average test scores across U.S. states, the Bayesian model provides more informative and reliable comparisons compared to classical multiple comparisons adjustments. Similarly, in an analysis of SAT coaching in eight schools, the Bayesian model shows strong pooling of estimates, reducing the need for multiple comparisons adjustments. The authors also discuss the practical implementation of multilevel models, emphasizing the ease of fitting these models using software like R, Stata, and SAS. They conclude by addressing challenges such as multiple outcomes and other complexities, suggesting that multilevel models can be extended to handle these additional dimensions. Overall, the paper advocates for a Bayesian hierarchical modeling approach as a more robust and flexible solution to the multiple comparisons problem.
Reach us at info@study.space