Why we (usually) don’t have to worry about multiple comparisons*

Why we (usually) don’t have to worry about multiple comparisons*

July 13, 2009 | Andrew Gelman†, Jennifer Hill‡, Masanao Yajima§
The paper challenges the traditional approach to multiple comparisons in statistical inference, arguing that the problem is often overstated and can be resolved through a hierarchical Bayesian perspective. It highlights that classical methods, such as the Bonferroni correction, adjust for multiple comparisons by increasing the threshold for significance, which can reduce power to detect true effects. In contrast, multilevel models, which are part of a Bayesian framework, naturally account for the relationships between groups and provide more accurate and efficient estimates by partial pooling. This approach shrinks estimates toward a common mean, reducing uncertainty and improving the reliability of comparisons. The paper illustrates that multilevel models can yield more conservative and accurate results, especially in settings with low group-level variation, where multiple comparisons are a concern. It also discusses the limitations of classical methods, such as the Bonferroni correction, and the benefits of Bayesian approaches in controlling Type S and Type M errors. The paper provides examples, including the Infant Health and Development Program and SAT coaching studies, to demonstrate how multilevel models can effectively address multiple comparisons without the need for traditional corrections. The authors argue that by building multiplicity into the model from the start, researchers can obtain more reliable and meaningful results. The paper concludes that multilevel models offer a more robust and flexible approach to handling multiple comparisons, particularly in complex settings where traditional methods may fail.The paper challenges the traditional approach to multiple comparisons in statistical inference, arguing that the problem is often overstated and can be resolved through a hierarchical Bayesian perspective. It highlights that classical methods, such as the Bonferroni correction, adjust for multiple comparisons by increasing the threshold for significance, which can reduce power to detect true effects. In contrast, multilevel models, which are part of a Bayesian framework, naturally account for the relationships between groups and provide more accurate and efficient estimates by partial pooling. This approach shrinks estimates toward a common mean, reducing uncertainty and improving the reliability of comparisons. The paper illustrates that multilevel models can yield more conservative and accurate results, especially in settings with low group-level variation, where multiple comparisons are a concern. It also discusses the limitations of classical methods, such as the Bonferroni correction, and the benefits of Bayesian approaches in controlling Type S and Type M errors. The paper provides examples, including the Infant Health and Development Program and SAT coaching studies, to demonstrate how multilevel models can effectively address multiple comparisons without the need for traditional corrections. The authors argue that by building multiplicity into the model from the start, researchers can obtain more reliable and meaningful results. The paper concludes that multilevel models offer a more robust and flexible approach to handling multiple comparisons, particularly in complex settings where traditional methods may fail.
Reach us at info@study.space
Understanding Why We (Usually) Don't Have to Worry About Multiple Comparisons