This paper examines the impact of replications published as comments in the American Economic Review (AER) between 2010 and 2020. The study analyzes how often these comments are cited and whether they influence the citations of the original papers (OPs). The results show that comments are rarely cited and do not significantly affect the citations of the OPs, even when the comments identify substantive issues. A survey of replicators and authors reveals no consensus on whether the OPs' contributions remain valid. The study concludes that the economics literature does not self-correct, and that robustness and replicability are difficult to define in economics.
The AER has a long history of publishing comments, which challenge previously published papers, often based on robustness replications. The AER has also implemented strict data-sharing policies. The study found that comments published in the AER do not affect the citations of the OPs, and their influence on the literature is minimal. The average citation ratio of comments to OPs is 14%, indicating that comments are not cited much in absolute terms. The study also found that the publication of a comment does not affect the citation trend of the OP.
The study further examines citations in the AER, Journal of Economic Literature (JEL), and Journal of Economic Perspectives (JEP). It found that most citations to OPs do not mention the comment, and the majority of high-quality citations still ignore the comment. The study also finds that comments rated as "must-cite" or "sometimes-cite" do not significantly affect the citation trends of the OPs.
The study concludes that economics does not self-correct, and that robustness and replicability are difficult to define in economics. The study highlights the need for more replications and a better understanding of how replications are received in the scientific community. The AER is praised for its systematic publication of comments and rigorous data-sharing policy. The study also notes that the absence of a clear definition of robustness and replicability raises questions about the extent to which empirical economics can live up to the Popperian definition of "science."This paper examines the impact of replications published as comments in the American Economic Review (AER) between 2010 and 2020. The study analyzes how often these comments are cited and whether they influence the citations of the original papers (OPs). The results show that comments are rarely cited and do not significantly affect the citations of the OPs, even when the comments identify substantive issues. A survey of replicators and authors reveals no consensus on whether the OPs' contributions remain valid. The study concludes that the economics literature does not self-correct, and that robustness and replicability are difficult to define in economics.
The AER has a long history of publishing comments, which challenge previously published papers, often based on robustness replications. The AER has also implemented strict data-sharing policies. The study found that comments published in the AER do not affect the citations of the OPs, and their influence on the literature is minimal. The average citation ratio of comments to OPs is 14%, indicating that comments are not cited much in absolute terms. The study also found that the publication of a comment does not affect the citation trend of the OP.
The study further examines citations in the AER, Journal of Economic Literature (JEL), and Journal of Economic Perspectives (JEP). It found that most citations to OPs do not mention the comment, and the majority of high-quality citations still ignore the comment. The study also finds that comments rated as "must-cite" or "sometimes-cite" do not significantly affect the citation trends of the OPs.
The study concludes that economics does not self-correct, and that robustness and replicability are difficult to define in economics. The study highlights the need for more replications and a better understanding of how replications are received in the scientific community. The AER is praised for its systematic publication of comments and rigorous data-sharing policy. The study also notes that the absence of a clear definition of robustness and replicability raises questions about the extent to which empirical economics can live up to the Popperian definition of "science."