Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics: Reply

S-Tier
Journal: American Economic Review
Year: 2022
Volume: 112
Issue: 9
Pages: 3137-39

Score contribution per author:

2.681 = (α=2.01 / 3 authors) × 4.0x S-tier

α: calibrated so average coauthorship-adjusted count equals average raw count

Abstract

In Brodeur, Cook, and Heyes (2020) we present evidence that instrumental variable (and to a lesser extent difference-in-difference) articles are more p-hacked than randomized controlled trial and regression discontinuity design articles. We also find no evidence that (i) articles published in the top five journals are different; (ii) the "revise and resubmit" process mitigates the problem; (iii) things are improving through time. Kranz and Pütz (2022) apply a novel adjustment to address rounding errors. They successfully replicate our results with the exception of our shakiest finding: after adjusting for rounding errors, bunching of test statistics for difference-in-difference articles is now smaller around the 5 percent level (and coincidentally larger at the 10 percent level).

Technical Details

RePEc Handle
repec:aea:aecrev:v:112:y:2022:i:9:p:3137-39
Journal Field
General
Author Count
3
Added to Database
2026-01-24