Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics

S-Tier
Journal: American Economic Review
Year: 2020
Volume: 110
Issue: 11
Pages: 3634-60

Score contribution per author:

2.681 = (α=2.01 / 3 authors) × 4.0x S-tier

α: calibrated so average coauthorship-adjusted count equals average raw count

Abstract

The credibility revolution in economics has promoted causal identification using randomized control trials (RCT), difference-in-differences (DID), instrumental variables (IV) and regression discontinuity design (RDD). Applying multiple approaches to over 21,000 hypothesis tests published in 25 leading economics journals, we find that the extent of p-hacking and publication bias varies greatly by method. IV (and to a lesser extent DID) are particularly problematic. We find no evidence that (i) papers published in the Top 5 journals are different to others; (ii) the journal "revise and resubmit" process mitigates the problem; (iii) things are improving through time.

Technical Details

RePEc Handle
repec:aea:aecrev:v:110:y:2020:i:11:p:3634-60
Journal Field
General
Author Count
3
Added to Database
2026-01-24