Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics: Comment

S-Tier
Journal: American Economic Review
Year: 2022
Volume: 112
Issue: 9
Pages: 3124-36

Authors (2)

Sebastian Kranz (Universität Ulm) Peter Pütz (not in RePEc)

Score contribution per author:

4.022 = (α=2.01 / 2 authors) × 4.0x S-tier

α: calibrated so average coauthorship-adjusted count equals average raw count

Abstract

Brodeur, Cook, and Heyes (2020) study hypothesis tests from economic articles and find evidence for p-hacking and publication bias, in particular for instrumental variable and difference-in-difference studies. When adjusting for rounding errors (introducing a novel method), statistical evidence for p-hacking from randomization tests and caliper tests at the 5 percent significance threshold vanishes for difference-in-difference studies but remains for instrumental variable studies. Results at the 1 percent and 10 percent significance thresholds remain largely similar. In addition, Brodeur, Cook, and Heyes derive latent distributions of z-statistics absent publication bias using two different approaches. We establish for each approach a result that challenges its applicability.

Technical Details

RePEc Handle
repec:aea:aecrev:v:112:y:2022:i:9:p:3124-36
Journal Field
General
Author Count
2
Added to Database
2026-01-25