Naguib, Cos anza
Wo king Pape
Does single-blind e iew encou age o discou age p-
hacking?
Discussion Pape s, No. 25-04
P o ided in Coope a ion wi h:
Depa men o Economics, Uni e si y o Be n
Sugges ed Ci a ion: Naguib, Cos anza (2025) : Does single-blind e iew encou age o discou age p-
hacking?, Discussion Pape s, No. 25-04, Uni e si y o Be n, Depa men o Economics, Be n
This Ve sion is a ailable a :
h ps://hdl.handle.ne /10419/324321
S anda d-Nu zungsbedingungen:
Die Dokumen e au EconS o dü en zu eigenen wissenscha lichen
Zwecken und zum P i a geb auch gespeiche und kopie we den.
Sie dü en die Dokumen e nich ü ö en liche ode komme zielle
Zwecke e iel äl igen, ö en lich auss ellen, ö en lich zugänglich
machen, e eiben ode ande wei ig nu zen.
So e n die Ve asse die Dokumen e un e Open-Con en -Lizenzen
(insbesonde e CC-Lizenzen) zu Ve ügung ges ell haben soll en,
gel en abweichend on diesen Nu zungsbedingungen die in de do
genann en Lizenz gewäh en Nu zungs ech e.
Te ms o use:
Documen s in EconS o may be sa ed and copied o you pe sonal
and schola ly pu poses.
You a e no o copy documen s o public o comme cial pu poses, o
exhibi he documen s publicly, o make hem publicly a ailable on he
in e ne , o o dis ibu e o o he wise use he documen s in public.
I he documen s ha e been made a ailable unde an Open Con en
Licence (especially C ea i e Commons Licences), you may exe cise
u he usage igh s as speci ied in he indica ed licence.
h ps://c ea i ecommons.o g/licenses/by/4.0/
Facul y o Business, Economics
and Social Sciences
Depa men o
Economics
Does single-blind e iew encou age
o discou age p-hacking?
Cos anza Naguib
25-04
July, 2025
Schanzenecks asse 1
CH-3012 Be n, Swi ze land
h p://www. wi.unibe.ch
DISCUSSION PAPERS
Does single-blind e iew encou age
o discou age p-hacking?
Cos anza Naguib∗
Abs ac
In 2011, he Ame ican Economic Associa ion (AEA) changed i s pee e iew policy
o all hei jou nals, shi ing om a double-blind p ocess o a single-blind pee - e iew
p ocess. Unde his new sys em, e e ees became awa e o he au ho s’ iden i ies. In
his pape , I explo e whe he his policy change in luenced he p e alence o p-hacking
in published pape s a The Ame ican Economic Re iew.
JEL codes: A11, A14, C13
Keywo ds: p-hacking, single-blind e iew, double-blind e iew, di e ence-in-di e ence
1 In oduc ion
1The e is an ongoing deba e on whe he single-blind o double-blind e iew is mo e
sui able o he pee e iew o scien i ic a icles in economics. In a single-blind e iew,
he au ho emains unawa e o he e e ee’s iden i y, whe eas in a double-blind e iew,
he e e ee is also no in o med abou he au ho ’s iden i y. P oponen s o double-blind
e iew a gue ha i helps mi iga e biases om e e ees. They highligh wo p ima y
conce ns. Fi s , e e ees migh un ai ly ejec pape s om lesse -known au ho s o hose
a ilia ed wi h lowe - anked ins i u ions, i espec i e o he pape ’s quali y. Second, i
gende o e hnic disc imina ion is p esen , pape s au ho ed by women o indi iduals wi h
o eign-sounding names migh ace biased e iews (Blank (1991))2.
Suppo e s o single-blind e iewing usually p esen he ollowing h ee a gumen s.
Fi s , hey claim ha e e ees can o en iden i y he au ho o a pape h ough i s con en
o ci a ions, so double-blind sys ems a e seldom uly anonymous.3Second, hey a gue
∗Uni e si y o Be n
1I hank S e an Egli, Re o Ho s and Thibaud Lau en o he excellen esea ch assis ance.
2I is howe e also possible ha e e ees a e s ic e on mo e p oli ic au ho s, as hey wish o lea e
space also o newcome s and o p e en al eady es ablished schola s om publishing ma ginal pape s, as
sugges ed by Ca d and DellaVigna (2020).
3Acco ding o Blank (1991), e e ees we e able o iden i y he au ho s o a ound 50% o pape s a he
end o he 80s. Despi e he imp o emen in sea ch engines, his sha e seems o ha e emained ai ly s able
up o now (see C essey (2014), Hill and P o os (2003)).
1
ha knowing he au ho ’s name and ins i u ion p o ides aluable con ex ha can a ec
how he pape is ead and e alua ed. Thi d, edi o s o en no e ha double-blind e iew-
ing in ol es addi ional adminis a i e e o , equi ing mo e me iculous p ocedu es in he
edi o ial o ice.
In his pape , I aim o assess whe he he single-blind e iew sys em encou ages o
discou ages p-hacking. P-hacking e e s o p ac ices, such as speci ica ion sea ching, ha
esea che s may use o ob ain mo e a o able p- alues. This o en occu s in esponse o
he challenges o publishing null esul s, as null indings a e o en pe cei ed o be o lowe
quali y (Imbens (2021), Chop a e al. (2024)). Such p ac ices inc ease he p e alence o
alse posi i es in he li e a u e, skewing he esea ch p esen ed o policymake s. Unde
a single-blind sys em, young esea che s and hose om less p es igious ins i u ions may
eel p essu ed o imp ess e e ees wi h eye-ca ching s a is ically signi ican esul s. I his
is he case, i would p o ide ye ano he a gumen agains adop ing a single-blind sys em.
In Ma ch 2011, he Ame ican Economic Associa ion (AEA) decided o swi ch om
a double-blind o a single-blind pee e iew s anda d.4. A he ime, i was he only
one among he so-called op-5 jou nals o use a double-blind sys em, whe eas all he
o he s we e using single-blind e iews. This con as p o ides a amewo k o s udying a
quasi-na u al expe imen . I aim o analyze whe he he policy change by The Ame ican
Economic Re iew was ollowed by a change in p-hacking p ac ices5.
I is wo h no ing ha , as men ioned abo e, in he age o Google, e iewe s a e able o
iden i y he au ho s o a pape om i s ex o ci a ions wi h app oxima ely 50% accu acy
(C essey (2014), Hill and P o os (2003)). This means ha I ac ually es ima e he impac
on p-hacking o an inc ease in he p obabili y o au ho iden i ica ion by he e e ees om
a ound 50% o 100%.
4The s a emen ead: ”Upon a join ecommenda ion o he edi o s o he Ame ican Economic Re iew
and he ou Ame ican Economic Jou nals, he Execu i e Commi ee has o ed o d op he ’double-blind’
e e eeing p ocess o all jou nals o he Ame ican Economic Associa ion. The change o ’single-blind’
e e eeing will become e ec i e on July 1, 2011. Easy access o sea ch engines inc easingly limi s he
e ec i eness o he double-blind p ocess in main aining anonymi y. Fu he , i inc eases he adminis-
a i e cos o he jou nals and makes i ha de o e e ees o iden i y an au ho ’s po en ial con lic s o
in e es a ising, o example, om consul ing.” Sou ce: h ps://c ooked imbe .o g/2011/06/05/should- he-
ame ican-economic- e iew-d op-double-anonymous- e iew/ and Goldbe g (2012). In he p esen pape I
only conside he Ame ican Economic Re iew and no he ou Ame ican Economic Jou nals, as he la e
only s a ed o publish issues in 2009, i.e. wo yea s be o e he change in he pee - e iew s anda d only.
5Ano he change o policy in he pe iod unde sc u iny is he implemen a ion o a s ic page limi on
submissions in 2008 by AER alone among he op-5. While a s ic limi on page coun may also encou age
p-hacking p ac ices, Ca d and DellaVigna (2014) ind ha his in e en ion had essen ially no impac on
he leng h o published pape s, and au ho s mos ly adjus ed hei submission by means o pu ely aes he ic
o ma ing changes in o de o mee he page limi . Hence, i is likely ha his in e en ion did no
in luence he ex en o p-hacking.
2
Fi s , I apply a di e ence-in-di e ence app oach o de e mine whe he a double-blind
e iew policy is associa ed wi h a lowe p opo ion o s a is ically signi ican es s a is ics
being published han a single-blind one. The ea men is ha ing a single-blind policy.
Since no o he Top-5 jou nal was adop ing a double-blind policy in 2011, he e is no
con ol g oup, bu only an always ea ed g oup and a swi che g oup. I am hence in he
amewo k o ” ime- e e se di e ence-in-di e ence”. As shown by Kim and Lee (2018), he
es ima ion p ocedu e is essen ially he same as in he s anda d DiD amewo k. Second, I
in es iga e whe he a se ies o s a is ical es s can de ec p-hacking unde he single-blind
egime, espec i ely unde he double-blind egime in pape s published a The Ame ican
Economic Re iew.
I am he i s o e alua e he impac o di e en pee e iew s anda ds on he ex en o
p-hacking. I ind ha , unde a double-blind e iew sys em, au ho s om op ins i u ions
end o epo a highe p opo ion o s a is ically signi ican esul s, whe eas au ho s om
non- op ins i u ions, on a e age, epo a lowe sha e o s a is ically signi ican indings
han hey do unde a single-blind s anda d. These indings a e obus o a ange o
sensi i i y analyses, including p ocedu es such as de- ounding and weigh ing.
This s udy con ibu es o he g owing body o esea ch on p-hacking, building on
seminal wo k by B odeu e al. (2016), who documen ed p-hacking and publica ion bias
in h ee op economics jou nals (The Ame ican Economic Re iew (AER), The Qua e ly
Jou nal o Economics (QJE), and The Jou nal o Poli ical Economy (JPE)). B odeu e
al. (2020) la e demons a ed ha hese issues a y by es ima ion me hod, wi h K anz
and P¨u z (2022) no ing ha such indings may be ele an ly in luenced by ounding e o s.
Subsequen esea ch by B odeu e al. (2024a, 2024b) showed ha nei he da a a ailabili y
and eplica ion policies no p e- egis a ion and p e-analysis plans signi ican ly educe p-
hacking.
Blanco-Pe ez and B odeu (2020) assessed he impac o a 2015 edi o ial s a emen
om eigh heal h economics jou nals encou aging he publica ion o s a is ically insigni -
ican bu economically ele an esul s. This in e en ion e ec i ely educed p-hacking
and publica ion bias, lowe ing he p opo ion o es s ejec ing he null hypo hesis by
app oxima ely 18 pe cen age poin s. Simila ly, Naguib (2024) s udies he impac o he
omission o signi icance as e isks implemen ed by he AEA jou nals in mid-2016 on he
ex en o p-hacking and publica ion bias, inding essen ially no impac o he policy. Fi-
nally, McCloskey and Michailla (2024) de i e c i ical alues o hypo hesis es ing ha
a e obus o p-hacking.
S udies e alua ing he cos s and bene i s o single-blind s double-blind pee e iew
3
o scien i ic a icles in economics a e sca ce. The ew exis ing pape s p o ide sugges i e
e idence o edi o ial a o i ism in he single-blind e iew p ocess. Blank (1991) analyzes
submissions o The Ame ican Economic Re iew om 1987 o 1989. In his pe iod and
expe imen ook place, whe e some pape s we e andomly assigned o single-blind and
o he s o double-blind e iew. Blank (1991) inds ha he double-blind e iew p ocess
esul s in lowe accep ance a es and mo e c i ical e e ee commen s o au ho s a ilia ed
wi h mid- anking op uni e si ies ( anks 6–50). Howe e , she does no ind any speci ic
ad e se e ec o single-blind e iew on emale au ho s. This is u he con i med by
Ca lsson e al. (2012), s udying double and single-blind accep ance decisions o a Swedish
con e ence held in 20086. Ne e heless, Laband and Pie e (1994) analyzed mo e han
1,000 a icles published in 28 leading economics jou nals in 1984 and disco e ed ha
a icles om jou nals using double-blind e iew ecei ed mo e ci a ions o e a i e-yea
pe iod, indica ing highe quali y compa ed o hose published in jou nals using single-blind
e iew.
1.1 Po en ial mechanisms
Unde a single-blind pee e iew sys em, au ho s a ilia ed wi h less p es igious ins i u ions
migh eel inc eased p essu e o p esen s a is ically signi ican esul s o enhance hei
chances o publica ion. Consequen ly, hey may be mo e inclined o engage in p-hacking
p ac ices compa ed o when hey wo k unde a double-blind e iew sys em.
Con e sely, in a double-blind e iew sys em, au ho s om epu able ins i u ions may
expe ience heigh ened p essu e o p esen pa icula ly compelling esul s, as hei iden-
i ies a e anonymized and canno in luence e iewe s. Thus, he shi om single-blind
o double-blind e iew c ea es opposing incen i es o di e en g oups o au ho s. I is
no clea , a p io i, wha he ne e ec o his ansi ion on he p e alence o p-hacking
would be. No ably, B odeu e al. (2020) ind ha p-hacking p ac ices a e no ele an ly
in luenced by au ho s’ expe ience le els o hei ins i u ions’ ankings.
Rega ding publica ion bias, his e m e e s o he p e e ence exhibi ed by edi o s and
e e ees o s a is ically signi ican esul s. Gi en ha edi o s always know he au ho s’
iden i ies, I an icipa e no signi ican change in hei a i udes owa ds null esul s be ween
single-blind and double-blind e iew sys ems.
Howe e , e e ees may display g ea e bias agains null indings submi ed by au ho s
om less p es igious ins i u ions unde a single-blind sys em. Con e sely, e e ees may also
6Howe e , Hengel (2022) inds ha , unde a single-blind sys em, women a e held o highe w i ing
s anda ds han men.
4
demons a e mo e leniency owa ds null esul s when hey o igina e om well-es ablished
au ho s a op ins i u ions in he same sys em. In summa y, swi ching om a double-blind
o a single-blind e iew policy is expec ed o:
•Reduce p-hacking p ac ices and publica ion bias among pape s au ho ed by p omi-
nen esea che s om epu able ins i u ions.
•Inc ease p-hacking p ac ices and publica ion bias among pape s au ho ed by less-
known esea che s om lowe - anked ins i u ions.
The o e all ne impac o hese wo con as ing e ec s on p-hacking and publica ion
bias emains unclea a p io i.
2 Da a Desc ip ion
I exploi he quasi-na u al expe imen o The Ame ican Economic Re iew (AER) passing
om a double-blind o a single-blind pee e iew s anda d o hei pape s in July 20117.
I conside da a o he pe iod 2005-2015. The analysis pe iod s ops in 2015, because in
mid-2016 he AEA in oduced ano he policy ha may po en ially impac he ex en o p-
hacking, e.g. he omission o signi icance s a s om he eg ession ables. This in e en ion
is desc ibed in de ail in Naguib (2024). I compa e he ex en o p-hacking in he AER
wi h ha in compa able op-5 jou nals in economics such as The Qua e ly Jou nal o
Economics (QJE) and The Jou nal o Poli ical Economy (JPE), which ha e bo h been
adop ing a single-blind e iew s anda d o he ull pe iod o analysis8.
In my da ase , I exclude co igenda, commen s and eplies o esea ch pape s om
he analysis. I u he exclude pape s ha do no include any es ima ed coe icien . Fol-
lowing B odeu e al. (2020), I only collec es ima es om esul s ables and only o
he coe icien s o in e es , o main esul s, excluding eg ession con ols, cons an e ms,
balance and obus ness checks, he e ogenei y o e ec s, and placebo es s. I howe e col-
lec coe icien s d awn om mul iple speci ica ions o he same hypo hesis. Mo eo e , I
collec he es ima ed coe icien s o in e ac ion e ms only i such e ms a e he a iable o
in e es , o example in he case o he in e ac ion be ween he pos - ea men pe iod and
he ea ed dummy in a di e ence-in-di e ence se up. I he main indings o a pape a e
exp essed by means o a Figu e, e.g. impulse esponse unc ions, hen I d op he pape
7The decision was announced in Ma ch 2011, and become e ec i e on 1s July 2011.
8As men ioned abo e, we a e hence in he amewo k o an ” e e se- ime di e ence-in-di e ence se up”,
whe e one g oup was ea ed (i.e. adop ing a single-blind s anda d) o he whole pe iod o analysis and
he o he g oup s a ed being ea ed only a a ce ain poin in ime, see Kim and Lee (2018).
5
om he collec ion. I collec all epo ed decimal places. I mo e han one s anda d e o
pe es ima ed coe icien is epo ed (e.g. ob ained wi h di e en me hods o clus e ing),
I only collec he i s one.
The e is no able o e lap be ween my da ase and he one collec ed by B odeu e al.
(2016), in pa icula o he yea s 2005-2011. Fo hose yea s, I only collec he main
esul s, whe eas hey also collec obus ness checks and simila addi ional esul s. Fo
he yea s 2012-2016, I collec es ima ed coe icien s om all he a icles published in he
h ee jou nals unde s udy, whe eas B odeu e al. (2024a) only collec s coe icien s om
a andom sample o a icles. In Figu e 5 in he Appendix I show he dis ibu ion o z-
s a is ics, espec i ely in my sample and in he sample collec ed by B odeu e al. (2024a).
Bo h his og ams clea ly exhibi wha B odeu e al. (2016) calls a ” wo-humped camel
shape”, which sugges s he p esence o p-hacking and/o publica ion bias.
Fu he , di e en ly om B odeu e al. (2020), I do no es ic he analysis o a icles
ha use one o a p e-de ined se o es ima ion me hods (DID, RDD, RTC, IV), bu I
collec esul s om all me hods ha p oduce es ima ed coe icien s and s anda d e o s
(o -s a is ics, o p- alues). No ably, his means ha I also include OLS es ima es.
Howe e , i OLS es ima es a e only used o p esen co ela ions o as a so o desc ip i e
s a is ics, and hence hey a e no in he Resul s Sec ion o he pape , bu a he in he
Da a desc ip ion, hen I do no collec hem. In case o IV es ima ions, ollowing B odeu
e al. (2020), I only collec he coe icien (s) o he ins umen ed a iable(s) in he second
s age.
Da a ha e been coded independen ly by a leas wo o he ollowing: he au ho and
h ee esea ch assis an s. We discussed and cla i ied disco dan cases. Finally, ollowing
B odeu e al. (2020), since all o he es s a is ics in he sample ela e o wo- ailed es s
and deg ees o eedom a e no always epo ed, I ea coe icien and s anda d e o a ios
as i hey ollow an asymp o ically s anda d no mal dis ibu ion. When a icles epo -
s a is ics o p- alues, I ans o m hem in o equi alen z-s a is ics. This can be a ough
app oxima ion o signi icance o wo easons. Fi s , he e ec i e numbe o deg ees o
eedom may be modes , especially in case o esul s ob ained wi h clus e - obus s anda d
e o s and a limi ed numbe o clus e s. Second, di e en jou nals may adop di e en
ounding con en ions o hei epo ed esul s. I hese con en ions di e ac oss jou nals
and/o ac oss ime, his may no cancel ou in my di -in-di se ing and hence bias he
esul s.
6
Figu e 1: Pe cen age o es s signi ican , espec i ely a he 1% le el (uppe panel), a he 5%
le el (middle panel), and a he 10% le el (bo om panel), by yea o publica ion.
7
as in Ellio e al. (2022). Fu he , ollowing Ellio e al. (2022), I in e p e any p- alue
in Table 2 lowe han 0.1 as e idence o he p esence o p-hacking and publica ion bias.
Name o es Bin. Disc. CS1 CS2B LCM N. Obs. N. a icles
Panel A: AER, Double-Blind Re iew (2005-2011)
Full sample 0.201 0.994 0.647 0.137 0.959 4483 222
Theo y model 0.209 0.128 0.027 0.002 1.000 2557 129
No heo y model 0.434 0.570 0.050 0.024 0.994 1926 93
Single au ho 0.412 0.594 0.000 0.000 1.000 975 51
No single au ho 0.238 0.590 0.238 0.131 0.968 3508 171
Top ins i u ion 0.314 0.890 0.000 0.000 1.000 1924 92
No op ins i u ion 0.292 0.082 0.474 0.607 1.000 2559 130
Panel B: AER, Single-Blind Re iew (2012-2015)
Full sample 0.939 0.748 0.102 0.070 0.568 4785 200
Theo y model 0.959 0.605 0.161 0.013 0.750 3144 131
No heo y model 0.678 0.608 0.000 0.000 1.000 1641 74
Single au ho 0.685 0.528 0.004 0.000 1.000 754 32
No single au ho 0.943 0.589 0.006 0.000 0.591 4031 168
Top ins i u ion 0.849 0.207 0.026 0.003 0.947 1886 74
No op ins i u ion 0.900 0.172 0.243 0.040 0.957 2899 126
Table 2: P- alues by Subsample: AER, Double-Blind s. Single-Blind Re iew
F om Table 2 I deduce ha , unde double-blind e iew (Panel A, 2005–2011), I can
de ec p-hacking in he subsamples o pape s wi h and wi hou a heo y model, as well as
single-au ho ed and wi h au ho s coming om a op ins i u ions (in all hese cases, only
h e CS1 and CS2B es s a e able o ejec he null o no p-hacking and no publica ion
bias). No es ejec s he null in he subsample o pape s wi h mul iple au ho s, as well as
in he o e all sample, and only he discon inui y es ejec s he null among pape s whose
au ho s a e no a ilia ed wi h op ins i u ions. In Panel B (AER unde a single-blind
e iew s anda d), he null hypo hesis is ejec ed by a leas one es in all he subsamples
as well as in he o e all ull sample. Simila ly o panel A, only he CS1 and CS2B es s
a e able o de ec p-hacking. I seems ha i is easie o de ec p-hacking unde he
single-blind egime. Howe e , a wo d o cau ion is necessa y, as some o ms o selec i e
epo ing migh no be de ec able by he es s p esen ed he e and di e ence in signi icance
do no co espond o signi ican di e ences (see Gelman and S e n (2006)).
In Appendix D I p esen he esul s o he es s epo ed in Table 2 when he wo
abo e-men ioned me hods o de ounding a e applied. My esul s a e b oadly consis en
o de ounding. In Table 9 I apply he de ounding me hod used in B odeu e al. (2016)
14
and I ind ha , unde double-blind e iew, I can de ec p-hacking in he subsamples
o pape s wi hou a heo y model, bo h single-au ho ed and wi h mul iple au ho s, and
wi h au ho s coming om op ins i u ions. On he o he hand, unde single-blind e iew
s anda d I can de ec p-hacking in he subsamples o pape s wi h and wi hou a heo y
model, single-au ho ed and wi h au ho s coming om op ins i u ions.
In Table 11, whe e I apply he de ounding me hod p oposed by K anz and P¨u z (2022)
I ind e idence o p-hacking/publica ion bias unde double-blind e iew s anda d in he ull
sample o pape s, as well as in he subsamples o bo h single-au ho ed and mul i-au ho ed
pape s, o pape s wi hou a heo y model and wi h au ho s coming om op ins i u ions.
Unde a single-blind e iew s anda d I am able o de ec he p esence o p-hacking in all
he six subsamples conside ed, bu no in he ull sample.
5 Conclusion
This s udy explo es he impac o single-blind e sus double-blind pee e iew sys ems on
he p e alence o p-hacking in economics jou nals, ocusing on The Ame ican Economic
Re iew’s (AER) ansi ion o a single-blind e iew policy in 2011. Using a di e ence-in-
di e ences app oach and a se ies o s a is ical es s, he esea ch p o ided sugges i e hin s
ha he change in he e iew sys em did no ele an ly in luence o e all he ex en o
p-hacking in published pape s a AER.
Howe e , esul s no ably di e ac oss subg oups. In pa icula , he hypo hesis acco d-
ing o which au ho s coming om op ins i u ions engage mo e in p-hacking- ype p ac ices
and au ho s coming om non op-ins i u ions engage less in hem unde double-blind e-
iew and ice e sa unde a single-blind e iew sys em is con i med empi ically.
The esul s ha e b oade implica ions o he ongoing deba e be ween single-blind and
double-blind e iew sys ems. Al hough p oponen s o double-blind e iews a gue ha
hey educe biases and p omo e ai ness, i migh ha e he unin ended consequence o
inc easing incen i es o p-hacking o some g oups o au ho s, in pa icula hose who a e
single au ho s and come om op ins i u ions. A he same ime, a double-blind e iew
sys em appea s o educe incen i es o p-hacking- ype p ac ices ac oss au ho s who a e
no a ilia ed wi h op uni e si ies and who coau ho pape s wi h o he s.
Re e ences
1. Blanco-Pe ez, C., & B odeu , A. (2020). Publica ion bias and edi o ial s a emen
on nega i e indings. The Economic Jou nal, 130(629), 1226-1247.
15
2. Blank, R. M. (1991). The e ec s o double-blind e sus single-blind e iewing: Ex-
pe imen al e idence om he Ame ican Economic Re iew. The Ame ican Economic
Re iew, 1041-1067.
3. B odeu , A., Cook, N., & Neisse , C. (2024a). P-hacking, da a ype and da a-sha ing
policy. The Economic Jou nal, 134(659), 985-1018.
4. B odeu , A., Cook, N. M., Ha ley, J. S., & Heyes, A. (2024b). Do P e egis a-
ion and P eanalysis Plans Reduce p-Hacking and Publica ion Bias? E idence om
15,992 Tes S a is ics and Sugges ions o Imp o emen . Jou nal o Poli ical Econ-
omy Mic oeconomics, 2(3), 527-561.
5. B odeu , A., Ca ell, S., Figlio, D., & Lushe , L. (2023). Unpacking p-hacking and
publica ion bias. Ame ican Economic Re iew, 113(11), 2974-3002.
6. B odeu , A., Cook, N., & Heyes, A. (2020). Me hods ma e : P-hacking and pub-
lica ion bias in causal analysis in economics. Ame ican Economic Re iew, 110(11),
3634-3660.
7. B odeu , A., L´e, M., Sangnie , M., & Zylbe be g, Y. (2016). S a wa s: The empi ics
s ike back. Ame ican Economic Jou nal: Applied Economics, 8(1), 1-32.
8. Ca d, D., & DellaVigna, S. (2020). Wha do edi o s maximize? E idence om ou
economics jou nals. Re iew o Economics and S a is ics, 102(1), 195-217.
9. Ca d, D., & DellaVigna, S. (2014). Page limi s on economics a icles: E idence om
wo jou nals. Jou nal o Economic Pe spec i es, 28(3), 149-168.
10. Ca lsson, F., L¨o g en, ˚
A., & S e ne , T. (2012). Disc imina ion in scien i ic e iew:
A na u al ield expe imen on blind e sus non-blind e iews. The Scandina ian
Jou nal o Economics, 114(2), 500-519.
11. Chop a, F., Haaland, I., Ro h, C., & S egmann, A. (2024). The null esul penal y.
The Economic Jou nal, 134(657), 193-219.
12. C essey, D. (2014, July 14). Jou nals weigh up double-blind pee e iew. Na u e.
Re ie ed om h ps://www.na u e.com/news/jou nals-weigh-up-double-blind-pee -
e iew-1.15564
13. Ellio , G., Kud in, N., & W¨u h ich, K. (2022a). The Powe o Tes s o De ec ing
p-Hacking. a Xi p ep in a Xi :2205.07950.
16
14. Ellio , G., Kud in, N., & W¨u h ich, K. (2022b). De ec ing p-hacking. Econome -
ica, 90(2), 887-906.
15. Gelman, A., & S e n, H. (2006). The di e ence be ween “signi ican ” and “no
signi ican ” is no i sel s a is ically signi ican . The Ame ican S a is ician, 60(4),
328-331.
16. Goldbe g, P. (2012): “Repo o he Edi o : Ame ican Economic Re iew,” Ame ican
Economic Re iew: Pape s & P oceedings, 102, 653–665.
17. Hada and, A., Hame mesh, D. S., & Wilson, W. W. (2024). Publishing economics:
How slow? Why slow? Is slow p oduc i e? How o ix slow?. Jou nal o Economic
Li e a u e, 62(1), 269-293.
18. Hengel, E. (2022). Publishing while emale: A e women held o highe s anda ds?
E idence om pee e iew. The Economic Jou nal, 132(648), 2951-2991.
19. Hill, S., & P o os , F. (2003). The my h o he double-blind e iew? Au ho iden-
i ica ion using only ci a ions. ACM SIGKDD Explo a ions Newsle e , 5, 179-184.
20. Imbens, G. W. (2021). S a is ical signi icance, p- alues, and he epo ing o unce -
ain y. Jou nal o Economic Pe spec i es, 35(3), 157-174.
21. Kim, K., & Lee, M. J. (2019). Di e ence in di e ences in e e se. Empi ical Eco-
nomics, 57, 705-725.
22. K anz, S., & P¨u z, P. (2022). Me hods ma e : P-hacking and publica ion bias
in causal analysis in economics: Commen . Ame ican Economic Re iew, 112(9),
3124-3136.
23. Laband, D. N., & Pie e, M. J. (1994). Does he” blindness” o pee e iew in luence
manusc ip selec ion e iciency?. Sou he n Economic Jou nal, 896-906.
24. E zo F.P. Lu me , (2024). Repo o he Edi o Ame ican Economic Re iew. AEA
Pape s and P oceedings 2024, 114: 734–750.
25. McCloskey, A., & Michailla , P. (2024). C i ical alues obus o p-hacking. Re iew
o Economics and S a is ics, 1-35.
26. Naguib, C. (2024). P-hacking and Signi icance S a s. Discussion pape se ies. Uni-
e si y o Be n.
17
Appendix ( o online publica ion only)
A. Desc ip i e s a is ics
Figu e 3: His og am (100 bins) o he z-s a is ic alues collec ed om jou nals in he
AER, QJE and JPE o he pe iod 2012-2015, i.e. when all hese jou nals we e adop ing
single-blind e iew, by subg oups. Z-s a is ics la ge han 10 ha e been immed in
o de o imp o e g aph eadabili y.
18
Figu e 4: His og am (100 bins) o he z-s a is ic alues collec ed om jou nals in he AER
o he pe iod 2005-2011, i.e. unde a egime o double-blind e iew, by subg oups.
Z-s a is ics la ge han 10 ha e been immed in o de o imp o e g aph eadabili y.
19
% o A icles % o Tes s N. A icles N. Tes s
AER 51.97% 50.82% 422 9268
QJE 32.88% 33.75% 267 6154
JPE 15.15% 15.43% 123 2814
Single-au ho ed 19.58% 18.26% 159 3329
Wi h a heo e ical model 57.14% 54.05% 464 9857
Sha e o au ho s om op ins 50.25% 48.68% 408 8878
Table 3: Desc ip i e s a is ics o he ea ed and con ol g oup samples. Pe iod 2005-2015.
Figu e 5: His og am (125 bins) o he z-s a is ic alues collec ed om QJE, JPE and AER, o
he pe iod 2005-2015 by Naguib (2024) and B odeu e al. (2024a). Z-s a is ics la ge han 10
ha e been immed in o de o imp o e g aph eadabili y. In he da a collec ed by Naguib (2024)
N= 18,236, whe eas in he da a collec ed by B odeu e al. (2024a) N= 7,658. * = 1.65, ** =
1.96, and *** = 2.58.
20
B. Addi ional empi ical esul s
(1) (2) (3) (4) (5) (6) (7)
O e all Theo y No heo y Single au No single au No op Top ins
Double x 2005-11 0.032 -0.004 0.104 0.064 0.032 -0.055 0.107
(0.016) (0.022) (0.024) (0.042) (0.017) (0.021) (0.024)
Cons an YES YES YES YES YES YES YES
Yea FEs YES YES YES YES YES YES YES
Adj R-sq 0.005 0.010 0.017 0.037 0.007 0.010 0.013
Obs 18,236 9,857 8,379 3,329 14907 9,358 8,878
A icles 812 469 351 159 653 404 408
Table 4: This able shows OLS es ima es o equa ion (1). The dependen a iable is a dummy o whe he
he es s a is ic is signi ican a he 1% le el. S anda d e o s in pa en heses. Top ins i u ions a e he
20 de ined by B odeu e al. (2020). The dummy is equal o one i a leas hal o he au ho s belong o a
op ins i u ion a he ime o he a icle publica ion.
(1) (2) (3) (4) (5) (6) (7)
O e all Theo y No heo y Single au No single au No op Top ins
Double x 2005-11 -0.024 -0.062 0.026 0.075 -0.034 -0.104 0.053
(0.015) (0.020) (0.023) (0.039) (0.016) (0.020) (0.022)
Cons an YES YES YES YES YES YES YES
Yea FEs YES YES YES YES YES YES YES
Adj R-sq 0.006 0.011 0.016 0.038 0.009 0.010 0.018
Obs 18,236 9,857 8,379 3,329 14907 9,358 8,878
A icles 812 469 351 159 653 404 408
Table 5: This able shows OLS es ima es o equa ion (1). The dependen a iable is a dummy o whe he
he es s a is ic is signi ican a he 10% le el. S anda d e o s in pa en heses. Top ins i u ions a e he
20 de ined by B odeu e al. (2020). The dummy is equal o one i a leas hal o he au ho s belong o a
op ins i u ion a he ime o he a icle publica ion.
The median lag in economics om pape submission o publica ion is a ound wo yea s.
Fo his eason, in his Sec ion I p esen some o he baseline es ima es by se ing he
s a ing da e o he change in he pee - e iew s anda d o 2013, ins ead han o 2011.
Indeed, he i s pape s subjec o he double e iew s anda d a AER we e mos likely
no published un il 2013. I ocus he e on he 5% signi icance h eshold only.
21
(1) (2) (3) (4) (5) (6) (7)
O e all Theo y No heo y Single au No single au No op Top ins
Double x ’05-13 -0.003 -0.005 0.005 0.091 -0.006 -0.074 0.063
(0.016) (0.023) (0.025) (0.045) (0.018) (0.022) (0.025)
Cons an YES YES YES YES YES YES YES
Yea FEs YES YES YES YES YES YES YES
Adj R-sq 0.006 0.012 0.016 0.040 0.009 0.010 0.016
Obs 18,236 9,857 8,379 3,329 14907 9,358 8,878
A icles 812 469 351 159 653 404 408
Table 6: This able shows OLS es ima es o equa ion (1). The dependen a iable is a dummy o whe he
he es s a is ic is signi ican a he 5% le el. S anda d e o s in pa en heses. Top ins i u ions a e he
20 de ined by B odeu e al. (2020). The dummy is equal o one i a leas hal o he au ho s belong o a
op ins i u ion a he ime o he a icle publica ion.
22
C. Robus ness checks wi h weigh ing
In his Sec ion, I eplica e he main ables and igu es o he pape using a icle weigh s, in
o de o a oid ha a icles wi h mo e es s ha e a disp opo iona e in luence on he esul s.
Following B odeu e al. (2016), o ob ain he esul s wi h a icle weigh s I associa e o
each es s a is ic he in e se o he o al numbe o es s ha a e epo ed in he same
a icle. The esul is ha each a icle con ibu es in he same way o he dis ibu ion.
Since I ocus on he main esul s o each pape , o mos o he pape s in my sample
I collec coe icien s om one able only. Fo his eason I e ain om epo ing esul s
ob ained wi h a icle and able weig hs. Indeed, hey would be essen ially iden ical o he
ones ob ained wi h a icle weigh s, which a e epo ed in he ollowing.
Simila o B odeu e al. (2016) he ”camel shape” o z-s a is ics is mo e e iden o
he weigh ed han o he unweigh ed dis ibu ions. This suppo s he hypo hesis ha
esea che s a e likely o epo mo e es ima ed coe icien s i hei esul s a e s a is ically
signi ican and con e sely hey only epo a ew i o he speci ica ions would ail o yield
s a is ically signi ican esul s. Weigh ed dis ibu ions gi e less weigh o a icles and
ables in which many es s a e epo ed.
(1) (2) (3) (4) (5) (6) (7)
O e all Theo y No heo y Single au No single au No op Top ins
Double x 2005-11 -0.050 -0.027 -0.081 0.060 -0.054 -0.168 0.045
(0.019) (0.025) (0.029) (0.045) (0.021) (0.026) (0.028)
Cons an YES YES YES YES YES YES YES
Yea FEs YES YES YES YES YES YES YES
Adj R-sq 0.017 0.022 0.031 0.039 0.022 0.034 0.034
Obs 18,236 9,857 8,379 3,329 14907 9,358 8,878
A icles 812 469 351 159 653 404 408
Table 7: This able shows OLS es ima es o equa ion (1). The dependen a iable is a dummy o whe he
he es s a is ic is signi ican a he 5% le el,wi h a icle weigh s. S anda d e o s in pa en heses.
Top ins i u ions a e he 20 de ined by B odeu e al. (2020). The dummy is equal o one i a leas hal o
he au ho s belong o a op ins i u ion a he ime o he a icle publica ion.
23
