B eznau, Na e e al.
A icle — Published Ve sion
Obse ing many esea che s using he same da a and
hypo hesis e eals a hidden uni e se o unce ain y
PNAS - P oceedings o he Na ional Academy o Sciences
P o ided in Coope a ion wi h:
WZB Be lin Social Science Cen e
Sugges ed Ci a ion: B eznau, Na e e al. (2022) : Obse ing many esea che s using he same da a
and hypo hesis e eals a hidden uni e se o unce ain y, PNAS - P oceedings o he Na ional
Academy o Sciences, ISSN 1091-6490, Na ional Academy o Sciences, Washing on, DC, Vol. 119, Iss.
44, pp. 1-8,
h ps://doi.o g/10.1073/pnas.2203150119
This Ve sion is a ailable a :
h ps://hdl.handle.ne /10419/266342
S anda d-Nu zungsbedingungen:
Die Dokumen e au EconS o dü en zu eigenen wissenscha lichen
Zwecken und zum P i a geb auch gespeiche und kopie we den.
Sie dü en die Dokumen e nich ü ö en liche ode komme zielle
Zwecke e iel äl igen, ö en lich auss ellen, ö en lich zugänglich
machen, e eiben ode ande wei ig nu zen.
So e n die Ve asse die Dokumen e un e Open-Con en -Lizenzen
(insbesonde e CC-Lizenzen) zu Ve ügung ges ell haben soll en,
gel en abweichend on diesen Nu zungsbedingungen die in de do
genann en Lizenz gewäh en Nu zungs ech e.
Te ms o use:
Documen s in EconS o may be sa ed and copied o you pe sonal
and schola ly pu poses.
You a e no o copy documen s o public o comme cial pu poses, o
exhibi he documen s publicly, o make hem publicly a ailable on he
in e ne , o o dis ibu e o o he wise use he documen s in public.
I he documen s ha e been made a ailable unde an Open Con en
Licence (especially C ea i e Commons Licences), you may exe cise
u he usage igh s as speci ied in he indica ed licence.
h ps://c ea i ecommons.o g/licenses/by/4.0/
Obse ing many esea che s using he same da a and
hypo hesis e eals a hidden uni e se o unce ain y
Edi ed by Douglas Massey, P ince on Uni e si y, P ince on, NJ; ecei ed Ma ch 6, 2022; accep ed Augus 22, 2022
This s udy explo es how esea che s’analy ical choices a ec he eliabili y o scien ific
findings. Mos discussions o eliabili y p oblems in science ocus on sys ema ic biases.
We b oaden he lens o emphasize he idiosync asy o conscious and unconscious
decisions ha esea che s make du ing da a analysis. We coo dina ed 161 esea che s in
73 esea ch eams and obse ed hei esea ch decisions as hey used he same da a o
independen ly es he same p ominen social science hypo hesis: ha g ea e immig a-
ion educes suppo o social policies among he public. In his ypical case o social
science esea ch, esea ch eams epo ed bo h widely di e ging nume ical findings and
subs an i e conclusions despi e iden ical s a condi ions. Resea che s’expe ise, p io
belie s, and expec a ions ba ely p edic he wide a ia ion in esea ch ou comes. Mo e
han 95% o he o al a iance in nume ical esul s emains unexplained e en a e
quali a i e coding o all iden ifiable decisions in each eam’swo kflow. This e eals a
uni e se o unce ain y ha emains hidden when conside ing a single s udy in isola-
ion. The idiosync a ic na u e o how esea che s’ esul s and conclusions a ied is a
p e iously unde app ecia ed explana ion o why many scien ific hypo heses emain
con es ed. These esul s call o g ea e epis emic humili y and cla i y in epo ing sci-
en ificfindings.
me ascience jmany analys s j esea che deg ees o eedom janaly ical flexibili y jimmig a ion and
policy p e e ences
O ganized scien ific knowledge p oduc ion in ol es ins i u ionalized checks, such as edi-
o ial e ing, pee e iew, and me hodological s anda ds, o ensu e ha findings a e
independen o he cha ac e is ics o p edisposi ions o any single esea che (1, 2). These
p ocedu es should gene a e in e esea che eliabili y, o e ing consume s o scien ific
findings assu ance ha hey a e no a bi a y flukes and ha o he esea che s would gen-
e a e simila findings gi en he same da a. Recen me ascience esea ch challenges his
assump ion as se e al a emp s o ep oduce findings om p e ious s udies ailed (3, 4).
In esponse, scien is s ha e discussed a ious h ea s o he eliabili y o he scien ific
p ocess wi h a ocus on biases inhe en in he p oduc ion o science. Poin ing o bo h
misaligned s uc u al incen i es and he cogni i e endencies o esea che s (5–7), his bias-
ocused pe spec i e a gues ha sys ema ic dis o ions o he esea ch p ocess push he
published li e a u e away om u h seeking and accu a e obse a ion. This hen educes
he p obabili y ha a ca e ully execu ed eplica ion will a i e a he same findings.
He e, we a gue ha some oo s o eliabili y issues in science un deepe han sys em-
a ically dis o ed esea ch p ac ices. We p opose ha o be e unde s and why esea ch is
o en non eplicable o lacking in e esea che eliabili y, we need o accoun o idiosyn-
c a ic a ia ion inhe en in he scien ific p ocess. Ou main a gumen is ha a iabili y
in esea ch ou comes be ween esea che s can occu e en unde igid adhe ence o he
scien ific me hod, high e hical s anda ds, and s a e-o - he-a app oaches o maximizing
ep oducibili y. As we epo below, e en well-meaning scien is s p o ided wi h iden ical
da a and eed om p essu es o dis o esul s may no eliably con e ge in hei findings
because o he complexi y and ambigui y inhe en o he p ocess o scien ificanalysis.
Va iabili y in Resea ch Ou comes
The scien ific p ocess con on s esea che s wi h a mul iplici y o seemingly mino , ye
non i ial, decision poin s, each o which may in oduce a iabili y in esea ch ou -
comes. An impo an bu unde app ecia ed ac is ha his e en holds o wha is o en
seen as he mos objec i e s ep in he esea ch p ocess: wo king wi h he da a a e i
has come in. Resea che s can ake li e ally millions o di e en pa hs in w angling, ana-
lyzing, p esen ing, and in e p e ing hei da a. The numbe o choices g ows exponen-
ially wi h he numbe o cases and a iables included (8–10).
A bias- ocused pe spec i e implici ly assumes ha educing “pe e se”incen i es o
gene a e su p ising and sleek esul s would ins ead lead esea che s o gene a e alid
Significance
Will di e en esea che s
con e ge on simila findings when
analyzing he same da a? Se en y-
h ee independen esea ch
eams used iden ical c oss-
coun y su ey da a o es a
p ominen social science
hypo hesis: ha mo e
immig a ion will educe public
suppo o go e nmen p o ision
o social policies. Ins ead o
con e gence, eams’ esul s a ied
g ea ly, anging om la ge
nega i e o la ge posi i e e ec s o
immig a ion on social policy
suppo . The choices made by he
esea ch eams in designing hei
s a is ical es s explain e y li le o
his a ia ion; a hidden uni e se o
unce ain y emains. Conside ing
his a ia ion, scien is s, especially
hose wo king wi h he
complexi ies o human socie ies
and beha io , should exe cise
humili y and s i e o be e
accoun o he unce ain y in
hei wo k.
The au ho s decla e no compe ing in e es .
This a icle is a PNAS Di ec Submission.
Copy igh © 2022 he Au ho (s). Published by PNAS.
This open access a icle is dis ibu ed unde C ea i e
Commons A ibu ion-NonComme cial-NoDe i a i es
License 4.0 (CC BY).
See online o ela ed con en such as Commen a ies.
1
To whom co espondence may be add essed. Email:
[email p o ec ed].
2
N.B., E.M.R., and A.W. we e he P incipal In es iga o s,
equally esponsible o concep ualiza ion and da a
collec ion. P ima y me a-analysis o da a analys s’
esul s and p epa a ion o me ada a o public
consump ion p e o med by N.B., wi h assis ance om
H.H.V.N.
This a icle con ains suppo ing in o ma ion online a
h p://www.pnas.o g/lookup/suppl/doi:10.1073/pnas.
2203150119/-/DCSupplemen al.
Published Oc obe 28, 2022.
PNAS 2022 Vol. 119 No. 44 e2203150119 h ps://doi.o g/10.1073/pnas.2203150119 1o 8
RESEARCH ARTICLE
|
SOCIAL SCIENCES OPEN ACCESS
conclusions. This may be oo op imis ic. While emo ing hese
ba ie s leads esea che s away om sys ema ically aking in alid
o biased analy ical pa hs (8–11), his alone does no gua an ee
alidi y and eliabili y. Fo easons less ne a ious, esea che s can
dispe se in di e en di ec ions in wha Gelman and Loken call a
“ga den o o king pa hs”in analy ical decision-making (8).
The e a e wo p ima y explana ions o a ia ion in o king
decisions. The compe ency hypo hesis posi s ha esea che s
may make di e en analy ical decisions because o a ying le els
o s a is ical and subjec expe ise ha lead o di e en judg-
men s as o wha cons i u es he “ideal”analysis in a gi en
esea ch si ua ion. The confi ma ion bias hypo hesis holds
ha esea che s may make eliably di e en analy ical choices
because o di e ences in p eexis ing belie s and a i udes, which
may lead o jus ifica ion o analy ical app oaches a o ing ce -
ain ou comes pos hoc. Howe e , many o he co e o idio-
sync a ic influences, la ge and small, may also lead o un eliable
and unexplainable a ia ion in analy ical decision pa hways
(10). Some imes e en he inies o hese di e ences may add
up and in e ac o p oduce widely a ying ou comes.
The e is g owing awa eness o he dependence o findings on
s a is ical modeling decisions and he impo ance o analy ical
obus ness (9, 11–13). Howe e , only ecen ly scien is s began
o assess whe he esea che a iabili y a ec s scien ific ou -
comes in eali y, some imes employing “many analys s”
app oaches whe e many esea che s o eams independen ly es
he same hypo hesis wi h he same da a. The fi s such s udy
showed ha when 29 esea che s es ed i socce e e ees we e
biased owa d da ke -skin playe s using he same da a, hey
epo ed 29 unique model specifica ions, wi h empi ical esul s
anging om modes ly nega i e o s ongly posi i e (14). Thus
a , mos many-analys s s udies ha e been small in scale o
ocused on na ow, field-specific analysis me hods (15, 16).
Recen s udies by Bo inik-Neze e al. (17) and Menk eld e al.
(18) we e la ge , in ol ing 65 and 164 eams, espec i ely.
C i ically, despi e hei size, hese s udies also ound la ge in e -
esea che a ia ion in epo ed esul s. They also made fi s
s eps a explaining he amoun o a ia ion in epo ed esul s
using a small se o a iables, such as he compu a ional ep o-
ducibili y and pee a ings o submi ed analyses o he s a is ical
so wa e ha analys s used. Ye , hey had li le success in explain-
ing he a ia ion in esul s o he dis ance o esul s om he
o e all mean o he esul s (i.e., e o ). We expanded on hese
explana o y a emp s by obse ing e e y s ep o each indepen-
den esea ch eam’swo kflow, expec ing ha such close obse a-
ion should explain a mo e a iance in esea ch ou comes.
Mo eo e , when coupled wi h measu es o ele an analy ical
compe encies and subs an i e belie s o he analys s as explana-
o y a iables, we expec ed o a i e a a deepe unde s anding o
he esul s and c ucially, which key decisions d i e hem.
Me hods
The p incipal in es iga o s (PIs) coo dina ed a g oup o 161 esea che s in
73 eams o comple e he same ask o independen ly es ing a hypo hesis
cen al o an “ex ensi e body o schola ship”(19) in he social sciences: ha
immig a ion educes suppo o social policies among he public.* Ou en i e
ep oducible wo kflow o his s udy is a ailable online in ou P ojec Reposi o y.
†
The ask gi en o he pa icipan s is ypical o esea ch on human socie ies, in
which he cen al concep s and quan i ies o in e es a e open o b oad and
complex in e p e a ions (20, 21). In classic poli ical economy esea ch, o exam-
ple, Albe o Alesina and Edwa d Glaese (22, 23) hypo hesized ha di e ences
in No h Ame ican and Eu opean social secu i y sys ems a e a esul o
immig a ion-gene a ed e hnic di e si y o a lack he eo . Mo e ecen ly, o he
schola s see immig a ion and e ugee c ises as ca alys s o e enchmen o social
secu i y sys ems in Wes e n Eu ope and ac oss he globe. Pu simply, his
hypo hesis was gi en o pa icipa ing eams because i is influen ial, long s and-
ing, and ypical o con empo a y social esea ch in poli ical science, sociology,
economics, geog aphy, and beyond (24–29).
We ec ui ed pa icipan s by ci cula ing a call ac oss academic ne wo ks,
social media, and o ficial communica ion channels o academic associa ions
ac oss social science disciplines (SI Appendix,Resea ch Design). Al hough
106 eams exp essed in e es in pa icipa ion, we coun ou ini ial sample as
88 ha comple ed he p es udy ques ionnai e. In he end, 73 o hose
88 eams (a o al o 161 esea che s wi h an a e age o 2.24 esea che s pe
eam) comple ed he s udy. O hese, 46% had a backg ound in sociology;
25% had a backg ound in poli ical science; and he es had economics, com-
munica ion, in e disciplina y, o me hods- ocused deg ee backg ounds.
Eigh y- h ee pe cen had expe ience eaching cou ses on da a analysis, and
70% had published a leas one a icle o chap e on he subs an i e opic o
he s udy o he usage o a ele an me hod (SI Appendix,III. Pa icipan
Su ey Codebook has mo e pa icipan de ails).
The PIs p o ided eams wi h da a om he In e na ional Social Su ey P o-
g am (ISSP), a long- unning la ge-scale c oss-na ionally compa a i e su ey o
poli ical and economic a i udes used in o e 10,000 published s udies.
‡
The
ISSP includes a six-ques ion module on he ole o go e nmen in p o iding
di e en social policies, such as old-age, labo ma ke , and heal h ca e p o i-
sions. This six-ques ion module is also he sou ce o he da a used by Da id
B ady and Ryan Finnigan (19) in one o he mos ci ed in es iga ions o he
subs an i e hypo hesis pa icipan s we e ins uc ed o es . The PIs also p o-
ided yea ly indica o da a o coun ies on immig an ’s ock’as a pe cen age
o he popula ion and on ’flow’as a ne change in s ock, aken om he Wo ld
Bank, he Uni ed Na ions, and he O ganiza ion o Economic Co-Ope a ion
and De elopmen . Rele an ISSP and immig a ion da a we e a ailable o
31 mos ly ich and some middle-income coun ies. The e we e up o fi e su -
ey wa es om 1985, 1990, 1996, 2006, and 2016. All p o ided da a come
om publicly a ailable sou ces.
To emo e po en ially biasing incen i es, all esea che s om eams ha com-
ple ed he s udy we e ensu ed coau ho ship on he final pape ega dless o
hei esul s. Because he “pa icipan s” hemsel es we e esea che s and all
asks assigned o hem we e s anda d esea ch p ac ices heo e ically wo hy o
coau ho ship, ins i u ional e iew p io o conduc ing his s udy was no neces-
sa y. The esea che s pa icipa ed in su eys o measu e expe ise and s udy-
ele an belie s and a i udes be o e and du ing he esea ch p ocess. Mo eo e ,
hey ook pa in online delibe a ions be o e (a andomized hal o eams) and
a e hey had un hei main analyses (all eams) (SI Appendix, III. Pa icipan
Su ey Codebook). To amilia ize pa icipa ing esea che s wi h he da a, hei
fi s ask was o nume ically ep oduce esul s om he B ady and Finnigan
(19) s udy on a subse o he ISSP da a. This was ollowed by a eques ha
he eams de elop hei own ideal models o es ing he same hypo hesis
using po en ially all o he p o ided da a, bu ha hey submi hei analysis
plan p io o unning he models. To enhance ecological alidi y, we
allowed he eams o include addi ional da a sou ces o measu ing inde-
penden a iables. Each eam was hen ins uc ed o un hei model(s) and
epo dependen a iable–s anda dized e ec es ima es equal o he
change in policy p e e ences (in SD uni s) p edic ed by a one-poin change
in he espec i e independen immig a ion a iable. We also asked hem o
d aw one o h ee subjec i e conclusions: whe he hei esul s o e ed e i-
dence ha suppo ed he hypo hesis ha immig a ion educes suppo o
social policies among he public, whe he hei esul s o e ed e idence ha
ejec ed he hypo hesis, o ins ead, whe he hey belie ed he hypo hesis
was no es able gi en hese da a.
O he 73 eams, 1 conduc ed p elimina y measu emen scaling es s, con-
cluded ha he hypo hesis could no be eliably es ed, and hus, did no design
o ca y ou any u he es s. This le 72 eams submi ing a o al o 1,261
*SI Appendix, Figs. S1 and S2 and Tables S1 and S2 ha e he ime line and mo e pa ici-
pan de ails.
†
I is a ailable a h ps://gi hub.com/nb eznau/CRI.
‡
In o ma ion is a ailable a h ps://issp.o g/abou -issp/.
2o 8 h ps://doi.o g/10.1073/pnas.2203150119 pnas.o g
models. One eam’s p e egis e ed models ailed o con e ge and hus, had no
nume ical esul s. This le a o al o 71 eams wi h nume ical esul s om 1,253
models. In hei subjec i e conclusions, 16 eams de e mined ha he wo di e -
en measu es o immig a ion should be conside ed independen hypo hesis
es s and he e o e, submi ed di e en conclusions o each. This changed he
p ima y uni o analysis o subjec i e conclusions om 73 eam conclusions o
89 eam-le el conclusions.
The e was an a e age o 17.5 models pe eam among he 71 eams submi -
ing nume ical esul s ( anging om 1 o 124 models pe eam). Mos eams sub-
mi ed a leas 12 models because hey used each o he six ISSP ques ions as a
single dependen ou come wice in hei s a is ical models, once o each e sion
o he immig a ion measu e (s ock and flow). Se e al eams submi ed 18 models
because hey an an addi ional 6 models wi h bo h immig a ion a iables
included. The eams o en adjus ed o he nes ed na u e o he da a, accoun ing
o a iance a he indi idual, coun y, yea , and/o coun y-yea le els. Some
made no such hie a chical adjus men s wi h mul ile el models, and o he s used
clus e ing o he SEs a he coun y, wa e, and/o coun y-wa e le els. Some used
dummy in e ac ions o example wo ld egion indica o s (such as Eas e n Eu ope)
o poli ical pa y p e e ences wi h immig a ion a iables, leading o nonlinea p e-
dic ed alues. O he s used al e na i e es ima o s based on maximum likelihood
o Bayesian es ima ion as opposed o o dina y leas squa es (SI Appendix,Table
S3 shows he mos common model decisions). In all, esea che s’modeling deci-
sions eflec ed he di e si y o simila , bu echnically dis inc , me hodological
app oaches cu en ly used in con empo a y esea ch.
Each eam’s code was checked and hen anonymized o public sha ing by
he PIs. Some eams ailed o epo a s anda dized es ima e. Also, di e en scal-
ing o he wo independen immig a ion a iables mean ha esul s we e no
always dis ibu ionally compa able. The e o e, we s anda dized he eams’ esul s
o each coe ficien o s ock and flow o immig a ion pos hoc. We also ans-
o med he eams’ esul s in o a e age ma ginal e ec s (AMEs), which a e he
s anda dized a e age e ec s o a one-uni change in he espec i e independen
(immig a ion) a iable on he espec i e dependen (policy suppo ) a iable,
whe e his a e age is based on p edic ions o each obse a ion in he da ase .
The ad an age o using AMEs is ha hey allow o a single ma ginal es ima e in
he p esence o nonlinea i ies and p esen p edic ed p obabili ies ha eflec he
eali y o he da a sample a he han he mean o each independen a iable
(Fig. 1 shows esul s). A e submi ing hei own esul s bu p io o seeing he
o he eams’ esul s, each pa icipan was andomly gi en a ough desc ip ion o
he models employed by ou o fi e o he eams and asked o ank hem on
hei quali y o es ing he hypo hesis. Wi h six o se en ankings pe eam, he
PIs cons uc ed model ankings (SI Appendix,Model Ranking).
A any poin a e hey submi ed hei esul s, including a e esul s o
he o he eams we e e ealed, he eams could change hei p e e ed mod-
els and esubmi hei esul s and conclusion. No eam olun a ily op ed o
do his. Howe e , some eams’ esul s and conclusions changed a e hey
we e in o med ha he PIs we e unable o ep oduce hei findings due o
coding mis akes o a misma ch be ween hei in ended models and hose
ha appea ed in he code.
Nex , we examined all 1,261 models and iden ified 166 dis inc esea ch
design decisions associa ed wi h hose models. “Decision” e e s o any aspec
in he design o a s a is ical model: o example, he measu emen s a egy,
es ima o , hie a chical s uc u e, choice o independen a iables, and po en ial
subse ing o he da a (SI Appendix, Table S12). Fo simplici y, decision also
e e s o a iables measu ing eam cha ac e is ics, such as so wa e used, o e -
all amilia i y wi h he subjec o me hods, and p eexis ing belie s as measu ed
in ou pa icipan su ey (SI Appendix, Table S1). O he 166 decisions, 107
we e aken by a leas h ee eams. We used hese 107 as a iables ha migh
s a is ically explain he a ia ion in he esul s and conclusions because he
o he 59 we e unique o one o wo eams and would hus impede s a is ical
iden ifica ion. In o he wo ds, uniquely iden i ying one o wo eams’ esul s
ia he a iance in a single independen a iable in he eg ession would in e -
up he pa simonious es ima ion o esidual, unexplained a iance calcula ed
in he le el 2 equa ion. A dissimila i y ma ix e ealed ha no wo models o
1,261 we e 100% iden ical.
To explo e he sou ces o a iance in esul s, we eg essed he nume ical
poin es ima es and subjec i e conclusions on all di e en combina ions and
in e ac ions o he 107 decisions. We used mul ile el eg ession models,
allowing us o accoun o models nes ed in eams and o explain o al, wi hin-
eam, and be ween- eam a iance in nume ical esul s. Fo subjec i e conclu-
sions, we used mul inomial logis ic eg essions p edic ing eams’conclusions o
1) suppo o 2) ejec he a ge hypo hesis, o 3) ega d i as no es able. Ou
analyses p oceeded in se e al s ages. A each s age, decision a iables and hei
in e ac ions we e es ed, and only e ms ha explained he mos a iance using
he leas deg ees o eedom (o de iance in he case o subjec i e conclusions)
we e ca ied o he nex phase.
Explo ing he a iance in esul s uns he isk o o e fi ing. I is s a is ically
inapp op ia e o use 107 a iables when he e a e 87 eam- es cases ( om
71 eams wi h nume ical esul s). The e o e, we en e ed he a iables in g oups
and only kep a iables om each g oup ha showed an inc ease in he
explained a iance wi hou a loss in fi as measu ed by Akaike’sIn o ma ionC i-
e ion (AIC) and log likelihood. This s a ed wi h “design”decisions, including
which o he six su ey ques ions he eams used as he dependen a iable in a
gi en model and dummies indica ing he wo andom expe imen al ea men s
(which we e included in he s udy bu a e un ela ed o he main modeling ask
assigned o he eams). The nex s age added “measu emen ”decisions, includ-
ing which immig a ion measu e he eam used in a gi en model and how he
dependen a iable was measu ed (dicho omous, o dinal, mul inomial, o con-
inuous). The ollowing s ages added “da a and sample”and “model design”
decisions and concluded wi h he addi ion o “ esea che aspec s”(P ojec Repos-
i o y, 04_CRI_Main_Analyses). We also e an he phase-wise analysis sepa a ely
o each o he six su ey ques ions used as dependen a iables by he eams
(P ojec Reposi o y, 07_CRI_DVspecific_Analyses) (SI Appendix, Tables S4 and
S9–S11). The exac a iables in ou final analysis plus a ious models leading up
o hem a e ound in SI Appendix, Tables S5 and S7. Fig. 2 epo s he explained
a iance om ou p e e ed model m13.
To check he obus ness o ou phase-wise s a egy, we used an algo i hm o
analyze all possible a iable combina ions om ou p ima y a iables o in e -
es — hose ha showed any capaci y o explain a iance in he main analyses
(P ojec Reposi o y, 06_CRI_Mul i e se). This led us o a sligh ly di e en ideal
model hanm13(Au o_1inSI Appendix, Table S10). Al hough his al e na i e
model had he bes AIC and could explain sligh ly mo e model-le el a iance, i
could no explain as much o al a iance. We hen combined all a iables om
ou main model (m13) and he algo i hm-de i ed model (Au o_1) o gene a e a
new model (Au o_1_m13). Al hough his new model explained mo e a iance
and had lowe AIC, we we e ca e ul no o o e fi because i had 22 a iables,
whe eas m13 and Au o_1 had 18 and 15, espec i ely.
Nex , based on PNAS pee e iew eedback, we gene a ed a lis o e e y pos-
sible in e ac ion pai o all 107 a iables. O hese 5,565 in e ac ions, 2,637
ha e nonze o a iance and hus, we e usable in a eg ession wi hou au oma i-
cally being d opped. Including 8 o mo e in e ac ion a iables plus hei main
e ec s (i.e., 24 o mo e a iables, many ha we e c oss-le el in e ac ions) led o
Fig. 1. B oad a ia ion in he findings om 73 eams es ing he same
hypo hesis wi h he same da a. The dis ibu ion o es ima ed AMEs ac oss
all con e ged models (n=1,253) includes esul s ha a e nega i e (yellow;
in he di ec ion p edic ed by he gi en hypo hesis he eams we e es ing),
no di e en om ze o (g ay), o posi i e (blue) using a 95% CI. AME a e xy
s anda dized. The yaxis con ains wo scaling b eaks a ±0.05. Numbe s
inside ci cles ep esen he pe cen ages o he dis ibu ion o each ou -
come in e sely weigh ed by he numbe o models pe eam.
PNAS 2022 Vol. 119 No. 44 e2203150119 h ps://doi.o g/10.1073/pnas.2203150119 3o 8
con e gence o o e iden ifica ion p oblems. E en wi h 87 le el 2 cases, his
lea es us wi h oughly 4 cases pe a iable. Resea ch e iewing di e en simula-
ion s udies on le el 2 case numbe s sugges s ha 10 cases pe a iable a e a
ba e minimum, and we should ha e close o 50 ideally (30). The e o e, we se -
led on 7 in e ac ed a iables as ou absolu e maximum (which co esponds o
21 a iables, including wo main e ec s o each in e ac ion). We hen andomly
sampled 1,000 se s o 7 a iables om he lis o all and le he algo i hm un
e e y subcombina ion o hese, which led o jus o e 1 million models. We
hen ook he 2 models wi h he lowes AIC sco e om each o he 1,000 i e a-
ions and ex ac ed all a iables om hose models. The e we e 19 unique a ia-
bles among he 2,000 bes -fi ing models in o al, which we hen analyzed using
he same “ andom-se en”sampling me hod. The bes -fi ing models om his
second i e a ion le us wi h ou in e ac ion a iables as candida es o explain
mo e a iance while a oiding sac ificing he simplici y o he model and o e fi -
ing (as indica ed by model AIC). We added each o hese a iables sepa a ely o
ou ini ial algo i hm-gene a ed esul s. None o hese models could explain
mo e a iance in esea ch ou comes han m13 o Au o_1_m13 (SI Appendix,
Table S10, Au o_2 o Au o_5).
Main Resul s
Fig. 1 isualizes he subs an ial a ia ion o nume ical esul s
epo ed by 71 esea che eams ha analyzed he same da a.
Resul s a e di use. Li le mo e han hal he epo ed es ima es
we e s a is ically no significan ly di e en om ze o a 95%
CI, while a qua e we e significan ly di e en and nega i e,
and 16.9% we e s a is ically significan and posi i e.
We obse e he same pa e n o di e gen esea ch ou comes
when we use he eams’subjec i e conclusions a he han hei
s a is ical esul s. O e all, 13.5% (12 o 89) o he eam conclu-
sions we e ha he hypo hesis was no es able gi en hese da a,
60.7% (54 o 89) we e ha he hypo hesis should be ejec ed,
and 28.5% (23 o 89) we e ha he hypo hesis was suppo ed
(SI Appendix, Figs. S5, S9, and S10).
§
We find ha compe encies and po en ial confi ma ion biases
do no explain he b oad a ia ion in ou comes; esea che
cha ac e is ics show a s a is ically significan associa ion wi h
nei he s a is ical esul s no subs an i e conclusions (Fig. 3).
Hence, he da a a e no consis en wi h he expec a ion ha
ou come a iabili y simply eflec s a lack o knowledge among
some pa icipan s o p eexis ing p e e ences o pa icula
esul s.
In p inciple, a ia ion in ou comes mus eflec p io decisions
o he esea che s. Ye , Fig. 2 shows ha he 107 iden ified deci-
sion poin s explain li le o he a ia ion. The majo componen s
o he iden ified esea che decisions explain less han a qua e o
he a ia ion in ou measu es o esea ch ou comes. Mos a iance
also emains unexplained a e accoun ing o esea che cha ac e -
is ics o assignmen o a small expe imen (no epo ed in his
s udy) (“assigned condi ions”in Fig. 2). Looking a o al a iance
in he nume ical esul s ( op ba ), iden ified componen s o he
esea ch design explain 2.6% (g een segmen ), and esea che
cha ac e is ics only accoun o a maximum o 1.2% o he a i-
ance ( iole segmen ). In o he wo ds, 95.2% o he o al a iance
in esul s is le unexplained, sugges ing ha massi e a ia ion in
epo ed esul s o igina ed om idiosync a ic decisions in he da a
analysis p ocess.
The sha e o explained a iance is somewha highe when
looking a be ween- eam esul s (second ba ), bu s ill, 82.4%
emained unexplained. Va iance con inues o emain mos ly
unexplained when mo ing away om he nume ical esul s and
conside ing esea che s’subs an i e conclusions (bo om ba ;
80.1% unexplained). I is no ewo hy ha e en he pe cen age
o es esul s pe eam ha s a is ically suppo hei conclu-
sions explains only abou a hi d o he de iance in conclusions
(salmon-colo ed segmen in he bo om ba ), which poin s a
he a ia ion in how di e en esea che s in e p e he same se
o nume ical esul s. O e all, he complexi y o he da a-
analy ic p ocess leads o a ia ion ha canno be easily
explained, e en wi h a close look a esea che cha ac e is ics
and esea che decisions.
Fig. 2. Va iance in s a is ical esul s and subs an i e conclusions be ween and wi hin eams is mos ly unexplained by condi ions, esea ch design, and
esea che cha ac e is ics. Decomposi ion o nume ical a iance aken om gene alized linea mul ile el eg ession models’AMEs ( he op h ee ows).
Explained de iance aken om mul inomial logis ic eg essions using he subs an i e conclusions abou he a ge hypo hesis as he ou come submi ed
by he esea ch eams (bo om ow). We used in o med s epwise addi ion and emo al o p edic o s o iden i y which specifica ions could explain he
mos nume ic a iance (SI Appendix,TableS6) and o he s ha could explain he mos subjec i e conclusion de iance (SI Appendix,TableS7) while sac ific-
ing he ewes deg ees o eedom and main aining he highes le el o model fi based on log likelihood and AIC. We also used algo i hms o es a iable
combina ions, bu hese could no explain mo e meaning ul a ia ion (Me hods). Assigned condi ions we e he di ision o pa icipan s in o wo di e en
ask g oups and wo di e en delibe a ion g oups du ing he p epa a o y phase. Iden ified esea che decisions a e he 107 common decisions aken in
da a p epa a ion and s a is ical modeling ac oss eams and hei models. Resea che cha ac e is ics we e iden ified h ough a su ey o pa icipan s and
mul ii em scaling using ac o analysis (SI Appendix,Fig.S3). The eade will find many o he de ails in SI Appendix.
§
This is a eminde ha 16 eams epo ed wo di e ing conclusions based on hei in e -
p e a ion o di e en model speci ica ions, causing he N o jump om 72 eams o 89
eam-conclusion uni s o analysis (Me hods).
4o 8 h ps://doi.o g/10.1073/pnas.2203150119 pnas.o g

Finally, we ollowed p e ious many-analys s esea ch (31) by
c ea ing a benchma k o he abo e esul s ia a mul i e se sim-
ula ion o possible model specifica ions. Using he same
app oach, we ound ha 23 decisions could explain jus o e
16% o he a iance in nume ical ou comes among 2,304 sim-
ula ed models (SI Appendix, Table S8). In con as o ou eco-
logical esea ch se ing obse ing ac ual esea che beha io s, we
all a sho o his 16% simula ed o al explained a iance by
almos 12 pe cen age poin s. E en wi h he use o an algo i hm
o sample all possible a iable combina ions, we emain o e 11
pe cen age poin s sho .
Discussion
Resul s om ou con olled esea ch design in a la ge-scale
c owdsou ced esea ch e o in ol ing 73 eams demons a e
ha analyzing he same hypo hesis wi h he same da a can lead
o subs an ial di e ences in s a is ical es ima es and subs an i e
conclusions. In ac , no wo eams a i ed a he same se o
nume ical esul s o ook he same majo decisions du ing da a
analysis. Ou finding o ou come a iabili y echoes hose o
ecen s udies in ol ing many analys s unde aken ac oss scien-
ific disciplines. The s udy epo ed he e di e s om hese p e-
ious e o s because i a emp ed o ca alog e e y decision in
he esea ch p ocess wi hin each eam and use hose decisions
and p edic i e modeling o explain why he e is so much ou -
come a iabili y. Despi e his highly g anula decomposi ion o
he analy ical p ocess, we could only explain less han 2.6% o
he o al a iance in nume ical ou comes. We also es ed i
expe ise, belie s, and a i udes obse ed among he eams
biased esul s, bu hey explained li le. E en highly skilled sci-
en is s mo i a ed o come o accu a e esul s a ied emen-
dously in wha hey ound when p o ided wi h he same da a
and hypo hesis o es . The s anda d p esen a ion and con-
sump ion o scien ific esul s did no disclose he o ali y o
esea ch decisions in he esea ch p ocess. Ou conclusion is
ha we ha e apped in o a hidden uni e se o idiosync a ic
esea che a iabili y.
This finding was a o ded by a many-analys s design as an
app oach o scien ific inqui y. Some schola s ha e p oposed
mul i e se analysis o simula e analy ic decisions ac oss
esea che s (31), a me hod o p o ide a many-analys s se o
ou comes wi hou he massi e coo dina ion and human capi-
al commi men o he wise necessa y o such a s udy. The
d awback o a simula ion app oach is ha i is cons uc ed
based on a single da a analysis pipeline om one esea ch
eam and may no eflec he complex eali y o di e en
esea ch p ocesses ca ied ou by di e en eams in di e en
con ex s. This s udy obse ed esea che s in a con olled ye
ecological wo k en i onmen . In doing so, i e ealed he hid-
den uni e se o consequen ial decisions and con ex ual ac o s
ha a y ac oss esea che s and ha a simula ion, hus a ,
canno cap u e.
Implica ions. Resea che s mus make analy ical decisions so
minu e ha hey o en do no e en egis e as decisions.
Ins ead, hey go unno iced as nondelibe a e ac ions ollowing
os ensibly s anda d ope a ing p ocedu es. Ou s udy shows
ha , when aken as a whole, hese hund eds o decisions
combine o be a om i ial. Howe e , his unde s anding
only a ises om he uniqueness o each o he 1,253 models
analyzed he ein. Ou findings sugges eliabili y ac oss
esea che s may emain low e en when hei accu acy mo i a-
ion is high and biasing incen i es a e emo ed. Highe le els
o me hodological expe ise, ano he equen ly sugges ed
emedy, did no lead o lowe a iance ei he . Hence, we a e
Fig. 3. Resea che cha ac e is ics do no explain ou come a iance be ween eams o wi hin eams. The dis ibu ion o eam a e age o AMEs (Le ) and
wi hin- eam a iance in AMEs (Righ ) ac oss esea che s g ouped acco ding o mean spli s (“lowe ”and “highe ”) on me hodological and opic expe ise
(po en ial compe encies bias) and on p io a i udes owa d immig a ion and belie s abou whe he he hypo hesis is ue (po en ial confi ma ion bias). Log
a iance was shi ed so ha he minimum log alue equals ze o. Teams submi ing only one model assigned a a iance o ze o. Pea son co ela ions along
wi h a P alue (“R”) a e calcula ed using con inuous sco es o each esea che cha ac e is ic a iable.
PNAS 2022 Vol. 119 No. 44 e2203150119 h ps://doi.o g/10.1073/pnas.2203150119 5o 8
le o belie e ha idiosync a ic unce ain y is a undamen al
ea u e o he scien ific p ocess ha is no easily explained
by ypically obse ed esea che cha ac e is ics o analy i-
cal decisions.
These findings add a pe spec i e o he me ascience con-
e sa ion, emphasizing unce ain y in addi ion o bias. The
conclusion wa an ed om much o he me ascience wo k
ca ied ou in he wake o he “ eplica ion c isis”in psychol-
ogy and o he fields has been ha published esea ch findings
a e mo e biased han p e iously hough . The conclusion
wa an ed om his and o he simila s udies is ha pub-
lished esea ch findings a e also mo e unce ain han p e i-
ously hough .
As esea che s, we bea he esponsibili y o accu a ely
desc ibe and explain he wo ld as i is bu also, o communica e
he unce ain y associa ed wi h ou knowledge claims.
Al hough he academic sys em p i ileges inno a ion o e epli-
ca ion, p o iding a no el answe o a ques ion is jus as essen-
ial as in o ming abou how much us we can place in ha
answe . Ou s udy has shown ha o ully assess and add ess
unce ain y, eplica ions a e aluable bu insu ficien . Only
la ge numbe s o analyses may show whe he in a specific
field, independen esea che s eliably a i e a simila conclu-
sions, hus enhancing o unde mining ou confidence in a
gi en knowledge claim.
Specifically, we belie e ha se ious acknowledgmen o
idiosync a ic a ia ion in esea ch findings has a leas ou
implica ions o imp o ing he p esen a ion and in e p e a ion
o empi ical e idence. Fi s , con empla ing ha esul s migh
a y g ea ly i a gi en s udy had been conduc ed by a di e en
se o esea che s o e en he same esea che s a a di e en
ime, calls o epis emic humili y when d awing conclusions
based on seemingly objec i e quan i a i e p ocedu es. Second,
he findings emind us o ca e ully documen e e y hing
because, in combina ion, e en he mos seemingly minu e deci-
sions could d i e esul s in di e en di ec ions; and only awa e-
ness o hese minu ae could lead o p oduc i e he oe ical
discussions o empi ical es s o hei legi imacy. Thi d, coun-
e ing a de ea is iew o he scien ific en e p ise, his s udy
helps us app ecia e he knowledge accumula ed in a eas whe e
scien is s do con e ge on expe consensus—such as he human
impac on he global clima e o a no able inc ease in poli ical
pola iza ion in he Uni ed S a es o e he pas decades. Fou h,
ou s udy sugges s ha i we eed scien is s om bias caused
by he “pe e se incen i es”inhe en in he ins i u ions o sci-
ence, hei own p eexis ing biases o belie s migh no ma e as
much o he ou comes hey gene a e as some may ea . In ai -
ness, he eams indica ed wha models hey in ended o un in
ad ance. Al hough hey we e ee o upda e and change hese a
any ime, we can assume ha his may ha e educed po en ial
confi ma ion bias.
Limi a ions and Ou look. Ou s udy has limi a ions ha wa -
an discussion. Fi s , we do no know he gene alizabili y o
ou s udy o di e en opics, disciplines, o e en da ase s.
A majo segmen o social science wo ks wi h su ey da a, and
ou esul s eflec his ype o esea ch. In expe imen al
esea ch, he da a-gene a ing model is o en clea e o in ol es
ewe decisions. Mo eo e , in social esea ch, he e a e no
New onian laws o defini e quan um s a is ical likelihoods o
wo k wi h, sugges ing ha ou case migh o e es ima e a i-
abili y compa ed wi h he na u al sciences. On he o he
hand, unc ional magne ic esonance imaging ( MRI), gene,
and space elescope da a, o example, a e a mo e complex
han wha we ga he in social su eys. The complexi y o da a
analysis pipelines is co espondingly g ea e in hese fields,
bu i is possible ha ha ing mo e analy ical decisions allows
us o accoun o mo e o he a ia ion in esea ch ou comes.
We belie e ha he numbe o decision poin s in da a analysis
and he ex en o which esea che s unde s and he da a-
gene a ing p ocess may de e mine he deg ee o ou come a i-
a ion in a field, bu i emains an open ques ion i and how
design decisions play a smalle o la ge ole ac oss fields.
Second, al hough we hoped o o e deepe insigh s on he
subs an i e hypo hesis unde obse a ion, we did no ob ain
e idence ha mo es conclusions in any di ec ion. These lessons
combined wi h he ac ha a subs an ial po ion o pa ici-
pan s conside ed he hypo hesis no es able wi h hese da a
o e a po en ial explana ion o why his is such a con es ed
hypo hesis in he social sciences (19, 24, 32).
Looking o wa d, we ake he ac ha 13.5% o pa icipa -
ing analys s claimed ha he a ge hypo hesis is “no es able”
wi h he p o ided da a as a powe ul eminde o he impo -
ance o design app op ia eness and clea specifica ion o
hypo heses. This implica es cla i y in he meaning o a conclu-
sion. In ou s udy, “suppo ”o he hypo hesis gene ally mean
ejec ion o he null, whe eas “ ejec ”mean consis ency wi h
he null o inconclusi e esul s. Teams we e le o hei own
de ices in deciding wha cons i u ed suppo o , e idence
agains , o non es abili y o he a ge hypo hesis, and his alone
in oduced a deg ee o inde e minacy in o he esea ch p ocess.
O e all, hese obse a ions call o mo e a en ion o concep-
ual, causal, and heo e ical cla i y in he social sciences as well
as o he ga he ing o new da a when esul s no longe appea
o mo e a subs an i e a ea o wa d (20, 21). They also sugges
ha i we wan o eap epis emic benefi s om he p esen
mo e owa d open esea ch p ac ices, we mus make mo e con-
scious e o s o complemen me hodological anspa ency wi h
heo e ical cla i y.
Finally, we no e ha he conclusions o his s udy we e
hemsel es de i ed om my iad seemingly mino (me a-)ana-
ly ical decisions, jus like hose we obse ed among ou
analys s. We, he e o e, encou age eade s o sc u inize ou ana-
ly ical p ocess by aking ad an age o SI Appendix, he ep o-
duc ion files, and he web-based in e ac i e app ha allow o
easy explo a ion o all da a unde lying his s udy.
¶
Da a, Ma e ials, and So wa e A ailabili y. Da aandcodeha ebeen
deposi ed in Gi Hub (h ps://gi hub.com/nb eznau/CRI) (33), and Ha a d Da a-
e se (h ps://doi.o g/10.7910/DVN/UUP8CX) (34).
ACKNOWLEDGMENTS. We hank he Mannheim Cen e o Eu opean
Social Resea ch a he Uni e si y o Mannheim (Mannheim, Ge many) o
p o iding ex a ins i u ional suppo o his la ge-scale p ojec and
Joachim Gassen a he Humbold Uni e si y (Be lin, Ge many) o inspi a-
ion and eedback o ou in e ac i e app. Addi ionally help ul we e
commen s p o ided a he S an o d Me a Resea ch Inno a ion Cen e a
S an o d (METRICS) In e na ional Fo um, he Ludwig Maximilian Uni e si y
o Munich Sociology Resea ch Colloquium, and he Technical Uni e si y
Chemni z “Open Science”Con e ence and in pe sonal communica ions
wi h S e e Velsko. A po ion o he la e phases o coding and shiny app
de elopmen was suppo ed by a F eies Wissen (“Open Knowledge”)Fel-
lowship o Wikimedia. The iews exp essed he ein a e hose o he au ho s
and do no eflec he posi ion o he US Mili a y Academy, he Depa men
o he A my, o he Depa men o De ense.
¶
In o ma ion is a ailable a h ps://na e-b eznau.shinyapps.io/shiny/ and h ps://gi hub.
com/nb eznau/CRI.
6o 8 h ps://doi.o g/10.1073/pnas.2203150119 pnas.o g
Na e B eznau
a,1,2
, Eike Ma k Rinke
b,2
,Alexande Wu ke
c,l,2
,HungH.V.
Nguyen
a,d
,MunaAdem
e
, Jule Ad iaans
, Amalia Al a ez-Benjumea
g
,Hen ikK.
Ande sen
h
, Daniel Aue
c
, Fla io Aze edo
j
,OkeBahnsen
i
,Da eBalze
k
,
Ge i Baue
oooo
,PaulC.Baue
c
, Ma kus Baumann
m,dd
, Sha on Bau e
n
, Ve ena
Benoi
l,o
, Julian Be naue
c
,Ca lBe ning
pppp
,AnnaBe hold
o
,FelixS.Be hke
p
,
Thomas Biege
q
, Ka ha ina Blinzle
, Johannes N. Blumenbe g
, Licia Bobzien
s
,
And ea Bohman
,ThijsBol
x,uuuu
,AmieBos ic
u
,ZuzannaB zozowska
,w
,Ka ha ina
Bu gdo
i
, Kaspa Bu ge
x,y,
, Ka h in B. Busch
z
, Juan Ca los-Cas illo
aa,mmmm
,
Na han Chan
nnnn
, Pablo Ch is mann
ssss
, Roxanne Connelly
cc
, Ch is ian S.
Czyma a
qqqq
,ElenaDamian
yyyy
, Alejand o Ecke
c
,AchimEdelmann
, Mau een A.
Ege
, Simon Elle b ock
c,i
, Anna Fo ke
z
,And eaFo s e
pp
, Ch is Gaasendam
ee
,
Kons an in Ga as
i
,Ve nonGayle
cc
, The esa Gessle
hhhh
,TimoGnambs
gg
,
Am
elie Gode oid
aaaaa
,MaxG
€
omping
hh
,Ma inG oß
ii
, S e an G ube
jj
, Tobias
Gumme
ssss
,And easHadja
kk,mm,llll,
, Jan Paul Heisig
iiii,mmmmm
,Sebas ian
Hellmeie
nnnnn
, S e anie Heyne
c
, Magdalena Hi sch
ooooo
,MikaelHje m
,Osh a
Hochman
ssss
, And eas H€
o e mann
mm,qq
,SophiaHunge
ppppp
, Ch is ian
Hunkle
nn
, No a Hu h
oo
,Zs

ofiaS.Ign

acz
qqqq
,Lau aJacobs
ll
,Jannes
Jacobsen
,jjjj
, Bas ian Jaege
, Sebas ian Jungkunz
ss,lll,iiiii
, Nils Jungmann
,
Ma hias Kau
uu
, Manuel Kleine
uuu
, Julia Klinge
jjjjj
, Jan-Philipp Kolb
,Ma a
Kołczy
nska
ww
,JohnKuk
xx
, Ka ha ina Kunißen
k
,Dafina Ku i Sina a
z
,Alexande
Langenkamp
qqqq
,PhilippM.Le sch
,ddddd
, Lea-Ma ia L€
obel
,PhilippLu sche
yy
,
Ma hias Made
zz
,JoanE.Madia
aaa,lllll
,Na aliaMalancu
bbb
,LuisMaldonado
ccc
,
Helge Ma ah ens
e
, Nicole Ma in
ddd
,PaulMa inez
eee
,JochenMaye l
h
,Osca J.
Mayo ga
, Pa icia McManus
e
, Kyle McWagne
bb
, Cecil Meeusen
ee
, Daniel
Meie ieks
ooooo
, Jona han Mellon
ddd
,F iedolinMe hou
ggg
,SamuelMe k
hhh
,
Daniel Meye
kkkkk
, Le icia Micheli
iii
,Jona hanMijs
jjj
,C is

obal Moya
kkk
,Ma cel
Neunhoe e
i
,DanielN
€
us
hhhhh
, Ola Nygå d
mmm
, Fabian Ochsen eld
nnn
, Gunna
O e
k
, Anna O. Pechenkina
ooo
, Ch is ophe P osse
ppp
,LouisRaes
,Ke in
Rals on
cc
, Miguel R. Ramos
qqq
,A neRoe s
,Jona hanRoge s
sss
,Guido
Rope s
i
, Robin Samuel
kk,
,G ego Sand
jj
,A ielaSchach e
,Me lin
Schae e
qqqqq
,Da idSchie e decke
xxxx
, Elma Schlue e
uuu
,RegineSchmid
o
,
Ka ja M. Schmid
, Alexande Schmid -Ca an
qqqq
, Claudia Schmiedebe g
oooo
,
J€
u gen Schneide
ccccc
, Ma ijn Schoon elde
,sssss
, Julia Schul e-Cloos
kkkk
,Sandy
Schumann
wwww
, Reinha d Schunck
oo
,J
€
u gen Schupp
, Julian Seu ing
bbbbb
,
Henning Silbe
, Willem Sleege s
, Nico Sonn ag
k
,Alexande S aud
z
,Nadia
S eibe
www
,NilsS eine
pppp
, Sebas ian S e nbe g
z
, Die e S ie s
zzzz
, D agana
S ojmeno ska
uuuu
,No aS o z
xxx
, E ich S iessnig
,Anne-Ka h inS oppe
,
Janna Tel emann
yyy
,And eyTibaje
mmm
, B ian Tung
, Giacomo Vagni
x
,Jaspe
Van Assche
,eeeee
, Me a an de Linden
xxx
, Jolanda an de Noll
zzz
,A noVan
Hoo egem
ee
, S e an Vog enhube
uuuuu
,BogdanVoicu
aaaa,bbbb
,Fieke
Wagemans
,cccc
, Nadja Wehl
dddd
,HannahWe ne
zzzz
,B en onM.Wie nik
eeee
,Fabian
Win e
g
, Ch is o Wol
c,i,wwwww
, Yuki Yamada
,NanZhang
c
,Con ad
Zille
ss,ggggg
, S e an Zins
gggg
and Tomasz _
Z
oł ak
ww
Au ho a filia ions:
a
Resea ch Cen e on Inequali y and Social Policy (SOCIUM), Uni e si y
o B emen, B emen, 28359, Ge many;
b
School o Poli ics and In e na ional S udies,
Uni e si y o Leeds, Leeds, LS2 9JT, Uni ed Kingdom;
c
Mannheim Cen e o Eu opean
Social Resea ch, Uni e si y o Mannheim, 68131 Mannheim, Ge many;
d
B emen
In e na ional G adua e School o Social Sciences, 28359 B emen, Ge many;
e
Depa men
o Sociology, Indiana Uni e si y, Blooming on, IN 47405;
Socio-Economic Panel S udy
(SOEP), Ge man Ins i u e o Economic Resea ch (DIW), 10117 Be lin, Ge many;
g
Mechanisms o No ma i e Change, Max Planck Ins i u e o Resea ch on Collec i e
Goods, 53113 Bonn, Ge many;
h
Ins i u e o Sociology, Chemni z Uni e si y o Technology,
09126 Chemni z, Ge many;
i
School o Social Sciences, Uni e si y o Mannheim, 68159
Mannheim, Ge many;
j
Depa men o Psychology, Uni e si y o Camb idge, Camb idge,
CB23RQ, Uni ed Kingdom;
k
Ins i u e o Sociology, Johannes Gu enbe g Uni e si y Mainz,
55128 Mainz, Ge many;
l
Depa men o Poli ical Science, Ludwig Maximilian Uni e si y,
80539 Munich, Ge many;
m
Heidelbe g Uni e si y, 69117 Heidelbe g, Ge many;
n
Compa a i e Poli ical Economy, Uni e si y o Kons anz, 78457 Kons anz, Ge many;
o
Facul y o Social Sciences, Economics, and Business Adminis a ion, Uni e si y o
Bambe g, 96052 Bambe g, Ge many;
p
Resea ch Depa men on In as a e Conflic , Peace
Resea ch Ins i u e F ank u , 60329 F ank u , Ge many;
q
Depa men o Social Policy,
London School o Economics and Poli ical Science, London, WC2A 2AE, Uni ed Kingdom;
Su ey Da a Cu a ion, Leibniz Ins i u e o he Social Sciences (GESIS), 50667 Cologne,
Ge many;
s
Jacques Delo s Cen e, He ie School, 10117 Be lin, Ge many;
Depa men o
Sociology, Umeå Uni e si y, 90187 Umeå, Sweden;
u
Depa men o Sociology, The
Uni e si y o Texas Rio G ande Valley, B owns ille, TX 78520;
Vienna Ins i u e o
Demog aphy, Aus ian Academy o Sciences, 1030 Vienna, Aus ia;
w
Aus ian Na ional
Public Heal h Ins i u e, Gesundhei €
Os e eich (G€
OG), 1030 Vienna, Aus ia;
x
Social
Resea ch Ins i u e, Ins i u e o Educa ion, Uni e si y College London, London, WC1H 0AL,
Uni ed Kingdom;
y
Depa men o Sociology, Uni e si y o Zu ich, 8050 Zu ich, Swi ze land;
z
Independen esea che ;
aa
Depa men o Sociology, Uni e si y o Chile, San iago,
7800284, Chile;
bb
Depa men o Poli ical Science, The Uni e si y o Cali o nia, I ine, CA
92617;
cc
School o Social and Poli ical Science, Uni e si y o Edinbu gh, Edinbu gh, EH8
9LD, Uni ed Kingdom;
dd
Ins i u e o Poli ical Science, Goe he Uni e si y F ank u , 60323
F ank u , Ge many;
ee
Depa men o Sociology, Cen e o Sociological Resea ch, KU
Leu en, 3000 Leu en, Belgium;
M
edialab, Sciences Po, 75007 Pa is, F ance;
gg
Educa ional
Measu emen , Leibniz Ins i u e o Educa ional T ajec o ies, 96047 Bambe g, Ge many;
hh
School o Go e nmen and In e na ional Rela ions, G i fi h Uni e si y, Na han, QLD,
4111, Aus alia;
ii
Depa men o Sociology, Uni e si y o T€
ubingen, 72074 T€
ubingen,
Ge many;
jj
Max Planck Ins i u e o Social Law and Social Policy, 80799 Munich, Ge many;
kk
Uni e si y o Luxembou g, 4365 Esch-su -Alze e, Luxembou g;
ll
Depa men o Poli ical
Science, Uni e si 
e Lib e de B uxelles, 1050 B uxelles, Belgium;
mm
Wi scha s- und
Sozialwissenscha liches Ins i u (WSI), Hans B€
ockle Founda ion, 40474 D€
usseldo ,
Ge many;
nn
Be lin Ins i u e o In eg a ion and Mig a ion Resea ch (BIM), Humbold
Uni e si y Be lin, 10099 Be lin, Ge many;
oo
School o Human and Social Sciences,
Uni e si y o Wuppe al, 42119 Wuppe al, Ge many;
pp
Empi ical Educa ional and Highe
Educa ion Resea ch, F eie Uni e si €
a Be lin, 14195 Be lin, Ge many;
qq
Ge man Socio-
Economic Panel Su ey, 10117 Be lin, Ge many;
Depa men o Social Psychology, Tilbu g
Uni e si y, 5037AB Tilbu g, The Ne he lands;
ss
Ins i u e o Socio-Economics, Uni e si y o
Duisbu g-Essen, 47057 Duisbu g, Ge many;
Zeppelin Uni e si y, 88045 F ied ichsha en,
Ge many;
uu
Depa men o Psychology, Medical School Hambu g, 20457 Hambu g,
Ge many;
Fede al S a is ics O fice Ge many, Des a is, 65189 Wiesbaden, Ge many;
ww
Depa men o Resea ch on Social and Ins i u ional T ans o ma ions, Ins i u e o
Poli ical S udies o he Polish Academy o Sciences, 00-625 Wa saw, Poland;
xx
Depa men
o Poli ical Science, Uni e si y o Oklahoma, No man, OK 73019;
yy
Depa men o Poli ical
Science, Uni e si y o Oslo, 0851 Oslo, No way;
zz
Depa men o Poli ics and Public
Adminis a ion, Uni e si y o Kons anz, 78457 Kons anz, Ge many;
aaa
Depa men o
Sociology, Nu field College, Uni e si y o Ox o d, Ox o d, OX1 1JD, Uni ed Kingdom;
bbb
The
Ins i u e o Ci izenship S udies (InCi e), Uni e si y o Gene a, 1205 Gene a, Swi ze land;
ccc
Ins i u o de Sociologia, Pon ifical Ca holic Uni e si y o Chile, San iago, 7820436, Chile;
ddd
Depa men o Poli ics, Uni e si y o Manches e , Manches e , M19 2JS, Uni ed
Kingdom;
eee
Depa men o Ins i u ional Resea ch, Wes e n Go e no s Uni e si y, Sal
Lake Ci y, UT 84107;
Depa men o Sociology, Uni e si y o Cali o nia, Los Angeles, CA
90095;
ggg
Depa men o Sociology and Cen e o Social Da a Science, Uni e si y o
Copenhagen, 1353 Copenhagen, Denma k;
hhh
Depa men o School De elopmen ,
Uni e si y o Educa ion Ka ls uhe, 76133 Ka ls uhe, Ge many;
iii
Depa men o Psychology
III, Julius-Maximilians Uni e si y W€
u zbu g, 97070 W€
u zbu g, Ge many;
jjj
Depa men o
Sociology, Bos on Uni e si y, Bos on, MA 02215;
kkk
Facul y o Sociology, Biele eld
Uni e si y, 33615 Biele eld, Ge many;
lll
Ins i u e o Poli ical Science, Uni e si y o M€
uns e ,
48149 M€
uns e , Ge many;
mmm
Di ision o Mig a ion, E hnici y and Socie y (REMESO),
Link€
oping Uni e si y, 60174 Link€
oping, Sweden;
nnn
Adminis a i e Headqua e s, Max
Planck Socie y, 80539 Be lin, Ge many;
ooo
Depa men o Poli ical Science, U ah S a e
Uni e si y, Logan, UT 84321;
ppp
Depa men o Poli ics, In e na ional Rela ions and
Philosophy, Royal Holloway Uni e si y o London, London, TW20 0EX, Uni ed Kingdom;
qqq
Depa men o Social Policy, Sociology and C iminology, Uni e si y o Bi mingham,
Bi mingham, B15 2TT, Uni ed Kingdom;
Depa men o De elopmen al, Pe sonali y and
Social Psychology, Ghen Uni e si y, 9000 Ghen , Belgium;
sss
Di ision o Social Science,
New Yo k Uni e si y Abu Dhabi, Abu Dhabi, 10276, Uni ed A ab Emi a es;
Depa men o
Sociology, Washing on Uni e si y in S . Louis, S . Louis, MO 63130;
uuu
Ins i u e o
Sociology, Jus us Liebig Uni e si y o Giessen, 35394 Giessen, Ge many;
Uni e si y
College Dublin, Dublin 4, I eland;
www
Depa men o Sociology, Uni e si y o Vienna, 1090
Vienna, Aus ia;
xxx
In e disciplina y Social Science, U ech Uni e si y, 3584 U ech , The
Ne he lands;
yyy
Ins i u e o Social Sciences, Uni e si y o Hildesheim, 31141 Hildesheim,
Ge many;
zzz
Depa men o Psychology, Uni e si y o Hagen, 58097 Hagen, Ge many;
aaaa
Resea ch Ins i u e o Quali y o Li e, Romanian Academy, 010071 Bucha es , Romania;
bbbb
Depa men o Sociology, Lucian Blaga Uni e si y o Sibiu, 550024 Sibiu, Romania;
cccc
Ne he lands Ins i u e o Social Resea ch, 2500 BD The Hague, he Ne he lands;
dddd
Resea ch Clus e "The Poli ics o Inequali y", Uni e si y o Kons anz, 78464 Kons anz,
Ge many;
eeee
Depa men o Psychology, Uni e si y o Sou h Flo ida, Tampa, FL 33620;
Facul y o A s and Science, Kyushu Uni e si y, Fukuoka, 819-0395, Japan;
gggg
Ins i u e
o Employmen Resea ch, Fede al Employmen Agency, 90478 Nu embe g, Ge many;
hhhh
Kul u wissenscha liche Fakul €
a , Eu opean Uni e si y Viad ina, 15230 F ank u (Ode ),
Ge many;
iiii
Uni e si y o G oningen, 9712 CP G oningen,The Ne he lands;
jjjj
Clus e "Da a-
Me hods-Moni o ing", Ge man Cen e o In eg a ion and Mig a ion Resea ch
(DeZIM),10117 Be lin, Ge many;
kkkk
Robe Schuman Cen e o Ad anced S udies,
Eu opean Uni e si y Ins i u e, 50133 Flo ence, I aly;
llll
Uni e si y o F ibou g, 1700 F ibou g,
Swi ze land;
mmmm
Cen e o Social Conflic and Cohesion S udies (COES), Pon ificia
Uni e sidad Ca 
olica de Chile, San iago, 8331150, Chile;
nnnn
Depa men o Poli ical
Science and In e na ional Rela ions, Loyola Ma ymoun Uni e si y, Los Angeles, CA 90045;
oooo
Depa men o Sociology, Ludwig Maximilian Uni e si y, 80801 Munich, Ge many;
pppp
Ins i u e o Poli ical Science, Johannes Gu enbe g Uni e si y Mainz, 55099 Mainz,
Ge many;
qqqq
Ins i u e o Sociology, Goe he Uni e si y F ank u , 60323 F ank u ,
Ge many;
Knowledge Exchange and Ou each, Leibniz Ins i u e o he Social Sciences
(GESIS), 68159 Mannheim, Ge many;
ssss
Da a and Resea ch on Socie y, Leibniz Ins i u e o
he Social Sciences, 68159 Mannheim, Ge many;
Depa men o Su ey Design and
Me hodology, Leibniz Ins i u e o he Social Sciences (GESIS), 68159 Mannheim, Ge many;
uuuu
Depa men o Sociology, Uni e si y o Ams e dam, 1001 Ams e dam, The
Ne he lands;
Jacobs Cen e o P oduc i e You h, Uni e si y o Zu ich, 8050 Zu ich,
Swi ze land;
wwww
Depa men o Secu i y and C ime Science, Uni e si y College London,
London,WC1E 6BT, Uni ed Kingdom;
xxxx
Ins i u e o Media and Communica ion S udies,
F eie Uni e si €
a Be lin, 14195 Be lin, Ge many;
yyyy
Li es yle and Ch onic Diseases,
Epidemiology and Public Heal h, Sciensano, 1000 B ussels, Belgium;
zzzz
Cen e o Poli ical
Science Resea ch, KU Leu en, 3000 Leu en, Belgium;
aaaaa
Cen e o Resea ch on Peace
and De elopmen , KU Leu en, 3000 Leu en, Belgium;
bbbbb
Depa men o Mig a ion,
Leibniz Ins i u e o Educa ional T ajec o ies, 96047 Bambe g, Ge many;
ccccc
T€
ubingen
School o Educa ion, Uni e si y o T€
ubingen, 72074 T€
ubingen, Ge many;
ddddd
Depa men
o Social Sciences, Humbold Uni e si y Be lin, 10099 Be lin, Ge many;
eeeee
Cen e o
Social and Cul u al Psychology, Uni e si 
e Lib e de B uxelles, 1050 B ussels, Belgium;
Depa men o Economics, Tilbu g Uni e si y, 5037AB Tilbu g, The Ne he lands;
ggggg
Depa men o Poli ical Science, Uni e si y o Duisbu g-Essen, 47057 Duisbu g,
Ge many;
hhhhh
Depa men o Geosciences, Uni e si y o M€
uns e , 49149 M€
uns e ,
Ge many;
iiiii
Chai o Poli ical Sociology, Uni e si y o Bambe g, 96052 Bambe g, Ge many;
jjjjj
Ins i u e o Sociology and Social Psychology, Uni e si y o Cologne, 50931 Cologne,
Ge many;
kkkkk
Depa men o Educa ion and Social Sciences, Uni e si y o Cologne, 50931
Cologne, Ge many;
lllll
Ins i u e o he E alua ion o Public Policies, Fondazione B uno
Kessle , 38122 T en o, I aly;
mmmmm
Resea ch G oup "Heal h and Social Inequali y", Be lin
Social Science Cen e (WZB), 10785 Be lin, Ge many;
nnnnn
T ans o ma ions o Democ acy
Uni , Be lin Social Science Cen e (WZB), 10785 Be lin, Ge many;
ooooo
Resea ch Uni
Mig a ion, In eg a ion, T ansna ionaliza ion, Be lin Social Science Cen e (WZB), 10785
Be lin, Ge many;
ppppp
Cen e o Ci il Socie y Resea ch, Be lin Social Science Cen e , 10785
Be lin, Ge many;
qqqqq
Depa men o Sociology, Uni e si y o Copenhagen, 1353
Copenhagen, Denma k;
Depa men o Social Sciences, Uni e si y o Luxembou g,
4366 Esch-su -Alze e, Luxembou g;
sssss
Depa men o Eu opean Languages and Cul u es,
Uni e si y o G oningen, 9712 EK G oningen, The Ne he lands;
Depa men o
Demog aphy, Uni e si y o Vienna, 1010 Vienna, Aus ia;
uuuuu
Educa ion and Employmen ,
Ins i u e o Ad anced S udies, Uni e si y o Vienna, Vienna, 1080 Aus ia; and
Policy
Pe spec i es, Ci izen Pe spec i es, and Beha io s, Ne he lands Ins i u e o Social
Resea ch, 2594 The Hague, The Ne he lands;
wwwww
P esiden , Leibniz Ins i u e o he
Social Sciences (GESIS), 68159 Mannheim, Ge many
Au ho con ibu ions: N.B., E.R., and A.W. designed esea ch; N.B., E.R., and A.W.
pe o med esea ch; N.B., H.N., and D.N. con ibu ed new eagen s/analy ic ools;
N.B., H.N., M.A., J.A., A.A., H.A., D.A., F.A., O.B., D.B., G.B., P.B., M.B., S.B., V.B., J.B.,
C.B., A.B., F.B., T.B., K.B., J.B., L.B., A.B., T.B., A.B., Z.B., K.B., K.B., K.B., J.C., N.C., P.C.,
R.C., C.C., E.D., A.E., A.E., M.E., S.E., A.F., A.F., C.G., K.G., V.G., T.G., T.G., A.G., M.G.,
M.G., S.G., T.G., A.H., J.H., S.H., S.H., M.H., M.H., O.H., A.H., S.H., C.H., N.H., Z.I., L.J.,
J.J., B.J., S.J., N.J., M.K., M.K., J.K., J.K., M.K., J.K., K.K., D.K., A.L., P.L., L.L., P.L., M.M.,
J.M., N.M., L.M., H.M., N.M., P.M., J.M., O.M., P.M., K.M., C.M., D.M., J.M., F.M., S.M.,
D.M., L.M., J.M., C.M., M.N., D.N., O.N., F.O., G.O., A.P., C.P., L.R., K.R., M.R., A.R., J.R.,
G.R., R.S., G.S., A.S., M.S., D.S., E.S., K.S., R.S., A.S., C.S., J.S., M.S., J.S., S.S., R.S., J.S.,
J.S., H.S., W.S., N.S., A.S., N.S., N.S., S.S., D.S., D.S., N.S., E.S., A.S., J.T., A.T., B.T., G.V.,
J.V., M.V., J.V., A.V., S.V., B.V., F.W., N.W., H.W., B.W., F.W., C.W., Y.Y., N.Z., C.Z., S.Z.,
and T.Å. analyzed da a; and N.B., E.R., and A.W. w o e he pape .
PNAS 2022 Vol. 119 No. 44 e2203150119 h ps://doi.o g/10.1073/pnas.2203150119 7o 8
1. M. Solomon, Social Empi icism (MIT P ess, 2007).
2. N. O eskes, Why T us Science? (P ince on Uni e si y P ess, 2019).
3. C. F. Came e e al., E alua ing he eplicabili y o social science expe imen s in Na u e and Science
be ween 2010 and 2015. Na . Hum. Beha . 2, 637–644 (2018).
4. Open Science Collabo a ion, PSYCHOLOGY. Es ima ing he ep oducibili y o psychological science.
Science 349, aac4716 (2015).
5. S. Ri chie, Science Fic ions: How F aud, Bias, Negligence, and Hype Unde mine he Sea ch o T u h
(Me opoli an Books, 2020).
6. A. B. Sø ensen, The s uc u al basis o social inequali y. Am. J. Sociol. 101, 1333–1365 (1996).
7. B. S. F ey, Publishing as p os i u ion? Choosing be ween one’s own ideas and academic success.
Public Choice 116, 205–223 (2003).
8. A. Gelman, E. Loken, The s a is ical c isis in science. Am. Sci. 102, 460 (2014).
9. A. O ben, A. K. P zybylski, The associa ion be ween adolescen well-being and digi al echnology
use. Na . Hum. Beha . 3, 173–182 (2019).
10. M. Del Giudice, S. Ganges ad, A a ele ’s guide o he mul i e se: P omises, pi alls, and a
amewo k o he e alua ion o analy ic decisions. Ad . Me hods P ac . Psychol. Sci., 10.1177/
2515245920954925 (2021).
11. J. P. Simmons, L. D. Nelson, U. Simonsohn, False-posi i e psychology: Undisclosed flexibili y in
da a collec ion and analysis allows p esen ing any hing as significan . Psychol. Sci. 22, 1359–1366
(2011).
12. U. Schimmack, A me a-psychological pe spec i e on he decade o eplica ion ailu es in social
psychology. Can. Psychol. 61, 364–376 (2020).
13. J. F eese, D. Pe e son, The eme gence o s a is ical objec i i y: Changing ideas o epis emic ice
and i ue in science. Sociol. Theo y 36, 289–313 (2018).
14. R. Silbe zahn e al., Many analys s, one da a se : Making anspa en how a ia ions in analy ic
choices a ec esul s. Ad . Me hods P ac . Psychol. Sci. 1, 337–356 (2018).
15. G. Du ilh e al., The quali y o esponse ime da a in e ence: A blinded, collabo a i e assessmen o
he alidi y o cogni i e models. Psychon. Bull. Re . 26, 1051–1069 (2019).
16. J. A. Bas iaansen e al., Time o ge pe sonal? The impac o esea che s choices on he selec ion o
ea men a ge s using he expe ience sampling me hodology. J. Psychosom. Res. 137, 110211 (2020).
17. R. Bo inik-Neze e al., Va iabili y in he analysis o a single neu oimaging da ase by many eams.
Na u e 582,84–88 (2020).
18. A. J. Menk eld e al., Non-s anda d e o s. Social Science Resea ch Ne wo k [P ep in ] (2022).
h ps://doi.o g/10.2139/ss n.3961574 (Accessed 4 Janua y 2022).
19. D. B ady, R. Finnigan, Does immig a ion unde mine public suppo o social policy? Am. Sociol.
Re . 79,17–42 (2014).
20. K. Auspu g, J. B €
ude l, Has he c edibili y o he social sciences been c edibly des oyed?
Reanalyzing he “Many Analys s, One Da a Se ”P ojec . Socius 86, 532–565 (2021).
21. I. Lundbe g, R. Johnson, B. M. S ewa , Wha is you es imand? Defining he a ge quan i y
connec s s a is ical e idence o heo y. Am. Sociol. Re . 86, 532–565 (2021).
22. A. Alesina, E. Glaese , Why a e wel a e s a es in he US and Eu ope so di e en ? Wha do we lea n?
Ho izons S a egiques 2,51–61 (2006).
23. A. Alesina, E. Glaese , Figh ing Po e y in he US and Eu ope: A Wo ld o Di e ence (Ox o d
Uni e si y P ess, 2004).
24. A. Alesina, E. Mu a d, H. Rapopo , Immig a ion and p e e ences o edis ibu ion in Eu ope.
J. Econ. Geog . 21, 925–954 (2021).
25. M. A. Ege , E en in Sweden: The e ec o immig a ion on suppo o wel a e s a e spending.
Eu . Sociol. Re . 26, 203–217 (2010).
26. J. C. Ga and, P. Xu, B. C. Da is, Immig a ion a i udes and suppo o he wel a e s a e in he
Ame ican mass public. Am. J. Pol. Sci. 61, 146–162 (2017).
27. B. Bu goon, Immig a ion, in eg a ion, and suppo o edis ibu ion in Eu ope. Wo ld Poli . 66,
365–405 (2014).
28. J. Al , T. I e sen, Inequali y, labo ma ke segmen a ion, and p e e ences o edis ibu ion.
Am. J. Pol. Sci. 61,21–36 (2017).
29. S. Pa dos-P ado, C. Xena, Immig a ion and suppo o social policy: An expe imen al compa ison
o uni e sal and means- es ed p og ams. Poli ical Sci. Res. Me hods 7, 717–735 (2019).
30. M. L. B yan, S. P. Jenkins, Mul ile el modelling o coun y e ec s: A cau iona y ale. Eu . Sociol.
Re . 32,3–22 (2016).
31. M. Schweinsbe g e al., Same da a, di e en conclusions: Radical dispe sion in empi ical esul s
when independen analys s ope a ionalize and es he same hypo hesis. O gan. Beha . Hum.
Decis. P ocess. 165, 228–249 (2021).
32. M. A. Ege , N. B eznau, Immig a ion and he wel a e s a e: A c oss- egional analysis o Eu opean
wel a e a i udes. In . J. Comp. Sociol. 58, 440–463 (2017).
33. N. B eznau e al., The Hidden Uni e se o Da a-Analysis. Gi Hub. h ps://gi hub.com/nb eznau/CRI.
Deposi ed 12 Ap il 2021.
34. N. B eznau e al., The C owdsou ced Replica ion Ini ia i e Pa icipan Su ey. Ha a d Da a e se.
h ps://da a e se.ha a d.edu/da ase .xh ml?pe sis en Id=doi:10.7910/DVN/UUP8CX. Deposi ed
24 Ma ch 2021.
8o 8 h ps://doi.o g/10.1073/pnas.2203150119 pnas.o g