Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

B eznau, Na e e al. A icle — Published Ve sion Obse ing many esea che s using he same da a and hypo hesis e eals a hidden uni e se o unce ain y PNAS - P oceedings o he Na ional Academy o Sciences P o ided in Coope a ion wi h: WZB Be lin Social Science Cen e Sugges ed Ci a ion: B eznau, Na e e al. (2022) : Obse ing many esea che s using he same da a and hypo hesis e eals a hidden uni e se o unce ain y, PNAS - P oceedings o he Na ional Academy o Sciences, ISSN 1091-6490, Na ional Academy o Sciences, Washing on, DC, Vol. 119, Iss. 44, pp. 1-8, h ps://doi.o g/10.1073/pnas.2203150119 This Ve sion is a ailable a : h ps://hdl.handle.ne /10419/266342 S anda d-Nu zungsbedingungen: Die Dokumen e au EconS o dü en zu eigenen wissenscha lichen Zwecken und zum P i a geb auch gespeiche und kopie we den. Sie dü en die Dokumen e nich ü ö en liche ode komme zielle Zwecke e iel äl igen, ö en lich auss ellen, ö en lich zugänglich machen, e eiben ode ande wei ig nu zen. So e n die Ve asse die Dokumen e un e Open-Con en -Lizenzen (insbesonde e CC-Lizenzen) zu Ve ügung ges ell haben soll en, gel en abweichend on diesen Nu zungsbedingungen die in de do genann en Lizenz gewäh en Nu zungs ech e. Te ms o use: Documen s in EconS o may be sa ed and copied o you pe sonal and schola ly pu poses. You a e no o copy documen s o public o comme cial pu poses, o exhibi he documen s publicly, o make hem publicly a ailable on he in e ne , o o dis ibu e o o he wise use he documen s in public. I he documen s ha e been made a ailable unde an Open Con en Licence (especially C ea i e Commons Licences), you may exe cise u he usage igh s as speci ied in he indica ed licence. h ps://c ea i ecommons.o g/licenses/by/4.0/ Obse ing many esea che s using he same da a and hypo hesis e eals a hidden uni e se o unce ain y Edi ed by Douglas Massey, P ince on Uni e si y, P ince on, NJ; ecei ed Ma ch 6, 2022; accep ed Augus 22, 2022 This s udy explo es how esea che s’analy ical choices a ec he eliabili y o scien ific findings. Mos discussions o eliabili y p oblems in science ocus on sys ema ic biases. We b oaden he lens o emphasize he idiosync asy o conscious and unconscious decisions ha esea che s make du ing da a analysis. We coo dina ed 161 esea che s in 73 esea ch eams and obse ed hei esea ch decisions as hey used he same da a o independen ly es he same p ominen social science hypo hesis: ha g ea e immig a- ion educes suppo o social policies among he public. In his ypical case o social science esea ch, esea ch eams epo ed bo h widely di e ging nume ical findings and subs an i e conclusions despi e iden ical s a condi ions. Resea che s’expe ise, p io belie s, and expec a ions ba ely p edic he wide a ia ion in esea ch ou comes. Mo e han 95% o he o al a iance in nume ical esul s emains unexplained e en a e quali a i e coding o all iden ifiable decisions in each eam’swo kflow. This e eals a uni e se o unce ain y ha emains hidden when conside ing a single s udy in isola- ion. The idiosync a ic na u e o how esea che s’ esul s and conclusions a ied is a p e iously unde app ecia ed explana ion o why many scien ific hypo heses emain con es ed. These esul s call o g ea e epis emic humili y and cla i y in epo ing sci- en ificfindings. me ascience jmany analys s j esea che deg ees o eedom janaly ical flexibili y jimmig a ion and policy p e e ences O ganized scien ific knowledge p oduc ion in ol es ins i u ionalized checks, such as edi- o ial e ing, pee e iew, and me hodological s anda ds, o ensu e ha findings a e independen o he cha ac e is ics o p edisposi ions o any single esea che (1, 2). These p ocedu es should gene a e in e esea che eliabili y, o e ing consume s o scien ific findings assu ance ha hey a e no a bi a y flukes and ha o he esea che s would gen- e a e simila findings gi en he same da a. Recen me ascience esea ch challenges his assump ion as se e al a emp s o ep oduce findings om p e ious s udies ailed (3, 4). In esponse, scien is s ha e discussed a ious h ea s o he eliabili y o he scien ific p ocess wi h a ocus on biases inhe en in he p oduc ion o science. Poin ing o bo h misaligned s uc u al incen i es and he cogni i e endencies o esea che s (5–7), his bias- ocused pe spec i e a gues ha sys ema ic dis o ions o he esea ch p ocess push he published li e a u e away om u h seeking and accu a e obse a ion. This hen educes he p obabili y ha a ca e ully execu ed eplica ion will a i e a he same findings. He e, we a gue ha some oo s o eliabili y issues in science un deepe han sys em- a ically dis o ed esea ch p ac ices. We p opose ha o be e unde s and why esea ch is o en non eplicable o lacking in e esea che eliabili y, we need o accoun o idiosyn- c a ic a ia ion inhe en in he scien ific p ocess. Ou main a gumen is ha a iabili y in esea ch ou comes be ween esea che s can occu e en unde igid adhe ence o he scien ific me hod, high e hical s anda ds, and s a e-o - he-a app oaches o maximizing ep oducibili y. As we epo below, e en well-meaning scien is s p o ided wi h iden ical da a and eed om p essu es o dis o esul s may no eliably con e ge in hei findings because o he complexi y and ambigui y inhe en o he p ocess o scien ificanalysis. Va iabili y in Resea ch Ou comes The scien ific p ocess con on s esea che s wi h a mul iplici y o seemingly mino , ye non i ial, decision poin s, each o which may in oduce a iabili y in esea ch ou - comes. An impo an bu unde app ecia ed ac is ha his e en holds o wha is o en seen as he mos objec i e s ep in he esea ch p ocess: wo king wi h he da a a e i has come in. Resea che s can ake li e ally millions o di e en pa hs in w angling, ana- lyzing, p esen ing, and in e p e ing hei da a. The numbe o choices g ows exponen- ially wi h he numbe o cases and a iables included (8–10). A bias- ocused pe spec i e implici ly assumes ha educing “pe e se”incen i es o gene a e su p ising and sleek esul s would ins ead lead esea che s o gene a e alid Significance Will di e en esea che s con e ge on simila findings when analyzing he same da a? Se en y- h ee independen esea ch eams used iden ical c oss- coun y su ey da a o es a p ominen social science hypo hesis: ha mo e immig a ion will educe public suppo o go e nmen p o ision o social policies. Ins ead o con e gence, eams’ esul s a ied g ea ly, anging om la ge nega i e o la ge posi i e e ec s o immig a ion on social policy suppo . The choices made by he esea ch eams in designing hei s a is ical es s explain e y li le o his a ia ion; a hidden uni e se o unce ain y emains. Conside ing his a ia ion, scien is s, especially hose wo king wi h he complexi ies o human socie ies and beha io , should exe cise humili y and s i e o be e accoun o he unce ain y in hei wo k. The au ho s decla e no compe ing in e es . This a icle is a PNAS Di ec Submission. Copy igh © 2022 he Au ho (s). Published by PNAS. This open access a icle is dis ibu ed unde C ea i e Commons A ibu ion-NonComme cial-NoDe i a i es License 4.0 (CC BY). See online o ela ed con en such as Commen a ies. 1 To whom co espondence may be add essed. Email: [email p o ec ed]. 2 N.B., E.M.R., and A.W. we e he P incipal In es iga o s, equally esponsible o concep ualiza ion and da a collec ion. P ima y me a-analysis o da a analys s’ esul s and p epa a ion o me ada a o public consump ion p e o med by N.B., wi h assis ance om H.H.V.N. This a icle con ains suppo ing in o ma ion online a h p://www.pnas.o g/lookup/suppl/doi:10.1073/pnas. 2203150119/-/DCSupplemen al. Published Oc obe 28, 2022. PNAS 2022 Vol. 119 No. 44 e2203150119 h ps://doi.o g/10.1073/pnas.2203150119 1o 8 RESEARCH ARTICLE | SOCIAL SCIENCES OPEN ACCESS conclusions. This may be oo op imis ic. While emo ing hese ba ie s leads esea che s away om sys ema ically aking in alid o biased analy ical pa hs (8–11), his alone does no gua an ee alidi y and eliabili y. Fo easons less ne a ious, esea che s can dispe se in di e en di ec ions in wha Gelman and Loken call a “ga den o o king pa hs”in analy ical decision-making (8). The e a e wo p ima y explana ions o a ia ion in o king decisions. The compe ency hypo hesis posi s ha esea che s may make di e en analy ical decisions because o a ying le els o s a is ical and subjec expe ise ha lead o di e en judg- men s as o wha cons i u es he “ideal”analysis in a gi en esea ch si ua ion. The confi ma ion bias hypo hesis holds ha esea che s may make eliably di e en analy ical choices because o di e ences in p eexis ing belie s and a i udes, which may lead o jus ifica ion o analy ical app oaches a o ing ce - ain ou comes pos hoc. Howe e , many o he co e o idio- sync a ic influences, la ge and small, may also lead o un eliable and unexplainable a ia ion in analy ical decision pa hways (10). Some imes e en he inies o hese di e ences may add up and in e ac o p oduce widely a ying ou comes. The e is g owing awa eness o he dependence o findings on s a is ical modeling decisions and he impo ance o analy ical obus ness (9, 11–13). Howe e , only ecen ly scien is s began o assess whe he esea che a iabili y a ec s scien ific ou - comes in eali y, some imes employing “many analys s” app oaches whe e many esea che s o eams independen ly es he same hypo hesis wi h he same da a. The fi s such s udy showed ha when 29 esea che s es ed i socce e e ees we e biased owa d da ke -skin playe s using he same da a, hey epo ed 29 unique model specifica ions, wi h empi ical esul s anging om modes ly nega i e o s ongly posi i e (14). Thus a , mos many-analys s s udies ha e been small in scale o ocused on na ow, field-specific analysis me hods (15, 16). Recen s udies by Bo inik-Neze e al. (17) and Menk eld e al. (18) we e la ge , in ol ing 65 and 164 eams, espec i ely. C i ically, despi e hei size, hese s udies also ound la ge in e - esea che a ia ion in epo ed esul s. They also made fi s s eps a explaining he amoun o a ia ion in epo ed esul s using a small se o a iables, such as he compu a ional ep o- ducibili y and pee a ings o submi ed analyses o he s a is ical so wa e ha analys s used. Ye , hey had li le success in explain- ing he a ia ion in esul s o he dis ance o esul s om he o e all mean o he esul s (i.e., e o ). We expanded on hese explana o y a emp s by obse ing e e y s ep o each indepen- den esea ch eam’swo kflow, expec ing ha such close obse a- ion should explain a mo e a iance in esea ch ou comes. Mo eo e , when coupled wi h measu es o ele an analy ical compe encies and subs an i e belie s o he analys s as explana- o y a iables, we expec ed o a i e a a deepe unde s anding o he esul s and c ucially, which key decisions d i e hem. Me hods The p incipal in es iga o s (PIs) coo dina ed a g oup o 161 esea che s in 73 eams o comple e he same ask o independen ly es ing a hypo hesis cen al o an “ex ensi e body o schola ship”(19) in he social sciences: ha immig a ion educes suppo o social policies among he public.* Ou en i e ep oducible wo kflow o his s udy is a ailable online in ou P ojec Reposi o y. † The ask gi en o he pa icipan s is ypical o esea ch on human socie ies, in which he cen al concep s and quan i ies o in e es a e open o b oad and complex in e p e a ions (20, 21). In classic poli ical economy esea ch, o exam- ple, Albe o Alesina and Edwa d Glaese (22, 23) hypo hesized ha di e ences in No h Ame ican and Eu opean social secu i y sys ems a e a esul o immig a ion-gene a ed e hnic di e si y o a lack he eo . Mo e ecen ly, o he schola s see immig a ion and e ugee c ises as ca alys s o e enchmen o social secu i y sys ems in Wes e n Eu ope and ac oss he globe. Pu simply, his hypo hesis was gi en o pa icipa ing eams because i is influen ial, long s and- ing, and ypical o con empo a y social esea ch in poli ical science, sociology, economics, geog aphy, and beyond (24–29). We ec ui ed pa icipan s by ci cula ing a call ac oss academic ne wo ks, social media, and o ficial communica ion channels o academic associa ions ac oss social science disciplines (SI Appendix,Resea ch Design). Al hough 106 eams exp essed in e es in pa icipa ion, we coun ou ini ial sample as 88 ha comple ed he p es udy ques ionnai e. In he end, 73 o hose 88 eams (a o al o 161 esea che s wi h an a e age o 2.24 esea che s pe eam) comple ed he s udy. O hese, 46% had a backg ound in sociology; 25% had a backg ound in poli ical science; and he es had economics, com- munica ion, in e disciplina y, o me hods- ocused deg ee backg ounds. Eigh y- h ee pe cen had expe ience eaching cou ses on da a analysis, and 70% had published a leas one a icle o chap e on he subs an i e opic o he s udy o he usage o a ele an me hod (SI Appendix,III. Pa icipan Su ey Codebook has mo e pa icipan de ails). The PIs p o ided eams wi h da a om he In e na ional Social Su ey P o- g am (ISSP), a long- unning la ge-scale c oss-na ionally compa a i e su ey o poli ical and economic a i udes used in o e 10,000 published s udies. ‡ The ISSP includes a six-ques ion module on he ole o go e nmen in p o iding di e en social policies, such as old-age, labo ma ke , and heal h ca e p o i- sions. This six-ques ion module is also he sou ce o he da a used by Da id B ady and Ryan Finnigan (19) in one o he mos ci ed in es iga ions o he subs an i e hypo hesis pa icipan s we e ins uc ed o es . The PIs also p o- ided yea ly indica o da a o coun ies on immig an ’s ock’as a pe cen age o he popula ion and on ’flow’as a ne change in s ock, aken om he Wo ld Bank, he Uni ed Na ions, and he O ganiza ion o Economic Co-Ope a ion and De elopmen . Rele an ISSP and immig a ion da a we e a ailable o 31 mos ly ich and some middle-income coun ies. The e we e up o fi e su - ey wa es om 1985, 1990, 1996, 2006, and 2016. All p o ided da a come om publicly a ailable sou ces. To emo e po en ially biasing incen i es, all esea che s om eams ha com- ple ed he s udy we e ensu ed coau ho ship on he final pape ega dless o hei esul s. Because he “pa icipan s” hemsel es we e esea che s and all asks assigned o hem we e s anda d esea ch p ac ices heo e ically wo hy o coau ho ship, ins i u ional e iew p io o conduc ing his s udy was no neces- sa y. The esea che s pa icipa ed in su eys o measu e expe ise and s udy- ele an belie s and a i udes be o e and du ing he esea ch p ocess. Mo eo e , hey ook pa in online delibe a ions be o e (a andomized hal o eams) and a e hey had un hei main analyses (all eams) (SI Appendix, III. Pa icipan Su ey Codebook). To amilia ize pa icipa ing esea che s wi h he da a, hei fi s ask was o nume ically ep oduce esul s om he B ady and Finnigan (19) s udy on a subse o he ISSP da a. This was ollowed by a eques ha he eams de elop hei own ideal models o es ing he same hypo hesis using po en ially all o he p o ided da a, bu ha hey submi hei analysis plan p io o unning he models. To enhance ecological alidi y, we allowed he eams o include addi ional da a sou ces o measu ing inde- penden a iables. Each eam was hen ins uc ed o un hei model(s) and epo dependen a iable–s anda dized e ec es ima es equal o he change in policy p e e ences (in SD uni s) p edic ed by a one-poin change in he espec i e independen immig a ion a iable. We also asked hem o d aw one o h ee subjec i e conclusions: whe he hei esul s o e ed e i- dence ha suppo ed he hypo hesis ha immig a ion educes suppo o social policies among he public, whe he hei esul s o e ed e idence ha ejec ed he hypo hesis, o ins ead, whe he hey belie ed he hypo hesis was no es able gi en hese da a. O he 73 eams, 1 conduc ed p elimina y measu emen scaling es s, con- cluded ha he hypo hesis could no be eliably es ed, and hus, did no design o ca y ou any u he es s. This le 72 eams submi ing a o al o 1,261 *SI Appendix, Figs. S1 and S2 and Tables S1 and S2 ha e he ime line and mo e pa ici- pan de ails. † I is a ailable a h ps://gi hub.com/nb eznau/CRI. ‡ In o ma ion is a ailable a h ps://issp.o g/abou -issp/. 2o 8 h ps://doi.o g/10.1073/pnas.2203150119 pnas.o g models. One eam’s p e egis e ed models ailed o con e ge and hus, had no nume ical esul s. This le a o al o 71 eams wi h nume ical esul s om 1,253 models. In hei subjec i e conclusions, 16 eams de e mined ha he wo di e - en measu es o immig a ion should be conside ed independen hypo hesis es s and he e o e, submi ed di e en conclusions o each. This changed he p ima y uni o analysis o subjec i e conclusions om 73 eam conclusions o 89 eam-le el conclusions. The e was an a e age o 17.5 models pe eam among he 71 eams submi - ing nume ical esul s ( anging om 1 o 124 models pe eam). Mos eams sub- mi ed a leas 12 models because hey used each o he six ISSP ques ions as a single dependen ou come wice in hei s a is ical models, once o each e sion o he immig a ion measu e (s ock and flow). Se e al eams submi ed 18 models because hey an an addi ional 6 models wi h bo h immig a ion a iables included. The eams o en adjus ed o he nes ed na u e o he da a, accoun ing o a iance a he indi idual, coun y, yea , and/o coun y-yea le els. Some made no such hie a chical adjus men s wi h mul ile el models, and o he s used clus e ing o he SEs a he coun y, wa e, and/o coun y-wa e le els. Some used dummy in e ac ions o example wo ld egion indica o s (such as Eas e n Eu ope) o poli ical pa y p e e ences wi h immig a ion a iables, leading o nonlinea p e- dic ed alues. O he s used al e na i e es ima o s based on maximum likelihood o Bayesian es ima ion as opposed o o dina y leas squa es (SI Appendix,Table S3 shows he mos common model decisions). In all, esea che s’modeling deci- sions eflec ed he di e si y o simila , bu echnically dis inc , me hodological app oaches cu en ly used in con empo a y esea ch. Each eam’s code was checked and hen anonymized o public sha ing by he PIs. Some eams ailed o epo a s anda dized es ima e. Also, di e en scal- ing o he wo independen immig a ion a iables mean ha esul s we e no always dis ibu ionally compa able. The e o e, we s anda dized he eams’ esul s o each coe ficien o s ock and flow o immig a ion pos hoc. We also ans- o med he eams’ esul s in o a e age ma ginal e ec s (AMEs), which a e he s anda dized a e age e ec s o a one-uni change in he espec i e independen (immig a ion) a iable on he espec i e dependen (policy suppo ) a iable, whe e his a e age is based on p edic ions o each obse a ion in he da ase . The ad an age o using AMEs is ha hey allow o a single ma ginal es ima e in he p esence o nonlinea i ies and p esen p edic ed p obabili ies ha eflec he eali y o he da a sample a he han he mean o each independen a iable (Fig. 1 shows esul s). A e submi ing hei own esul s bu p io o seeing he o he eams’ esul s, each pa icipan was andomly gi en a ough desc ip ion o he models employed by ou o fi e o he eams and asked o ank hem on hei quali y o es ing he hypo hesis. Wi h six o se en ankings pe eam, he PIs cons uc ed model ankings (SI Appendix,Model Ranking). A any poin a e hey submi ed hei esul s, including a e esul s o he o he eams we e e ealed, he eams could change hei p e e ed mod- els and esubmi hei esul s and conclusion. No eam olun a ily op ed o do his. Howe e , some eams’ esul s and conclusions changed a e hey we e in o med ha he PIs we e unable o ep oduce hei findings due o coding mis akes o a misma ch be ween hei in ended models and hose ha appea ed in he code. Nex , we examined all 1,261 models and iden ified 166 dis inc esea ch design decisions associa ed wi h hose models. “Decision” e e s o any aspec in he design o a s a is ical model: o example, he measu emen s a egy, es ima o , hie a chical s uc u e, choice o independen a iables, and po en ial subse ing o he da a (SI Appendix, Table S12). Fo simplici y, decision also e e s o a iables measu ing eam cha ac e is ics, such as so wa e used, o e - all amilia i y wi h he subjec o me hods, and p eexis ing belie s as measu ed in ou pa icipan su ey (SI Appendix, Table S1). O he 166 decisions, 107 we e aken by a leas h ee eams. We used hese 107 as a iables ha migh s a is ically explain he a ia ion in he esul s and conclusions because he o he 59 we e unique o one o wo eams and would hus impede s a is ical iden ifica ion. In o he wo ds, uniquely iden i ying one o wo eams’ esul s ia he a iance in a single independen a iable in he eg ession would in e - up he pa simonious es ima ion o esidual, unexplained a iance calcula ed in he le el 2 equa ion. A dissimila i y ma ix e ealed ha no wo models o 1,261 we e 100% iden ical. To explo e he sou ces o a iance in esul s, we eg essed he nume ical poin es ima es and subjec i e conclusions on all di e en combina ions and in e ac ions o he 107 decisions. We used mul ile el eg ession models, allowing us o accoun o models nes ed in eams and o explain o al, wi hin- eam, and be ween- eam a iance in nume ical esul s. Fo subjec i e conclu- sions, we used mul inomial logis ic eg essions p edic ing eams’conclusions o 1) suppo o 2) ejec he a ge hypo hesis, o 3) ega d i as no es able. Ou analyses p oceeded in se e al s ages. A each s age, decision a iables and hei in e ac ions we e es ed, and only e ms ha explained he mos a iance using he leas deg ees o eedom (o de iance in he case o subjec i e conclusions) we e ca ied o he nex phase. Explo ing he a iance in esul s uns he isk o o e fi ing. I is s a is ically inapp op ia e o use 107 a iables when he e a e 87 eam- es cases ( om 71 eams wi h nume ical esul s). The e o e, we en e ed he a iables in g oups and only kep a iables om each g oup ha showed an inc ease in he explained a iance wi hou a loss in fi as measu ed by Akaike’sIn o ma ionC i- e ion (AIC) and log likelihood. This s a ed wi h “design”decisions, including which o he six su ey ques ions he eams used as he dependen a iable in a gi en model and dummies indica ing he wo andom expe imen al ea men s (which we e included in he s udy bu a e un ela ed o he main modeling ask assigned o he eams). The nex s age added “measu emen ”decisions, includ- ing which immig a ion measu e he eam used in a gi en model and how he dependen a iable was measu ed (dicho omous, o dinal, mul inomial, o con- inuous). The ollowing s ages added “da a and sample”and “model design” decisions and concluded wi h he addi ion o “ esea che aspec s”(P ojec Repos- i o y, 04_CRI_Main_Analyses). We also e an he phase-wise analysis sepa a ely o each o he six su ey ques ions used as dependen a iables by he eams (P ojec Reposi o y, 07_CRI_DVspecific_Analyses) (SI Appendix, Tables S4 and S9–S11). The exac a iables in ou final analysis plus a ious models leading up o hem a e ound in SI Appendix, Tables S5 and S7. Fig. 2 epo s he explained a iance om ou p e e ed model m13. To check he obus ness o ou phase-wise s a egy, we used an algo i hm o analyze all possible a iable combina ions om ou p ima y a iables o in e - es — hose ha showed any capaci y o explain a iance in he main analyses (P ojec Reposi o y, 06_CRI_Mul i e se). This led us o a sligh ly di e en ideal model hanm13(Au o_1inSI Appendix, Table S10). Al hough his al e na i e model had he bes AIC and could explain sligh ly mo e model-le el a iance, i could no explain as much o al a iance. We hen combined all a iables om ou main model (m13) and he algo i hm-de i ed model (Au o_1) o gene a e a new model (Au o_1_m13). Al hough his new model explained mo e a iance and had lowe AIC, we we e ca e ul no o o e fi because i had 22 a iables, whe eas m13 and Au o_1 had 18 and 15, espec i ely. Nex , based on PNAS pee e iew eedback, we gene a ed a lis o e e y pos- sible in e ac ion pai o all 107 a iables. O hese 5,565 in e ac ions, 2,637 ha e nonze o a iance and hus, we e usable in a eg ession wi hou au oma i- cally being d opped. Including 8 o mo e in e ac ion a iables plus hei main e ec s (i.e., 24 o mo e a iables, many ha we e c oss-le el in e ac ions) led o Fig. 1. B oad a ia ion in he findings om 73 eams es ing he same hypo hesis wi h he same da a. The dis ibu ion o es ima ed AMEs ac oss all con e ged models (n=1,253) includes esul s ha a e nega i e (yellow; in he di ec ion p edic ed by he gi en hypo hesis he eams we e es ing), no di e en om ze o (g ay), o posi i e (blue) using a 95% CI. AME a e xy s anda dized. The yaxis con ains wo scaling b eaks a ±0.05. Numbe s inside ci cles ep esen he pe cen ages o he dis ibu ion o each ou - come in e sely weigh ed by he numbe o models pe eam. PNAS 2022 Vol. 119 No. 44 e2203150119 h ps://doi.o g/10.1073/pnas.2203150119 3o 8 con e gence o o e iden ifica ion p oblems. E en wi h 87 le el 2 cases, his lea es us wi h oughly 4 cases pe a iable. Resea ch e iewing di e en simula- ion s udies on le el 2 case numbe s sugges s ha 10 cases pe a iable a e a ba e minimum, and we should ha e close o 50 ideally (30). The e o e, we se - led on 7 in e ac ed a iables as ou absolu e maximum (which co esponds o 21 a iables, including wo main e ec s o each in e ac ion). We hen andomly sampled 1,000 se s o 7 a iables om he lis o all and le he algo i hm un e e y subcombina ion o hese, which led o jus o e 1 million models. We hen ook he 2 models wi h he lowes AIC sco e om each o he 1,000 i e a- ions and ex ac ed all a iables om hose models. The e we e 19 unique a ia- bles among he 2,000 bes -fi ing models in o al, which we hen analyzed using he same “ andom-se en”sampling me hod. The bes -fi ing models om his second i e a ion le us wi h ou in e ac ion a iables as candida es o explain mo e a iance while a oiding sac ificing he simplici y o he model and o e fi - ing (as indica ed by model AIC). We added each o hese a iables sepa a ely o ou ini ial algo i hm-gene a ed esul s. None o hese models could explain mo e a iance in esea ch ou comes han m13 o Au o_1_m13 (SI Appendix, Table S10, Au o_2 o Au o_5). Main Resul s Fig. 1 isualizes he subs an ial a ia ion o nume ical esul s epo ed by 71 esea che eams ha analyzed he same da a. Resul s a e di use. Li le mo e han hal he epo ed es ima es we e s a is ically no significan ly di e en om ze o a 95% CI, while a qua e we e significan ly di e en and nega i e, and 16.9% we e s a is ically significan and posi i e. We obse e he same pa e n o di e gen esea ch ou comes when we use he eams’subjec i e conclusions a he han hei s a is ical esul s. O e all, 13.5% (12 o 89) o he eam conclu- sions we e ha he hypo hesis was no es able gi en hese da a, 60.7% (54 o 89) we e ha he hypo hesis should be ejec ed, and 28.5% (23 o 89) we e ha he hypo hesis was suppo ed (SI Appendix, Figs. S5, S9, and S10). § We find ha compe encies and po en ial confi ma ion biases do no explain he b oad a ia ion in ou comes; esea che cha ac e is ics show a s a is ically significan associa ion wi h nei he s a is ical esul s no subs an i e conclusions (Fig. 3). Hence, he da a a e no consis en wi h he expec a ion ha ou come a iabili y simply eflec s a lack o knowledge among some pa icipan s o p eexis ing p e e ences o pa icula esul s. In p inciple, a ia ion in ou comes mus eflec p io decisions o he esea che s. Ye , Fig. 2 shows ha he 107 iden ified deci- sion poin s explain li le o he a ia ion. The majo componen s o he iden ified esea che decisions explain less han a qua e o he a ia ion in ou measu es o esea ch ou comes. Mos a iance also emains unexplained a e accoun ing o esea che cha ac e - is ics o assignmen o a small expe imen (no epo ed in his s udy) (“assigned condi ions”in Fig. 2). Looking a o al a iance in he nume ical esul s ( op ba ), iden ified componen s o he esea ch design explain 2.6% (g een segmen ), and esea che cha ac e is ics only accoun o a maximum o 1.2% o he a i- ance ( iole segmen ). In o he wo ds, 95.2% o he o al a iance in esul s is le unexplained, sugges ing ha massi e a ia ion in epo ed esul s o igina ed om idiosync a ic decisions in he da a analysis p ocess. The sha e o explained a iance is somewha highe when looking a be ween- eam esul s (second ba ), bu s ill, 82.4% emained unexplained. Va iance con inues o emain mos ly unexplained when mo ing away om he nume ical esul s and conside ing esea che s’subs an i e conclusions (bo om ba ; 80.1% unexplained). I is no ewo hy ha e en he pe cen age o es esul s pe eam ha s a is ically suppo hei conclu- sions explains only abou a hi d o he de iance in conclusions (salmon-colo ed segmen in he bo om ba ), which poin s a he a ia ion in how di e en esea che s in e p e he same se o nume ical esul s. O e all, he complexi y o he da a- analy ic p ocess leads o a ia ion ha canno be easily explained, e en wi h a close look a esea che cha ac e is ics and esea che decisions. Fig. 2. Va iance in s a is ical esul s and subs an i e conclusions be ween and wi hin eams is mos ly unexplained by condi ions, esea ch design, and esea che cha ac e is ics. Decomposi ion o nume ical a iance aken om gene alized linea mul ile el eg ession models’AMEs ( he op h ee ows). Explained de iance aken om mul inomial logis ic eg essions using he subs an i e conclusions abou he a ge hypo hesis as he ou come submi ed by he esea ch eams (bo om ow). We used in o med s epwise addi ion and emo al o p edic o s o iden i y which specifica ions could explain he mos nume ic a iance (SI Appendix,TableS6) and o he s ha could explain he mos subjec i e conclusion de iance (SI Appendix,TableS7) while sac ific- ing he ewes deg ees o eedom and main aining he highes le el o model fi based on log likelihood and AIC. We also used algo i hms o es a iable combina ions, bu hese could no explain mo e meaning ul a ia ion (Me hods). Assigned condi ions we e he di ision o pa icipan s in o wo di e en ask g oups and wo di e en delibe a ion g oups du ing he p epa a o y phase. Iden ified esea che decisions a e he 107 common decisions aken in da a p epa a ion and s a is ical modeling ac oss eams and hei models. Resea che cha ac e is ics we e iden ified h ough a su ey o pa icipan s and mul ii em scaling using ac o analysis (SI Appendix,Fig.S3). The eade will find many o he de ails in SI Appendix. § This is a eminde ha 16 eams epo ed wo di e ing conclusions based on hei in e - p e a ion o di e en model speci ica ions, causing he N o jump om 72 eams o 89 eam-conclusion uni s o analysis (Me hods). 4o 8 h ps://doi.o g/10.1073/pnas.2203150119 pnas.o g Finally, we ollowed p e ious many-analys s esea ch (31) by c ea ing a benchma k o he abo e esul s ia a mul i e se sim- ula ion o possible model specifica ions. Using he same app oach, we ound ha 23 decisions could explain jus o e 16% o he a iance in nume ical ou comes among 2,304 sim- ula ed models (SI Appendix, Table S8). In con as o ou eco- logical esea ch se ing obse ing ac ual esea che beha io s, we all a sho o his 16% simula ed o al explained a iance by almos 12 pe cen age poin s. E en wi h he use o an algo i hm o sample all possible a iable combina ions, we emain o e 11 pe cen age poin s sho . Discussion Resul s om ou con olled esea ch design in a la ge-scale c owdsou ced esea ch e o in ol ing 73 eams demons a e ha analyzing he same hypo hesis wi h he same da a can lead o subs an ial di e ences in s a is ical es ima es and subs an i e conclusions. In ac , no wo eams a i ed a he same se o nume ical esul s o ook he same majo decisions du ing da a analysis. Ou finding o ou come a iabili y echoes hose o ecen s udies in ol ing many analys s unde aken ac oss scien- ific disciplines. The s udy epo ed he e di e s om hese p e- ious e o s because i a emp ed o ca alog e e y decision in he esea ch p ocess wi hin each eam and use hose decisions and p edic i e modeling o explain why he e is so much ou - come a iabili y. Despi e his highly g anula decomposi ion o he analy ical p ocess, we could only explain less han 2.6% o he o al a iance in nume ical ou comes. We also es ed i expe ise, belie s, and a i udes obse ed among he eams biased esul s, bu hey explained li le. E en highly skilled sci- en is s mo i a ed o come o accu a e esul s a ied emen- dously in wha hey ound when p o ided wi h he same da a and hypo hesis o es . The s anda d p esen a ion and con- sump ion o scien ific esul s did no disclose he o ali y o esea ch decisions in he esea ch p ocess. Ou conclusion is ha we ha e apped in o a hidden uni e se o idiosync a ic esea che a iabili y. This finding was a o ded by a many-analys s design as an app oach o scien ific inqui y. Some schola s ha e p oposed mul i e se analysis o simula e analy ic decisions ac oss esea che s (31), a me hod o p o ide a many-analys s se o ou comes wi hou he massi e coo dina ion and human capi- al commi men o he wise necessa y o such a s udy. The d awback o a simula ion app oach is ha i is cons uc ed based on a single da a analysis pipeline om one esea ch eam and may no eflec he complex eali y o di e en esea ch p ocesses ca ied ou by di e en eams in di e en con ex s. This s udy obse ed esea che s in a con olled ye ecological wo k en i onmen . In doing so, i e ealed he hid- den uni e se o consequen ial decisions and con ex ual ac o s ha a y ac oss esea che s and ha a simula ion, hus a , canno cap u e. Implica ions. Resea che s mus make analy ical decisions so minu e ha hey o en do no e en egis e as decisions. Ins ead, hey go unno iced as nondelibe a e ac ions ollowing os ensibly s anda d ope a ing p ocedu es. Ou s udy shows ha , when aken as a whole, hese hund eds o decisions combine o be a om i ial. Howe e , his unde s anding only a ises om he uniqueness o each o he 1,253 models analyzed he ein. Ou findings sugges eliabili y ac oss esea che s may emain low e en when hei accu acy mo i a- ion is high and biasing incen i es a e emo ed. Highe le els o me hodological expe ise, ano he equen ly sugges ed emedy, did no lead o lowe a iance ei he . Hence, we a e Fig. 3. Resea che cha ac e is ics do no explain ou come a iance be ween eams o wi hin eams. The dis ibu ion o eam a e age o AMEs (Le ) and wi hin- eam a iance in AMEs (Righ ) ac oss esea che s g ouped acco ding o mean spli s (“lowe ”and “highe ”) on me hodological and opic expe ise (po en ial compe encies bias) and on p io a i udes owa d immig a ion and belie s abou whe he he hypo hesis is ue (po en ial confi ma ion bias). Log a iance was shi ed so ha he minimum log alue equals ze o. Teams submi ing only one model assigned a a iance o ze o. Pea son co ela ions along wi h a P alue (“R”) a e calcula ed using con inuous sco es o each esea che cha ac e is ic a iable. PNAS 2022 Vol. 119 No. 44 e2203150119 h ps://doi.o g/10.1073/pnas.2203150119 5o 8 le o belie e ha idiosync a ic unce ain y is a undamen al ea u e o he scien ific p ocess ha is no easily explained by ypically obse ed esea che cha ac e is ics o analy i- cal decisions. These findings add a pe spec i e o he me ascience con- e sa ion, emphasizing unce ain y in addi ion o bias. The conclusion wa an ed om much o he me ascience wo k ca ied ou in he wake o he “ eplica ion c isis”in psychol- ogy and o he fields has been ha published esea ch findings a e mo e biased han p e iously hough . The conclusion wa an ed om his and o he simila s udies is ha pub- lished esea ch findings a e also mo e unce ain han p e i- ously hough . As esea che s, we bea he esponsibili y o accu a ely desc ibe and explain he wo ld as i is bu also, o communica e he unce ain y associa ed wi h ou knowledge claims. Al hough he academic sys em p i ileges inno a ion o e epli- ca ion, p o iding a no el answe o a ques ion is jus as essen- ial as in o ming abou how much us we can place in ha answe . Ou s udy has shown ha o ully assess and add ess unce ain y, eplica ions a e aluable bu insu ficien . Only la ge numbe s o analyses may show whe he in a specific field, independen esea che s eliably a i e a simila conclu- sions, hus enhancing o unde mining ou confidence in a gi en knowledge claim. Specifically, we belie e ha se ious acknowledgmen o idiosync a ic a ia ion in esea ch findings has a leas ou implica ions o imp o ing he p esen a ion and in e p e a ion o empi ical e idence. Fi s , con empla ing ha esul s migh a y g ea ly i a gi en s udy had been conduc ed by a di e en se o esea che s o e en he same esea che s a a di e en ime, calls o epis emic humili y when d awing conclusions based on seemingly objec i e quan i a i e p ocedu es. Second, he findings emind us o ca e ully documen e e y hing because, in combina ion, e en he mos seemingly minu e deci- sions could d i e esul s in di e en di ec ions; and only awa e- ness o hese minu ae could lead o p oduc i e he oe ical discussions o empi ical es s o hei legi imacy. Thi d, coun- e ing a de ea is iew o he scien ific en e p ise, his s udy helps us app ecia e he knowledge accumula ed in a eas whe e scien is s do con e ge on expe consensus—such as he human impac on he global clima e o a no able inc ease in poli ical pola iza ion in he Uni ed S a es o e he pas decades. Fou h, ou s udy sugges s ha i we eed scien is s om bias caused by he “pe e se incen i es”inhe en in he ins i u ions o sci- ence, hei own p eexis ing biases o belie s migh no ma e as much o he ou comes hey gene a e as some may ea . In ai - ness, he eams indica ed wha models hey in ended o un in ad ance. Al hough hey we e ee o upda e and change hese a any ime, we can assume ha his may ha e educed po en ial confi ma ion bias. Limi a ions and Ou look. Ou s udy has limi a ions ha wa - an discussion. Fi s , we do no know he gene alizabili y o ou s udy o di e en opics, disciplines, o e en da ase s. A majo segmen o social science wo ks wi h su ey da a, and ou esul s eflec his ype o esea ch. In expe imen al esea ch, he da a-gene a ing model is o en clea e o in ol es ewe decisions. Mo eo e , in social esea ch, he e a e no New onian laws o defini e quan um s a is ical likelihoods o wo k wi h, sugges ing ha ou case migh o e es ima e a i- abili y compa ed wi h he na u al sciences. On he o he hand, unc ional magne ic esonance imaging ( MRI), gene, and space elescope da a, o example, a e a mo e complex han wha we ga he in social su eys. The complexi y o da a analysis pipelines is co espondingly g ea e in hese fields, bu i is possible ha ha ing mo e analy ical decisions allows us o accoun o mo e o he a ia ion in esea ch ou comes. We belie e ha he numbe o decision poin s in da a analysis and he ex en o which esea che s unde s and he da a- gene a ing p ocess may de e mine he deg ee o ou come a i- a ion in a field, bu i emains an open ques ion i and how design decisions play a smalle o la ge ole ac oss fields. Second, al hough we hoped o o e deepe insigh s on he subs an i e hypo hesis unde obse a ion, we did no ob ain e idence ha mo es conclusions in any di ec ion. These lessons combined wi h he ac ha a subs an ial po ion o pa ici- pan s conside ed he hypo hesis no es able wi h hese da a o e a po en ial explana ion o why his is such a con es ed hypo hesis in he social sciences (19, 24, 32). Looking o wa d, we ake he ac ha 13.5% o pa icipa - ing analys s claimed ha he a ge hypo hesis is “no es able” wi h he p o ided da a as a powe ul eminde o he impo - ance o design app op ia eness and clea specifica ion o hypo heses. This implica es cla i y in he meaning o a conclu- sion. In ou s udy, “suppo ”o he hypo hesis gene ally mean ejec ion o he null, whe eas “ ejec ”mean consis ency wi h he null o inconclusi e esul s. Teams we e le o hei own de ices in deciding wha cons i u ed suppo o , e idence agains , o non es abili y o he a ge hypo hesis, and his alone in oduced a deg ee o inde e minacy in o he esea ch p ocess. O e all, hese obse a ions call o mo e a en ion o concep- ual, causal, and heo e ical cla i y in he social sciences as well as o he ga he ing o new da a when esul s no longe appea o mo e a subs an i e a ea o wa d (20, 21). They also sugges ha i we wan o eap epis emic benefi s om he p esen mo e owa d open esea ch p ac ices, we mus make mo e con- scious e o s o complemen me hodological anspa ency wi h heo e ical cla i y. Finally, we no e ha he conclusions o his s udy we e hemsel es de i ed om my iad seemingly mino (me a-)ana- ly ical decisions, jus like hose we obse ed among ou analys s. We, he e o e, encou age eade s o sc u inize ou ana- ly ical p ocess by aking ad an age o SI Appendix, he ep o- duc ion files, and he web-based in e ac i e app ha allow o easy explo a ion o all da a unde lying his s udy. ¶ Da a, Ma e ials, and So wa e A ailabili y. Da aandcodeha ebeen deposi ed in Gi Hub (h ps://gi hub.com/nb eznau/CRI) (33), and Ha a d Da a- e se (h ps://doi.o g/10.7910/DVN/UUP8CX) (34). ACKNOWLEDGMENTS. We hank he Mannheim Cen e o Eu opean Social Resea ch a he Uni e si y o Mannheim (Mannheim, Ge many) o p o iding ex a ins i u ional suppo o his la ge-scale p ojec and Joachim Gassen a he Humbold Uni e si y (Be lin, Ge many) o inspi a- ion and eedback o ou in e ac i e app. Addi ionally help ul we e commen s p o ided a he S an o d Me a Resea ch Inno a ion Cen e a S an o d (METRICS) In e na ional Fo um, he Ludwig Maximilian Uni e si y o Munich Sociology Resea ch Colloquium, and he Technical Uni e si y Chemni z “Open Science”Con e ence and in pe sonal communica ions wi h S e e Velsko. A po ion o he la e phases o coding and shiny app de elopmen was suppo ed by a F eies Wissen (“Open Knowledge”)Fel- lowship o Wikimedia. The iews exp essed he ein a e hose o he au ho s and do no eflec he posi ion o he US Mili a y Academy, he Depa men o he A my, o he Depa men o De ense. ¶ In o ma ion is a ailable a h ps://na e-b eznau.shinyapps.io/shiny/ and h ps://gi hub. com/nb eznau/CRI. 6o 8 h ps://doi.o g/10.1073/pnas.2203150119 pnas.o g Na e B eznau a,1,2 , Eike Ma k Rinke b,2 ,Alexande Wu ke c,l,2 ,HungH.V. Nguyen a,d ,MunaAdem e , Jule Ad iaans , Amalia Al a ez-Benjumea g ,Hen ikK. Ande sen h , Daniel Aue c , Fla io Aze edo j ,OkeBahnsen i ,Da eBalze k , Ge i Baue oooo ,PaulC.Baue c , Ma kus Baumann m,dd , Sha on Bau e n , Ve ena Benoi l,o , Julian Be naue c ,Ca lBe ning pppp ,AnnaBe hold o ,FelixS.Be hke p , Thomas Biege q , Ka ha ina Blinzle , Johannes N. Blumenbe g , Licia Bobzien s , And ea Bohman ,ThijsBol x,uuuu ,AmieBos ic u ,ZuzannaB zozowska ,w ,Ka ha ina Bu gdo i , Kaspa Bu ge x,y, , Ka h in B. Busch z , Juan Ca los-Cas illo aa,mmmm , Na han Chan nnnn , Pablo Ch is mann ssss , Roxanne Connelly cc , Ch is ian S. Czyma a qqqq ,ElenaDamian yyyy , Alejand o Ecke c ,AchimEdelmann , Mau een A. Ege , Simon Elle b ock c,i , Anna Fo ke z ,And eaFo s e pp , Ch is Gaasendam ee , Kons an in Ga as i ,Ve nonGayle cc , The esa Gessle hhhh ,TimoGnambs gg , Am elie Gode oid aaaaa ,MaxG € omping hh ,Ma inG oß ii , S e an G ube jj , Tobias Gumme ssss ,And easHadja kk,mm,llll, , Jan Paul Heisig iiii,mmmmm ,Sebas ian Hellmeie nnnnn , S e anie Heyne c , Magdalena Hi sch ooooo ,MikaelHje m ,Osh a Hochman ssss , And eas H€ o e mann mm,qq ,SophiaHunge ppppp , Ch is ian Hunkle nn , No a Hu h oo ,Zs  ofiaS.Ign  acz qqqq ,Lau aJacobs ll ,Jannes Jacobsen ,jjjj , Bas ian Jaege , Sebas ian Jungkunz ss,lll,iiiii , Nils Jungmann , Ma hias Kau uu , Manuel Kleine uuu , Julia Klinge jjjjj , Jan-Philipp Kolb ,Ma a Kołczy nska ww ,JohnKuk xx , Ka ha ina Kunißen k ,Dafina Ku i Sina a z ,Alexande Langenkamp qqqq ,PhilippM.Le sch ,ddddd , Lea-Ma ia L€ obel ,PhilippLu sche yy , Ma hias Made zz ,JoanE.Madia aaa,lllll ,Na aliaMalancu bbb ,LuisMaldonado ccc , Helge Ma ah ens e , Nicole Ma in ddd ,PaulMa inez eee ,JochenMaye l h ,Osca J. Mayo ga , Pa icia McManus e , Kyle McWagne bb , Cecil Meeusen ee , Daniel Meie ieks ooooo , Jona han Mellon ddd ,F iedolinMe hou ggg ,SamuelMe k hhh , Daniel Meye kkkkk , Le icia Micheli iii ,Jona hanMijs jjj ,C is  obal Moya kkk ,Ma cel Neunhoe e i ,DanielN € us hhhhh , Ola Nygå d mmm , Fabian Ochsen eld nnn , Gunna O e k , Anna O. Pechenkina ooo , Ch is ophe P osse ppp ,LouisRaes ,Ke in Rals on cc , Miguel R. Ramos qqq ,A neRoe s ,Jona hanRoge s sss ,Guido Rope s i , Robin Samuel kk, ,G ego Sand jj ,A ielaSchach e ,Me lin Schae e qqqqq ,Da idSchie e decke xxxx , Elma Schlue e uuu ,RegineSchmid o , Ka ja M. Schmid , Alexande Schmid -Ca an qqqq , Claudia Schmiedebe g oooo , J€ u gen Schneide ccccc , Ma ijn Schoon elde ,sssss , Julia Schul e-Cloos kkkk ,Sandy Schumann wwww , Reinha d Schunck oo ,J € u gen Schupp , Julian Seu ing bbbbb , Henning Silbe , Willem Sleege s , Nico Sonn ag k ,Alexande S aud z ,Nadia S eibe www ,NilsS eine pppp , Sebas ian S e nbe g z , Die e S ie s zzzz , D agana S ojmeno ska uuuu ,No aS o z xxx , E ich S iessnig ,Anne-Ka h inS oppe , Janna Tel emann yyy ,And eyTibaje mmm , B ian Tung , Giacomo Vagni x ,Jaspe Van Assche ,eeeee , Me a an de Linden xxx , Jolanda an de Noll zzz ,A noVan Hoo egem ee , S e an Vog enhube uuuuu ,BogdanVoicu aaaa,bbbb ,Fieke Wagemans ,cccc , Nadja Wehl dddd ,HannahWe ne zzzz ,B en onM.Wie nik eeee ,Fabian Win e g , Ch is o Wol c,i,wwwww , Yuki Yamada ,NanZhang c ,Con ad Zille ss,ggggg , S e an Zins gggg and Tomasz _ Z oł ak ww Au ho a filia ions: a Resea ch Cen e on Inequali y and Social Policy (SOCIUM), Uni e si y o B emen, B emen, 28359, Ge many; b School o Poli ics and In e na ional S udies, Uni e si y o Leeds, Leeds, LS2 9JT, Uni ed Kingdom; c Mannheim Cen e o Eu opean Social Resea ch, Uni e si y o Mannheim, 68131 Mannheim, Ge many; d B emen In e na ional G adua e School o Social Sciences, 28359 B emen, Ge many; e Depa men o Sociology, Indiana Uni e si y, Blooming on, IN 47405; Socio-Economic Panel S udy (SOEP), Ge man Ins i u e o Economic Resea ch (DIW), 10117 Be lin, Ge many; g Mechanisms o No ma i e Change, Max Planck Ins i u e o Resea ch on Collec i e Goods, 53113 Bonn, Ge many; h Ins i u e o Sociology, Chemni z Uni e si y o Technology, 09126 Chemni z, Ge many; i School o Social Sciences, Uni e si y o Mannheim, 68159 Mannheim, Ge many; j Depa men o Psychology, Uni e si y o Camb idge, Camb idge, CB23RQ, Uni ed Kingdom; k Ins i u e o Sociology, Johannes Gu enbe g Uni e si y Mainz, 55128 Mainz, Ge many; l Depa men o Poli ical Science, Ludwig Maximilian Uni e si y, 80539 Munich, Ge many; m Heidelbe g Uni e si y, 69117 Heidelbe g, Ge many; n Compa a i e Poli ical Economy, Uni e si y o Kons anz, 78457 Kons anz, Ge many; o Facul y o Social Sciences, Economics, and Business Adminis a ion, Uni e si y o Bambe g, 96052 Bambe g, Ge many; p Resea ch Depa men on In as a e Conflic , Peace Resea ch Ins i u e F ank u , 60329 F ank u , Ge many; q Depa men o Social Policy, London School o Economics and Poli ical Science, London, WC2A 2AE, Uni ed Kingdom; Su ey Da a Cu a ion, Leibniz Ins i u e o he Social Sciences (GESIS), 50667 Cologne, Ge many; s Jacques Delo s Cen e, He ie School, 10117 Be lin, Ge many; Depa men o Sociology, Umeå Uni e si y, 90187 Umeå, Sweden; u Depa men o Sociology, The Uni e si y o Texas Rio G ande Valley, B owns ille, TX 78520; Vienna Ins i u e o Demog aphy, Aus ian Academy o Sciences, 1030 Vienna, Aus ia; w Aus ian Na ional Public Heal h Ins i u e, Gesundhei € Os e eich (G€ OG), 1030 Vienna, Aus ia; x Social Resea ch Ins i u e, Ins i u e o Educa ion, Uni e si y College London, London, WC1H 0AL, Uni ed Kingdom; y Depa men o Sociology, Uni e si y o Zu ich, 8050 Zu ich, Swi ze land; z Independen esea che ; aa Depa men o Sociology, Uni e si y o Chile, San iago, 7800284, Chile; bb Depa men o Poli ical Science, The Uni e si y o Cali o nia, I ine, CA 92617; cc School o Social and Poli ical Science, Uni e si y o Edinbu gh, Edinbu gh, EH8 9LD, Uni ed Kingdom; dd Ins i u e o Poli ical Science, Goe he Uni e si y F ank u , 60323 F ank u , Ge many; ee Depa men o Sociology, Cen e o Sociological Resea ch, KU Leu en, 3000 Leu en, Belgium; M edialab, Sciences Po, 75007 Pa is, F ance; gg Educa ional Measu emen , Leibniz Ins i u e o Educa ional T ajec o ies, 96047 Bambe g, Ge many; hh School o Go e nmen and In e na ional Rela ions, G i fi h Uni e si y, Na han, QLD, 4111, Aus alia; ii Depa men o Sociology, Uni e si y o T€ ubingen, 72074 T€ ubingen, Ge many; jj Max Planck Ins i u e o Social Law and Social Policy, 80799 Munich, Ge many; kk Uni e si y o Luxembou g, 4365 Esch-su -Alze e, Luxembou g; ll Depa men o Poli ical Science, Uni e si  e Lib e de B uxelles, 1050 B uxelles, Belgium; mm Wi scha s- und Sozialwissenscha liches Ins i u (WSI), Hans B€ ockle Founda ion, 40474 D€ usseldo , Ge many; nn Be lin Ins i u e o In eg a ion and Mig a ion Resea ch (BIM), Humbold Uni e si y Be lin, 10099 Be lin, Ge many; oo School o Human and Social Sciences, Uni e si y o Wuppe al, 42119 Wuppe al, Ge many; pp Empi ical Educa ional and Highe Educa ion Resea ch, F eie Uni e si € a Be lin, 14195 Be lin, Ge many; qq Ge man Socio- Economic Panel Su ey, 10117 Be lin, Ge many; Depa men o Social Psychology, Tilbu g Uni e si y, 5037AB Tilbu g, The Ne he lands; ss Ins i u e o Socio-Economics, Uni e si y o Duisbu g-Essen, 47057 Duisbu g, Ge many; Zeppelin Uni e si y, 88045 F ied ichsha en, Ge many; uu Depa men o Psychology, Medical School Hambu g, 20457 Hambu g, Ge many; Fede al S a is ics O fice Ge many, Des a is, 65189 Wiesbaden, Ge many; ww Depa men o Resea ch on Social and Ins i u ional T ans o ma ions, Ins i u e o Poli ical S udies o he Polish Academy o Sciences, 00-625 Wa saw, Poland; xx Depa men o Poli ical Science, Uni e si y o Oklahoma, No man, OK 73019; yy Depa men o Poli ical Science, Uni e si y o Oslo, 0851 Oslo, No way; zz Depa men o Poli ics and Public Adminis a ion, Uni e si y o Kons anz, 78457 Kons anz, Ge many; aaa Depa men o Sociology, Nu field College, Uni e si y o Ox o d, Ox o d, OX1 1JD, Uni ed Kingdom; bbb The Ins i u e o Ci izenship S udies (InCi e), Uni e si y o Gene a, 1205 Gene a, Swi ze land; ccc Ins i u o de Sociologia, Pon ifical Ca holic Uni e si y o Chile, San iago, 7820436, Chile; ddd Depa men o Poli ics, Uni e si y o Manches e , Manches e , M19 2JS, Uni ed Kingdom; eee Depa men o Ins i u ional Resea ch, Wes e n Go e no s Uni e si y, Sal Lake Ci y, UT 84107; Depa men o Sociology, Uni e si y o Cali o nia, Los Angeles, CA 90095; ggg Depa men o Sociology and Cen e o Social Da a Science, Uni e si y o Copenhagen, 1353 Copenhagen, Denma k; hhh Depa men o School De elopmen , Uni e si y o Educa ion Ka ls uhe, 76133 Ka ls uhe, Ge many; iii Depa men o Psychology III, Julius-Maximilians Uni e si y W€ u zbu g, 97070 W€ u zbu g, Ge many; jjj Depa men o Sociology, Bos on Uni e si y, Bos on, MA 02215; kkk Facul y o Sociology, Biele eld Uni e si y, 33615 Biele eld, Ge many; lll Ins i u e o Poli ical Science, Uni e si y o M€ uns e , 48149 M€ uns e , Ge many; mmm Di ision o Mig a ion, E hnici y and Socie y (REMESO), Link€ oping Uni e si y, 60174 Link€ oping, Sweden; nnn Adminis a i e Headqua e s, Max Planck Socie y, 80539 Be lin, Ge many; ooo Depa men o Poli ical Science, U ah S a e Uni e si y, Logan, UT 84321; ppp Depa men o Poli ics, In e na ional Rela ions and Philosophy, Royal Holloway Uni e si y o London, London, TW20 0EX, Uni ed Kingdom; qqq Depa men o Social Policy, Sociology and C iminology, Uni e si y o Bi mingham, Bi mingham, B15 2TT, Uni ed Kingdom; Depa men o De elopmen al, Pe sonali y and Social Psychology, Ghen Uni e si y, 9000 Ghen , Belgium; sss Di ision o Social Science, New Yo k Uni e si y Abu Dhabi, Abu Dhabi, 10276, Uni ed A ab Emi a es; Depa men o Sociology, Washing on Uni e si y in S . Louis, S . Louis, MO 63130; uuu Ins i u e o Sociology, Jus us Liebig Uni e si y o Giessen, 35394 Giessen, Ge many; Uni e si y College Dublin, Dublin 4, I eland; www Depa men o Sociology, Uni e si y o Vienna, 1090 Vienna, Aus ia; xxx In e disciplina y Social Science, U ech Uni e si y, 3584 U ech , The Ne he lands; yyy Ins i u e o Social Sciences, Uni e si y o Hildesheim, 31141 Hildesheim, Ge many; zzz Depa men o Psychology, Uni e si y o Hagen, 58097 Hagen, Ge many; aaaa Resea ch Ins i u e o Quali y o Li e, Romanian Academy, 010071 Bucha es , Romania; bbbb Depa men o Sociology, Lucian Blaga Uni e si y o Sibiu, 550024 Sibiu, Romania; cccc Ne he lands Ins i u e o Social Resea ch, 2500 BD The Hague, he Ne he lands; dddd Resea ch Clus e "The Poli ics o Inequali y", Uni e si y o Kons anz, 78464 Kons anz, Ge many; eeee Depa men o Psychology, Uni e si y o Sou h Flo ida, Tampa, FL 33620; Facul y o A s and Science, Kyushu Uni e si y, Fukuoka, 819-0395, Japan; gggg Ins i u e o Employmen Resea ch, Fede al Employmen Agency, 90478 Nu embe g, Ge many; hhhh Kul u wissenscha liche Fakul € a , Eu opean Uni e si y Viad ina, 15230 F ank u (Ode ), Ge many; iiii Uni e si y o G oningen, 9712 CP G oningen,The Ne he lands; jjjj Clus e "Da a- Me hods-Moni o ing", Ge man Cen e o In eg a ion and Mig a ion Resea ch (DeZIM),10117 Be lin, Ge many; kkkk Robe Schuman Cen e o Ad anced S udies, Eu opean Uni e si y Ins i u e, 50133 Flo ence, I aly; llll Uni e si y o F ibou g, 1700 F ibou g, Swi ze land; mmmm Cen e o Social Conflic and Cohesion S udies (COES), Pon ificia Uni e sidad Ca  olica de Chile, San iago, 8331150, Chile; nnnn Depa men o Poli ical Science and In e na ional Rela ions, Loyola Ma ymoun Uni e si y, Los Angeles, CA 90045; oooo Depa men o Sociology, Ludwig Maximilian Uni e si y, 80801 Munich, Ge many; pppp Ins i u e o Poli ical Science, Johannes Gu enbe g Uni e si y Mainz, 55099 Mainz, Ge many; qqqq Ins i u e o Sociology, Goe he Uni e si y F ank u , 60323 F ank u , Ge many; Knowledge Exchange and Ou each, Leibniz Ins i u e o he Social Sciences (GESIS), 68159 Mannheim, Ge many; ssss Da a and Resea ch on Socie y, Leibniz Ins i u e o he Social Sciences, 68159 Mannheim, Ge many; Depa men o Su ey Design and Me hodology, Leibniz Ins i u e o he Social Sciences (GESIS), 68159 Mannheim, Ge many; uuuu Depa men o Sociology, Uni e si y o Ams e dam, 1001 Ams e dam, The Ne he lands; Jacobs Cen e o P oduc i e You h, Uni e si y o Zu ich, 8050 Zu ich, Swi ze land; wwww Depa men o Secu i y and C ime Science, Uni e si y College London, London,WC1E 6BT, Uni ed Kingdom; xxxx Ins i u e o Media and Communica ion S udies, F eie Uni e si € a Be lin, 14195 Be lin, Ge many; yyyy Li es yle and Ch onic Diseases, Epidemiology and Public Heal h, Sciensano, 1000 B ussels, Belgium; zzzz Cen e o Poli ical Science Resea ch, KU Leu en, 3000 Leu en, Belgium; aaaaa Cen e o Resea ch on Peace and De elopmen , KU Leu en, 3000 Leu en, Belgium; bbbbb Depa men o Mig a ion, Leibniz Ins i u e o Educa ional T ajec o ies, 96047 Bambe g, Ge many; ccccc T€ ubingen School o Educa ion, Uni e si y o T€ ubingen, 72074 T€ ubingen, Ge many; ddddd Depa men o Social Sciences, Humbold Uni e si y Be lin, 10099 Be lin, Ge many; eeeee Cen e o Social and Cul u al Psychology, Uni e si  e Lib e de B uxelles, 1050 B ussels, Belgium; Depa men o Economics, Tilbu g Uni e si y, 5037AB Tilbu g, The Ne he lands; ggggg Depa men o Poli ical Science, Uni e si y o Duisbu g-Essen, 47057 Duisbu g, Ge many; hhhhh Depa men o Geosciences, Uni e si y o M€ uns e , 49149 M€ uns e , Ge many; iiiii Chai o Poli ical Sociology, Uni e si y o Bambe g, 96052 Bambe g, Ge many; jjjjj Ins i u e o Sociology and Social Psychology, Uni e si y o Cologne, 50931 Cologne, Ge many; kkkkk Depa men o Educa ion and Social Sciences, Uni e si y o Cologne, 50931 Cologne, Ge many; lllll Ins i u e o he E alua ion o Public Policies, Fondazione B uno Kessle , 38122 T en o, I aly; mmmmm Resea ch G oup "Heal h and Social Inequali y", Be lin Social Science Cen e (WZB), 10785 Be lin, Ge many; nnnnn T ans o ma ions o Democ acy Uni , Be lin Social Science Cen e (WZB), 10785 Be lin, Ge many; ooooo Resea ch Uni Mig a ion, In eg a ion, T ansna ionaliza ion, Be lin Social Science Cen e (WZB), 10785 Be lin, Ge many; ppppp Cen e o Ci il Socie y Resea ch, Be lin Social Science Cen e , 10785 Be lin, Ge many; qqqqq Depa men o Sociology, Uni e si y o Copenhagen, 1353 Copenhagen, Denma k; Depa men o Social Sciences, Uni e si y o Luxembou g, 4366 Esch-su -Alze e, Luxembou g; sssss Depa men o Eu opean Languages and Cul u es, Uni e si y o G oningen, 9712 EK G oningen, The Ne he lands; Depa men o Demog aphy, Uni e si y o Vienna, 1010 Vienna, Aus ia; uuuuu Educa ion and Employmen , Ins i u e o Ad anced S udies, Uni e si y o Vienna, Vienna, 1080 Aus ia; and Policy Pe spec i es, Ci izen Pe spec i es, and Beha io s, Ne he lands Ins i u e o Social Resea ch, 2594 The Hague, The Ne he lands; wwwww P esiden , Leibniz Ins i u e o he Social Sciences (GESIS), 68159 Mannheim, Ge many Au ho con ibu ions: N.B., E.R., and A.W. designed esea ch; N.B., E.R., and A.W. pe o med esea ch; N.B., H.N., and D.N. con ibu ed new eagen s/analy ic ools; N.B., H.N., M.A., J.A., A.A., H.A., D.A., F.A., O.B., D.B., G.B., P.B., M.B., S.B., V.B., J.B., C.B., A.B., F.B., T.B., K.B., J.B., L.B., A.B., T.B., A.B., Z.B., K.B., K.B., K.B., J.C., N.C., P.C., R.C., C.C., E.D., A.E., A.E., M.E., S.E., A.F., A.F., C.G., K.G., V.G., T.G., T.G., A.G., M.G., M.G., S.G., T.G., A.H., J.H., S.H., S.H., M.H., M.H., O.H., A.H., S.H., C.H., N.H., Z.I., L.J., J.J., B.J., S.J., N.J., M.K., M.K., J.K., J.K., M.K., J.K., K.K., D.K., A.L., P.L., L.L., P.L., M.M., J.M., N.M., L.M., H.M., N.M., P.M., J.M., O.M., P.M., K.M., C.M., D.M., J.M., F.M., S.M., D.M., L.M., J.M., C.M., M.N., D.N., O.N., F.O., G.O., A.P., C.P., L.R., K.R., M.R., A.R., J.R., G.R., R.S., G.S., A.S., M.S., D.S., E.S., K.S., R.S., A.S., C.S., J.S., M.S., J.S., S.S., R.S., J.S., J.S., H.S., W.S., N.S., A.S., N.S., N.S., S.S., D.S., D.S., N.S., E.S., A.S., J.T., A.T., B.T., G.V., J.V., M.V., J.V., A.V., S.V., B.V., F.W., N.W., H.W., B.W., F.W., C.W., Y.Y., N.Z., C.Z., S.Z., and T.Å. analyzed da a; and N.B., E.R., and A.W. w o e he pape . PNAS 2022 Vol. 119 No. 44 e2203150119 h ps://doi.o g/10.1073/pnas.2203150119 7o 8 1. M. Solomon, Social Empi icism (MIT P ess, 2007). 2. N. O eskes, Why T us Science? (P ince on Uni e si y P ess, 2019). 3. C. F. Came e e al., E alua ing he eplicabili y o social science expe imen s in Na u e and Science be ween 2010 and 2015. Na . Hum. Beha . 2, 637–644 (2018). 4. Open Science Collabo a ion, PSYCHOLOGY. Es ima ing he ep oducibili y o psychological science. Science 349, aac4716 (2015). 5. S. Ri chie, Science Fic ions: How F aud, Bias, Negligence, and Hype Unde mine he Sea ch o T u h (Me opoli an Books, 2020). 6. A. B. Sø ensen, The s uc u al basis o social inequali y. Am. J. Sociol. 101, 1333–1365 (1996). 7. B. S. F ey, Publishing as p os i u ion? Choosing be ween one’s own ideas and academic success. Public Choice 116, 205–223 (2003). 8. A. Gelman, E. Loken, The s a is ical c isis in science. Am. Sci. 102, 460 (2014). 9. A. O ben, A. K. P zybylski, The associa ion be ween adolescen well-being and digi al echnology use. Na . Hum. Beha . 3, 173–182 (2019). 10. M. Del Giudice, S. Ganges ad, A a ele ’s guide o he mul i e se: P omises, pi alls, and a amewo k o he e alua ion o analy ic decisions. Ad . Me hods P ac . Psychol. Sci., 10.1177/ 2515245920954925 (2021). 11. J. P. Simmons, L. D. Nelson, U. Simonsohn, False-posi i e psychology: Undisclosed flexibili y in da a collec ion and analysis allows p esen ing any hing as significan . Psychol. Sci. 22, 1359–1366 (2011). 12. U. Schimmack, A me a-psychological pe spec i e on he decade o eplica ion ailu es in social psychology. Can. Psychol. 61, 364–376 (2020). 13. J. F eese, D. Pe e son, The eme gence o s a is ical objec i i y: Changing ideas o epis emic ice and i ue in science. Sociol. Theo y 36, 289–313 (2018). 14. R. Silbe zahn e al., Many analys s, one da a se : Making anspa en how a ia ions in analy ic choices a ec esul s. Ad . Me hods P ac . Psychol. Sci. 1, 337–356 (2018). 15. G. Du ilh e al., The quali y o esponse ime da a in e ence: A blinded, collabo a i e assessmen o he alidi y o cogni i e models. Psychon. Bull. Re . 26, 1051–1069 (2019). 16. J. A. Bas iaansen e al., Time o ge pe sonal? The impac o esea che s choices on he selec ion o ea men a ge s using he expe ience sampling me hodology. J. Psychosom. Res. 137, 110211 (2020). 17. R. Bo inik-Neze e al., Va iabili y in he analysis o a single neu oimaging da ase by many eams. Na u e 582,84–88 (2020). 18. A. J. Menk eld e al., Non-s anda d e o s. Social Science Resea ch Ne wo k [P ep in ] (2022). h ps://doi.o g/10.2139/ss n.3961574 (Accessed 4 Janua y 2022). 19. D. B ady, R. Finnigan, Does immig a ion unde mine public suppo o social policy? Am. Sociol. Re . 79,17–42 (2014). 20. K. Auspu g, J. B € ude l, Has he c edibili y o he social sciences been c edibly des oyed? Reanalyzing he “Many Analys s, One Da a Se ”P ojec . Socius 86, 532–565 (2021). 21. I. Lundbe g, R. Johnson, B. M. S ewa , Wha is you es imand? Defining he a ge quan i y connec s s a is ical e idence o heo y. Am. Sociol. Re . 86, 532–565 (2021). 22. A. Alesina, E. Glaese , Why a e wel a e s a es in he US and Eu ope so di e en ? Wha do we lea n? Ho izons S a egiques 2,51–61 (2006). 23. A. Alesina, E. Glaese , Figh ing Po e y in he US and Eu ope: A Wo ld o Di e ence (Ox o d Uni e si y P ess, 2004). 24. A. Alesina, E. Mu a d, H. Rapopo , Immig a ion and p e e ences o edis ibu ion in Eu ope. J. Econ. Geog . 21, 925–954 (2021). 25. M. A. Ege , E en in Sweden: The e ec o immig a ion on suppo o wel a e s a e spending. Eu . Sociol. Re . 26, 203–217 (2010). 26. J. C. Ga and, P. Xu, B. C. Da is, Immig a ion a i udes and suppo o he wel a e s a e in he Ame ican mass public. Am. J. Pol. Sci. 61, 146–162 (2017). 27. B. Bu goon, Immig a ion, in eg a ion, and suppo o edis ibu ion in Eu ope. Wo ld Poli . 66, 365–405 (2014). 28. J. Al , T. I e sen, Inequali y, labo ma ke segmen a ion, and p e e ences o edis ibu ion. Am. J. Pol. Sci. 61,21–36 (2017). 29. S. Pa dos-P ado, C. Xena, Immig a ion and suppo o social policy: An expe imen al compa ison o uni e sal and means- es ed p og ams. Poli ical Sci. Res. Me hods 7, 717–735 (2019). 30. M. L. B yan, S. P. Jenkins, Mul ile el modelling o coun y e ec s: A cau iona y ale. Eu . Sociol. Re . 32,3–22 (2016). 31. M. Schweinsbe g e al., Same da a, di e en conclusions: Radical dispe sion in empi ical esul s when independen analys s ope a ionalize and es he same hypo hesis. O gan. Beha . Hum. Decis. P ocess. 165, 228–249 (2021). 32. M. A. Ege , N. B eznau, Immig a ion and he wel a e s a e: A c oss- egional analysis o Eu opean wel a e a i udes. In . J. Comp. Sociol. 58, 440–463 (2017). 33. N. B eznau e al., The Hidden Uni e se o Da a-Analysis. Gi Hub. h ps://gi hub.com/nb eznau/CRI. Deposi ed 12 Ap il 2021. 34. N. B eznau e al., The C owdsou ced Replica ion Ini ia i e Pa icipan Su ey. Ha a d Da a e se. h ps://da a e se.ha a d.edu/da ase .xh ml?pe sis en Id=doi:10.7910/DVN/UUP8CX. Deposi ed 24 Ma ch 2021. 8o 8 h ps://doi.o g/10.1073/pnas.2203150119 pnas.o g