B eznau, Na e e al.
A icle — Published Ve sion
The eliabili y o eplica ions: a s udy in compu a ional
ep oduc ions
Royal Socie y Open Science
P o ided in Coope a ion wi h:
WZB Be lin Social Science Cen e
Sugges ed Ci a ion: B eznau, Na e e al. (2025) : The eliabili y o eplica ions: a s udy in
compu a ional ep oduc ions, Royal Socie y Open Science, ISSN 2054-5703, The Royal Socie y
Publishing, London, Vol. 12, Iss. 3, pp. 1-23,
h ps://doi.o g/10.1098/ sos.241038
This Ve sion is a ailable a :
h ps://hdl.handle.ne /10419/315353
S anda d-Nu zungsbedingungen:
Die Dokumen e au EconS o dü en zu eigenen wissenscha lichen
Zwecken und zum P i a geb auch gespeiche und kopie we den.
Sie dü en die Dokumen e nich ü ö en liche ode komme zielle
Zwecke e iel äl igen, ö en lich auss ellen, ö en lich zugänglich
machen, e eiben ode ande wei ig nu zen.
So e n die Ve asse die Dokumen e un e Open-Con en -Lizenzen
(insbesonde e CC-Lizenzen) zu Ve ügung ges ell haben soll en,
gel en abweichend on diesen Nu zungsbedingungen die in de do
genann en Lizenz gewäh en Nu zungs ech e.
Te ms o use:
Documen s in EconS o may be sa ed and copied o you pe sonal
and schola ly pu poses.
You a e no o copy documen s o public o comme cial pu poses, o
exhibi he documen s publicly, o make hem publicly a ailable on he
in e ne , o o dis ibu e o o he wise use he documen s in public.
I he documen s ha e been made a ailable unde an Open Con en
Licence (especially C ea i e Commons Licences), you may exe cise
u he usage igh s as speci ied in he indica ed licence.
h p://c ea i ecommons.o g/licenses/by/4.0/
The eliabili y o
eplica ions: a s udy in
compu a ional ep oduc ions
Na e B eznau1, Eike Ma k Rinke2, Alexande Wu ke3,
Muna Adem4, Jule Ad iaans5, Es a Akdeniz6, Amalia
Al a ez-Benjumea7, Hen ik K. Ande sen8, Daniel Aue 9,
Fla io Aze edo10, Oke Bahnsen11, Ling Bai, Da e Balze 12,
Paul C. Baue 13,14, Ge i Baue 15, Ma kus Baumann16,
Sha on Bau e17, Ve ena Benoi 18, Julian Be naue 19,
Ca l Be ning20, Anna Be hold18, Felix S. Be hke21,
Thomas Biege 22, Ka ha ina Blinzle 23, Johannes N.
Blumenbe g24, Licia Bobzien25, And ea Bohman26, Thijs
Bol28, Amie Bos ic29, Zuzanna B zozowska30, Ka ha ina
Bu gdo 31, Kaspa Bu ge 32,33,34, Ka h in Busch35, Juan-
Ca los Cas illo36, Na han Chan37, Pablo Ch is mann38,
Roxanne Connelly39, Ch is ian S. Czyma a40, Elena
Damian41, Eline A. de Rooij42, Alejand o Ecke 43,
Achim Edelmann44, Ch is ina Ede 23, Mau een A.
Ege 27,45, Simon Elle b ock19, Anna Fo ke, And ea
Fo s e 46, Danilo F ei e47, Ch is Gaasendam48, Kons an in
Ga as11, Ve non Gayle39, The esa Gessle 49, Timo
Gnambs50, Amélie Gode oid 52, Max G ömping54,
Ma in G oß55, S e an G ube 56, Tobias Gumme 57,
And eas Hadja 58, Ve ena Halbhe 59, Jan Paul Heisig60,
Sebas ian Hellmeie 61, S e anie Heyne19, Magdalena
Hi sch62, Mikael Hje m26, Osh a Hochman63, Jan H.
© 2025 The Au ho (s). Published by he Royal Socie y unde he e ms o he C ea i e
Commons A ibu ion License h p://c ea i ecommons.o g/licenses/by/4.0/, which pe mi s
un es ic ed use, p o ided he o iginal au ho and sou ce a e c edi ed.
Resea ch
Ci e his a icle: B eznau N e al. 2025 The
eliabili y o eplica ions: a s udy in compu a ional
ep oduc ions. R. Soc. Open Sci. 12: 241038.
h ps://doi.o g/10.1098/ sos.241038
Recei ed: 12 Augus 2024
Accep ed: 2 Decembe 2024
Subjec Ca ego y:
Science, Socie y and Policy
Subjec A eas:
beha iou , compu a ional ma hema ics, compu e
modelling and simula ion
Keywo ds:
eliabili y, eplica ions, compu a ional
ep oduc ion, social and beha iou al sciences
Au ho o co espondence:
Na e B eznau
e-mail: b eznau.na [email protected]
Elec onic supplemen a y ma e ial is a ailable
online a h ps://doi.o g/10.6084/
m9. igsha e.c.7655134.
Hö le 64, And eas Hö e mann65, Sophia Hunge 66, Ch is ian Hunkle 67, No a Hu h-
S öckle68, Zsó ia S. Ignácz40, Sabine Is ael35, Lau a Jacobs70, Jannes Jacobsen71, Bas ian
Jaege 72, Sebas ian Jungkunz73, Nils Jungmann74, Jenni e Kanjana, Ma hias Kau 75,
Salman Khan76, Sayak Kha ua77, Manuel Kleine 78, Julia Klinge 35, Jan-Philipp Kolb79,
Ma a Kołczyńska80, John Kuk81, Ka ha ina Kunißen12, Da ina Ku i Sina a82, Alexande
Langenkamp40, Robin C. Lee83, Philipp M. Le sch84, Da id Liu, Lea-Ma ia Löbel85, Philipp
Lu sche 86, Ma hias Made 87, Joan E. Madia88, Na alia Malancu89, Luis Maldonado90,
Helge Ma ah ens91, Nicole Ma in92, Paul Ma inez93, Jochen Maye l8, Osca J. Mayo ga94,
Robe McDonnell, Pa icia McManus95, Kyle McWagne 96, Cecil Meeusen97, Daniel
Meie ieks62, Jona han Mellon98, F iedolin Me hou 99, Samuel Me k100, Daniel Meye 101,
Le icia Micheli102, Jona han Mijs103, C is óbal Moya104, Ma cel Neunhoe e 105, Daniel
Nüs 106, Ola Nygå d107, Fabian Ochsen eld108, Gunna O e12, Anna Pechenkina109, Ma k
Pickup110, Ch is ophe P osse , Louis Raes111, Ke in Rals on39, Miguel Ramos112, F ank
Reiche , A ne Roe s113, Jona han Roge s114, Guido Rope s35, Robin Samuel115, G ego
Sand116, Cons anza Sanhueza Pe a ca117, A iela Schach e 118, Me lin Schae e 99, Da id
Schie e decke 119, Elma Schlue e 78, Ka ja Schmid 120,121, Regine Schmid 18, Alexande
Schmid -Ca an40, Claudia Schmiedebe g15, Jü gen Schneide 122, Ma ijn Schoon elde123,
Julia Schul e-Cloos124, Sandy Schumann125, Reinha d Schunck69, Julian Seu ing51, Henning
Silbe 126, Willem Sleege s72, Nico Sonn ag12, Alexande S aud , Nadia S eibe 127, Nils D.
S eine 20, Sebas ian S e nbe g35, Die e S ie s53, D agana S ojmeno ska28, No a S o z128,
E ich S iessnig127, Anne-Ka h in S oppe23, Jo dan W. Suchow129, Janna Tel emann130,
And ey Tibaje 131, B ian Tung132, Giacomo Vagni133, Jaspe Van Assche134,135, Me a an de
Linden10, Jolanda an de Noll136, A no Van Hoo egem137, S e an Vog enhube 138, Bogdan
Voicu139,140, Fieke Wagemans141, Nadja Wehl142, Hannah We ne 143, B en on M. Wie nik35,
Fabian Win e 144, Ch is o Wol 145, Ca y Wu146, Yuki Yamada18, Bjö n Zakula, Nan Zhang147,
Con ad Zille 148, S e an Zins149, Tomasz Żół ak150 and Hung H.V. Nguyen151
1O ganiza ion and P og am Planning, Ge man Ins i u e o Adul Educa ion—Leibniz Cen e o Li elong Lea ning, Bonn 53175, Ge many
2School o Poli ics and In e na ional S udies, Uni e si y o Leeds, Leeds LS2 9JT, UK
3Geschwis e Scholl Ins i u e, LMU Munich, Munich 81541, Ge many
4Depa men o Sociology, Uni e si y o Ma yland, College Pa k 47405, USA
5Facul y o Sociology, Uni e si y o Biele eld, Biele eld 33615, Ge many
6School o Medicine, Ma ma a Uni e si y, Is anbul 34722, Tu key
7The Ins i u e o Public Goods and Policies (IPP), Cen o de Ciencias Humanas y Sociales—Consejo Supe io de In es igaciones Cien í icas, Mad id
28003, Spain
2
oyalsocie ypublishing.o g/jou nal/ sos R. Soc. Open Sci. 12: 241038
8Ins i u e o Sociology, Chemni z Uni e si y o Technology, Chemni z 09126, Ge many
9Social and Poli ical Science, Collegio Ca lo Albe o, Tu in 10122, I aly
10Depa men o In e disciplina y Social Science, U ech Uni e si y, U ech 3584CH, Ne he lands
11School o Social Sciences, Uni e si y o Mannheim, Mannheim 68159, Ge many
12Ins i u e o Sociology, Johannes Gu enbe g Uni e si y Mainz, Mainz 55128, Ge many
13Ins i u e o Poli ical Science, Uni e si y o F eibu g, F eibu g im B eisgau 79085, Ge many
14Ins i u e o S a is ics, LMU Munich, Munich, 79098, Ge many
15Depa men o Sociology, LMU Munich, Munich 80801, Ge many
16O ice o he Execu i e Boa d, GESIS—Leibniz Ins i u e o he Social Sciences, Mannheim 68159, Ge many
17Depa men o Poli ics and Public Adminis a ion, Uni e si y o Kons anz, Kons anz 78457, Ge many
18 Facul y o Social Sciences, Economics, and Business Adminis a ion, Uni e si y o Bambe g, Bambe g 96052, Ge many
19Mannheim Cen e o Eu opean Social Resea ch (MZES), Uni e si y o Mannheim, Mannheim 68131, Ge many
20Ins i u e o Poli ical Science, Johannes Gu enbe g Uni e si y Mainz, Mainz 55099, Ge many
21Resea ch Depa men IV: In as a e Con lic , Peace Resea ch Ins i u e F ank u (PRIF), F ank u 60329, Ge many
22Depa men o Social Policy, London School o Economics and Poli ical Science, London WC2A 2AE, UK
23Su ey Da a Cu a ion, GESIS—Leibniz Ins i u e o he Social Sciences, Mannheim 50667, Ge many
24 Knowledge Exchange and Ou each, GESIS—Leibniz-Ins i u e o he Social Sciences, Mannheim 67067, Ge many
25 Facul y o Economics and Social Sciences, Uni e si y o Po sdam, Po sdam 10117, Ge many
26Depa men o Sociology, and 27 Depa men o Sociology, Umeå Uni e si y, Umeå 901 87, Sweden
28 Depa men o Sociology, Uni e si y o Ams e dam, Ams e dam 1012 WP, Ne he lands
29 Depa men o Sociology, The Uni e si y o Texas Rio G ande Valley, Edinbu g TX 78539, USA
30 Vienna Ins i u e o Demog aphy, Aus ian Academy o Sciences, Vienna, Aus ia & Aus ian Na ional Public Heal h Ins i u e (Gesundhei Ös e eich
GmbH, GÖG), Vienna, Aus ia
31 School o Social Sciences, Uni e si y o B emen, B emen 28359, Ge many
32Depa men o Educa ion, Uni e si y o Po sdam, Po sdam, 14469, Ge many
33Jacobs Cen e o P oduc i e You h De elopmen , Uni e si y o Zu ich, Zü ich, 8006, Swi ze land
34Social Resea ch Ins i u e, Uni e si y College London, London WC1H 0AL, UK
35Independen Resea che
36 Depa men o Sociology Uni e sidad de Chile, Millennium Nucleus on Digi al Inequali ies and Oppo uni ies (NUDOS NCS2022_046), and Cen e
o Social Con lic and Cohesion S udies (COES), Uni e si y o Chile, San iago 7800284, Chile
37 Depa men o Poli ical Science and In e na ional Rela ions, Loyola Ma ymoun Uni e si y, Los Angeles CA 90045, USA
38 Da a and Resea ch on Socie y, GESIS—Leibniz-Ins i u e o he Social Sciences, Mannheim 68159, Ge many
39 School o Social and Poli ical Science, Uni e si y o Edinbu gh, Edinbu gh EH8 9LD, Sco land
40 Ins i u e o Sociology, Goe he-Uni e si y F ank u , F ank u 60323, Ge many
41 Epidemiology and Public Heal h, Sciensano, B ussels 1050, Belgium
42 Poli ical Science, Simon F ase Uni e si y, Bu naby V5A 1S6, Canada
43 Ins i u e o Poli ical Science, Heidelbe g Uni e si y, Heidelbe g 69115, Ge many
44Médialab Sciences Po, Pa is 75007, F ance
45Cen e o Ad anced S udy in he Beha io al Sciences, S an o d, CA 94305, USA
46 Depa men o Sociology, U ech Uni e si y, U ech 3584 CS, Ne he lands
47 Depa men o Quan i a i e Theo y and Me hods, Emo y Uni e si y, A lan a 30306, USA
48 Depa men o Wo k and Social Economy, Go e nmen o Flande s-Belgium
49 Kul u wissenscha liche Fakul ä , Eu opa Uni e si ä Viad ina, F ank u 15230, Ge many
50Educa ional Measu emen , and 51 Depa men Mig a ion, Leibniz Ins i u e o Educa ional T ajec o ies, Bambe g 96047, Ge many
52Cen e o Resea ch on Peace and De elopmen , and 53 Cen e o Poli ical Science Resea ch, KU Leu en, Leu en 3000, Belgium
54 School o Go e nmen and In e na ional Rela ions, G i i h Uni e si y, Na han, Queensland 4111, Aus alia
55 Depa men o Sociology, Uni e si y o Tübingen, Tübingen 72074, Ge many
56 Resea ch Da a Cen e and Communica ion, SHARE BERLIN Ins i u e, Be lin 10115, Ge many
57 Da a and Resea ch on Socie y, GESIS—Leibniz Ins i u e o he Social Sciences, Mannheim 68159, Ge many
58 Di ision Sociology, Social Policy and Social Wo k, Uni e si y o F ibou g, F ibou g CH-1700, Swi ze land
59Associa ion o Doc o al S udies Baden-Wue embe g 70174, Ge many
60Resea ch G oup ‘Heal h and Social Inequali y’, 61T ans o ma ions o Democ acy Uni , and 62 Resea ch Uni Mig a ion, In eg a ion,
T ansna ionaliza ion, WZB Be lin Social Science Cen e , Be lin 10785, Ge many
63 Da a and Resea ch on Socie y, GESIS—Leibniz Ins i ue o he Social Sciences, Mannheim 68159, Ge many
64 Facul ad de Emp endimien o, Negocios y Economía, Uni e sidad Espí i u San o, Replica ionWiki, and EQ-Lab
65 Wi scha s- und Sozialwissenscha liches Ins i u (WSI), Hans-Böckle -Founda ion, Düsseldo 40474, Ge many
66 SOCIUM—Resea ch Cen e on Inequali y and Social Policy, Uni e si y o B emen, B emen 10785, Ge many
67 Be lin Ins i u e o In eg a ion and Mig a ion Resea ch (BIM), Humbold Uni e si ä zu Be lin, Be lin 10099, Ge many
68School o Human and Social Sciences, and 69 School o Human and Social Sciences, Uni e si y o Wuppe al, Wuppe al 42119, Ge many
70 Depa men o Poli ical Science, Uni e si y o An we p, An we pen 2000, Belgium
71 Clus e ‘Da a-Me hods-Moni o ing’, Ge man Cen e o In eg a ion and Mig a ion Resea ch (DeZIM)
72 Depa men o Social Psychology, Tilbu g Uni e si y, Tilbu g 5037AB, Ne he lands
73 Ins i u e o Poli ical Science and Sociology, Uni e si y o Bonn, Bonn 53111, Ge many
3
oyalsocie ypublishing.o g/jou nal/ sos R. Soc. Open Sci. 12: 241038
74 Su ey Da a Cu a ion, GESIS—Leibniz Ins i u e o he Social Sciences, Mannheim 50667, Ge many
75 Depa men o Psychology, Medical School Hambu g, Hambu g 20457, Ge many
76 Economics, Uni e si y o Illinois, Chicago, Chicago, IL, USA
77 School o Public Policy, O egon S a e Uni e si y, Co allis 97330, USA
78 Ins i u e o Sociology, Jus us Liebig Uni e si y Giessen, Giessen 35394, Ge many
79S a is isches Bundesam , S a is isches Bundesam Wiesbaden, Wiesbaden 67549, Ge many
80 Depa men o Resea ch on Social and Ins i u ional T ans o ma ions, Ins i u e o Poli ical S udies o he Polish Academy o Sciences, Wa saw
00-625, Poland
81 Depa men o Poli ical Science, Michigan S a e Uni e si y, Eas Lansing 48823, USA
82 Cen e o E alua ion, Independen Resea che (Fo me ly Uni Cologne)
83 Depa men o Sociology, P ince on Uni e si y, P ince on, USA
84 Socio-Economic Panel, Ge man Ins i u e o Economic Resea ch, Be lin 10117, Ge many
85Socio-Economic Panel, Ge man Ins i u e o Economic Resea ch, Be lin 10117, Ge many
86 Depa men o Poli ical Science, Uni e si y o Oslo, Oslo 0851, No way
87 Depa men o Philosophy, Poli ics and Economics, Wi en/He decke Uni e si y, Wi en 58488, Ge many
88 Depa men o P ima y Ca e and Heal h Sciences, Uni e si y o Ox o d, Ox o d OX11JD, England
89 Swiss Fo um o Mig a ion and Popula ion S udies, Uni e si y o Neucha el, Neuchâ el 1205, Swi ze land
90 Ins i u o de Sociologia, Pon i ical Ca holic Uni e si y o Chile, San iago 7820436, Chile
91 Massi e Da a Ins i u e, Geo ge own Uni e si y, Washing on D.C. 20057, USA
92 Depa men o Poli ics, Uni e si y o Manches e , Manches e M19JS, UK
93 Depa men o Ins i u ional Resea ch, Wes e n Go e no s Uni e si y, Millc eek 84107, USA
94 Di ec o o Da a o F eedom, Equi y Resea ch Coope a i e 19107
95 Depa men o Sociology, Indiana Uni e si y Blooming on, Blooming on, IN 47405, USA
96 Depa men o Poli ical Science, Uni e si y o Wisconsin-Milwaukee, Milwaukee, WI 53211, USA
97 Depa men o Sociology, Cen e o Sociological Resea ch, KU Leu en 3000
98 Depa men o Poli ics, Wes poin Depa men o Sys ems Enginee ing M19 2JS
99 Depa men o Sociology and Cen e o Social Da a Science, Uni e si y o Copenhagen 1353
100 Depa men o School De elopmen , Ka ls uhe Uni e si y o Educa ion, Ka ls uhe 76133, Ge many
101 Compe ence Cen e o Regional De elopmen , Fede al Ins i u e o Resea ch on Building, U ban A ai s and Spa ial De elopmen (BBSR) 03048
102 Depa men o Social, Economic & O ganisa ional Psychology, Leiden Uni e si y, Leiden 2333AK, The Ne he lands
103 Depa men o Sociology, Bos on Uni e si y, Bos on, MA 02215, USA
104 Socio-Economic Panel, Ins i u e o Economic Resea ch, Be lin 10117, Ge many
105 School o Social Sciences, LMU Munich, Munich, Ge many
106 Depa men o Geosciences, Technische Uni e si ä D esden, D esden 01069, Ge many
107 Di ision o Mig a ion, E hnici y and Socie y (REMESO), Linköping Uni e si y 60174
108 Adminis a i e Headqua e s, Max Planck Socie y, Munich 80539, Ge many
109 Depa men o Poli ical Science, U ah S a e Uni e si y, Logan, UT 84321, USA
110 Poli ical Science, Simon F ase Uni e si y, Canada
111 Depa men o Economics, Tilbu g Uni e si y, Tilbu g 5037 AB, The Ne he lands
112 Depa men o Social Policy, Sociology and C iminology, Uni e si y o Bi mingham, Bi mingham B15 2TT, UK
113 Depa men o De elopmen al, Pe sonali y, and Social Psychology, Ghen Uni e si y, Sin -Pie e snieuws aa , B-9000, Belgium
114 School o Law, Empi ical Resea ch G oup, Uni e si y o Cali o nia, Los Angeles, Los Angeles, CA, USA
115 Depa men o Social Sciences, Uni e si y o Luxembou g 4366, Luxembou g
116 SHARE Ope a ions, SHARE Be lin Ins i u e, Be lin 10115, Ge many
117 School o Poli ics and In e na ional Rela ions, Aus alian Na ional Uni e si y, Canbe a 2132, Aus alia
118 Depa men o Sociology, Washing on Uni e si y in S . Louis, S . Louis MO 63130, USA
119 Ins i u e o Media and Communica ion S udies, F eie Uni e si ä Be lin, Be lin 14195, Ge many
120Depa men o Social Sciences, Humbold Uni e si y Be lin, Be lin, 10117, Ge many
121Depa men o Social Sciences, Socio-Economic Panel, Be lin, 10117, Ge many
122 Teache and Teaching Quali y, Leibniz Ins i u e o Resea ch and In o ma ion in Educa ion, F ank u 60323, Ge many
123 Chai g oup Eu opean Poli ics and Socie y, Uni e si y o G oningen, G oningen 9712 EK, Ne he lands
124 Depa men o Poli ical Science, Uni e si y o Ma bu g, Ma bu g 35037, Ge many
125 Depa men o Secu i y and C ime Science, Uni e si y College London, London WC1E 6BT, UK
126 Ins i u e o Social Resea ch, Uni e si y o Michigan, Ann A bo MI 48109, USA
127 Depa men o Sociology, Uni e si y o Vienna, Vienna 1090, Aus ia
128Expe Council on In e g a ion and Mig a ion, Be lin 10178, Ge many
129 School o Business, S e ens Ins i u e o Technology, Hoboken 07030, USA
130 Ins i u e o Social Sciences, Uni e si y o Hildesheim, Hildesheim 31141, Ge many
131 Depa men o Women's and Child en's Heal h, Uppsala Uni e si y, Uppsala SE-751 05, Sweden
132 Depa men o Sociology, Washing on Uni e si y in S . Louis, S . Louis 63130, USA
133 Social Resea ch Ins i u e (UCL), Uni e si y College London, London WC1E 6BT, USA
134Cen e o Social and Cul u al Psychology (CESCUP), Uni e si é Lib e de B uxelles, B ussels, BE-1050, Belgium
135Op en ia Resea ch Uni , No h-Wes Uni e si y, Po che s oom, 2531, Sou h A ica
4
oyalsocie ypublishing.o g/jou nal/ sos R. Soc. Open Sci. 12: 241038
136 Depa men o Psychology, Uni e si y o Hagen, Hagen 58097, Ge many
137 Depa men o Sociology and Human Geog aphy, Uni e si y o Oslo, Oslo 851, No way
138 Educa ion and Employmen , Ins i u e o Ad anced S udies, Vienna 1080, Aus ia
139Resea ch Ins i u e o Quali y o Li e, Romanian Academy, Bucha es , 010071, Romania
140Depa men o Sociology, Lucian Blaga Uni e si y o Sibiu, Sibiu, 550024, Romania
141 Beleids isies, Bu ge isies en Ged agingen (Policy Pe spec i es, Ci izen Pe spec i es, and Beha io s), Ne he lands Ins i u e o Social Resea ch, The
Hague 2594, Ne he lands
142 Resea ch Clus e ‘The Poli ics o Inequali y’, Uni e si y o Kons anz, Kons anz 78464, Ge many
143 Depa men o Poli ical Science, Uni e si y o Zu ich, Zü ich 8050, Swi ze land
144 Mechanisms o No ma i e Change, Max-Planck-Ins i u e o Resea ch on Collec i e Goods, Bonn 53113, Ge many
145GESIS Leibniz-Ins i u e o he Social Science & Uni e si y o Mannheim, Mannheim 68159, Ge many
146 Depa men o Sociology, Yo k Uni e si y, To on o M3J 1P3, Canada
147 Facul y o A s and Science, Kyushu Uni e si y, Fukuoka 819-0395, Japan
148 Depa men o Poli ical Science, Uni e si y o Duisbu g-Essen, Duisbu g-Essen 47057, Ge many
149 Ins i u e o Employmen Resea ch, Fede al Employmen Agency, Nu embe g 90478, Ge many
150 Depa men o Compu a ional Social Sciences, Ins i u e o Philosophy and Sociology o he Polish Academy o Sciences, Wa saw 00-330, Poland
151Poli ical Science, Uni e si y o B emen, B emen 28359, Ge many
NB,0000-0003-4983-3137; EMR,0000-0002-5330-7634; AW,0000-0002-9579-5357; MA,0000-0002-3560-9858;
JA,0000-0001-7782-505X; EA,0000-0001-5022-2216 ; AA-B,0000-0002-5829-2099; HKA,0000-0001-6842-5337;
DA,0000-0003-4454-2365; FA,0000-0001-9000-8513; OB,0000-0003-3198-2804; DB,0000-0001-8345-7169;
PCB,0000-0002-8382-9724; GB,0000-0002-3682-8323; MB,0000-0003-4783-868X; SB,0000-0003-2931-935X;
VB,0000-0002-8596-9202; JB,0000-0001-5699-5543; AB,0000-0002-1017-5731; FSB,0000-0002-4259-6071;
TB,0000-0001-5437-2561; JNB,0000-0003-0943-2283; LB,0000-0003-2274-509X; AB,0000-0001-8335-9235;
TB,0000-0001-9509-8423; AB,0000-0002-9809-5014; ZB,0000-0002-0235-991X; KB,0000-0002-0928-3313;
KB,0000-0001-5582-7062; KB,0000-0002-6951-0776; J-CC,0000-0003-1265-7854; NC,0000-0001-7793-3157;
PC,0000-0003-0458-9572; RC,0000-0002-3886-1506; CSC,0000-0002-9535-3559; ED,0000-0002-3776-6988;
EAdR,0000-0002-5808-920X; AE,0009-0009-1792-2080; AE,0000-0001-8293-674X; CE,0000-0002-7703-4108;
MAE,0000-0001-9023-7316; SE,0000-0002-9099-1420; AF,0000-0002-5201-1452; DF,0000-0002-4712-6810;
CG,0000-0002-9431-5833 ; KG,0000-0002-9222-0101; VG,0000-0002-1929-5983; TG,0000-0003-2339-6266;
TG,0000-0002-6984-1276; AG,0000-0002-5010-2860; MG,0000-0003-1488-4436; MG,0000-0002-5193-9865;
SG,0000-0002-3459-421X; TG,0000-0001-6469-7802; AH,0000-0002-2641-010X; VH,0000-0001-7995-1738;
JPH,0000-0001-8228-1907; SH,0000-0002-9423-7150; SH,0000-0002-1546-9421; MH,0000-0002-9709-9259;
MH,0000-0003-4203-5394; OH,0000-0002-4941-0815; JHH,0000-0002-8382-2071; AH,0000-0002-6774-6128;
SH,0000-0002-3859-5674; CH,0000-0002-1632-9834; NH-S,0000-0002-1651-9101; ZSI,0000-0002-2288-5757;
LJ,0000-0001-5094-3531; JJ,0000-0003-4358-0458; BJ,0000-0002-4398-9731; SJ,0000-0003-1040-8635;
NJ,0000-0001-8849-8373; MK,0000-0003-3803-3521; SK,0000-0003-3432-3150; JK,0000-0001-8120-5785;
J-PK,0000-0001-6982-2115; MK,0000-0003-4981-0437; JK,0000-0003-0772-1110; KK,0000-0002-8330-6392;
DKS,0000-0001-7268-661X; AL,0000-0002-3359-7179; PML,0000-0003-3863-8301; L-ML,0000-0002-1541-6514;
PL,0000-0001-6176-7297; MM,0000-0002-4593-2392; JEM,0000-0001-8398-8859; NM,0000-0002-3576-2422;
LM,0000-0002-0028-4766; HM,0000-0002-1729-9104; NM,0000-0001-8480-7175; PM,0000-0002-7041-4466;
JM,0000-0002-4599-9976; OJM,0000-0002-5299-8955; RMD,0000-0002-6440-2776; PMM,0000-0003-0954-4517;
KMW,0000-0002-8144-2569; CM,0000-0003-3071-9529; DM,0000-0003-2058-8385; JM,0000-0001-6754-203X;
FM,0000-0003-3703-7651; SM,0000-0003-2594-5337; DM,0000-0002-1057-6498; LM,0000-0003-0066-8222;
JM,0000-0002-7895-0028; CM,0000-0002-7176-4775; MN,0000-0002-9137-5785; DN,0000-0002-0024-5046;
ON,0000-0003-2272-8150; GO,0000-0001-7025-2543; AP,0000-0002-7934-9832; MP,0000-0003-0539-1138;
CP,0000-0002-2992-8190; LR,0000-0003-2640-7493; KR,0000-0003-4344-7120; MR,0000-0001-6821-3692;
FR,0000-0003-0692-5082; AR,0000-0001-5814-1189; JR,0000-0002-0039-608X; GR,0000-0001-5069-2699;
RS,0000-0002-7598-197X; GS,0000-0002-4475-0757; CSP,0000-0002-8884-4771; AS,0000-0002-7404-4140;
MS,0000-0003-1969-8974; DS,0000-0003-2376-0929; ES,0000-0003-3880-4111; KS,0000-0003-3695-1054;
RS,0000-0002-8484-5646; AS-C,0000-0002-9485-6314; CS,0000-0002-6015-0460; JS,0000-0002-3772-4198;
MS,0000-0003-4370-2654; JS-C,0000-0001-7223-3602; SS,0000-0002-0900-5356; RS,0000-0002-8185-8919;
JS,0000-0001-5567-596X; HS,0000-0002-3568-3257; WS,0000-0001-9058-3817; NS,0000-0001-9951-9117;
NS,0000-0002-9425-8840; NDS,0000-0003-3433-4079; SS,0000-0003-4225-6402; DS,0000-0001-7242-8477;
DS,0000-0002-9805-7229; NS,0000-0001-5262-4024; ES,0000-0001-5419-9498; A-KS,0000-0002-1251-9235;
JWS,0000-0001-9848-4872; JT,0000-0003-0329-3104; AT,0000-0001-7348-1632; BT,0000-0003-2630-6115;
GV,0000-0002-8140-1361; JVA,0000-0002-2570-2928; M dL,0000-0003-3944-3354; J dN,0000-0001-7929-466X;
AVH,0000-0002-9559-8038; SV,0000-0003-0839-4481; BV,0000-0002-2221-2499; FW,0000-0002-8537-0187;
NW,0000-0002-8828-4399; HW,0000-0003-4248-5037; BMW,0000-0001-9560-6336; FW,0000-0002-4838-4504;
CW,0000-0002-9364-9524; CW,0000-0003-2652-5684; YY,0000-0003-1431-568X ; BZ,0000-0002-4191-2239;
5
oyalsocie ypublishing.o g/jou nal/ sos R. Soc. Open Sci. 12: 241038
NZ,0009-0001-6883-1359; CZ,0000-0002-2282-636X; SZ,0000-0002-3097-5939; TŻ,0000-0003-1354-4472;
HH.VN,0000-0001-9496-6217
This s udy in es iga es esea che a iabili y in compu a ional ep oduc ion, an ac i i y o which i
is leas expec ed. Eigh y- i e independen eams a emp ed nume ical eplica ion o esul s om an
o iginal s udy o policy p e e ences and immig a ion. Rep oduc ion eams we e andomly g ouped
in o a ‘ anspa en g oup’ ecei ing o iginal s udy and code o ‘opaque g oup’ ecei ing only a
me hod and esul s desc ip ion and no code. The anspa en g oup mos ly e i ied o iginal esul s
(95.7% same sign and p- alue cu o ), while he opaque g oup had less success (89.3%). Second-
decimal place exac nume ical ep oduc ions we e less common (76.9 and 48.1%). Quali a i e
in es iga ion o he wo k lows e ealed many causes o e o , including mis akes and p ocedu al
a ia ions. When cu a ing mis akes, we s ill ind ha only he anspa en g oup was eliably
success ul. Ou indings imply a need o anspa ency, bu also mo e. Ins i u ional checks and less
subjec i e di icul y o esea che s ‘doing ep oduc ion’ would help, implying a need o be e
aining. We also u ge inc eased awa eness o complexi y in he esea ch p ocess and in ‘push
bu on’ eplica ions.
1. In oduc ion
A basic equi emen o science being eliable is compu a ional ep oducibili y [1]: he capaci y ‘ o
assessing he alue o accu acy o scien i ic claims based on he o iginal me hods, da a and code’ [2].
Compu a ional ep oduc ion is a special case o scien i ic eliabili y checking because i in ol es no
esea ch design decision-making. The e is no need o speci y me hods, empi ical esea ch ques ions
o de ine es imands [3,4]. Mo eo e , he da a a e p e-exis ing and os ensibly iden ical. Compu a ion-
ally ep oducing exis ing nume ical esul s should hus be s aigh o wa d, ye ecen indings in
me a-science sugges his is o en no he case. Compu a ional ep oduc ions a e subjec o unce ain y
esul ing om he in anspa ency o an o iginal s udy; some imes wha should be iden ical da a a ies
because o ead-in so wa e o e sion changes. Also, idiosync asies ac oss esea che s migh lead
hem o p ocess he da a in ways ha cause di e en alues in he compu ing en i onmen . In his
s udy, we look a compu a ional ep oducibili y ia an expe imen in which 85 eams o 1–3 esea che s
we e andomly spli in o wo g oups wi h mo e o less access o eplica ion ma e ials om a published
s udy. They we e asked o eplica e he nume ical esul s o he o iginal s udy using he same s a ing
da a and same me hods. We obse ed hese esea che s wi h he goal o unde s anding he eliabili y
o compu a ional ep oduc ions, and iden i ying he sou ces o ala mingly high unce ain y ound in
o he ep oducibili y s udies.
Rep oducibili y is cu en ly an in ense opic in he academic communi y [5–7]. The p ac ice o
public code sha ing is essen ial o ep oducibili y, bu access o o he s’ en i e esea ch pipelines is
s ill somewha o a pipe d eam in many social and beha iou al sciences. In a ecen su ey o ac i e
esea che s, only 18% in social science (n = 733) and 17% in business and economics (n = 592) p o ided
code o hei published s a is ical esul s [8].1 Ano he s udy ound ha o all social science publica-
ions in he jou nals Science and Na u e be ween 2000 and 2019, only 20% came wi h ep oducible
ma e ials (usually da a and code), and his only inc eased o 40% when he au ho s we e con ac ed
pe sonally [9]. In a simila ein, only 38% o au ho s om o e a housand s udies using da a om he
Eu opean Social Su ey sha ed hei code a e ecei ing a eques [10]. Al hough a ecen s udy shows
ha social and beha iou al scien is s o e whelmingly suppo code sha ing, e idence sugges s ha in
mo e han hal o s udies i is no p ac ised [11].
Code sha ing alone does no sol e all p oblems o ep oducibili y. E en wi h access o eplica-
ion ma e ials, compu a ional eplica ion egula ly ails [12–14]. Fo example, he Ame ican Jou nal
o Poli ical Science (AJPS) s a ed checking he ep oducibili y o all quan i a i e esea ch esul s in
pape s accep ed o publica ion in 2014. The i s 15 s udies’ esul s could no be compu a ionally
ep oduced wi h he ma e ials p o ided, and i o en ook mul iple communica ions wi h au ho s
be o e ep oduc ion was possible [15,16]. In he same ein, many s udies a emp ing o compu a-
ionally eplica e p e iously published esul s ound s iking a es o ailu e. Ha dwicke e al. [17]
1Calcula ed om hei own igu e 1.
6
oyalsocie ypublishing.o g/jou nal/ sos R. Soc. Open Sci. 12: 241038
a emp ed o ep oduce he nume ic esul s o 35 s udies published in he jou nal Cogni ion, and e en
wi h au ho assis ance, 37% had a leas one e ec no s a is ically ep oducible wi hin 10% o he
o iginal. S ockeme e al. [18] ailed o ep oduce one- hi d o esul s among majo poli ical beha iou
publica ions in 2015, wi h one-qua e no p oducing any nume ical esul s because he code was
so poo ly o ganized. Mo e ecen ly, Pé ignon e al. [19] looked a 168 s udies in inance and could
ep oduce only 52% o he epo ed nume ical e ec s. These indings demons a e ha he e is s ill
much o lea n and do be o e compu a ional ep oducibili y is he no m.
These p e ious compu a ional eplica ion a emp s demons a e ha e i ying he nume ical
indings o a s udy is no pu ely a mechanis ic p ocess. I is o en possible o achie e eplica ion,
bu no in a ‘push-bu on’ o ma —no wi hou addi ional communica ion, ma e ials and suppo .
Al hough push-bu on eplica ion is echnically possible using i ual compu ing en i onmen s, he
skills o build such applica ions a e a e, in pa icula in he social and beha iou al sciences [20]. I
push-bu on eplica ion o nume ical esea ch esul s is only i ially possible on a e age, his calls in o
ques ion he cu en eliabili y o social and beha iou al science a a basic le el. The goal o his p ojec
is o unde s and why ep oduc ion ails; hope ully, i holds keys o suppo de elopmen s among
academics, jou nals and ins i u ions seeking o imp o e he eliabili y o science.
2. Me hods
The h ee p incipal in es iga o s (PIs) launched his expe imen in 2018 wi h he a ge o a compu a-
ional eplica ion o a high- isibili y inding om a s udy using a la ge mul i-le el da ase combining
su ey da a and coun y-le el indica o s, one equi ing ela i ely s ong compu a ional skills [21]. We
c owdsou ced olun ee eams o a maximum h ee eplica o s and obse ed hem as hey a emp ed
o e i y nume ical esul s om Da id B ady & Ryan Finnigan’s 2014 a icle, ‘Does Immig a ion
Unde mine Public Suppo o Social Policy?’ [22]. This a icle me se e al c i e ia: i is highly ci ed,
o e s eely a ailable da a and code, was independen ly eplicable by wo o he s udy’s PIs in S a a
and R and he o iginal au ho s consen ed o he use o hei wo k.
We p e- egis e ed ou expe imen al design and plans o quali a i ely code he esea che s’
wo k lows on he Open Science F amewo k [23]. Powe analysis o achie e powe o 0.95 unde a
condi ion o a small (0.382), medium (0.463) o la ge (0.518) s anda dized (Cohen’s d: XY-s anda dized)
di e ence in a e age e ec size o one g oup compa ed o he poin es ima es o he o iginal s udy’s
indings using a one- ailed 95% con idence in e al de e mined ha we need a leas 76, 52 o 42 o al
esea che s, espec i ely. We assumed ha we would need only he nume ical dis ance o o iginal and
eplica ion esul s as he ou come a iable, bu in wha ollows we p esen wo addi ional dicho omous
measu es o a success ul eplica ion de eloped pos hoc. We also we e unable o imagine in ad ance
all ypes o e o s esea che s migh make; he e o e, ou quali a i e coding o hese eme ged di ec ly
om he eplica ion eams’ wo k lows and includes e en s beyond ou heo e ical lis o p edic ed
mis akes in he p e- egis a ion plan.
All pa icipan s we e o e ed co-au ho ship on he inal s udy i hey comple ed all asks. O he
ini ial 105 eams ha egis e ed, 99 success ully comple ed he ini ial su ey. Random assignmen o
hese 99 eams placed 50 in o a anspa en g oup (TG) ha ecei ed he B ady & Finnigan a icle,
he o iginal S a a code and published echnical appendix. The o he 49 eams, he opaque g oup
(OG), go an anonymized and less anspa en e sion o he s udy (see elec onic supplemen a y
ma e ial, appendix A). Compa ison o means o eam ea u es e eals balanced g oup assignmen (see
elec onic supplemen a y ma e ial, able S1 in appendix B). Thi een eams d opped ou be o e s a ing
he eplica ion and one du ing he eplica ion, lea ing 39 eams in he TG and 46 in he OG. All s udy
ma e ials ha can be sha ed publicly a e a ailable in ou P ojec Reposi o y.2
The B ady & Finnigan s udy used wo wa es o In e na ional Social Su ey P og am (ISSP) da a
con aining esponses o ques ions abou he go e nmen ’s esponsibili y o p o ide a ious o ms o
social secu i y and wel a e. These da a we e agg ega ed o he coun y-wa e le el and eg essed on
s ock and low o immig a ion measu es ac oss di e en model speci ica ions. To c ea e an in anspa -
en condi ion o he s udy o he OG o eplica e, he PIs emo ed wo ou o six o he dependen
a iables and he indi idual-le el independen a iable measu ing income (selec ed because i had no
impac on any e ec s o in e es ). The esul s we e p esen ed o he OG in a Me hods sec ion w i en by
2Gi Hub P ojec Reposi o y: h ps://gi hub.com/nb eznau/how_many_ eplica o s.
7
oyalsocie ypublishing.o g/jou nal/ sos R. Soc. Open Sci. 12: 241038
he PIs desc ibing he models, and di ec ion and signi icance o coe icien s wi hou he o iginal pape ,
nume ical esul s o code (see elec onic supplemen a y ma e ial, appendix A3). Ou wo expe imen al
condi ions we e in ended o simula e pola ex emes in anspa ency.3 Fo he pu pose o simula ing a
eal esea ch endea ou , he pa icipan s we e ins uc ed o use he so wa e hey no mally use a he
han lea n S a a. In he TG, he S a a use s we e asked o w i e hei own code based on he S a a ile
om he o iginal au ho s.4
Pa icipan s had h ee weeks o comple e he eplica ion, wi h ex ensions g an ed upon eques .
They we e asked o p esen odds a ios as hese we e he nume ical ou comes epo ed in he
o iginal s udy. All eams ecei ed an Excel empla e o help s anda dize epo ing. We eco ded eams’
nume ical ep oduc ions o ou dependen a iables eg essed on di e en co a ia es in a o al o 26
models ( he i s ou columns o B ady & Finnigan [22] 5). Fou models included bo h s ock and low
measu es o immig a ion simul aneously (pe cen o eign-bo n and ne mig a ion), bu hese we e no
gi en o he OG in ano he s ep o disguise he o iginal s udy. Thus, a o al o 48 odds a ios in he
TG and 40 in he OG we e epo ed o analysis. A ew models an in o con e gence issues, and a ew
eams made mis akes ha p e en ed hem om a i ing a es ima es; he e o e, no all epo ed all
e ec s. The inal N was 3695 odds a ios om 85 eams.
No all s udy pa icipan s consen ed o ha e hei names e ealed in connec ion wi h hei code, so
we we e e hically obliga ed o edac all iden i ying ea u es be o e making i all public (see elec onic
supplemen a y ma e ial, appendix C). In ou esea ch design, we in en ionally did no engage in
quali y con ol o p o ide wo k low guidance o he han he empla e o epo ing esul s. Some
eams submi ed in Wo d o RTF documen o ma s, and o he s used Ge man-language Excel wi h
commas ins ead o decimal poin s. Hence, we cons uc ed a ma ix o all esul s wi h some ine i able
copy-pas ing om incompa ible ile o ma s. We checked on h ee independen occasions ha hei
submi ed code p oduced hese esul s. In ou eams, pa s o he code we e missing due o a poin -
and-click me hod o esea che s no sa ing hei wo k lows.6 To incu minimal ecological bias, we did
no demand ha hese eams p oduce new code o us. S a ing wi h he eams’ submi ed wo k lows
and esul s, all wo k conduc ed o his a icle, including analysis o eams’ submissions, p oduc ion
o igu es and analysis o a pa icipan su ey, is a ailable in ou P ojec Reposi o y. In addi ion o
quan i ying he unce ain y o compu a ional eplica ions, we quali a i ely in es iga ed he con en o
each eam’s wo k low o de e mine he sou ces o his unce ain y.
Rep oducibili y, some imes labelled as compu a ional eplica ion, o compu a ional o analy ic
ep oduc ion, means ob aining he same esul s as he o iginal s udy using he same da a and code
[2,24]. P ac ically speaking, his is no always easible o wo easons. The i s is ha no all eplica-
o s will know how o use o e en ha e access o he so wa e used in an o iginal s udy, and he second
is ha di e en compu ing en i onmen s may p oduce di e en le els o decimal place p ecision by
de aul . None heless, a a basic le el, ep oducibili y should occu a leas wi hin a ew decimal places
and should no depend on he so wa e, so long as iden ical me hods a e implemen ed.
Gi en he unce ain y in he de ini ion o a success ul compu a ional eplica ion [25], we de eloped
h ee di e en measu es o quan i y ou esul s. The i s we call a Di ec ional Rep oduc ion, a dicho -
omy whe e esul s simply poin in he same di ec ion and ma ch a null hypo hesis signi icance es ha
he coe icien is exac ly ze o wi h a cu -o o p < 0.05 o no . In his scena io, he exac numbe s need
no ma ch o success. This is impo an , because he discussion o scien i ic indings o en e ol es
a ound he exis ence o an e ec o no . Nex , we de ine a s ic e dicho omy o Exac Replica ion
whe e esul s mus be wi hin 1% o he o iginal. This e lec s p ecision, an impo an aspec o science
such ha wi hou i , we migh no claim eliable esul s. In his case, ou es imand is a nume ical
odds a io. Because odds a ios a e nume ically asymme ic on ei he side o 1,7 we di ide o iginal
odd a ios by he eplica ed odds a io in cases whe e he eplica ed odds a io is smalle han 1
and hen mul iply by nega i e one and add one, and we di ide he eplica ed by he o iginal and
3Al hough he OG condi ion may seem ex eme, we a e awa e o many published s udies in which au ho s claim hey conduc ed
addi ional analyses wi h simila indings wi hou o e ing any nume ical esul s o code o hese.
4This se ed he ollowing wo pu poses: (i) i ga e he au ho s some oppo uni y o make ecologically alid mis akes, and (ii) he
o iginal code con ained a mo e a iable cons uc ions and models han we e o in e es o his s udy; hus a esh w i e-up se ed
o educe con usion in he code submi ed by he eplica ion eams.
5This e e s o hei main esul s in hei o iginal ables 4 and 5, which eade s can iew in ou wo k low ile 01_Da a_P ep.h ml.
6In wo cases, u he exchanges wi h he eams we e necessa y o ge hei code unning because i con ained p ocedu al elemen s
he PIs we e no amilia wi h— once in an R p ojec and ano he unning MLwiN in a S a a shell.
7Fo example, 0.5 and 2 ep esen iden ical changes in odds (1/2 and 2/1, espec i ely) bu a e di e en nume ical dis ances om he
null odds o one.
8
oyalsocie ypublishing.o g/jou nal/ sos R. Soc. Open Sci. 12: 241038
3.2. Co ela es o e o s
E o s we e dis ibu ed ac oss mos eams ( igu e 2) and abou one- hi d o he a iance in whe he
a model was a success ul ep oduc ion o no ook place be ween eams ( igu e 1). This a iance
allows us o s a is ically analyse he sou ces o unce ain y we ound in ou h ee eplica ion ou come
measu es. This includes a en ion o he s a is ical skills and expe ience o he eams, hei pe cei ed
di icul y in comple ing he ask and o cou se he expe imen al condi ion i sel ( anspa ency o
ma e ials). able 4 p esen s he aw esul s (elec onic supplemen a y ma e ial, ables S8 and S9 in
appendix B p esen cu a ed and immed). Pooled esul s a e in he i s le column. Al hough eams
using S a a had a highe Di ec ional Rep oduc ion a e on a e age, mul i a ia e analysis adjus ing o
he po en ial co ela ions o o he a iables sugges s e y b oad con idence in e als (b = 0.14, s.e. =
0.08); ha is, hese da a a e as likely o be obse ed i he s a is ical e ec was uly ze o. S a is ics skills
and s udying o ha ing acqui ed a sociology deg ee appea o ha e no associa ion, nei he a sizeable
coe icien no a ejec ed null hypo hesis. Teams epo ing ha he ask was mo e di icul we e less
likely o succeed ( ailed NHST; b = −0 .14, s.e. = 0.05, x-s anda dized). Finally, eams we e oughly
25% mo e likely o succeed i hey we e in he TG (b = 0.25, s.e. = 0.06 , x dicho omous) all else equal.
We place no in e p e a ional weigh on coe icien s ha a e bo h small and lack a p- alue below 0.05.
The adjus ed R-squa e sugges s ha we can explain abou 25% o he eam-le el a ia ion in esul s.
G oup-speci ic esul s sugges ha S a a use s we e a mo e likely o di ec ionally ep oduce han
non-S a a use s in he TG, bu his e ec was absen in he OG. Highe pe cei ed di icul y is also
associa ed wi h lowe ep oducibili y in bo h g oups, and gi en i s co ela ion wi h s a is ics skills,
we assume ha i abso bs he e ec s. The expe imen al condi ion, pe cei ed di icul y and s a is ics
skills a e all endogenous. These esul s a e simila o Exac Replica ion, al hough a less a iance is
explained and e ec sizes a e also smalle . I is impo an o no e ha a low-N weakens ou capaci y
o s a is ical in e ence in g oup-le el analyses. Ou p e- egis e ed powe analysis was only designed
o de ec an expe imen al g oup di e ence, no o de ec mul i a ia e e ec s, so his analysis should be
conside ed explo a o y.
Wha we mos ly canno explain is he deg ee o e o p esen on a e age pe eam. Reg essions
on Replica ion E o yield adjus ed R-squa ed o ze o o he pooled sample and he OG. The TG
eg ession p oduces esul s wi h an adjus ed R-squa ed o 0.21 ( oughly 21% a iance explained), and
his seems en i ely a ibu able o he iny e ec o using S a a, which on a e age is associa ed wi h a
2% lowe e o ma gin; his migh be, o example, he di e ence be ween an exac eplica ion (e o =
0) and an odds a io ha is 2% la ge o smalle han he o iginal.
I seems clea ha anspa en ma e ials a e a cause o eplica ion success likelihood. Howe e ,
he e is a signi ican nega i e Pea son co ela ion o S a a wi h Di icul in bo h g oups ( = −0.17 TG, =
−0.39 OG; see elec onic supplemen a y ma e ial, able S11, appendix B). Wi h such low case numbe s,
we a e unlikely o be able o adjudica e clea ly be ween hese wo a iables. We no e ha he signs o
bo h mos ly pa e n as expec ed despi e wide con idence in e als.
Tu ning o he immed and cu a ed esul s, some s a is ical associa ions emain simila . Howe e ,
cu a ion ende ed he TG o ha e e y li le explained a iance and no signi ican coe icien s o all
h ee eplica ion ou come measu es. This may ela e o he ac ha a e cu a ion, 98.2% o he a iable
Di ec ional Rep oduc ion a e ze os, lea ing li le a iance o explain. This is no he case o he
immed da a whe e we see a high R-squa ed. Mo e s iking in he cu a ion is a much highe deg ee o
explained a iance in he OG. We a ibu e his o he cu a ion o majo mis akes, which we assume a e
somewha andom, and once we emo e hem, we a e le wi h clea e associa ions be ween pe cei ed
di icul y o he ask and he accu acy o he ou come. I he cu a ed esul s a e mo e ecologically
alid han he aw esul s, we would conclude ha e o is a p oduc o he esea che ’s abili ies and
challenges encoun e ed in hei esea ch, whe eas he aw esul s sugges ha e o is mos ly andom
i hey lack access o he o iginal code.
Finally, we in es iga e ou quali a i e ca ego ies Mis ake and P ocedu al using eg ession analyses.
Fo he Mis ake ou come, we include only hose eams ha had a leas one ins ance o he ca ego y
Mis ake o Mis ake-P ocedu al (= 1) and compa e hem o all o he eams (= 0), wi h hose ha ing
only P ocedu al being d opped. Fo he P ocedu al ou come, we e e se his and d op eams wi h any
Mis ake. Respec i ely, we d op eams wi h any P ocedu al o Mis ake e o s om he analyses because
we wan o isola e he likelihood o commi ing Mis ake o P ocedu al e o s ela i e o no o he wise
making e o s.
Table 5 shows ha he pe cei ed di icul y o he eplica ion, and being in he OG led o a much
highe likelihood o Mis ake and P ocedu al e o s alike. Keeping in mind he high co ela ion o S a a
15
oyalsocie ypublishing.o g/jou nal/ sos R. Soc. Open Sci. 12: 241038
and Di icul , i is unsu p ising ha he p- alues a e abo e ou cu o s when we un g oup-speci ic
eg essions. We p o ide p- alues owing o con en ion, bu again do no place a s ong s ake in hem
gi en sample sizes and a lack o p e- egis a ion. The explained a iance in he TG is high and d i en
mos ly by whe he he eam used S a a o no . The e ec o Di icul is he only a iable ha ma hema -
ically explains a iance in he OG, bu con idence in e als s ill o e lap ze o e en i we d op o 90%
con idence, and o e all, he eg ession explains e y li le a iance. We conclude ha bo h Mis akes
and P ocedu al e o s a e mo e likely i esea che s ace g ea e subjec i e di icul y in comple ing
hei eplica ion asks, ega dless o he anspa ency o ma e ials. As we doub ha S a a use s a e
mo e o less skilled han R use s, we specula e ha he co ela ion o S a a and Di icul migh esul
Table 4. Mul i a ia e analysis o compu a ional ep oducibili y in 85 eams, aw esul s.
di ec ional ep oduc ion exac eplica ion eplica ion e o
a iable pooled TG OG pooled TG OG pooled TG OG
(in e cep ) 0.45***
(0.06)
0.60***
(0.08)
0.51***
(0.07)
0.89***
(0.03)
0.91***
(0.03)
0.91***
(0.04)
0.44
(0.23)
0.03***
(0.01)
0.56
(0.36)
S a a 0.14
(0.08)
0.24*
(0.10)
0.01
(0.09)
0.04
(0.04)
0.07*
(0.03)
−0 .01
(0.06)
−0 .19
(0.27)
−0 .02*
(0.01)
−0 .33
(0.47)
s a -skill −0 .03
(0.02)
−0 .06
(0.03)
−0 .00
(0.03)
−0 .01
(0.01)
−0 .02
(0.01)
−0 .01
(0.02)
−0 .07
(0.07)
0.00
(0.00)
−0 .14
(0.14)
di icul −0 .14**
(0.05)
−0 .17*
(0.08)
−0 .13*
(0.05)
−0 .06*
(0.02)
−0 .04
(0.03)
−0 .08*
(0.03)
−0 .01
(0.02)
0.01
(0.01)
−0 .02
(0.04)
sociology deg ee −0 .03
(0.07)
−0 .02
(0.03)
0.01
(0.04)
TG 0.25***
(0.06)
0.05
(0.03)
−0 .06
(0.03)
obse a ions 85 39 46 85 39 46 85 39 46
R20.297 0.265 0.164 0.163 0.214 0.138 0.050 0.267 0.049
R2 adjus ed 0.252 0.202 0.104 0.110 0.147 0.077 0.000 0.205 0.000
No e: Uns anda dized OLS eg ession coe icien s p edic ing ou comes agg ega ed o hei mean by eam; s anda d e o s in
pa en heses. Deg ee omi ed om g oup-speci ic eg essions due o low p edic i e powe and smalle sample sizes. TG =
anspa en g oup wi h access o all ma e ials and OG = opaque g oup wi h no code and less me hodological in o ma ion.
*p < 0.05, **p < 0.01, ***p < 0.001
Table 5. Mul i a ia e analysis p edic ing quali a i ely ca ego ized sou ces o e o .
mis ake p ocedu al
pooled TG OG pooled TG OG
(in e cep ) 0.67***
(0.10)
0.51***
(0.11)
0.58***
(0.14)
0.63***
(0.10)
0.42***
(0.09)
0.51**
(0.15)
S a a −0 .14
(0.11)
−0 .31*
(0.13)
0.01
(0.18)
−0 .07
(0.11)
−0 .29*
(0.10)
0.13
(0.20)
s a skill 0.05
(0.03)
0.10**
(0.04)
−0 .02
(0.05)
0.05
(0.03)
0.10**
(0.03)
−0 .01
(0.06)
di icul 0.20**
(0.07)
0.28**
(0.10)
0.16
(0.10)
0.22**
(0.08)
0.35***
(0.08)
0.21
(0.12)
exp −0 .29**
(0.10)
−0 .38**
(0.10)
obse a ions 77 38 39 72 34 38
R20.240 0.366 0.102 0.291 0.543 0.086
R2 adjus ed 0.198 0.310 0.026 0.249 0.497 0.006
No e: Linea p obabili y models. Uns anda dized OLS eg ession coe icien s p edic ing ou comes agg ega ed o hei mean by eam;
s anda d e o s in pa en heses. TG = anspa en g oup wi h access o all ma e ials and OG = opaque g oup wi h no code and less
me hodological in o ma ion.
*p < 0.05, **p < 0.01, ***p < 0.001
16
oyalsocie ypublishing.o g/jou nal/ sos R. Soc. Open Sci. 12: 241038
om S a a use s ha ing access o he code; howe e , when hey do no , hey migh be ei he be e
ained o ha e mo e expe ience a his s age in his o y, because R is much newe in social science.
4. Discussion
When a emp ing o ep oduce he nume ical esul s o a published s udy, di e en eplica o s
ob ained di e en esul s wi h a ying sou ces o e o . The e o a e was exace ba ed when he e
was less anspa ency. In o he wo ds, less in o ma ion abou da a p epa a ion and analy ical choices
is a ailable o hem. We see no eason o assume ou sample is any mo e o less echnically capable
o eplica ion han he popula ion o all social and beha iou al scien is s; howe e , i is admi edly
plausible. I we none heless hold o ou assump ion ha hey a e ep esen a i e o social science
esea che s using compu a ionally in ensi e hie a chical seconda y da a analysis, hen we ace a eali y
whe e i migh ake mo e han one independen a emp o p oduce eliable ep oduc ion esul s.
A e ing his eali y equi es much highe s anda ds o anspa ency and possibly lowe equi emen s
o p ecision in nume ical compu a ional eplica ion.
We would hope ha eplica o s communica e wi h he o iginal au ho s when hey do no a i e a
he same esul s [26]. Bu some imes such communica ion is no possible. As a hough exe cise, le us
assume ha wha we mean by eliabili y is 95% con idence ha he conclusion indica ed by a majo i y
o he obse ed numbe o eplica ions ma chs he ‘co ec ’ conclusion. This would mean ha o a
single eplica ion o su ice, i would ha e o be co ec a leas 95% o he ime. I he e is a lowe
success a e, hen mo e han one eplica ion is necessa y o each he 95% h eshold. We simula e his
p oblem by plo ing binomial p obabili ies o esea che s coming o a co ec compu a ional eplica ion
by he numbe o eplica ions needed o achie e a ce ain c i ical p obabili y ( igu e 3).
In eal-wo ld esea ch, we do no know in ad ance wha is ‘co ec ’ o ‘ ue’, as his would
unde mine he need o conduc esea ch in he i s place. This is why compu a ional ep oduc ion
is an impo an es case, as we can know he co ec esul in ad ance, a leas wi hin a deg ee o
ounding e o . We assign 51% o n eplica ions a i ing a x, he co ec answe , as ou minimum
de ini ion o majo i y. Thus, in one eplica ion, x mus be equal o 1; in wo eplica ions, x = 2; in h ee
eplica ions, x ≥ 2; in ou eplica ions, x ≥ 3 and so o h. We can hen calcula e he p obabili y P
o a i ing a x success ul ou comes among n eplica ions when he ue likelihood o a success ul
eplica ion as p using he Be noulli ials o mula PX≥x= x =x
npx1−pn−x. Then we can
calcula e he minimum alue o n o ou c i ical h eshold o p = 0.95 (i.e. 95% con idence), and
his ells us how many independen eplica ions i akes o achie e eliabili y a di e en heo e ical
Figu e 3. Simula ion o he numbe o independen eplica ions equi ed o achie e a majo i y co ec a di e en P cu o s
17
oyalsocie ypublishing.o g/jou nal/ sos R. Soc. Open Sci. 12: 241038
alues o p eplicabili y. Figu e 3 p esen s di e en alues o n a heo e ical alues o p a di e en P
h esholds.
Figu e 3 is a simula ion, and p is a mos ly unknown p ope y o eplica ions, bu i we use he
pooled di ec ional ep oduc ion esul s o his s udy (92.5%) as a po en ial alue o p, hen i would
ake a leas h ee independen eplica o s o achie e eliabili y, when his is de ined as a majo i y o
eplica ions e i ying a co ec esul 95% o he ime. We encou age use o he pooled a e as ou
cu en bes app oxima ion because i lies be ween wo ex emes o anspa ency, possibly some hing
close o eali y. I we demand 95% o wi hin- eam e ec s be di ec ional ep oduc ions, hen i would
ake mo e han 10 eplica ions pe s udy because less han 70% o eams could achie e such h ee-way
eliabili y—a binomial p obabili y o a majo i y o eplica ions, 95% o he ime, and 95% o wi hin-
eam e ec s e i ied. This may sound ex eme, bu looking back a ou li e a u e e iew, using a 92.5%
alue o p is gene ous when compa ed o o he s udies which ended o ha e i be ween 50 and 75%
[17–19].
5. Conclusion
How eliable a e compu a ional ep oduc ions? Ou s udy makes clea ha he answe o his ques ion
depends on se e al ac o s. O e all, he a e o di ec ional ep oduc ion o he o iginal esul s appea s
eliable a 92.4%, bu does no necessa ily mee a common s anda d alpha o 95%. Only when cu a ing
mis akes made by he eplica ion eams is he 95% h eshold c ossed. We unco e ed much abou he
easons o unce ain y in compu a ional eplica ion. The mos impo an is anspa ency. A success ul
single compu a ional ep oduc ion o a gi en model is a mo e likely i he comple e, well-anno a ed
o iginal code is a ailable. The a e o di ec ional ep oduc ion ha he esul s a e he same di ec ion
and sign in ou expe imen was 95.7% o he TG, no iceably highe han he OG a 89.2%, which
ecei ed less in o ma ion abou he o iginal s udy’s me hods (see igu e 1 ). The p ecision o he
eplica ion demanded also ma e s. When demanding exac nume ical eplica ion wi hin 1% o he
o iginal odds a io, he TG success a e was nea ly 30 pe cen age poin s highe (77.4 e sus 50.1%).
The skills o he esea che and he subjec i e di icul y o he eplica ion a e also impo an . In ou
sample o esea che s, we could no eliably disen angle hese wo ac o s gi en hei co ela ion (a
a ound −0 .3) and ha he expe imen al condi ion ga e a seemingly mo e di icul ask o one g oup
( he OG); howe e , a one s.d. mo e di icul eplica ion expe ience educed he likelihood o eplica ion
by anywhe e om 5 o 15% depending on he demanded p ecision and handling o ou lie s. Finally,
i he eplica o is ‘ luen ’ in he same so wa e as he o iginal s udy, his inc eases he likelihood o
a success ul eplica ion by 7 o 24%, owa ds he highe end o he dis ibu ion wi h mo e anspa en
ma e ials (see able 4 and elec onic supplemen a y ma e ial, ables S8 and S9). C ucially, he ea men
e ec o ou expe imen is shown o hold s ong a e adjus ing o o he aspec s o he eplica ion
eams.
In ou example, wo o he PIs independen ly eplica ed he o iginal s udy’s e ec s wi hin wo
decimal places in bo h S a a and R. Thei goal was o come o his esul , and he one wo king in R
spen se e al hou s ge ing o his poin . This mean ha he e was a e y high deg ee o mo i a ion
o a i e a hese esul s ha may no ha e been p esen in he pa icipan s o his s udy, gi en ha
hey we e p omised publica ion as long as hey comple ed all asks assigned o hem, ega dless o
hei ou comes. Al hough hey we e asked o app oach his s udy as hey would hei usual wo k, ou
s udy migh no exhibi ideal ecological alidi y. We ied o con ol o less mo i a ed o less skilled
ou lie s by o e ing cu a ed and immed se s o esul s. These s eps inc eased he pooled di ec ional
ep oduc ion a e ages o 95.2 and 94.1% espec i ely, while he exac eplica ion a e a e ages came
up o 71.7 and 66.4%. The highes g oup-speci ic exac eplica ion a e was only 83.9% in he TG in he
cu a ed se o esul s. This is a le el ha is much lowe han he 90 o 95% cu -o o en o en used as a
s anda d o a esul o be conside ed eliable.
The s udies we e iewed in §1 sugges compu a ional ep oducibili y migh lie be ween 50 and 75%
o esul s wi hin and be ween s udies. Thus, ou inding looks p omising in his ligh . Howe e , we
need o be cau ious abou how we de ine ep oduc ion. I we de ined i a he eam-le el a he han
model-by-model, demanding ha a leas 95% o all esul s wi hin a gi en eam a e wi hou e o is a
necessa y condi ion o a ‘success ul’ eplica ion, his yields only a 64.9% success a e (see igu e 1 ).
This means ha mos eams had a leas a ew models ha ailed o eplica e. Mos eams had e o s.
Only 14 eams had 100% exac nume ical eplica ion, all om he g oup wi h anspa en access o he
o iginal code. This may owe o ano he ac o ha we unco e ed in ou quali a i e coding o he eams’
18
oyalsocie ypublishing.o g/jou nal/ sos R. Soc. Open Sci. 12: 241038
wo k lows. We disco e ed ha he e was in o ma ion in he code o he o iginal s udy ha is necessa y
o achie e an exac eplica ion ha o he wise canno be ound in he pape o supplemen a y ma e ials. This
was mos e iden in minu e de ails ela ing o he ecoding o socio-economic s a us a iables. In his
s udy, like many in he social sciences, socio-economic s a us was a key adjus men a iable, and i s
cons uc ion by he o iginal au ho s equi ed combining esponses om a ious ques ions om he
o iginal su ey da a (see able 2). I po en ial eplica o s do no ha e access o o use he same coding
language as he o iginal, i is di icul o hem o unde s and all s eps in he code. This means ha
S a a use s in he TG had he easies access o addi ional, bu c ucial, me hodological in o ma ion abou
he s udy. An a gumen o mo e anspa ency, i no clea e p esen a ion o me hods, in u u e s udies.
Wi hou e e ence o any nume ic ep oducibili y a es, ou quali a i e in es iga ion shows he e is
analy ic lexibili y, o wha Gelman & Loken [27] e e o as ‘ esea che deg ees o eedom’ leading
hem h ough a ga den o o king pa hs, e en in esea ch so na ow as a compu a ional ep oduc ion.
This su ely is he case in he TG who we e p o ided wi h he o iginal analy ical code. Ga y King
[28] e e s o p o ision o o iginal code as he mos elegan way o engage in ep oducible esea ch.
Ou esul s demons a e ha uly elegan code should be well-s uc u ed, well-documen ed and
comp ehensible o esea che s who do no use he so wa e, bu e en so, migh s ill lea e eplica o s
subjec o unce ain y. Ou esul s sugges ha his unce ain y may be p ocedu al, as in, idiosync a ic
o he esea che s o hei esea ch p ocesses. We obse ed unce ain y in esul s ha we e no based
on conscious analy ical decisions o mis akes bu occu ed as he eams engaged in hei s anda d
idiosync a ic esea ch p ac ices and p e iously lea ned so wa e ou ines.
The e a e h ee clea implica ions o ou indings. The i s is ha anspa ency is c i ical o
inc easing he eliabili y o science. We unde s and science o mean epea ed es ing o scien i ic
claims o o m a se o esul s ha can be us ed as communica ing accu a e in o ma ion abou
he wo ld [29]. Like many social and beha iou al science jou nals, we did no con ol he esea ch
p ocess o esea che s. We asked hem o do wo k as hey no mally do and hen submi hei indings.
This sugges s ha ins i u ions, jou nals and eache s should place much highe quali y demands on
scien is s, because hei anspa ency beha iou s a e a om ideal [10,30,31]. Ou indings unde sco e
ha we would immedia ely make social and beha iou al science a mo e eliable: i all jou nals no
only equi ed bu checked code. Today some jou nals hi e hi d pa ies o p o ide code checking, bu
no all jou nals a e in a inancial posi ion o do his. The e o e, we sugges ha one e iewe , o elec ed
pe son pe pape , p o ides a compu a ional ep oducibili y check, simila o wha he AJPS s a ed in
2015 and wha Psychological Science ecen ly adop ed [32].
Second, we conclude ha anspa ency is no a cu e-all. I he quali y o anspa en ma e ials
is lowe , esea che s migh be o ced o make assump ions ha change he esul s. This was clea
om ou quali a i e in es iga ion whe e eams lacking speci ic s ep-by-s ep ins uc ions abou how
o ecode a iables made di e en ‘guesses’ along he way. Mo eo e , he e is an assump ion made
by many esea che s ha he gold s anda d o anspa ency should be a ‘push-bu on’ eplica ion.
Ou own esea ch and knowledge gained om his s udy sugges ha his is an o e simpli ica ion.
I a esea che does no know how o use he so wa e needed o a ‘push-bu on’ eplica ion, hey
canno un i . Fo example, a uly push-bu on eplica ion migh equi e ins alla ion o Py hon, R, S a a
and/o Jupy e No ebooks, plus he skills o ge hem wo king p ope ly wi h one ano he on any gi en
ope a ing sys em and e sion; a good example o push-bu on eplica ion ha equi es such skills and
ins alla ions is he Jupy e No ebook o a s udy by Connelly & Gayle, which is pe ec ly ep oducible bu
only ia mul iple so wa e ins alla ions and he knowledge equi ed o use hem [33]. Also, di e ences
be ween ope a ing sys ems, packages and p ocesso s can p oduce di e en esul s [34] like wi h he
di e en S a a e sus R ounding de aul s. In his s udy, eplica ion eams submi ed hei esul s wi h
di e en numbe s o decimals, and his may ha e been a p oduc o so wa e o package de aul s.
De aul se ings o en change ac oss e sions o so wa e packages. Thus, he ‘ aci knowledge’ ha
esea che s equi e o execu e hei s udies ‘canno be ully explica ed o absolu ely es ablished’ in
p ac ice [35].
Rela ed o his poin , i he da a canno be sha ed and mus be sough ou by he eplica o , his
in oduces po en ial e o in a supposed ‘push-bu on’ eplica ion. Da a in eposi o ies o en change
o e ime, and occasionally a chi is s make hese changes wi hou e sion con ol [36]. Remedies
o his poin a e a leas wo old. Imp o ed eaching o he many po en ial pi alls in scien i ic
ep oducibili y and eplica ion as a me hodology, e en as ea ly as du ing unde g adua e s udies,
would educe e o s and he pe cei ed di icul y o ep oduc ion asks [37]. I would bo h inc ease
a en ion o de ail and awa eness o key aspec s o p oducing anspa en and eliable wo k [1,38].
Expec ing knowledge o i ual compu ing en i onmen s migh be o e ly op imis ic, bu a leas
19
oyalsocie ypublishing.o g/jou nal/ sos R. Soc. Open Sci. 12: 241038
sha ing o en i onmen and dependencies wi hin he so wa e (some imes known as a ‘colophon’
o ‘session in o’) heo e ically gi es all necessa y in o ma ion o emo e in a-so wa e and package
a ia ion in esul s. This could be pa o eaching p og ams o cu ing-edge analy ical pipelines and
eplicable esea ch [26,39]. In ai ness, many jou nals a e g appling wi h his exac issue and can
be commended o pushing o adop code-sha ing and ideally push-bu on eplica ions [40,41]. No
only a e anspa ency and eplica ion policies di icul o de elop gi en he challenges in c ea ing
some hing like a push bu on eplica ion, bu cla i ying and en o cing hem adds an addi ional laye
o he p oblem [42]. We ce ainly suppo e o s o p oduce push-bu on eplica ions as hey can be
checked by a leas hose wi h equisi e knowledge, o example, how o implemen olde e sions o
so wa e and packages. An al e na i e solu ion is ha schola s hemsel es can p oduce ep oducible
esea ch ia a hi d-pa y pla o m such as Colab o Code Ocean, whe e use s can push-bu on un he
code in he i ual compu ing en i onmen . These also can ha e a DOI o make o easy linking. The e
a e no quali y con ols o gua an ees ha hese a e long- e m iable, bu hey a e an excellen op ion
when he e a e no o he al e na i es.
Finally, ou s udy implies ha we should be humble and mo e cau ious in ou communica ion o
science han cu en ly p ac ised. Social and beha iou al science, like all science, may no be as eliable as
p e iously hough . We ha e shown he e ha his may be ue e en in a ask as supposedly decision- ee as
compu a ional ep oduc ion. Looking back a ou simula ion in igu e 3, we will likely o en need mul iple
eplica ions o achie e eliable knowledge p oduc ion, which in many con ex s will no be a iable op ion.
This need no con ibu e o a ‘c isis’ na a i e bu a he a eminde o con inuously imp o e ou wo k and
he ins i u ions suppo ing i , in pa icula he jou nals, hei policies and he le el o en o cemen . I is, a e
all, ou job as scien is s o measu e and communica e unce ain y. The e o e and none heless, eliance on
a single model, o epo ing esul s as de ini i e o absolu ely u h ul based on a single ep oduc ion is
i esponsible, misleading and con a y o he scien i ic me hod.
He e ou s udy links o analogous e idence o in e - esea che a iabili y when esea che s conduc
simila o iginal esea ch asks as seen in ‘many analys s’ and mul i-analys s udies [14,43–46]. I is
di icul o es how well hese indings gene alize beyond simple compu a ional ep oduc ion, gi en
he challenge o ob aining a eliable p io p obabili y o coming o a co ec esul in any gi en s udy.
We do no know i an o iginal s udy is ‘co ec ’, mis ake- ee o using he mos heo e ically plausible
da a-gene a ing models [47]. P edic ion ma ke s o z-cu es a e sugges ed op ions o es ima e plausible
expec ed eplicabili y a es [48,49], bu any a emp o iden i y a ‘ ue’ eplicabili y a e can quickly
dig ess in o a philosophical deba e ega ding he na u e o u h.
Ou s udy is no wi hou limi a ions. I is possible ha he peculia i ies o a ask in ol ing ISSP da a
wi h a 10-ca ego y employmen a iable and a 7-ca ego y educa ion a iable (a leas in he 1996 wa e)
made esea che s especially p one o p ocedu al a iabili y. Howe e , we no e ha mac o-compa a i e
social scien is s do a g ea deal o su ey-based esea ch, and mos su eys gene a e da a on ISCO
codes, educa ion ca ego ies ( ha o en a y by coun y) and se e al labou ma ke s a uses ha a e
no always consis en (like esponden s epo ing being ‘unemployed’ in one ques ion and wo king
‘pa - ime’ in ano he ). Mo eo e , a s udy wi h odds a ios as an es imand may ha e peculia i ies.
Logis ic eg ession in ol es ans o ma ion om linea o logi , and hen again he coe icien s a e
ans o med in o odds a ios. This lea es mo e s eps o e o . Mo eo e , logis ic eg ession is i e a i e
a he han de ini i e, and he e a e a ious ways ha a esea che can engage in i e a i ely a i ing a
he bes unde lying linea combina ion, unlike o dina y leas squa es.
We a e also limi ed in ou abili y o d aw a popula ion in e ence om ou sample. The e is no
s aigh o wa d way o de ine he global popula ion o po en ial eplica o s. Ou sample-N a he
eam le el spli in o wo g oups does no gi e subs an ial s a is ical powe , only a he e ec le el as
p e- egis e ed; he e o e, we mus be cau ious abou in e ence a his le el. We in i ed anyone who
has wo king knowledge o mul i-le el modelling and expe ience wi h su ey and coun y-indica o
da a o join, ye no all had expe ience in he opical a ea and some may no ha e accu a ely ead o
me he quali ica ions. Follow-up s udies should conside limi ing pa icipa ion o expe s on he opic,
a pa icula discipline o o he c i e ia designed o elimina e addi ional noise likely gene a ed in ou
e o o measu e in e - esea che eliabili y.
In ou s udy, he opaque g oup a emp ed o eplica e unde excep ionally in anspa en condi ions,
wi hou code and wi hou e en nume ical esul s. I hus does no come as a su p ise ha his g oup
was a less likely o ep oduce he o iginal s udy. This is an ex eme case, and we do no expec
s udies o be published wi hou hei nume ical esul s. Ye i is ai in he sense ha many s udies
o e oo no ed ‘addi ional analyses’ which o en suppo he indings o he main analyses wi hou
nume ical e idence. Mo eo e , we know ha esea che s migh epo alse nume ical esul s [50,51],
20
oyalsocie ypublishing.o g/jou nal/ sos R. Soc. Open Sci. 12: 241038
which is a case whe e any eplica ion a emp wi hou also ha ing he code is essen ially a new s udy.
Ou s udy makes loud and clea he ac ha when o iginal code is no a ailable, eplica ion e o
a es inc ease ma kedly. T anspa ency is a low-cos al e na i e o la ge -scale me hods o ‘s abilizing’
es ima e unce ain y like he c owdsou ced eplica ion we conduc ed o e en la ge Me ake a e o s
[52]. I hei goal is he e icien and eliable p oduc ion o collec i e knowledge, a social scien is
should no need e hical ules o en o cemen mechanisms o wan o gene a e and sha e high quali y
code. Thei mo i a ion should only inc ease when in o med abou he cos s o e o s and po en ially
alse claims agains hei wo k, no o men ion eliabili y. This same logic applies o social science
jou nals when w i ing up and en o cing anspa ency policies. I is ue o all o science.
E hics. Al hough he e we e human subjec s in ol ed in his esea ch, no e hical commi ee o o e sigh was
necessa y. The subjec s we e collabo a i e co-au ho s pe o ming he same asks ha hey do in hei s anda d wo k
as academics. The goals and collabo a i e na u e o he p ojec we e cla i ied o all pa icipan s in ad ance, and hey
eely ag eed o p o ide esea ch wo k in exchange o co-au ho ship igh s. They did no ag ee o non-anonymous
code sha ing, so we ha e edac ed all iden i ying ea u es.
Da a accessibili y. We p o ide all da a and wo k low on Gi Hub. Fo ou las submission, he au ho s had p oblems
accessing Zenodo; he e o e, o ensu e scien i ic eliabili y, we can no longe us Zenodo in he pee e iew
p ocess. He e is ou Gi Hub link o ou en i e ep oducible eposi o y [53].
Supplemen a y ma e ial is a ailable online [54].
Decla a ion o AI use. We ha e no used AI-assis ed echnologies in c ea ing his a icle.
Au ho s’ con ibu ions. N.B.: concep ualiza ion, da a cu a ion, o mal analysis, in es iga ion, me hodology, p ojec
adminis a ion, so wa e, esou ces, supe ision, alida ion, isualiza ion, w i ing—o iginal d a , w i ing— e iew
and edi ing; E.M.R.: concep ualiza ion, in es iga ion, me hodology, p ojec adminis a ion, supe ision, alida ion,
isualiza ion, w i ing— e iew and edi ing; A.W.: concep ualiza ion, in es iga ion, me hodology, p ojec adminis-
a ion, supe ision, alida ion, isualiza ion, w i ing— e iew and edi ing; M.A.: o mal analysis; E.A.: o mal
analysis; A.A.-B.: o mal analysis; H.K.A.: o mal analysis; D.A.: o mal analysis; F.A.: o mal analysis; O.B.: o mal
analysis; L.B.: o mal analysis; D.B.: o mal analysis; P.C.B.: o mal analysis; G.B.: o mal analysis; M.B.: o mal
analysis; S.B.: o mal analysis; V.B.: o mal analysis; J.B.: o mal analysis; C.B.: o mal analysis; A.B.: o mal analysis;
F.S.B.: o mal analysis; T.B.: o mal analysis; K.B.: o mal analysis; J.N.B.: o mal analysis; L.B.: o mal analysis;
A.B.: o mal analysis; T.B.: o mal analysis; A.B.: o mal analysis; Z.B.: o mal analysis; K.B.: o mal analysis; K.B.:
o mal analysis; K.B.: o mal analysis; J.-C.C.: o mal analysis; N.C.: o mal analysis; P.C.: o mal analysis; R.C.:
o mal analysis; C.S.C.: o mal analysis; E.D.: o mal analysis; E.AdR.: o mal analysis; A.E.: o mal analysis; A.E.:
o mal analysis; C.E.: o mal analysis; M.A.E.: o mal analysis; S.E.: o mal analysis; A.F.: o mal analysis; A.F.:
o mal analysis; D.F.: o mal analysis; C.G.: o mal analysis; K.G.: o mal analysis; V.G.: o mal analysis; T.G.: o mal
analysis; T.G.: o mal analysis; A.G.: o mal analysis; M.G.: o mal analysis; M.G.: o mal analysis; S.G.: o mal
analysis; T.G.: o mal analysis; A.H.: o mal analysis; V.H.: o mal analysis; J.P.H.: o mal analysis; S.H.: o mal
analysis; S.H.: o mal analysis; M.H.: o mal analysis; M.H.: o mal analysis; O.H.: o mal analysis; J.H.H.: o mal
analysis; A.H.: o mal analysis; S.H.: o mal analysis; C.H.: o mal analysis; N.H.-S.: o mal analysis; Z.S.I.: o mal
analysis; S.I.: o mal analysis; L.J.: o mal analysis; J.J.: o mal analysis; B.J.: o mal analysis; S.J.: o mal analysis;
N.J.: o mal analysis; J.K.: o mal analysis; M.K.: o mal analysis; S.K.: o mal analysis; S.K.: o mal analysis; M.K.:
o mal analysis; J.K.: o mal analysis; J.-P.K.: o mal analysis; M.K.: o mal analysis; J.K.: o mal analysis; K.K.:
o mal analysis; D.K.S.: o mal analysis; A.L.: o mal analysis; R.C.L.: o mal analysis; P.M.L.: o mal analysis; D.L.:
o mal analysis; L.-M.L.: o mal analysis; P.L.: o mal analysis; M.M.: o mal analysis; J.E.M.: o mal analysis; N.M.:
o mal analysis; L.M.: o mal analysis; H.M.: o mal analysis; N.M.: o mal analysis; P.M.: o mal analysis; J.M.:
o mal analysis; O.J.M.: o mal analysis; R.McD.: o mal analysis; P.McM.: o mal analysis; K.McW.: o mal analysis;
C.M.: o mal analysis; D.M.: o mal analysis; J.M.: o mal analysis; F.M.: o mal analysis; S.M.: o mal analysis;
D.M.: o mal analysis; L.M.: o mal analysis; J.M.: o mal analysis; C.M.: o mal analysis; M.N.: o mal analysis;
D.N.: o mal analysis, da a cu a ion; O.N.: o mal analysis; F.O.: o mal analysis; G.O.: o mal analysis; A.P.: o mal
analysis; M.P.: o mal analysis; C.P.: o mal analysis; L.R.: o mal analysis; K.R.: o mal analysis; M.R.: o mal
analysis; F.R.: o mal analysis; A.R.: o mal analysis; J.R.: o mal analysis; G.R.: o mal analysis; R.S.: o mal analysis;
G.S.: o mal analysis; C.S.P.: o mal analysis; A.S.: o mal analysis; M.S.: o mal analysis; D.S.: o mal analysis;
E.S.: o mal analysis; K.S.: o mal analysis; R.S.: o mal analysis; A.S.-C.: o mal analysis; C.S.: o mal analysis;
J.S.: o mal analysis; M.S.: o mal analysis; J.S.-C.: o mal analysis; S.S.: o mal analysis; R.S.: o mal analysis; J.S.:
o mal analysis; H.S.: o mal analysis; W.S.: o mal analysis; N.S.: o mal analysis; A.S.: o mal analysis; N.S.:
o mal analysis; N.D.S.: o mal analysis; S.S.: o mal analysis; D.S.: o mal analysis; D.S.: o mal analysis; N.S.:
o mal analysis; E.S.: o mal analysis; A.-K.S.: o mal analysis; J.W.S.: o mal analysis; J.T.: o mal analysis; A.T.:
o mal analysis; B.T.: o mal analysis; G.V.: o mal analysis; J.V.A.: o mal analysis; M dL.: o mal analysis; J dN.:
o mal analysis; A.V.H.: o mal analysis; S.V.: o mal analysis; B.V.: o mal analysis; F.W.: o mal analysis; N.W.:
o mal analysis; H.W.: o mal analysis; B.M.W.: o mal analysis; F.W.: o mal analysis; C.W.: o mal analysis; C.W.:
o mal analysis; Y.Y.: o mal analysis; B.Z.: o mal analysis; N.Z.: o mal analysis; C.Z.: o mal analysis; S.Z.: o mal
analysis; TŻ.: o mal analysis; H.H.V.N.: da a cu a ion, isualiza ion, me hodology.
All au ho s ga e inal app o al o publica ion and ag eed o be held accoun able o he wo k pe o med
he ein.
Con lic o in e es decla a ion. We decla e we ha e no compe ing in e es s.
Funding. This esea ch was unded by a g an om he Ge man Science Founda ion – Deu sche
Fo schungsgemeinscha (DFG) – P ojec Numbe 464546557, and om he Chilean Na ional Resea ch and
De elopmen Agency – Agencia Nacional de In es igación y Desa ollo – ANID/FONDAP/1523A0005. We a e
21
oyalsocie ypublishing.o g/jou nal/ sos R. Soc. Open Sci. 12: 241038
also g a e ul o he Mannheim Cen e o Eu opean Social Resea ch, Ge many o suppo ing he Open Science
Con e ence 2018 ha buil he idea o his p ojec .
Re e ences
1. Willis C, S odden V. 2020 T us bu Ve i y: How o Le e age Policies, Wo k lows, and In as uc u e o Ensu e Compu a ional Rep oducibili y in
Publica ion. Ha . Da a Sci. Re . 2. (doi:10.1162/99608 92.25982dc )
2. Pa sons S. 2022 A communi y-sou ced glossa y o open schola ship e ms. Na . Hum. Beha . 6, 312–318. (doi:%2010.1038/s41562-021-01269-
4)
3. Auspu g K, B üde l J. 2021 Has he C edibili y o he Social Sciences Been C edibly Des oyed? Reanalyzing he ‘Many Analys s, One Da a Se ’
P ojec . Socius. Sociol. Res. Dyn. Wo ld. 7, 237802312110244. (doi:10.1177/23780231211024421)
4. Lundbe g I, Johnson R, S ewa BM. 2021 Wha Is You Es imand? De ining he Ta ge Quan i y Connec s S a is ical E idence o Theo y. Am.
Sociol. Re . 86, 532–565. (doi:10.1177/00031224211004187)
5. Bake M. 2016 1,500 scien is s li he lid on ep oducibili y. Na . New Biol. 533, 452–454. (doi:10.1038/533452a)
6. Fanelli D. 2018 Is science eally acing a ep oducibili y c isis, and do we need i o? P oc. Na l Acad. Sci. USA 115, 2628 – 2631. (doi:10.1073/
pnas.1708272114)
7. Sh ou PE, Rodge s JL. 2018 Psychology, Science, and Knowledge Cons uc ion: B oadening Pe spec i es om he Replica ion C isis. Annu. Re .
Psychol. 69, 487 – 510. (doi:10.1146/annu e -psych-122216-011845)
8. Khan N, Thelwall M, Kousha K. 2023 Da a sha ing and euse p ac ices: disciplina y di e ences and imp o emen s needed. Online In . Re . 47,
1036 – 1064. (doi:10.1108/oi -08-2021-0423)
9. Tede soo L e al. 2021 Da a sha ing p ac ices and da a a ailabili y upon eques di e ac oss scien i ic disciplines. Sci. Da a 8, 192. (doi:10.1038/
s41597-021-00981-0)
10. K ähme D, Schäch ele L, Schneck A. 2023 Ca e o sha e? Expe imen al e idence on code sha ing beha io in he social sciences. PloS One 18,
e0289380. (doi:10.1371/jou nal.pone.0289380)
11. Thoege sen JL, Bo lund P. 2022 Resea che a i udes owa d da a sha ing in public da a eposi o ies: a me a-e alua ion o s udies on esea che
da a sha ing. J. Doc. 78, 1 – 17. (doi:10.1108/jd-01-2021-0015)
12. Eubank N. 2016 Lessons om a Decade o Replica ions a he Qua e ly Jou nal o Poli ical Science. PS 49, 273 – 276. (doi:10.1017/
s1049096516000196)
13. T iso ic A, Lau MK, Pasquie T, C osas M. 2022 A la ge-scale s udy on esea ch code quali y and execu ion. Sci. Da a 9, 60. (doi:10.1038/s41597-
022-01143-6)
14. B eznau N e al. 2022 Obse ing many esea che s using he same da a and hypo hesis e eals a hidden uni e se o unce ain y. P oc. Na l Acad.
Sci. USA 119, e2203150119. (doi:10.1073/pnas.2203150119)
15. Jacoby WG, La e y-Hess S, Ch is ian TM. 2017 Should Jou nals Be Responsible o Rep oducibili y? | Inside Highe . ( ed. EHE Blog ), See h ps://
www.insidehighe ed.com/blogs/ e hinking- esea ch/should-jou nals-be- esponsible- ep oducibili y.
16. Janz N. 2015 Leading Jou nal Ve i ies A icles be o e Publica ion – So Fa , All Replica ions Failed. Poli . Sci. Replic. Blog h ps://
poli icalscience eplica ion.wo dp ess.com/2015/05/04/leading-jou nal- e i ies-a icles-be o e-publica ion-so- a -all- eplica ions- ailed/
17. Ha dwicke TE e al. 2018 Da a a ailabili y, eusabili y, and analy ic ep oducibili y: e alua ing he impac o a manda o y open da a policy a he
jou nal Cogni ion. R. Soc. Open Sci. 5, 180448. (doi:10.1098/ sos.180448)
18. S ockeme D, Koehle S, Len z T. 2018 Da a Access, T anspa ency, and Replica ion: New Insigh s om he Poli ical Beha io Li e a u e. PS 51, 799
– 803. (doi:10.1017/s1049096518000926)
19. Pé ignon C, Akmansoy O, Hu lin C, D ebe A, Holzmeis e F, Hube J. Compu a ional Rep oducibili y in Finance: E idence om 1,000 Tes s 2023.
(doi:10.2139/ss n.4064172). See h ps://doi.o g/10.2139/ss n.4064172.
20. Liu DM, Salganik MJ. 2019 Successes and S uggles wi h Compu a ional Rep oducibili y: Lessons om he F agile Families Challenge. Socius 5.
(doi:10.1177/2378023119849803)
21. B eznau N, Rinke EM, Wu ke A. 2019 C owdsou ced eplica ion ini ia i e: execu i e epo . soca xi . Mannheim Cen e o Eu opean Social
Resea ch. See h ps://os .io/p ep in s/soca xi /6j9qb.
22. B ady D, Finnigan R. 2014 Does Immig a ion Unde mine Public Suppo o Social Policy? Am. Sociol. Re . 79, 17 – 42. (doi:10.1177/
0003122413513022)
23. B eznau N, Rinke EM, Wu ke A. 2018 P e-Regis e ed Repo o ‘How Reliable A e Replica ions? Measu ing Rou ine Resea che Va iabili y in
Mac o-Compa a i e Seconda y Da a Analyses’ OSF Regis ies. h ps://os .io/s uq3
24. F eese J, Pe e son D. 2017 Replica ion in Social Science. Annu. Re . Sociol. 43, 147 – 165. (doi:10.1146/annu e -soc-060116-053450)
25. Schoch D, Chan C, Wagne C, Bleie A. Compu a ional Rep oducibili y in Compu a ional Social Science 2023. (doi:10.48550/a Xi .2307.01918). See
h ps://doi.o g/10.48550/a Xi .2307.01918.
26. Baue G, B eznau N, Ge eke J, Hö le JH, Janz N, Rahal RM, Renns ich JK, Soiné H. 2023 Teaching Cons uc i e Replica ions in he Beha io al and
Social Sciences Using Quan i a i e Da a. Teach. Psychol. 00986283231219503. (doi:10.1177/00986283231219503)
27. Gelman A, Loken E. 2014 The S a is ical C isis in Science. Am. Sci. 102, 460. (doi:10.1511/2014.111.460)
22
oyalsocie ypublishing.o g/jou nal/ sos R. Soc. Open Sci. 12: 241038
28. King G. 1995 Replica ion, Replica ion. PS (Wash DC) 28, 444 – 452. (doi:10.2307/420301)
29. Me on RK. 1973 The sociology o science: heo e ical and empi ical in es iga ions. Chicago, IL : Uni e si y o Chicago P ess.
30. F eese J. 2007 Replica ion S anda ds o Quan i a i e Social Science. Sociol. Me hods Res. 36, 153 – 172. (doi:10.1177/0049124107306659)
31. Zenk-Möl gen W, Akdeniz E, Ka sanidou A, Naßho en V, Balaban E. 2018 Fac o s in luencing he da a sha ing beha io o esea che s in
sociology and poli ical science. J. Doc. 74, 1053 – 1073. (doi:10.1108/jd-09-2017-0126)
32. Ha dwicke TE, Vazi e S. 2023 T anspa ency Is Now he De aul a Psychological Science. Psychol. Sci. 9567976231221573. (doi:10.1177/
09567976231221573)
33. Connelly R, Gayle V. 2019 An in es iga ion o social class inequali ies in gene al cogni i e abili y in wo B i ish bi h coho s. B . J. Sociol. 70, 90
– 108. (doi:10.1111/1468-4446.12343)
34. McCoach DB, Ri enba k GG, New on SD, Li X, Kooken J, Yom o D, Gambino AJ, Bella a A. 2018 Does he Package Ma e ? A Compa ison o Fi e
Common Mul ile el Modeling So wa e Packages. J. Educ. Beha . S a . 43, 594 – 627. (doi:10.3102/1076998618776348)
35. Collins HM. 1985 Changing o de : eplica ion and induc ion in scien i ic p ac ice. London, Be e ly Hills & New Delhi : Sage Publica ions.
36. B eznau N. 2016 Seconda y obse e e ec s: idiosync a ic e o s in small-N seconda y da a analysis. In . J. Soc. Res. Me hodol. 19, 301 – 318.
(doi:10.1080/13645579.2014.1001221)
37. Vilhube L, Son HH, Welch M, Wasse DN, Da isse M. 2022 Teaching o La ge-Scale Rep oducibili y Ve i ica ion. J. S a . Da a Sci. Educ. 30, 274 –
281. (doi:10.1080/26939169.2022.2074582)
38. S ojmeno ska D, Bol T, Leopold T. 2019 Teaching Replica ion o G adua e S uden s. Teach. Sociol. 47, 303 – 313. (doi:10.1177/
0092055x19867996)
39. Chan CH, Schoch D. 2023 ang: Recons uc ing ep oducible R compu a ional en i onmen s. PloS One 18, e0286761. (doi:10.1371/jou nal.pone.
0286761)
40. Hö le JH. 2017 Replica ion and Economics Jou nal Policies. Am. Econ. Re . 107, 52 – 55. (doi:10.1257/ae .p20171032)
41. Vilhube L. 2020 Rep oducibili y and Replicabili y in Economics. Issue 2 4 Fall 2020 Ha . Da a Sci. Re . 2. (doi:10.1162/99608 92.4 6b9e67)
42. Ch is ian TM, Gooch A, Vision T, Hull E. 2020 Jou nal da a policies: Explo ing how he unde s anding o edi o s and au ho s co esponds o he
policies hemsel es. PloS One 15, e0230281. (doi:10.1371/jou nal.pone.0230281)
43. Bas iaansen JA e al. 2020 Time o ge pe sonal? The impac o esea che s choices on he selec ion o ea men a ge s using he expe ience
sampling me hodology. J. Psychosom. Res. 137, 110211. (doi:10.1016/j.jpsycho es.2020.110211)
44. Du ilh G e al. 2019 The Quali y o Response Time Da a In e ence: A Blinded, Collabo a i e Assessmen o he Validi y o Cogni i e Models.
Psychon. Bull. Re . 26, 1051 – 1069. (doi:10.3758/s13423-017-1417-2)
45. Landy JF e al. 2020 C owdsou cing hypo hesis es s: Making anspa en how design choices shape esea ch esul s. Psychol. Bull. 146, 451 –
479. (doi:10.1037/bul0000220)
46. Silbe zahn R e al. 2018 Many Analys s, One Da a Se : Making T anspa en How Va ia ions in Analy ic Choices A ec Resul s. Ad . Me hods P ac .
Psychol. Sci. 1, 337 – 356. (doi:10.1177/2515245917747646 )
47. Young C. 2018 Model Unce ain y and he C isis in Science. Socius 4. (doi:10.1177/2378023117737206)
48. D ebe A, P ei e T, Almenbe g J, Isaksson S, Wilson B, Chen Y, Nosek BA, Johannesson M. 2015 Using p edic ion ma ke s o es ima e he
ep oducibili y o scien i ic esea ch. P oc. Na l Acad. Sci. US A 112, 15343 – 15347. (doi:10.1073/pnas.1516179112)
49. Schimmack U. 2020 A me a-psychological pe spec i e on he decade o eplica ion ailu es in social psychology. Can. Psychol. / Psychol. Can. 61,
364 – 376. (doi:10.1037/cap0000246)
50. Nuij en MB, Ha ge ink CHJ, an Assen MALM, Epskamp S, Wiche s JM. 2016 The p e alence o s a is ical epo ing e o s in psychology (1985–
2013). Beha . Res. Me hods 48, 1205 – 1226. (doi:10.3758/s13428-015-0664-2)
51. Picke JT. 2020 The S ewa Re ac ions: A Quan i a i e and Quali a i e Analysis. Econ J. Wa ch 17, 152.
52. Dunning T, G ossman G, Humph eys M, Hyde SD, McIn osh C. 2019 In o ma ion, accoun abili y, and cumula i e lea ning. (ed. G Nellis).
Camb idge Uni e si y P ess. (doi:%2010.1017/9781108381390)
53. B eznau N, Rinke EM, Wu ke A. 2024 The Reliabili y o Compu a ional Replica ions. See h ps://gi hub.com/nb eznau/how_many_ eplica o s.
54. B eznau N, Wu ke A, Rinke EM, Adem M, Ad iaans J, Akdeniz E. 2025 Supplemen a y Ma e ial om: The Reliabili y o Replica ions: A S udy in
Compu a ional Rep oduc ions. Figsha e. (doi:10.6084/m9. igsha e.c.7655134)
23
oyalsocie ypublishing.o g/jou nal/ sos R. Soc. Open Sci. 12: 241038
