Innovation and forward-thinking are needed to improve traditional synthesis methods: A response to Pescott and Stewart

This documen is he Accep ed Manusc ip e sion o a Published Wo k ha appea ed in inal o m in: Ch is ie, A.P.; Amano, T.; Ma in, P.A.; Shackel o d, G.E.; Simmons, B.I.; Su he land, W.J.2022. Plu al alua ion o na u e o equi y and sus ainabili y: Insigh s om he Global Sou h. Jou nal o Applied Ecology. 59. DOI (10.1111/1365-2664.14154). © 2022 B i ish Ecological Socie y. This manusc ip e sion is made a ailable unde he CC-BY-NC-ND 3.0 license h p://c ea i ecommons.o g/licenses/by-nc-nd/3.0/ Inno a ion and o wa d- hinking a e needed o imp o e adi ional 1 syn hesis me hods: a esponse o Pesco & S ewa 2 3 Alec P. Ch is ie1,4,7*, Ta suya Amano1,2,3, Philip A. Ma in1,4,8, Go m E. Shackel o d1,4,4 Benno I. Simmons1,5,6, William J. Su he land1,4 5 1Conse a ion Science G oup, Depa men o Zoology, Uni e si y o Camb idge, The Da id 6 A enbo ough Building, Downing S ee , Camb idge, UK. 7 2Cen e o he S udy o Exis en ial Risk, Uni e si y o Camb idge, 16 Mill Lane, Camb idge, UK. 8 3School o Biological Sciences, Uni e si y o Queensland, B isbane, 4072 Queensland, Aus alia 9 4BioRISC, S Ca ha ine’s College, Camb idge, UK. 10 5Depa men o Animal and Plan Sciences, Uni e si y o She ield, She ield, UK. 11 6Cen e o Ecology and Conse a ion, College o Li e and En i onmen al Sciences, Uni e si y o 12 Exe e , Pen yn, UK. 13 7Downing College, Regen S ee , Camb idge, UK. 14 8Basque Cen e o Clima e Change (BC3), Edi icio sede no 1, plan a 1, Pa que cien í ico 15 UPV/EHU, Ba io Sa iena s/n, 48940, Leioa, Bizkaia, Spain16 17 *Co esponding au ho , a[email p o ec ed].uk18 19 20 21 22 23 Abs ac 24 1. In Ch is ie e al. (2019), we used simula ions o quan i a i ely compa e he bias o 25 commonly used s udy designs in ecology and conse a ion. Based on hese simula ions, 26 we p oposed ‘accu acy weigh s’ as a po en ial way o accoun o s udy design alidi y in 27 me a-analy ic weigh ing me hods. Pesco & S ewa (2021) aised conce ns ha hese 28 weigh s may no be gene alisable and s ill lead o biased me a-es ima es. He e we 29 espond o hei conce ns and demons a e why de eloping al e na i e weigh ing 30 me hods is key o he u u e o e idence syn hesis. 31 2. We acknowledge ha ou simple simula ion un ai ly penalised Randomised Con olled 32 T ial (RCT) ela i e o Be o e-A e Con ol-Impac (BACI) designs as we assumed ha 33 he pa allel ends assump ion held o BACI designs. We poin o an empi ical ollow-up 34 s udy in which we mo e ai ly quan i y di e ences in biases be ween di e en s udy 35 designs. Howe e , we s and by ou main indings ha Be o e-A e (BA), Con ol-Impac 36 (CI), and A e designs a e quan i iably mo e biased han BACI and RCT designs. We 37 also emphasise ha ou 'accu acy weigh ing’ me hod was p elimina y and welcome 38 u u e esea ch o inco po a e mo e dimensions o s udy quali y. 39 3. We u he show ha o e a decade o ad ances in quali y e ec modelling, which 40 Pesco & S ewa (2021) omi , highligh s he impo ance o esea ch such as ou s in 41 be e unde s anding how o quan i a i ely in eg a e da a on s udy quali y di ec ly in o 42 me a-analyses. We u he a gue ha he adi ional me hods ad oca ed o by Pesco & 43 S ewa (2021) (e.g., manual isk-o -bias assessmen s and in e se- a iance weigh ing) 44 a e subjec i e, was e ul, and po en ially biased hemsel es. They also lack scalabili y o 45 use in la ge syn heses ha keep up- o-da e wi h he apidly g owing scien i ic li e a u e. 46 4. Syn hesis and applica ions. We sugges , con a y o Pesco & S ewa ’s na a i e, ha 47 mo ing owa ds al e na i e weigh ing me hods is key o u u e-p oo ing e idence 48 syn hesis h ough g ea e au oma ion, lexibili y, and upda ing o espond o decision- 49 make s needs – pa icula ly in c isis disciplines in conse a ion science whe e 50 p oblema ic biases and a iabili y exis in s udy designs, con ex s, and me ics used. 51 Whils we mus be cau ious o a oid misin o ming decision-make s, his should no s op 52 us in es iga ing al e na i e weigh ing me hods ha in eg a e s udy quali y da a di ec ly 53 in o me a-analyses. To eliably and p agma ically in o m decision-make s wi h science, 54 we need e icien , scalable, eadily au oma ed, and easible me hods o app aise and 55 weigh s udies o p oduce la ge-scale li ing syn heses o he u u e. 56 57 Keywo ds: e idence syn hesis, me a-analysis, dynamic me a-analysis, li ing e iews, 58 au oma ion, quali y e ec s modelling, me a-analyses, isk-o -bias, c i ical app aisal, bias 59 adjus men . 60 61 In oduc ion 62 63 Pesco & S ewa (2021) ou lined hei conce ns o e an al e na i e me hod o weigh ing in 64 me a-analysis we p oposed called “accu acy weigh s” in Ch is ie e al. (2019). These weigh s 65 we e de i ed om ou simula ion s udy ha aimed o quan i a i ely compa e he pe o mance o 66 di e en expe imen al and obse a ional s udy designs (Ch is ie e al., 2019). Thei wo majo 67 conce ns we e ha ou accu acy weigh s we e no gene alisable and ha quali y sco e 68 weigh ings, such as ou s, may s ill lead o biased es ima es in me a-analyses. He e we espond 69 o hei conce ns and discuss why we belie e al e na i e me hods o weigh ing a e cen al o he 70 u u e o e idence syn hesis. 71 72 1. Accu acy weigh s need imp o ing and combining wi h o he quali y measu es 73 74 As Pesco & S ewa sugges , we acknowledge ha ou simula ion may ha e un ai ly penalised 75 Randomised Con olled T ial (RCT) designs, depending on whe he esea che s in ecology and 76 conse a ion do ake in o accoun p e-impac sampling. Howe e , in ou expe ience, ew 77 Randomised Con olled T ials in conse a ion ake accoun o p e-impac baseline da a; his is 78 suppo ed by a ecen s udy quan i ying he use o di e en s udy designs in he en i onmen al 79 and social sciences (Ch is ie e al., 2020a). We acknowledge ha we did no discuss mo e o 80 he sho comings o Be o e-A e Con ol-Impac (BACI) designs in e ms o he bias ha can be 81 in oduced by iola ing he ‘pa allel ends’ assump ion (Dimick and Ryan, 2014; Unde wood, 82 1991; Wauchope e al., 2020). The e o e, wi h espec o compa ing BACI and RCT designs, we 83 acknowledge ou simula ion has limi a ions. 84 85 Ne e heless, ou majo mo i a ion was o demons a e he di e ence in s udy design 86 pe o mance be ween simple designs (e.g., Be o e-A e (BA), Con ol-Impac (CI), and A e 87 designs) and mo e igo ous designs (RCT and BACI). Thus, we in en ionally made ou 88 simula ion ela i ely simple o engage a wide audience o esea che s. We ha e since buil on 89 ou simula ions in Ch is ie e al. (2020a), which uses an empi ical, model-based me hodology o 90 quan i y he di e ences in bias a ec ing di e en s udy designs using aw ( a he han 91 simula ed) da a om a la ge numbe o wi hin-s udy compa isons. This mo e ai ly quan i ies he 92 bias associa ed wi h RCT e sus BACI designs by making ewe , mo e s a is ically de ensible 93 assump ions abou he ‘ ue e ec ’ ( o es ima e bias) and inhe en ly accoun s o he pa allel 94 ends assump ion ha can bias BACI designs (Ch is ie e al., 2020a). 95 96 Pesco & S ewa also sugges ou simula ion weigh s do no cap u e he ull ange o po en ial 97 sou ces o bias a ec ing s udy designs and ad ise ha assessmen s o s udy quali y should 98 closely sc u inise he de ails o speci ic s udies being summa ised (e.g., using manual isk-o - 99 bias assessmen s). In ou s udy, we speci ically acknowledged ha ou weigh s we e ela i ely 100 simple and need o be buil upon o inco po a e a wide ange o s udy quali y indica o s; we 101 ou lined possible app oaches in he u u e ha could in eg a e sco es om c i ical app aisal 102 ools ha exis o ecology and conse a ion (Mupepele e al., 2016). We a e happy o see ha 103 o he s a e building on ou wo k and in es iga ing he use o a b oade se o quali y o alidi y 104 measu es o weigh s udies in me a-analyses (e.g., Scha e al. 2021, Mupepele e al. 2021). In 105 he nex sec ions, we add ess Pesco & S ewa ’s c i icisms o weigh ing by quali y sco es and 106 discuss s a is ical ad ances in applying quali y sco e weigh ings o me a-analyses. We also 107 discuss he p oblems associa ed wi h he adi ional me hods ad oca ed o by Pesco & 108 S ewa (such as in e se- a iance weigh ing and manual isk-o -bias assessmen s). 109 110 2. Recen ad ances in di ec ly in eg a ing da a on s udy quali y in o me a- 111 analyses 112 113 In Pesco & S ewa 's discussion on why hey ad oca e agains weigh ing by quali y sco es in 114 me a-analyses, hey omi o e a decade o esea ch in epidemiology on al e na i e quali y sco e 115 weigh ing me hods ha ha e o e come many o he p oblems hey discuss (Doi, Ba end eg 116 and Mozu kewich, 2011; Doi e al., 2015a, 2015b; Doi and Thalib, 2008; Rhodes e al., 2020; 117 S one e al., 2020). In pa icula , ‘bias adjus men ’ me hods, such as quali y e ec s models, 118 ep esen an ac i e and p omising a ea o esea ch in e idence syn hesis in epidemiology (Doi, 119 Ba end eg and Mozu kewich, 2011; Doi and Thalib, 2008; Rhodes e al., 2020; S one e al., 120 2020). 121 122 C i ical app aisal is adi ionally used o desc ip i ely epo he isk o bias o di e en s udies, 123 a he han ying o quan i a i ely inco po a e hose assessmen s wi hin he analyses 124 hemsel es (Johnson, Low and MacDonald, 2015). Ins ead, ou accu acy weigh s a e ela ed o 125 he ield o ‘bias-adjus men ’ me hods which seek o di ec ly in eg a e isk-o -bias assessmen s 126 in o me a-analy ic esul s (S one e al., 2020). C i icisms o quali y sco e weigh ings ha e 127 cen e ed a ound ou majo issues: 1.) he choice o quali y scale in luences he weigh o 128 indi idual s udies; 2.) he me a-es ima e and i s con idence in e al depends on he scale; 3.) 129 he e is no eason why s udy quali y should modi y he p ecision o es ima es; and 4.) poo 130 s udies a e no excluded (S one e al., 2020). The e o e, as Pesco & S ewa also appea o 131 a gue, any bias associa ed wi h poo quali y s udies can only be educed a bes , and no 132 emo ed (S one e al., 2020). 133 134 Whils p oponen s o quali y sco e app oaches accep ed hese c i icisms and ceased hei 135 de elopmen , an al e na i e, imp o ed me hodology called ‘quali y e ec s models’ ha e 136 subsequen ly been de eloped and e ined in ecen yea s. This app oach uses a ela i e scale 137 and ‘syn he ic weigh s’ (yielding ela i e c edibili y anks o di e en s udies) ha o e came he 138 majo issues ha a ec ed quali y sco e app oaches, and has been shown o yield an es ima o 139 wi h supe io e o and co e age o con en ional es ima o s (Doi e al., 2015b, 2017). The e a e 140 a ange o possible ways, each wi h ad an ages o disad an ages, o de i e he ela i e 141 c edibili y weigh s o s udies using nume ical da a gene a ed by expe opinion (Tu ne e al., 142 2009), da a-based dis ibu ions, o s a is ically combining expe opinion and da a-based 143 dis ibu ions (Rhodes e al., 2020). The e o e, esul s om u he e ining and imp o ing ou 144 simula ions and empi ical analyses (Ch is ie e al., 2019; Ch is ie e al., 2020a) could p o ide 145 aluable con ibu ions o he ac i e de elopmen o hese me hods o in eg a e da a on s udy 146 quali y di ec ly in o me a-analyses. 147 148 Pesco & S ewa ocus on he possibili y o inco po a ing s udy quali y sco es in o me a- 149 eg ession app oaches. Thei c i icism o ou weigh s in hei cu en o m is ha hey a e oo 150 unidimensional and no s udy-speci ic; his is a c i icism ha we pa ially accep . Indeed, we 151 speci ically discussed he need o expand and imp o e ou weigh s o in eg a e o he aspec s o 152 s udy quali y (e.g., using expe opinion, da a-based dis ibu ions, o c i ical app aisal ools o 153 adjus ela i e c edibili y anks; Rhodes e al., 2020). In hindsigh , we should ha e dedica ed 154 mo e a en ion o how we would u he de elop and mo e obus ly apply ou accu acy weigh s 155 alongside discussing ad ances in quali y e ec s models. 156 157 Pesco & S ewa also sugges ha we igno e issues ela ing o ex e nal alidi y. Gi en ha 158 adi ional weigh ings, such as sample size o in e se a iance, also ail o conside ex e nal 159 alidi y, we ind his an odd c i icism, pa icula ly gi en ou simula ion was clea ly ocused on 160 add essing issues o s udy design quali y and in e nal alidi y. We a e in ac de eloping an 161 al e na i e me a-analy ic me hod, dynamic me a-analysis (Shackel o d e al., 2021), based on 162 he Me ada ase pla o m (www.me ada ase .com), which we plan o use o es di e en 163 weigh ing me hods, including ‘ ecalib a ion’ om he medical sciences (Kneale e al., 2019) 164 which aims o adjus s udies’ in luence in me a-analyses based on hei ex e nal alidi y (o 165 ele ance o decision-make s). Again his wo k is in he ea ly s ages o de elopmen and he e 166 a e many me hodological challenges o o e come, pa icula ly in how o in eg a e ‘ ecalib a ion’ 167 me hods in o andom e ec s models and how o ensu e such in e ac i e me a-analy ic ools a e 168 used obus ly (Shackel o d e al. 2021). The e o e, as Pesco & S ewa sugges , we belie e i 169 should be possible o in eg a e in e nal alidi y o quali y i ems, and ex e nal alidi y i ems, in o a 170 hie a chical me a- eg ession amewo k, o o di ec ly weigh s udies using new ad ances in 171 quali y e ec s models as discussed p e iously (see S one e al. 2020 o a compa ison and 172 discussion o di e en app oaches). 173 174 175 3. In eg a ing da a on s udy quali y in o me a-analyses is essen ial o he u u e o 176 e idence syn hesis 177 178 We also belie e Pesco & S ewa ’s discussion p esen s a na ow ision o he challenges 179 aced by adi ional c i ical app aisal and weigh ing me hods. We belie e ha he adi ional 180 ‘medical-s yle’ app oaches (e.g., manual isk-o -bias assessmen s combined wi h in e se- 181 a iance weigh ing) ha Pesco & S ewa belie e should be adhe ed o a e ul ima ely 182 ine icien and was e ul. The ield o e idence syn hesis is ad ancing a pace o espond o he 183 challenges o apidly g owing e idence bases and as -mo ing c ises, which equi es new 184 me hodologies ha help o keep e idence bases ‘up- o-da e’ o ‘li ing’, cos -e icien by wo king 185 a massi e discipline-wide scales, and dynamically adjus able o be ele an o di e en 186 decision-make s’ needs. He e we elabo a e on why his is p oblema ic o Pesco & S ewa ’s 187 asse ion ha we should con inue o ely on adi ional me hods, a he han al e na i e 188 weigh ing me hods such as he one we p oposed in Ch is ie e al. (2019). 189 190 3.a. Al e na i e weigh ing me hods acili a e mo e e icien , au oma ed, li ing, la ge-scale 191 syn heses 192 193 Fi s , he e is g owing ecogni ion ha decision make s need cons an ly upda ed e idence 194 syn heses (Ellio e al., 2021) and ha adi ional syn hesis me hods (e.g., adi ional sys ema ic 195 e iews) a e o en oo ime-consuming, quickly go ou -o -da e, and can miss impo an 196 oppo uni ies o in luence p ac ice and policy (Bou on e al., 2020; G ainge e al., 2019; 197 Haddaway and Wes ga e, 2019; Ko iche a and Kulinskaya, 2019; Nakagawa e al., 2020; 198 Pa ani um e al., 2012; Shojania e al., 2007). Gi en ha he scien i ic li e a u e in mos 199 disciplines is g owing apidly (Bo nmann and Mu z, 2015; La sen and on Ins, 2010) and ha 200 Bo ah, R., B own, A.W., Cape s, P.L. and Kaise , K.A. (2017) “Analysis o he imeand wo ke s 356 needed o conduc sys ema ic e iews o medicalin e en ions using da a om he PROSPERO 357 egis y,” BMJ, Open7:e012545. 358 Bo nmann, L. and Mu z, R. (2015) “G ow h a es o mode n science: A bibliome ic analysis 359 based on he numbe o publica ions and ci ed e e ences,” Jou nal o he Associa ion o 360 In o ma ion Science and Technology, 66(11) John Wiley & Sons, L d, pp. 2215–2222. 361 Bou on, I., C équi , P., Williams, H., Mee pohl, J., C aig, J.C. and Ra aud, P. (2020) “Fu u e o 362 e idence ecosys em se ies: 1. In oduc ion — E idence syn hesis ecosys em needs d ama ic 363 change,” Jou nal o Clinical Epidemiology, Else ie Inc. 364 Ch is ie, A.P., Abecasis, D., Adje oud, M., Alonso, J.C., Amano, T., An on, A., Baldigo, B.P., 365 Ba ien os, R., Bicknell, J.E., Buhl, D.A., Ceb ian, J., Ceia, R.S., Cibils-Ma ina, L., Cla ke, S., 366 Claude , J., C aig, M.D., Da oul , D., de Backe , A., Dono an, M.K., Eddy, T.D., F ança, F.M., 367 Ga dne , J.P.A., Ha is, B.P., Huusko, A., Jones, I.L., Kelahe , B.P., Ko iaho, J.S., López- 368 Baucells, A., Majo , H.L., Mäki-Pe äys, A., Ma ín, B., Ma ín, C.A., Ma in, P.A., Ma eos-Molina, 369 D., McConnaughey, R.A., Me oni, M., Meye , C.F.J., Mills, K., Mon e alcone, M., No eika, N., 370 Palacín, C., Pande, A., Pi che , C.R., Ponce, C., Rinella, M., Rocha, R., Ruiz-Delgado, M.C., 371 Schmi e -So o, J.J., Sha e , J.A., Sha ma, S., She , A.A., S agnol, D., S anley, T.R., 372 S okesbu y, K.D.E., To es, A., Tully, O., Vehanen, T., Wa s, C., Zhao, Q. and Su he land, W.J. 373 (2020a) “Quan i ying and add essing he p e alence and bias o s udy designs in he 374 en i onmen al and social sciences,” Na u e Communica ions, 11(1), p. 6377. 375 Ch is ie, A.P., Amano, T., Ma in, P.A., Pe o an, S.O., Shackel o d, G.E., Simmons, B.I., Smi h, 376 R.K., Williams, D.R., Wo dley, C.F.R. and Su he land, W.J. (2020b) “Poo a ailabili y o con ex - 377 speci ic e idence hampe s decision-making in conse a ion,” Biological Conse a ion, 248 Cold 378 Sp ing Ha bo Labo a o y, p. 108666. 379 Ch is ie, A.P., Amano, T., Ma in, P.A., Pe o an, S.O., Shackel o d, G.E., Simmons, B.I., Smi h, 380 R.K., Williams, D.R., Wo dley, C.F.R. and Su he land, W.J. (2020c) “The challenge o biased 381 e idence in conse a ion,” Conse a ion Biology, Wiley o Socie y o Conse a ion Biology, p. 382 cobi.13577. 383 Ch is ie, A.P., Amano, T., Ma in, P.A., Shackel o d, G.E., Simmons, B.I. and Su he land, W.J. 384 (2019) “Simple s udy designs in ecology p oduce inaccu a e es ima es o biodi e si y 385 esponses,” Jou nal o Applied Ecology, 56(12) John Wiley & Sons, L d (10.1111), pp. 2742– 386 2754. 387 Ch is ie, A.P., Whi e, T.B., Ma in, P.A., Pe o an, S.O., Bladon, A.J., Bowke , A.E., Li lewood, 388 N.A., Mupepele, A.C., Rocha, R., Sainsbu y, K.A., Smi h, R.K., Taylo , N.G., Su he land, W.J., 389 2021. Reducing publica ion delay o imp o e he e iciency and impac o conse a ion science. 390 Pee J 9, e12245. h ps://doi.o g/10.7717/PEERJ.12245/SUPP-17 391 Co n o d, R., Deine , S., de Palma, A., Hill, S.L.L., McRae, L., Pe i , B., Ma coni, V., Pu is, A. 392 and F eeman, R. (2021) “Fas , scalable, and au oma ed iden i ica ion o a icles o biodi e si y 393 and mac oecological da ase s,” Global Ecology and Biogeog aphy, 30(1) John Wiley & Sons, 394 L d, pp. 339–347. 395 Da ies, G.M. and G ay, A. (2015) “Don’ le spu ious accusa ions o pseudo eplica ion limi ou 396 abili y o lea n om na u al expe imen s (and o he messy kinds o ecological moni o ing),” 397 Ecology and E olu ion, 5(22) John Wiley & Sons, L d, pp. 5295–5304. 398 Dimick, J.B. and Ryan, A.M. (2014) “Me hods o E alua ing Changes in Heal h Ca e Policy,” 399 JAMA, 312(22), p. 2401. 400 Doi, S.A.R., Ba end eg , J.J., Khan, S., Thalib, L. and Williams, G.M. (2015a) “Ad ances in he 401 me a-analysis o he e ogeneous clinical ials II: The quali y e ec s model,” Con empo a y 402 Clinical T ials, 45 Else ie Inc., pp. 123–129. 403 Doi, S.A.R., Ba end eg , J.J., Khan, S., Thalib, L. and Williams, G.M. (2015b) “Simula ion 404 Compa ison o he Quali y E ec s and Random E ec s Me hods o Me a-analysis,” 405 Epidemiology, 26(4) Lippinco Williams & Wilkins, pp. e42–e44. 406 Doi, S.A.R., Ba end eg , J.J. and Mozu kewich, E.L. (2011) “Me a-analysis o he e ogeneous 407 clinical ials: An empi ical example,” Con empo a y Clinical T ials, 32(2) Else ie Inc., pp. 288– 408 298. 409 Doi, S.A.R., Fu uya-Kanamo i, L., Thalib, L. and Ba end eg , J.J. (2017) “Me a-analysis in 410 e idence-based heal hca e: A pa adigm shi away om andom e ec s is o e due,” 411 In e na ional Jou nal o E idence-Based Heal hca e, 15(4) Lippinco Williams and Wilkins, pp. 412 152–160. A ailable a : 10.1097/XEB.0000000000000125 (Accessed: Janua y 25, 2021). 413 Doi, S.A.R. and Thalib, L. (2008) “A quali y-e ec s model o me a-analysis,” Epidemiology, 414 19(1), pp. 94–100. 415 Doncas e , C.P., Spake, R., 2018. Co ec ion o bias in me a-analysis o li le- eplica ed 416 s udies. Me hods Ecol. E ol. 9, 634–644. h ps://doi.o g/h ps://doi.o g/10.1111/2041- 417 210X.12927 418 Ellio , J., Law ence, R., Minx, J.C., Oladapo, O.T., Ra aud, P., Tendal Jeppesen, B., Thomas, 419 J., Tu ne , T., Vand ik, P.O., G imshaw, J.M., 2021. Decision make s need cons an ly upda ed 420 e idence syn hesis. Na . 2021 6007889 600, 383–385. h ps://doi.o g/10.1038/d41586-021- 421 03690-1 422 Gough, D. and Whi e, H. (2018) E idence s anda ds and e idence claims in web based 423 esea ch po als. London. 424 G ainge , M.J., Bolam, F.C., S ewa , G.B. and Nilsen, E.B. (2019) “E idence syn hesis o 425 ackling esea ch was e,” Na u e Ecology & E olu ion, Sp inge US 426 Gu za , F. and Do mann, C.F. (2020) “Explo a ion o Conce ns abou he E idence-Based 427 Guideline App oach in Conse a ion Managemen : Hin s om Medical P ac ice,” En i onmen al 428 Managemen , 66(3), pp. 435–449. 429 Haddaway, N.R. and Wes ga e, M.J. (2019) “P edic ing he ime needed o en i onmen al 430 sys ema ic e iews and sys ema ic maps,” Conse a ion Biology, 33(2) John Wiley & Sons, L d, 431 pp. 434–443. 432 Hamman, E.A., Pappala do, P., Bence, J.R., Peaco , S.D. and Osenbe g, C.W. (2018) “Bias in 433 me a-analyses using Hedges’ d,” Ecosphe e, 9(9) John Wiley & Sons, L d, p. e02419. 434 Jenicek, M. (1989) “Me a-analysis in medicine Whe e we a e and whe e we wan o go,” Jou nal 435 o Clinical Epidemiology, 42(1), pp. 35–44. 436 Johnson, B.T., Low, R.E. and MacDonald, H. . (2015) “Panning o he gold in heal h esea ch: 437 Inco po a ing s udies’ me hodological quali y in me a-analysis,” Psychology and Heal h, 30(1) 438 Rou ledge, pp. 135–152. A ailable a : 10.1080/08870446.2014.953533 (Accessed: Janua y 25, 439 2021). 440 Junke , J., Pe o an, S.O., A oyo-Rod Íguez, V., Boon a ana, R., Byle , D., Chapman, C.A., 441 Che y, D., Cheyne, S.M., Co nejo, F.M., Co És-O iz, L., Cowlishaw, G., Ch is ie, A.P., 442 C ock o d, C., To e, S.D. la, de Melo, F.R., Fan, P., G ue e , C.C., GuzmÁn-Ca o, D.C., 443 Heymann, E.W., He binge , I., Hoang, M.D., Ho wich, R.H., Humle, T., Ikemeh, R.A., Imong, 444 I.S., Je usalinsky, L., Johnson, S.E., Kappele , P.M., Kie ul , M.C.M., KonÉ, I., Ko mos, R., Le, 445 K.Q., Li, B., Ma shall, A.J., Meijaa d, E., Mi e meie , R.A., Mu oyama, Y., Neugebaue , E., O h, 446 L., Palacios, E., Papwo h, S.K., Plump e, A.J., Rawson, B.M., Re isch, J., Ra simbaza y, J., 447 Roos, C., Se chell, J.M., Smi h, R.K., Sop, T., Schwi ze , C., Sla e , K., S um, S.C., Su he land, 448 W.J., Talebi, M., Wallis, J., Wich, S., Williamson, E.A., Wi ig, R.M. and KÜhl, H.S. (2020) “A 449 Se e e Lack o E idence Limi s E ec i e Conse a ion o he Wo ld’s P ima es,” BioScience, 450 70(9), pp. 794–803. 451 Kneale, D., Thomas, J., O’Ma a‐E es, A. and Wiggins, R. (2019) “How can addi ional seconda y 452 da a analysis o obse a ional da a enhance he gene alisabili y o me a‐analy ic e idence o 453 local public heal h decision making?,” Resea ch Syn hesis Me hods, 10(1), pp. 44–56. 454 Ko iche a, J. and Kulinskaya, E. (2019) “Tempo al Ins abili y o E idence Base: A Th ea o 455 Policy Making?,” T ends in Ecology & E olu ion, 34(10), pp. 895–902. 456 La sen, P.O. and on Ins, M. (2010) “The a e o g ow h in scien i ic publica ion and he decline 457 in co e age p o ided by Science Ci a ion Index,” Scien ome ics, 84(3), pp. 575–603. 458 Lo ie, C.J., S ewa , G., Ro hs ein, H. and Lau, J. (2015) “How o c i ically ead ecological me a- 459 analyses,” Resea ch Syn hesis Me hods, 6(2) John Wiley & Sons, L d, pp. 124–133. 460 Ma shall, I.J., Johnson, B.T., Wang, Z., Rajaseka an, S. and Wallace, B.C. (2020) “Semi- 461 Au oma ed e idence syn hesis in heal h psychology: cu en me hods and u u e p ospec s,” 462 Heal h Psychology Re iew, 14(1) Rou ledge, pp. 145–158. 463 Ma shall, I.J., Kuipe , J. and Wallace, B.C. (2015) “Au oma ing Risk o Bias Assessmen o 464 Clinical T ials,” IEEE Jou nal o Biomedical and Heal h In o ma ics, 19(4), pp. 1406–1412. 465 Ma shall, I.J. and Wallace, B.C. (2019) “Towa d sys ema ic e iew au oma ion: a p ac ical guide 466 o using machine lea ning ools in esea ch syn hesis,” Sys ema ic Re iews, 8(1), p. 163. 467 Mupepele, A.-C., Walsh, J.C., Su he land, W.J. and Do mann, C.F. (2016) “An e idence 468 assessmen ool o ecosys em se ices and conse a ion s udies,” Ecological Applica ions, 469 26(5), pp. 1295–1301. 470 Mupepele, A.-C., Kelle , M., Do mann, C.F., (2021). Eu opean ag o o es y has no unequi ocal 471 e ec on biodi e si y: a ime-cumula i e me a-analysis. BMC Ecol. E ol. 21, 193. 472 h ps://doi.o g/10.1186/s12862-021-01911-9 473 Nakagawa, S., Dunn, A.G., Lagisz, M., Bannach-B own, A., G ames, E.M., Sánchez-Tója , A., 474 O’Dea, R.E., Noble, D.W.A., Wes ga e, M.J., A nold, P.A., Ba ow, S., Be hel, A., Coope , E., 475 Foo, Y.Z., Geange, S.R., Hennessy, E., Mapanga, W., Menge sen, K., Mune a, C., Page, M.J., 476 Welch, V., Ca e , M., Fo bes, O., Fu uya-Kanamo i, L., G ay, C.T., Hamil on, W.K., Ka , F., 477 Ko he, E., Kwong, J., McGuinness, L.A., Ma in, P., Ngwenya, M., Penkin, C., Pe ez, D., 478 Sche mann, M., Senio , A.M., Vásquez, J., Viech baue , W., Whi e, T.E., Whi elaw, M., 479 Haddaway, N.R. and Pa icipan s, E.S.H. 2019 (2020) “A new ecosys em o e idence 480 syn hesis,” Na u e Ecology & E olu ion, 4(4), pp. 498–501. 481 O’Conno , A.M., Tsa na , G., Gilbe , S.B., Thaye , K.A. and Wol e, M.S. (2018) “Mo ing owa d 482 he au oma ion o he sys ema ic e iew p ocess: a summa y o discussions a he second 483 mee ing o In e na ional Collabo a ion o he Au oma ion o Sys ema ic Re iews (ICASR),” 484 Sys ema ic Re iews, 7(1), p. 3. 485 Pa ani um, P., Laopaiboon, M., Mohe , D., Lumbiganon, P. and Ngamja us, C. (2012) “A 486 Compa ison o S a is ical Me hods o Iden i ying Ou -o -Da e Sys ema ic Re iews,” PLOS ONE, 487 7(11) Public Lib a y o Science, p. e48894. 488 Rhodes, K.M., Sa o ić, J., Elbe s, R., Jones, H.E., Higgins, J.P.T., S e ne, J.A.C., Wel on, N.J. 489 and Tu ne , R.M. (2020) “Adjus ing ial esul s o biases in me a-analysis: combining da a- 490 based e idence on bias wi h de ailed ial assessmen ,” Jou nal o he Royal S a is ical Socie y: 491 Se ies A (S a is ics in Socie y), 183(1) John Wiley & Sons, L d, pp. 193–209. 492 Rhodes, K.M., Tu ne , R.M., Sa o ić, J., Jones, H.E., Mawdsley, D. and Higgins, J.P.T. (2018) 493 “Be ween- ial he e ogenei y in me a-analyses may be pa ially explained by epo ed design 494 cha ac e is ics,” Jou nal o Clinical Epidemiology, 95, pp. 45–54. 495 Scha , M., Wegne , B., Meye , N., Wol e , C., A linghaus, R., (2021). Ecological impac s o 496 wa e -based ec ea ional ac i i ies on eshwa e ecosys ems: a global me a-analysis. P oc. R. 497 Soc. B Biol. Sci. 288, 20211623. h ps://doi.o g/10.1098/ spb.2021.1623 498 Shackel o d, G.E., Kemp, L., Rhodes, C., Sunda am, L., HÉigea aigh, S.S.Ó., Bea d, S., 499 Bel ield, H., Wei zdö e , J., A in, S., Sø ebø, D., Jones, E.M., Hume, J.B., P ice, D., Pyle, D., 500 Hu , D., S one, T., Wa kins, H., Collas, L., Cade, B.C., Johnson, T.F., F ei as-G o , Z., 501 Denkenbe ge , D., Le o , M. and Su he land, W.J. (2019) “Accumula ing e idence using 502 c owdsou cing and machine lea ning: a li ing bibliog aphy abou exis en ial isk and global 503 ca as ophic isk,” Fu u es, , p. 102508. 504 Shackel o d, G.E., Ma in, P.A., Hood, A.S.C., Ch is ie, A.P., Kulinskaya, E. and Su he land, 505 W.J. (2021). Dynamic me a-analysis: a me hod o using global e idence o local decision 506 making. BMC Biol 19(33) . h ps://doi.o g/10.1186/s12915-021-00974-w 507 Shojania, K.G., Sampson, M., Ansa i, M.T., Ji, J., Douce e, S. and Mohe , D. (2007) “How 508 Quickly Do Sys ema ic Re iews Go Ou o Da e? A Su i al Analysis,” Annals o In e nal 509 Medicine, 147(4) Ame ican College o Physicians, pp. 224–233. 510 Sla in, R.E. (1995) “Bes e idence syn hesis: An in elligen al e na i e o me a-analysis,” 511 Jou nal o Clinical Epidemiology, 48(1), pp. 9–18. 512 Sla in, R.E. (1986) “Bes -E idence Syn hesis: An Al e na i e o Me a-Analy ic and T adi ional 513 Re iews,” Educa ional Resea che , 15(9), pp. 5–11. 514 S one, J., Gu una han, U., Glass, K., Munn, Z., Tugwell, P. and Doi, S.A.R. (2019) “S a i ica ion 515 by quali y-induced selec ion bias in a me a-analysis o clinical ials,” Jou nal o Clinical 516 Epidemiology, 107 Else ie USA, pp. 51–59. A ailable a : 10.1016/j.jclinepi.2018.11.015 517 (Accessed: Janua y 25, 2021). 518 S one, J.C., Glass, K., Munn, Z., Tugwell, P. and Doi, S.A.R. (2020) “Compa ison o bias 519 adjus men me hods in me a-analysis sugges s ha quali y e ec s modeling may ha e less 520 limi a ions han o he app oaches,” Jou nal o Clinical Epidemiology, 117 Else ie Inc, pp. 36– 521 45. 522 Su he land, W.J. and Wo dley, C.F.R. (2017) “E idence complacency hampe s conse a ion,” 523 Na u e Ecology & E olu ion, 1(9), pp. 1215–1216. 524 Su he land, W.J. and Wo dley, C.F.R. (2018) “A esh app oach o e idence syn hesis 525 commen ,” Na u e, 558(7710), pp. 364–366. 526 The Royal Socie y (2020) Face masks and co e ings o he gene al public: Beha iou al 527 knowledge, e ec i eness o clo h co e ings and public messaging. 528 Thomas, J., Noel-S o , A., Ma shall, I., Wallace, B., McDonald, S., Ma e games, C., Glasziou, 529 P., Shemil , I., Synno , A., Tu ne , T., Ellio , J., Ago i sas, T., Hil on, J., Pe on, C., Akl, E., 530 Hodde , R., Pes idge, C., Alb ech , L., Ho sley, T., Pla , J., A ms ong, R., Nguyen, P.H., 531 Plo nick, R., A no, A., I e s, N., Quinn, G., Au, A., Johns on, R., Rada, G., Bagg, M., Jones, A., 532 Ra aud, P., Boden, C., Kahale, L., Rich e , B., Bois e , I., Kesha a z, H., Ryan, R., B and , L., 533 Kolakowsky-Hayne , S.A., Salama, D., B azino a, A., Nag aj, S.K., Salan i, G., Buchbinde , R., 534 Lasse son, T., San aguida, L., Champion, C., Law ence, R., San esso, N., Chandle , J., Les, Z., 535 Schünemann, H.J., Cha idimou, A., Leuch , S., Shemil , I., Chou, R., Low, N., She i ali, D., 536 Chu chill, R., Maas, A., Siemieniuk, R., Cnossen, M.C., MacLehose, H., Simmonds, M., Cossi, 537 M.-J., Macleod, M., Skoe z, N., Couno e, M., Ma shall, I., Soa es-Weise , K., C aigie, S., 538 Ma shall, R., S ikan h, V., Dahm, P., Ma in, N., Sulli an, K., Danilkewich, A., Ma ínez Ga cía, 539 L., Synno , A., Danko, K., Ma e games, C., Taylo , M., Donoghue, E., Maxwell, L.J., Thaye , K., 540 D essle , C., McAuley, J., Thomas, J., Egan, C., McDonald, S., T i on, R., Ellio , J., McKenzie, 541 J., Tsa na , G., Ellio , S.A., Mee pohl, J., Tugwell, P., E xeandia, I., Me ne , B., Tu geon, A., 542 Fea he s one, R., Mondello, S., Tu ne , T., Foxlee, R., Mo ley, R., an Valkenhoe , G., Ga ne , 543 P., Muna o, M., Vand ik, P., Ge i y, M., Munn, Z., Wallace, B., Glasziou, P., Mu ano, M., 544 Wallace, S.A., G een, S., Newman, K., Wa s, C., G imshaw, J., Nieuwlaa , R., Weeks, L., 545 Gu usamy, K., Nikolakopoulou, A., Weigl, A., Haddaway, N., Noel-S o , A., Wells, G., Ha ling, 546 L., O’Conno , A., Wie cioch, W., Hayden, J., Page, M., Wol enden, L., Hel and, M., Pahwa, M., 547 Yepes Nuñez, J.J., Higgins, J., Pa do, J.P., Yos , J., Hill, S. and Pea son, L. (2017) “Li ing 548 sys ema ic e iews: 2. Combining human and machine e o ,” Jou nal o Clinical Epidemiology, 549 91 Else ie , pp. 31–37. 550 Tsa na , G., Dunn, A., Glasziou, P. and Coie a, E. (2013) “The au oma ion o sys ema ic 551 e iews,” BMJ, 346(jan10 1), pp. 139– 139. 552 Tugwell, P. and Haynes, R.B. (2006) “Assessing claims o causa ion,” Clinical epidemiology: 553 how o do clinical p ac ice esea ch, Lippinco Williams & Wilkins Philadelphia, PA, pp. 356– 554 387. 555 Tu ne , R.M., Spiegelhal e , D.J., Smi h, G.C.S. and Thompson, S.G. (2009) “Bias modelling in 556 e idence syn hesis,” Jou nal o he Royal S a is ical Socie y: Se ies A (S a is ics in Socie y), 557 172(1) John Wiley & Sons, L d, pp. 21–47. 558 Unde wood, A.J. (1991) “Beyond BACI: Expe imen al designs o de ec ing human 559 en i onmen al impac s on empo al a ia ions in na u al popula ions,” Ma ine and F eshwa e 560 Resea ch, 42(5), pp. 569–587. 561 Wallace, B.C., Dahab eh, I.J., Schmid, C.H., Lau, J. and T ikalinos, T.A. (2014) “Mode nizing 562 E idence Syn hesis o E idence-Based Medicine,” in G eenes, R. A. B. T.-C. D. S. (Second E. 563 (ed.) Clinical Decision Suppo . Ox o d: Else ie , pp. 339–361. 564 Wauchope, H.S., Amano, T., Geldmann, J., Johns on, A., Simmons, B.I., Su he land, W.J. and 565 Jones, J.P.G. (2020) “E alua ing Impac Using Time-Se ies Da a,” T ends in Ecology & 566 E olu ion 567 568 569 570