This documen is he Accep ed Manusc ip e sion o a Published Wo k ha
appea ed in inal o m in:
Ch is ie, A.P.; Amano, T.; Ma in, P.A.; Shackel o d, G.E.; Simmons, B.I.; Su he land,
W.J.2022. Plu al alua ion o na u e o equi y and sus ainabili y: Insigh s om he
Global Sou h. Jou nal o Applied Ecology. 59. DOI (10.1111/1365-2664.14154).
© 2022 B i ish Ecological Socie y.
This manusc ip e sion is made a ailable unde he CC-BY-NC-ND 3.0 license
h p://c ea i ecommons.o g/licenses/by-nc-nd/3.0/
Inno a ion and o wa d- hinking a e needed o imp o e adi ional 1
syn hesis me hods: a esponse o Pesco & S ewa 2
3
Alec P. Ch is ie1,4,7*, Ta suya Amano1,2,3, Philip A. Ma in1,4,8, Go m E. Shackel o d1,4,4
Benno I. Simmons1,5,6, William J. Su he land1,4
5
1Conse a ion Science G oup, Depa men o Zoology, Uni e si y o Camb idge, The Da id 6
A enbo ough Building, Downing S ee , Camb idge, UK. 7
2Cen e o he S udy o Exis en ial Risk, Uni e si y o Camb idge, 16 Mill Lane, Camb idge, UK. 8
3School o Biological Sciences, Uni e si y o Queensland, B isbane, 4072 Queensland, Aus alia 9
4BioRISC, S Ca ha ine’s College, Camb idge, UK. 10
5Depa men o Animal and Plan Sciences, Uni e si y o She ield, She ield, UK. 11
6Cen e o Ecology and Conse a ion, College o Li e and En i onmen al Sciences, Uni e si y o 12
Exe e , Pen yn, UK. 13
7Downing College, Regen S ee , Camb idge, UK. 14
8Basque Cen e o Clima e Change (BC3), Edi icio sede no 1, plan a 1, Pa que cien í ico 15
UPV/EHU, Ba io Sa iena s/n, 48940, Leioa, Bizkaia, Spain16
17
*Co esponding au ho , a[email p o ec ed].uk18
19
20
21
22
23
Abs ac
24
1. In Ch is ie e al. (2019), we used simula ions o quan i a i ely compa e he bias o
25
commonly used s udy designs in ecology and conse a ion. Based on hese simula ions,
26
we p oposed ‘accu acy weigh s’ as a po en ial way o accoun o s udy design alidi y in
27
me a-analy ic weigh ing me hods. Pesco & S ewa (2021) aised conce ns ha hese
28
weigh s may no be gene alisable and s ill lead o biased me a-es ima es. He e we
29
espond o hei conce ns and demons a e why de eloping al e na i e weigh ing
30
me hods is key o he u u e o e idence syn hesis.
31
2. We acknowledge ha ou simple simula ion un ai ly penalised Randomised Con olled
32
T ial (RCT) ela i e o Be o e-A e Con ol-Impac (BACI) designs as we assumed ha
33
he pa allel ends assump ion held o BACI designs. We poin o an empi ical ollow-up
34
s udy in which we mo e ai ly quan i y di e ences in biases be ween di e en s udy
35
designs. Howe e , we s and by ou main indings ha Be o e-A e (BA), Con ol-Impac
36
(CI), and A e designs a e quan i iably mo e biased han BACI and RCT designs. We
37
also emphasise ha ou 'accu acy weigh ing’ me hod was p elimina y and welcome
38
u u e esea ch o inco po a e mo e dimensions o s udy quali y.
39
3. We u he show ha o e a decade o ad ances in quali y e ec modelling, which
40
Pesco & S ewa (2021) omi , highligh s he impo ance o esea ch such as ou s in
41
be e unde s anding how o quan i a i ely in eg a e da a on s udy quali y di ec ly in o
42
me a-analyses. We u he a gue ha he adi ional me hods ad oca ed o by Pesco &
43
S ewa (2021) (e.g., manual isk-o -bias assessmen s and in e se- a iance weigh ing)
44
a e subjec i e, was e ul, and po en ially biased hemsel es. They also lack scalabili y o
45
use in la ge syn heses ha keep up- o-da e wi h he apidly g owing scien i ic li e a u e.
46
4. Syn hesis and applica ions. We sugges , con a y o Pesco & S ewa ’s na a i e, ha
47
mo ing owa ds al e na i e weigh ing me hods is key o u u e-p oo ing e idence
48
syn hesis h ough g ea e au oma ion, lexibili y, and upda ing o espond o decision-
49
make s needs – pa icula ly in c isis disciplines in conse a ion science whe e
50
p oblema ic biases and a iabili y exis in s udy designs, con ex s, and me ics used.
51
Whils we mus be cau ious o a oid misin o ming decision-make s, his should no s op
52
us in es iga ing al e na i e weigh ing me hods ha in eg a e s udy quali y da a di ec ly
53
in o me a-analyses. To eliably and p agma ically in o m decision-make s wi h science,
54
we need e icien , scalable, eadily au oma ed, and easible me hods o app aise and
55
weigh s udies o p oduce la ge-scale li ing syn heses o he u u e.
56
57
Keywo ds: e idence syn hesis, me a-analysis, dynamic me a-analysis, li ing e iews,
58
au oma ion, quali y e ec s modelling, me a-analyses, isk-o -bias, c i ical app aisal, bias
59
adjus men .
60
61
In oduc ion
62
63
Pesco & S ewa (2021) ou lined hei conce ns o e an al e na i e me hod o weigh ing in
64
me a-analysis we p oposed called “accu acy weigh s” in Ch is ie e al. (2019). These weigh s
65
we e de i ed om ou simula ion s udy ha aimed o quan i a i ely compa e he pe o mance o
66
di e en expe imen al and obse a ional s udy designs (Ch is ie e al., 2019). Thei wo majo
67
conce ns we e ha ou accu acy weigh s we e no gene alisable and ha quali y sco e
68
weigh ings, such as ou s, may s ill lead o biased es ima es in me a-analyses. He e we espond
69
o hei conce ns and discuss why we belie e al e na i e me hods o weigh ing a e cen al o he
70
u u e o e idence syn hesis.
71
72
1. Accu acy weigh s need imp o ing and combining wi h o he quali y measu es
73
74
As Pesco & S ewa sugges , we acknowledge ha ou simula ion may ha e un ai ly penalised
75
Randomised Con olled T ial (RCT) designs, depending on whe he esea che s in ecology and
76
conse a ion do ake in o accoun p e-impac sampling. Howe e , in ou expe ience, ew
77
Randomised Con olled T ials in conse a ion ake accoun o p e-impac baseline da a; his is
78
suppo ed by a ecen s udy quan i ying he use o di e en s udy designs in he en i onmen al
79
and social sciences (Ch is ie e al., 2020a). We acknowledge ha we did no discuss mo e o
80
he sho comings o Be o e-A e Con ol-Impac (BACI) designs in e ms o he bias ha can be
81
in oduced by iola ing he ‘pa allel ends’ assump ion (Dimick and Ryan, 2014; Unde wood,
82
1991; Wauchope e al., 2020). The e o e, wi h espec o compa ing BACI and RCT designs, we
83
acknowledge ou simula ion has limi a ions.
84
85
Ne e heless, ou majo mo i a ion was o demons a e he di e ence in s udy design
86
pe o mance be ween simple designs (e.g., Be o e-A e (BA), Con ol-Impac (CI), and A e
87
designs) and mo e igo ous designs (RCT and BACI). Thus, we in en ionally made ou
88
simula ion ela i ely simple o engage a wide audience o esea che s. We ha e since buil on
89
ou simula ions in Ch is ie e al. (2020a), which uses an empi ical, model-based me hodology o
90
quan i y he di e ences in bias a ec ing di e en s udy designs using aw ( a he han
91
simula ed) da a om a la ge numbe o wi hin-s udy compa isons. This mo e ai ly quan i ies he
92
bias associa ed wi h RCT e sus BACI designs by making ewe , mo e s a is ically de ensible
93
assump ions abou he ‘ ue e ec ’ ( o es ima e bias) and inhe en ly accoun s o he pa allel
94
ends assump ion ha can bias BACI designs (Ch is ie e al., 2020a).
95
96
Pesco & S ewa also sugges ou simula ion weigh s do no cap u e he ull ange o po en ial
97
sou ces o bias a ec ing s udy designs and ad ise ha assessmen s o s udy quali y should
98
closely sc u inise he de ails o speci ic s udies being summa ised (e.g., using manual isk-o -
99
bias assessmen s). In ou s udy, we speci ically acknowledged ha ou weigh s we e ela i ely
100
simple and need o be buil upon o inco po a e a wide ange o s udy quali y indica o s; we
101
ou lined possible app oaches in he u u e ha could in eg a e sco es om c i ical app aisal
102
ools ha exis o ecology and conse a ion (Mupepele e al., 2016). We a e happy o see ha
103
o he s a e building on ou wo k and in es iga ing he use o a b oade se o quali y o alidi y
104
measu es o weigh s udies in me a-analyses (e.g., Scha e al. 2021, Mupepele e al. 2021). In
105
he nex sec ions, we add ess Pesco & S ewa ’s c i icisms o weigh ing by quali y sco es and
106
discuss s a is ical ad ances in applying quali y sco e weigh ings o me a-analyses. We also
107
discuss he p oblems associa ed wi h he adi ional me hods ad oca ed o by Pesco &
108
S ewa (such as in e se- a iance weigh ing and manual isk-o -bias assessmen s).
109
110
2. Recen ad ances in di ec ly in eg a ing da a on s udy quali y in o me a-
111
analyses
112
113
In Pesco & S ewa 's discussion on why hey ad oca e agains weigh ing by quali y sco es in
114
me a-analyses, hey omi o e a decade o esea ch in epidemiology on al e na i e quali y sco e
115
weigh ing me hods ha ha e o e come many o he p oblems hey discuss (Doi, Ba end eg
116
and Mozu kewich, 2011; Doi e al., 2015a, 2015b; Doi and Thalib, 2008; Rhodes e al., 2020;
117
S one e al., 2020). In pa icula , ‘bias adjus men ’ me hods, such as quali y e ec s models,
118
ep esen an ac i e and p omising a ea o esea ch in e idence syn hesis in epidemiology (Doi,
119
Ba end eg and Mozu kewich, 2011; Doi and Thalib, 2008; Rhodes e al., 2020; S one e al.,
120
2020).
121
122
C i ical app aisal is adi ionally used o desc ip i ely epo he isk o bias o di e en s udies,
123
a he han ying o quan i a i ely inco po a e hose assessmen s wi hin he analyses
124
hemsel es (Johnson, Low and MacDonald, 2015). Ins ead, ou accu acy weigh s a e ela ed o
125
he ield o ‘bias-adjus men ’ me hods which seek o di ec ly in eg a e isk-o -bias assessmen s
126
in o me a-analy ic esul s (S one e al., 2020). C i icisms o quali y sco e weigh ings ha e
127
cen e ed a ound ou majo issues: 1.) he choice o quali y scale in luences he weigh o
128
indi idual s udies; 2.) he me a-es ima e and i s con idence in e al depends on he scale; 3.)
129
he e is no eason why s udy quali y should modi y he p ecision o es ima es; and 4.) poo
130
s udies a e no excluded (S one e al., 2020). The e o e, as Pesco & S ewa also appea o
131
a gue, any bias associa ed wi h poo quali y s udies can only be educed a bes , and no
132
emo ed (S one e al., 2020).
133
134
Whils p oponen s o quali y sco e app oaches accep ed hese c i icisms and ceased hei
135
de elopmen , an al e na i e, imp o ed me hodology called ‘quali y e ec s models’ ha e
136
subsequen ly been de eloped and e ined in ecen yea s. This app oach uses a ela i e scale
137
and ‘syn he ic weigh s’ (yielding ela i e c edibili y anks o di e en s udies) ha o e came he
138
majo issues ha a ec ed quali y sco e app oaches, and has been shown o yield an es ima o
139
wi h supe io e o and co e age o con en ional es ima o s (Doi e al., 2015b, 2017). The e a e
140
a ange o possible ways, each wi h ad an ages o disad an ages, o de i e he ela i e
141
c edibili y weigh s o s udies using nume ical da a gene a ed by expe opinion (Tu ne e al.,
142
2009), da a-based dis ibu ions, o s a is ically combining expe opinion and da a-based
143
dis ibu ions (Rhodes e al., 2020). The e o e, esul s om u he e ining and imp o ing ou
144
simula ions and empi ical analyses (Ch is ie e al., 2019; Ch is ie e al., 2020a) could p o ide
145
aluable con ibu ions o he ac i e de elopmen o hese me hods o in eg a e da a on s udy
146
quali y di ec ly in o me a-analyses.
147
148
Pesco & S ewa ocus on he possibili y o inco po a ing s udy quali y sco es in o me a-
149
eg ession app oaches. Thei c i icism o ou weigh s in hei cu en o m is ha hey a e oo
150
unidimensional and no s udy-speci ic; his is a c i icism ha we pa ially accep . Indeed, we
151
speci ically discussed he need o expand and imp o e ou weigh s o in eg a e o he aspec s o
152
s udy quali y (e.g., using expe opinion, da a-based dis ibu ions, o c i ical app aisal ools o
153
adjus ela i e c edibili y anks; Rhodes e al., 2020). In hindsigh , we should ha e dedica ed
154
mo e a en ion o how we would u he de elop and mo e obus ly apply ou accu acy weigh s
155
alongside discussing ad ances in quali y e ec s models.
156
157
Pesco & S ewa also sugges ha we igno e issues ela ing o ex e nal alidi y. Gi en ha
158
adi ional weigh ings, such as sample size o in e se a iance, also ail o conside ex e nal
159
alidi y, we ind his an odd c i icism, pa icula ly gi en ou simula ion was clea ly ocused on
160
add essing issues o s udy design quali y and in e nal alidi y. We a e in ac de eloping an
161
al e na i e me a-analy ic me hod, dynamic me a-analysis (Shackel o d e al., 2021), based on
162
he Me ada ase pla o m (www.me ada ase .com), which we plan o use o es di e en
163
weigh ing me hods, including ‘ ecalib a ion’ om he medical sciences (Kneale e al., 2019)
164
which aims o adjus s udies’ in luence in me a-analyses based on hei ex e nal alidi y (o
165
ele ance o decision-make s). Again his wo k is in he ea ly s ages o de elopmen and he e
166
a e many me hodological challenges o o e come, pa icula ly in how o in eg a e ‘ ecalib a ion’
167
me hods in o andom e ec s models and how o ensu e such in e ac i e me a-analy ic ools a e
168
used obus ly (Shackel o d e al. 2021). The e o e, as Pesco & S ewa sugges , we belie e i
169
should be possible o in eg a e in e nal alidi y o quali y i ems, and ex e nal alidi y i ems, in o a
170
hie a chical me a- eg ession amewo k, o o di ec ly weigh s udies using new ad ances in
171
quali y e ec s models as discussed p e iously (see S one e al. 2020 o a compa ison and
172
discussion o di e en app oaches).
173
174
175
3. In eg a ing da a on s udy quali y in o me a-analyses is essen ial o he u u e o
176
e idence syn hesis
177
178
We also belie e Pesco & S ewa ’s discussion p esen s a na ow ision o he challenges
179
aced by adi ional c i ical app aisal and weigh ing me hods. We belie e ha he adi ional
180
‘medical-s yle’ app oaches (e.g., manual isk-o -bias assessmen s combined wi h in e se-
181
a iance weigh ing) ha Pesco & S ewa belie e should be adhe ed o a e ul ima ely
182
ine icien and was e ul. The ield o e idence syn hesis is ad ancing a pace o espond o he
183
challenges o apidly g owing e idence bases and as -mo ing c ises, which equi es new
184
me hodologies ha help o keep e idence bases ‘up- o-da e’ o ‘li ing’, cos -e icien by wo king
185
a massi e discipline-wide scales, and dynamically adjus able o be ele an o di e en
186
decision-make s’ needs. He e we elabo a e on why his is p oblema ic o Pesco & S ewa ’s
187
asse ion ha we should con inue o ely on adi ional me hods, a he han al e na i e
188
weigh ing me hods such as he one we p oposed in Ch is ie e al. (2019).
189
190
3.a. Al e na i e weigh ing me hods acili a e mo e e icien , au oma ed, li ing, la ge-scale
191
syn heses
192
193
Fi s , he e is g owing ecogni ion ha decision make s need cons an ly upda ed e idence
194
syn heses (Ellio e al., 2021) and ha adi ional syn hesis me hods (e.g., adi ional sys ema ic
195
e iews) a e o en oo ime-consuming, quickly go ou -o -da e, and can miss impo an
196
oppo uni ies o in luence p ac ice and policy (Bou on e al., 2020; G ainge e al., 2019;
197
Haddaway and Wes ga e, 2019; Ko iche a and Kulinskaya, 2019; Nakagawa e al., 2020;
198
Pa ani um e al., 2012; Shojania e al., 2007). Gi en ha he scien i ic li e a u e in mos
199
disciplines is g owing apidly (Bo nmann and Mu z, 2015; La sen and on Ins, 2010) and ha
200
Bo ah, R., B own, A.W., Cape s, P.L. and Kaise , K.A. (2017) “Analysis o he imeand wo ke s
356
needed o conduc sys ema ic e iews o medicalin e en ions using da a om he PROSPERO
357
egis y,” BMJ, Open7:e012545.
358
Bo nmann, L. and Mu z, R. (2015) “G ow h a es o mode n science: A bibliome ic analysis
359
based on he numbe o publica ions and ci ed e e ences,” Jou nal o he Associa ion o
360
In o ma ion Science and Technology, 66(11) John Wiley & Sons, L d, pp. 2215–2222.
361
Bou on, I., C équi , P., Williams, H., Mee pohl, J., C aig, J.C. and Ra aud, P. (2020) “Fu u e o
362
e idence ecosys em se ies: 1. In oduc ion — E idence syn hesis ecosys em needs d ama ic
363
change,” Jou nal o Clinical Epidemiology, Else ie Inc.
364
Ch is ie, A.P., Abecasis, D., Adje oud, M., Alonso, J.C., Amano, T., An on, A., Baldigo, B.P.,
365
Ba ien os, R., Bicknell, J.E., Buhl, D.A., Ceb ian, J., Ceia, R.S., Cibils-Ma ina, L., Cla ke, S.,
366
Claude , J., C aig, M.D., Da oul , D., de Backe , A., Dono an, M.K., Eddy, T.D., F ança, F.M.,
367
Ga dne , J.P.A., Ha is, B.P., Huusko, A., Jones, I.L., Kelahe , B.P., Ko iaho, J.S., López-
368
Baucells, A., Majo , H.L., Mäki-Pe äys, A., Ma ín, B., Ma ín, C.A., Ma in, P.A., Ma eos-Molina,
369
D., McConnaughey, R.A., Me oni, M., Meye , C.F.J., Mills, K., Mon e alcone, M., No eika, N.,
370
Palacín, C., Pande, A., Pi che , C.R., Ponce, C., Rinella, M., Rocha, R., Ruiz-Delgado, M.C.,
371
Schmi e -So o, J.J., Sha e , J.A., Sha ma, S., She , A.A., S agnol, D., S anley, T.R.,
372
S okesbu y, K.D.E., To es, A., Tully, O., Vehanen, T., Wa s, C., Zhao, Q. and Su he land, W.J.
373
(2020a) “Quan i ying and add essing he p e alence and bias o s udy designs in he
374
en i onmen al and social sciences,” Na u e Communica ions, 11(1), p. 6377.
375
Ch is ie, A.P., Amano, T., Ma in, P.A., Pe o an, S.O., Shackel o d, G.E., Simmons, B.I., Smi h,
376
R.K., Williams, D.R., Wo dley, C.F.R. and Su he land, W.J. (2020b) “Poo a ailabili y o con ex -
377
speci ic e idence hampe s decision-making in conse a ion,” Biological Conse a ion, 248 Cold
378
Sp ing Ha bo Labo a o y, p. 108666.
379
Ch is ie, A.P., Amano, T., Ma in, P.A., Pe o an, S.O., Shackel o d, G.E., Simmons, B.I., Smi h,
380
R.K., Williams, D.R., Wo dley, C.F.R. and Su he land, W.J. (2020c) “The challenge o biased
381
e idence in conse a ion,” Conse a ion Biology, Wiley o Socie y o Conse a ion Biology, p.
382
cobi.13577.
383
Ch is ie, A.P., Amano, T., Ma in, P.A., Shackel o d, G.E., Simmons, B.I. and Su he land, W.J.
384
(2019) “Simple s udy designs in ecology p oduce inaccu a e es ima es o biodi e si y
385
esponses,” Jou nal o Applied Ecology, 56(12) John Wiley & Sons, L d (10.1111), pp. 2742–
386
2754.
387
Ch is ie, A.P., Whi e, T.B., Ma in, P.A., Pe o an, S.O., Bladon, A.J., Bowke , A.E., Li lewood,
388
N.A., Mupepele, A.C., Rocha, R., Sainsbu y, K.A., Smi h, R.K., Taylo , N.G., Su he land, W.J.,
389
2021. Reducing publica ion delay o imp o e he e iciency and impac o conse a ion science.
390
Pee J 9, e12245. h ps://doi.o g/10.7717/PEERJ.12245/SUPP-17
391
Co n o d, R., Deine , S., de Palma, A., Hill, S.L.L., McRae, L., Pe i , B., Ma coni, V., Pu is, A.
392
and F eeman, R. (2021) “Fas , scalable, and au oma ed iden i ica ion o a icles o biodi e si y
393
and mac oecological da ase s,” Global Ecology and Biogeog aphy, 30(1) John Wiley & Sons,
394
L d, pp. 339–347.
395
Da ies, G.M. and G ay, A. (2015) “Don’ le spu ious accusa ions o pseudo eplica ion limi ou
396
abili y o lea n om na u al expe imen s (and o he messy kinds o ecological moni o ing),”
397
Ecology and E olu ion, 5(22) John Wiley & Sons, L d, pp. 5295–5304.
398
Dimick, J.B. and Ryan, A.M. (2014) “Me hods o E alua ing Changes in Heal h Ca e Policy,”
399
JAMA, 312(22), p. 2401.
400
Doi, S.A.R., Ba end eg , J.J., Khan, S., Thalib, L. and Williams, G.M. (2015a) “Ad ances in he
401
me a-analysis o he e ogeneous clinical ials II: The quali y e ec s model,” Con empo a y
402
Clinical T ials, 45 Else ie Inc., pp. 123–129.
403
Doi, S.A.R., Ba end eg , J.J., Khan, S., Thalib, L. and Williams, G.M. (2015b) “Simula ion
404
Compa ison o he Quali y E ec s and Random E ec s Me hods o Me a-analysis,”
405
Epidemiology, 26(4) Lippinco Williams & Wilkins, pp. e42–e44.
406
Doi, S.A.R., Ba end eg , J.J. and Mozu kewich, E.L. (2011) “Me a-analysis o he e ogeneous
407
clinical ials: An empi ical example,” Con empo a y Clinical T ials, 32(2) Else ie Inc., pp. 288–
408
298.
409
Doi, S.A.R., Fu uya-Kanamo i, L., Thalib, L. and Ba end eg , J.J. (2017) “Me a-analysis in
410
e idence-based heal hca e: A pa adigm shi away om andom e ec s is o e due,”
411
In e na ional Jou nal o E idence-Based Heal hca e, 15(4) Lippinco Williams and Wilkins, pp.
412
152–160. A ailable a : 10.1097/XEB.0000000000000125 (Accessed: Janua y 25, 2021).
413
Doi, S.A.R. and Thalib, L. (2008) “A quali y-e ec s model o me a-analysis,” Epidemiology,
414
19(1), pp. 94–100.
415
Doncas e , C.P., Spake, R., 2018. Co ec ion o bias in me a-analysis o li le- eplica ed
416
s udies. Me hods Ecol. E ol. 9, 634–644. h ps://doi.o g/h ps://doi.o g/10.1111/2041-
417
210X.12927
418
Ellio , J., Law ence, R., Minx, J.C., Oladapo, O.T., Ra aud, P., Tendal Jeppesen, B., Thomas,
419
J., Tu ne , T., Vand ik, P.O., G imshaw, J.M., 2021. Decision make s need cons an ly upda ed
420
e idence syn hesis. Na . 2021 6007889 600, 383–385. h ps://doi.o g/10.1038/d41586-021-
421
03690-1
422
Gough, D. and Whi e, H. (2018) E idence s anda ds and e idence claims in web based
423
esea ch po als. London.
424
G ainge , M.J., Bolam, F.C., S ewa , G.B. and Nilsen, E.B. (2019) “E idence syn hesis o
425
ackling esea ch was e,” Na u e Ecology & E olu ion, Sp inge US
426
Gu za , F. and Do mann, C.F. (2020) “Explo a ion o Conce ns abou he E idence-Based
427
Guideline App oach in Conse a ion Managemen : Hin s om Medical P ac ice,” En i onmen al
428
Managemen , 66(3), pp. 435–449.
429
Haddaway, N.R. and Wes ga e, M.J. (2019) “P edic ing he ime needed o en i onmen al
430
sys ema ic e iews and sys ema ic maps,” Conse a ion Biology, 33(2) John Wiley & Sons, L d,
431
pp. 434–443.
432
Hamman, E.A., Pappala do, P., Bence, J.R., Peaco , S.D. and Osenbe g, C.W. (2018) “Bias in
433
me a-analyses using Hedges’ d,” Ecosphe e, 9(9) John Wiley & Sons, L d, p. e02419.
434
Jenicek, M. (1989) “Me a-analysis in medicine Whe e we a e and whe e we wan o go,” Jou nal
435
o Clinical Epidemiology, 42(1), pp. 35–44.
436
Johnson, B.T., Low, R.E. and MacDonald, H. . (2015) “Panning o he gold in heal h esea ch:
437
Inco po a ing s udies’ me hodological quali y in me a-analysis,” Psychology and Heal h, 30(1)
438
Rou ledge, pp. 135–152. A ailable a : 10.1080/08870446.2014.953533 (Accessed: Janua y 25,
439
2021).
440
Junke , J., Pe o an, S.O., A oyo-Rod Íguez, V., Boon a ana, R., Byle , D., Chapman, C.A.,
441
Che y, D., Cheyne, S.M., Co nejo, F.M., Co És-O iz, L., Cowlishaw, G., Ch is ie, A.P.,
442
C ock o d, C., To e, S.D. la, de Melo, F.R., Fan, P., G ue e , C.C., GuzmÁn-Ca o, D.C.,
443
Heymann, E.W., He binge , I., Hoang, M.D., Ho wich, R.H., Humle, T., Ikemeh, R.A., Imong,
444
I.S., Je usalinsky, L., Johnson, S.E., Kappele , P.M., Kie ul , M.C.M., KonÉ, I., Ko mos, R., Le,
445
K.Q., Li, B., Ma shall, A.J., Meijaa d, E., Mi e meie , R.A., Mu oyama, Y., Neugebaue , E., O h,
446
L., Palacios, E., Papwo h, S.K., Plump e, A.J., Rawson, B.M., Re isch, J., Ra simbaza y, J.,
447
Roos, C., Se chell, J.M., Smi h, R.K., Sop, T., Schwi ze , C., Sla e , K., S um, S.C., Su he land,
448
W.J., Talebi, M., Wallis, J., Wich, S., Williamson, E.A., Wi ig, R.M. and KÜhl, H.S. (2020) “A
449
Se e e Lack o E idence Limi s E ec i e Conse a ion o he Wo ld’s P ima es,” BioScience,
450
70(9), pp. 794–803.
451
Kneale, D., Thomas, J., O’Ma a‐E es, A. and Wiggins, R. (2019) “How can addi ional seconda y
452
da a analysis o obse a ional da a enhance he gene alisabili y o me a‐analy ic e idence o
453
local public heal h decision making?,” Resea ch Syn hesis Me hods, 10(1), pp. 44–56.
454
Ko iche a, J. and Kulinskaya, E. (2019) “Tempo al Ins abili y o E idence Base: A Th ea o
455
Policy Making?,” T ends in Ecology & E olu ion, 34(10), pp. 895–902.
456
La sen, P.O. and on Ins, M. (2010) “The a e o g ow h in scien i ic publica ion and he decline
457
in co e age p o ided by Science Ci a ion Index,” Scien ome ics, 84(3), pp. 575–603.
458
Lo ie, C.J., S ewa , G., Ro hs ein, H. and Lau, J. (2015) “How o c i ically ead ecological me a-
459
analyses,” Resea ch Syn hesis Me hods, 6(2) John Wiley & Sons, L d, pp. 124–133.
460
Ma shall, I.J., Johnson, B.T., Wang, Z., Rajaseka an, S. and Wallace, B.C. (2020) “Semi-
461
Au oma ed e idence syn hesis in heal h psychology: cu en me hods and u u e p ospec s,”
462
Heal h Psychology Re iew, 14(1) Rou ledge, pp. 145–158.
463
Ma shall, I.J., Kuipe , J. and Wallace, B.C. (2015) “Au oma ing Risk o Bias Assessmen o
464
Clinical T ials,” IEEE Jou nal o Biomedical and Heal h In o ma ics, 19(4), pp. 1406–1412.
465
Ma shall, I.J. and Wallace, B.C. (2019) “Towa d sys ema ic e iew au oma ion: a p ac ical guide
466
o using machine lea ning ools in esea ch syn hesis,” Sys ema ic Re iews, 8(1), p. 163.
467
Mupepele, A.-C., Walsh, J.C., Su he land, W.J. and Do mann, C.F. (2016) “An e idence
468
assessmen ool o ecosys em se ices and conse a ion s udies,” Ecological Applica ions,
469
26(5), pp. 1295–1301.
470
Mupepele, A.-C., Kelle , M., Do mann, C.F., (2021). Eu opean ag o o es y has no unequi ocal
471
e ec on biodi e si y: a ime-cumula i e me a-analysis. BMC Ecol. E ol. 21, 193.
472
h ps://doi.o g/10.1186/s12862-021-01911-9
473
Nakagawa, S., Dunn, A.G., Lagisz, M., Bannach-B own, A., G ames, E.M., Sánchez-Tója , A.,
474
O’Dea, R.E., Noble, D.W.A., Wes ga e, M.J., A nold, P.A., Ba ow, S., Be hel, A., Coope , E.,
475
Foo, Y.Z., Geange, S.R., Hennessy, E., Mapanga, W., Menge sen, K., Mune a, C., Page, M.J.,
476
Welch, V., Ca e , M., Fo bes, O., Fu uya-Kanamo i, L., G ay, C.T., Hamil on, W.K., Ka , F.,
477
Ko he, E., Kwong, J., McGuinness, L.A., Ma in, P., Ngwenya, M., Penkin, C., Pe ez, D.,
478
Sche mann, M., Senio , A.M., Vásquez, J., Viech baue , W., Whi e, T.E., Whi elaw, M.,
479
Haddaway, N.R. and Pa icipan s, E.S.H. 2019 (2020) “A new ecosys em o e idence
480
syn hesis,” Na u e Ecology & E olu ion, 4(4), pp. 498–501.
481
O’Conno , A.M., Tsa na , G., Gilbe , S.B., Thaye , K.A. and Wol e, M.S. (2018) “Mo ing owa d
482
he au oma ion o he sys ema ic e iew p ocess: a summa y o discussions a he second
483
mee ing o In e na ional Collabo a ion o he Au oma ion o Sys ema ic Re iews (ICASR),”
484
Sys ema ic Re iews, 7(1), p. 3.
485
Pa ani um, P., Laopaiboon, M., Mohe , D., Lumbiganon, P. and Ngamja us, C. (2012) “A
486
Compa ison o S a is ical Me hods o Iden i ying Ou -o -Da e Sys ema ic Re iews,” PLOS ONE,
487
7(11) Public Lib a y o Science, p. e48894.
488
Rhodes, K.M., Sa o ić, J., Elbe s, R., Jones, H.E., Higgins, J.P.T., S e ne, J.A.C., Wel on, N.J.
489
and Tu ne , R.M. (2020) “Adjus ing ial esul s o biases in me a-analysis: combining da a-
490
based e idence on bias wi h de ailed ial assessmen ,” Jou nal o he Royal S a is ical Socie y:
491
Se ies A (S a is ics in Socie y), 183(1) John Wiley & Sons, L d, pp. 193–209.
492
Rhodes, K.M., Tu ne , R.M., Sa o ić, J., Jones, H.E., Mawdsley, D. and Higgins, J.P.T. (2018)
493
“Be ween- ial he e ogenei y in me a-analyses may be pa ially explained by epo ed design
494
cha ac e is ics,” Jou nal o Clinical Epidemiology, 95, pp. 45–54.
495
Scha , M., Wegne , B., Meye , N., Wol e , C., A linghaus, R., (2021). Ecological impac s o
496
wa e -based ec ea ional ac i i ies on eshwa e ecosys ems: a global me a-analysis. P oc. R.
497
Soc. B Biol. Sci. 288, 20211623. h ps://doi.o g/10.1098/ spb.2021.1623
498
Shackel o d, G.E., Kemp, L., Rhodes, C., Sunda am, L., HÉigea aigh, S.S.Ó., Bea d, S.,
499
Bel ield, H., Wei zdö e , J., A in, S., Sø ebø, D., Jones, E.M., Hume, J.B., P ice, D., Pyle, D.,
500
Hu , D., S one, T., Wa kins, H., Collas, L., Cade, B.C., Johnson, T.F., F ei as-G o , Z.,
501
Denkenbe ge , D., Le o , M. and Su he land, W.J. (2019) “Accumula ing e idence using
502
c owdsou cing and machine lea ning: a li ing bibliog aphy abou exis en ial isk and global
503
ca as ophic isk,” Fu u es, , p. 102508.
504
Shackel o d, G.E., Ma in, P.A., Hood, A.S.C., Ch is ie, A.P., Kulinskaya, E. and Su he land,
505
W.J. (2021). Dynamic me a-analysis: a me hod o using global e idence o local decision
506
making. BMC Biol 19(33) . h ps://doi.o g/10.1186/s12915-021-00974-w
507
Shojania, K.G., Sampson, M., Ansa i, M.T., Ji, J., Douce e, S. and Mohe , D. (2007) “How
508
Quickly Do Sys ema ic Re iews Go Ou o Da e? A Su i al Analysis,” Annals o In e nal
509
Medicine, 147(4) Ame ican College o Physicians, pp. 224–233.
510
Sla in, R.E. (1995) “Bes e idence syn hesis: An in elligen al e na i e o me a-analysis,”
511
Jou nal o Clinical Epidemiology, 48(1), pp. 9–18.
512
Sla in, R.E. (1986) “Bes -E idence Syn hesis: An Al e na i e o Me a-Analy ic and T adi ional
513
Re iews,” Educa ional Resea che , 15(9), pp. 5–11.
514
S one, J., Gu una han, U., Glass, K., Munn, Z., Tugwell, P. and Doi, S.A.R. (2019) “S a i ica ion
515
by quali y-induced selec ion bias in a me a-analysis o clinical ials,” Jou nal o Clinical
516
Epidemiology, 107 Else ie USA, pp. 51–59. A ailable a : 10.1016/j.jclinepi.2018.11.015
517
(Accessed: Janua y 25, 2021).
518
S one, J.C., Glass, K., Munn, Z., Tugwell, P. and Doi, S.A.R. (2020) “Compa ison o bias
519
adjus men me hods in me a-analysis sugges s ha quali y e ec s modeling may ha e less
520
limi a ions han o he app oaches,” Jou nal o Clinical Epidemiology, 117 Else ie Inc, pp. 36–
521
45.
522
Su he land, W.J. and Wo dley, C.F.R. (2017) “E idence complacency hampe s conse a ion,”
523
Na u e Ecology & E olu ion, 1(9), pp. 1215–1216.
524
Su he land, W.J. and Wo dley, C.F.R. (2018) “A esh app oach o e idence syn hesis
525
commen ,” Na u e, 558(7710), pp. 364–366.
526
The Royal Socie y (2020) Face masks and co e ings o he gene al public: Beha iou al
527
knowledge, e ec i eness o clo h co e ings and public messaging.
528
Thomas, J., Noel-S o , A., Ma shall, I., Wallace, B., McDonald, S., Ma e games, C., Glasziou,
529
P., Shemil , I., Synno , A., Tu ne , T., Ellio , J., Ago i sas, T., Hil on, J., Pe on, C., Akl, E.,
530
Hodde , R., Pes idge, C., Alb ech , L., Ho sley, T., Pla , J., A ms ong, R., Nguyen, P.H.,
531
Plo nick, R., A no, A., I e s, N., Quinn, G., Au, A., Johns on, R., Rada, G., Bagg, M., Jones, A.,
532
Ra aud, P., Boden, C., Kahale, L., Rich e , B., Bois e , I., Kesha a z, H., Ryan, R., B and , L.,
533
Kolakowsky-Hayne , S.A., Salama, D., B azino a, A., Nag aj, S.K., Salan i, G., Buchbinde , R.,
534
Lasse son, T., San aguida, L., Champion, C., Law ence, R., San esso, N., Chandle , J., Les, Z.,
535
Schünemann, H.J., Cha idimou, A., Leuch , S., Shemil , I., Chou, R., Low, N., She i ali, D.,
536
Chu chill, R., Maas, A., Siemieniuk, R., Cnossen, M.C., MacLehose, H., Simmonds, M., Cossi,
537
M.-J., Macleod, M., Skoe z, N., Couno e, M., Ma shall, I., Soa es-Weise , K., C aigie, S.,
538
Ma shall, R., S ikan h, V., Dahm, P., Ma in, N., Sulli an, K., Danilkewich, A., Ma ínez Ga cía,
539
L., Synno , A., Danko, K., Ma e games, C., Taylo , M., Donoghue, E., Maxwell, L.J., Thaye , K.,
540
D essle , C., McAuley, J., Thomas, J., Egan, C., McDonald, S., T i on, R., Ellio , J., McKenzie,
541
J., Tsa na , G., Ellio , S.A., Mee pohl, J., Tugwell, P., E xeandia, I., Me ne , B., Tu geon, A.,
542
Fea he s one, R., Mondello, S., Tu ne , T., Foxlee, R., Mo ley, R., an Valkenhoe , G., Ga ne ,
543
P., Muna o, M., Vand ik, P., Ge i y, M., Munn, Z., Wallace, B., Glasziou, P., Mu ano, M.,
544
Wallace, S.A., G een, S., Newman, K., Wa s, C., G imshaw, J., Nieuwlaa , R., Weeks, L.,
545
Gu usamy, K., Nikolakopoulou, A., Weigl, A., Haddaway, N., Noel-S o , A., Wells, G., Ha ling,
546
L., O’Conno , A., Wie cioch, W., Hayden, J., Page, M., Wol enden, L., Hel and, M., Pahwa, M.,
547
Yepes Nuñez, J.J., Higgins, J., Pa do, J.P., Yos , J., Hill, S. and Pea son, L. (2017) “Li ing
548
sys ema ic e iews: 2. Combining human and machine e o ,” Jou nal o Clinical Epidemiology,
549
91 Else ie , pp. 31–37.
550
Tsa na , G., Dunn, A., Glasziou, P. and Coie a, E. (2013) “The au oma ion o sys ema ic
551
e iews,” BMJ, 346(jan10 1), pp. 139– 139.
552
Tugwell, P. and Haynes, R.B. (2006) “Assessing claims o causa ion,” Clinical epidemiology:
553
how o do clinical p ac ice esea ch, Lippinco Williams & Wilkins Philadelphia, PA, pp. 356–
554
387.
555
Tu ne , R.M., Spiegelhal e , D.J., Smi h, G.C.S. and Thompson, S.G. (2009) “Bias modelling in
556
e idence syn hesis,” Jou nal o he Royal S a is ical Socie y: Se ies A (S a is ics in Socie y),
557
172(1) John Wiley & Sons, L d, pp. 21–47.
558
Unde wood, A.J. (1991) “Beyond BACI: Expe imen al designs o de ec ing human
559
en i onmen al impac s on empo al a ia ions in na u al popula ions,” Ma ine and F eshwa e
560
Resea ch, 42(5), pp. 569–587.
561
Wallace, B.C., Dahab eh, I.J., Schmid, C.H., Lau, J. and T ikalinos, T.A. (2014) “Mode nizing
562
E idence Syn hesis o E idence-Based Medicine,” in G eenes, R. A. B. T.-C. D. S. (Second E.
563
(ed.) Clinical Decision Suppo . Ox o d: Else ie , pp. 339–361.
564
Wauchope, H.S., Amano, T., Geldmann, J., Johns on, A., Simmons, B.I., Su he land, W.J. and
565
Jones, J.P.G. (2020) “E alua ing Impac Using Time-Se ies Da a,” T ends in Ecology &
566
E olu ion
567
568
569
570
