Synthesizing and standardizing criteria for the evaluation of sustainability indicators in the water sector

SYNTHESIZING AND STANDARDIZING CRITERIA
FOR THE EVALUATION OF SUSTAINABILITY
INDICATORS IN THE WATER SECTOR
Pi es, A. 1*; Mo a o, J.2; Peixo o, H.1; B adley, S.1; Mulle , A.3
ABSTRACT
Indica o s a e one o he ools a ailable in planning and managemen p ojec s ha aid in he
decision-making p ocess and he moni o ing o hose decisions on he pa h owa ds he sus ainable
use and managemen o na u al esou ces. Howe e , he quali y and eliabili y o he indica o s
depends on he cons an imp o emen o he means o assess and design c i e ia se s. The
iden i ica ion and selec ion o c i e ia o e alua e indica o s is no a i ial ask. The esea ch
iden i ied a p oli e a ion o unconsolida ed c i e ia in use in he sus ainabili y and wa e esou ce
managemen domains. In esponse, a p ocess o syn hesis and consolida ion was unde aken in
o de o educe he le el o edundancies and o iden i y possible candida es o “co e c i e ia” ha
a e iden i ied as being a ele an pa o mos e alua ion amewo ks. A ep esen a i e collec ion o
sou ces om he specialized li e a u e was sc eened o e alua ion c i e ia. In o al, 74 sou ces
we e examined, con aining a o al o 346 men ions o c i e ia used o indica o assessmen . An in-
dep h syn hesis was pe o med using a s uc u ed ma ix o o ganize and iden i y he edundancies
in he c i e ia being u ilized. The analysis pe mi ed a educ ion o he 346 c i e ia ound o 60
unique c i e ia. The s udy p oposes a s anda d name and a desc ip ion o each c i e ion, aiming o
p o ide mo e cla i y and educe ambigui y. The c i e ia we e also anked o iden i y which c i e ia
we e in mo e sys emic use. O he 60 c i e ia ound, he 12 mos ci ed we e iden i ied as possible
co e c i e ia o amewo k de elopmen . Also, in o de o acili a e he design o indica o se s, all
60 c i e ia we e di ided in o wo app oaches (scien i ic/ op-down o end-use/bo om-up). This
s udy iden i ied signi ican edundancies and a lack o s anda diza ion in he use o c i e ia and i also
anked c i e ia o acili a e mul i-me hod amewo k de elopmen . Thus, i is c ucial ha indica o
de elope s no only conside c i e ia ha ha e some le el o s anda diza ion o be able o compa e
and communica e wi h o he agencies and communi ies bu also conside how o u ilize co e-
c i e ia in he design o indica o se s.
Keywo ds: indica o ; c i e ia; selec ion; op-down and bo om-up app oach, syn hesis,
s anda diza ion.
1 Uni e sidade Fede al da Bahia
2 Uni e sidad Poli ecnica de Ca alunya
3 In e na ional Li ing Fu u e Ins i u e
* Co esponding au ho : alex.pi es@u ba.b / +5571991780250
Ti le Page W/All Au ho con ac In o
SYNTHESIZING AND STANDARDIZING
CRITERIA FOR THE EVALUATION OF
SUSTAINABILITY INDICATORS
IN THE WATER SECTOR
ABSTRACT
Indica o s a e one o he ools a ailable in planning and managemen p ojec s ha aid in he
decision-making p ocess and he moni o ing o hose decisions on he pa h owa ds he
sus ainable use and managemen o na u al esou ces. Howe e , he quali y and eliabili y o
he indica o s depends on he cons an imp o emen o he means o assess and design
c i e ia se s. The iden i ica ion and selec ion o c i e ia o e alua e indica o s is no a i ial
ask. The esea ch iden i ied a p oli e a ion o unconsolida ed c i e ia in use in he
sus ainabili y and wa e esou ce managemen domains. In esponse, a p ocess o syn hesis
and consolida ion was unde aken in o de o educe he le el o edundancies and o
iden i y possible candida es o “co e c i e ia” ha a e iden i ied as being a ele an pa o
mos e alua ion amewo ks. A ep esen a i e collec ion o sou ces om he specialized
li e a u e was sc eened o e alua ion c i e ia. In o al, 74 sou ces we e examined,
con aining a o al o 346 men ions o c i e ia used o indica o assessmen . An in-dep h
syn hesis was pe o med using a s uc u ed ma ix o o ganize and iden i y he edundancies
in he c i e ia being u ilized. The analysis pe mi ed a educ ion o he 346 c i e ia ound o
60 unique c i e ia. The s udy p oposes a s anda d name and a desc ip ion o each c i e ion,
aiming o p o ide mo e cla i y and educe ambigui y. The c i e ia we e also anked o
iden i y which c i e ia we e in mo e sys emic use. O he 60 c i e ia ound, he 12 mos
ci ed we e iden i ied as possible co e c i e ia o amewo k de elopmen . Also, in o de o
acili a e he design o indica o se s, all 60 c i e ia we e di ided in o wo app oaches
(scien i ic/ op-down o end-use/bo om-up). This s udy iden i ied signi ican edundancies
and a lack o s anda diza ion in he use o c i e ia and i also anked c i e ia o acili a e
mul i-me hod amewo k de elopmen . Thus, i is c ucial ha indica o de elope s no
only conside c i e ia ha ha e some le el o s anda diza ion o be able o compa e and
communica e wi h o he agencies and communi ies bu also conside how o u ilize co e-
c i e ia in he design o indica o se s.
Keywo ds: indica o ; c i e ia; selec ion; op-down and bo om-up app oach, syn hesis,
s anda diza ion.
Manusc ip (au ho in o ma ion mus no appea in he manusc ip ) Click he e o iew linked Re e ences
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
1. INTRODUCTION
Indica o s a e ele an ins umen s o measu e, communica e, and o ganize en i onmen al
in o ma ion (Heink & Kowa ik, 2010; Lyy imäki, e al., 2013). Nicholson e al (2012) show
ha sus ainabili y indica o s can be powe ul policy decision ools when well-in eg a ed in o
he managemen p ocess. The e o e, hey should p esen a ibu es conside ed ele an by
he di e en g oups, police make s, esea che s, e c. ha de elop and apply hem (Klug &
Kmoch, 2014).
Indica o s a e an in eg al pa o a cyclical managemen p ocess as hey suppo he decision-
making p ocess as well as enable socie y o moni o he p og ess o he policies and
p og ams u ilized o mo e owa ds highe le els o sus ainabili y (WWAP, 2003). Wo k on
de eloping sus ainabili y indica o s has imp o ed signi ican ly since he 1980s and he need
o mo e consis en and applicable indica o sys ems con inues o be e iden (Sp eng &
Wils, 2000; Becke , 2010). Juwana, e al., 2012; Spangenbe g, 2008; McCool & S ankey,
2004, among o he s con i m ha he ex ensi e and in ense applica ion o indica o s by a
wide ange o use s in di e en con ex s is leading o as a ay o app oaches,
me hodologies and e minologies, o en wi h much o e lap and edundancy. I indica o use
is o emain e ec i e, he e is a need o cycle be ween momen s o c ea i i y in which new
indica o s and me hodologies a e p oposed and de eloped, and momen s o analysis,
syn hesis, and s anda diza ion in o de e lec on he needs o sha ing in o ma ion and o
a oid he c ea ion o an o e whelming numbe o indica o s wi h high le els o edundancy
which may damage he e ec i eness o indica o use o e all.
2. INDICATOR EVALUATION METHODS
The e is o en a ension be ween he need o p oduce scien i ically e i iable and ela i ely
s anda dized indica o s ha allow egions o be compa ed e sus he use o communi y
de eloped c ea i e and local-speci ic indica o s (Bell and Mo se, 2003; Reed e al., 2005).
Indica o s o en need o be compa ed o sha ed by egions o inse ed in o he simila
p ojec managemen p ocesses o in e na ional de elopmen agencies. Thus, he e i ica ion
and consolida ion o indica o s is becoming a signi ican a ea o esea ch and imp o emen .
The e a e se e al me hods ha can be used check he quali y o an indica o o a se o
indica o s, each o which will con ibu e di e en ly o consolida ing phases o indica o
de elopmen (A eline e . al, 2009; Bocks alle & Gi a din, 2003; James e al., 2012;
Cloquell-Balles e e al., 2006). These include:
 Modeling.
 Expe e alua ion.
 Re e en ial c i e ia.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
 Pa icipa o y selec ion.
 3S me hod - sel , scien i ic and social alida ion.
 Simila eal da a compa ison.
 Me hodological e ision.
 Applica ion and obse a ion (use ulness es ).
This a icle ocuses on he consolida ion o e e en ial c i e ia as used in he
sus ainabili y/wa e esou ce nexus. The use o c i e ia is he mos common me hod o
e alua ing and alida ing indica o s and is he e o e a c i ical a ea o esea ch. In e es ing
s udies in o he combina ion o e alua ion me hods is unde going bu as long as c i e ia-
based e alua ion con inues o be impo an , no sec o can comple ely a oid he need o
pe iodic e ision and consolida ion.
3. CRITERIA EVALUATION
To con ex ualize he no ion o c i e ia, i is common o conside ha he e a e o e a ching
gene al p inciples o which c i e ia a e a mo e speci ic posi i e cha ac e is ic and inally he
indica o a e he measu es hemsel es ha syn hesize agen ela ionships and “enable us o
gain an unde s anding o a complex sys em […] so ha e ec i e managemen decisions can
be aken ha lead owa ds ini ial objec i es” (Mi chell e al., 1995). Mendoza & P abhu
(adap ed 2000) p esen his hie a chy as ollows:
 P inciple: A undamen al u h o law as a basis o easoning o ac ion ha p o ides he
jus i ica ion o he c i e ia as e i ie s.
 C i e ia: Cha ac e is ics o s anda ds ha con ey speci ic meaning and/o ope a ionali y
o a p inciple wi hou i sel being a di ec measu e o pe o mance.
 Indica o : A measu e o an agen ela ionship o suppo sys em pe o mance used o
in e a ibu es o he success o sus ainabili y o he ela ionship.
C i e ion a e, he e o e, mos o en u ilized as he means o desc ibing he cha ac e is ics
ha will assu e e ec i eness in indica o s o indica o s se s hough such cha ac e is ics a e
ela i e and espec i e o he use s o hose indica o s. The o al numbe o exis ing c i e ia
appea s o be in he o de o hund eds and s ill g owing (WWAP, 2006; UNEP, 2006; and
Niemeije & G oo , 2008).
The Bellagio P inciples (Ha di & Zdan, 1997) we e an ini ial a emp and making a
connec ion be ween p inciples and c i e ia in o de o aid in he de elopmen o c i e ia se s
and indica o choice. I was an impo an i s s ep in he p ocess o consolida ion o c i e ia
and hey ein o ce he no ion ha he iden i ica ion o co e c i e ia may wo k as an ini ial
guide o he de elopmen and use o indica o s. The p inciples se e as a gene al guide o
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
aid in he p ocess o syn hesizing and in eg a ing he wo k o di e en g oups p oducing
indica o s wo ld-wide.
The quali y and eliabili y o he indica o s can be di ec ly a ec ed by he applica ion
me hod and he app op ia e choice o he c i e ia used o assess hem (Niemeije & de
G oo , 2008). These au ho s, Gudmundsson (2010) and o he s a gue ha he iden i ica ion
and selec ion o c i e ia o e alua e indica o s is no a i ial ask. The p ocess o selec ion
should be done in a anspa en and scien i ically- alid way and i should inco po a e
signi ican s akeholde pa icipa ion bu no a he expense o becoming an o e ly
uns uc u ed p ocess, as a balance is needed be ween op-down and bo om-up app oaches
(Reed e al., 2005).
No only is i necessa y o wo k on he imp o emen o he indica o s hemsel es bu i is
also necessa y o e alua e and consolida e he quali y o he c i e ia being used. A signi ican
numbe o wo ks and s udies ha e been published wi h his goal in mind (Gudmundsson,
2010; Ku ka & Blackwood, 2013). The e iew o c i e ia-based me hods done by
Gudmundsson (2010) demons a ed ha “a ich pale e o c i e ia – mo e o less well-
de ined - is a ailable o pick om he li e a u e, bu a uni e sal lis o c i e ia o assessing
indica o s does no exis .” Fu he mo e, he e is s ill no scien i ic consensus ega ding
which c i e ia a e conside ed mos ele an , al hough many c i e ia appea mul iple imes
ac oss a wide ange o s udies and may soon gain he s a us o co e c i e ia o mos
applica ions.
Al hough he e a e many me hodologies ha conside he c i e ia selec ion p ocess and
quali y o he c i e ia hemsel es (Cas illo & Pi ield, 2010; Callie a e al., 2013; Khadka, &
Vacik, 2012; Tanguay e al., 2013; Masca enhas e al., 2015; B eslow e al., 2017), hey
p esen mo e simila i ies han di e ences. Cu en ly mos include some componen o
mul i-s akeholde consul a ions and can be conside ed hyb id expe /pa icipa i e
app oaches. Some o m o weigh ing o MCA (Mul i-C i e ia Analysis) sys ems a e ypically
applied. This s udy also a o s a hyb id app oach as he mos alid. A combina ion o he
ollowing me hods is conside ed a obus o m o designing a amewo k, bu his a icle
will es ic i sel essen ially o he compa a i e s udy me hod as a necessa y i s s age
be o e mo e complex combina o ial me hods can be explo ed o he sus ainabili y/ wa e
esou ce nexus:
 Expe and pee e iew me hods/ op-down app oaches.
 Compa a i e s udies.
 F amewo k selec ion.
 Bo om-up app oaches.
All o hese me hods can con ibu e o he imp o emen o c i e ia selec ion in di e en
manne s. This s udy will men ion each o hese me hods, bu p esen esul s ocused on a
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

needed e iew and consolida ion o c i e ia use in he wa e esou ce sec o . Some
implica ions o u he combina ion o hese e alua ion me hodologies will be ouched
upon bu only as needed o con ex ualize he p ocess o syn hesis and consolida ion.
Top-down expe and pee e iew me hods will end o p esen logical consis encies and
co e mo e complex hemes and p ocesses. Decision make s will o en au oma ically look
o hose wi h p o essional and echnical expe ience o e alua e and sugges c i e ia se s.
S uc u ed expe consul a ions wi h echniques like he Delphi me hod can be u ilized wi h
success (Khadka, & Vacik, 2012). S a is ical me hods such as clus e analysis, and p incipal
componen s analysis can also play an impo an pa . The key quali y ha op-down
me hods can b ing is he assu ance ha he indica o se is designed in such a manne o
accoun o mos o he obse ed changes and can he e o e supply decision make s wi h he
base in o ma ion needed (Reed e al., 2005: 411). The e a e cases whe e expe consensus
does no lead o me hodological cla i y, bu some signi ican componen o expe
s uc u ing is usually conside ed an impo an phase. In sum, op-down app oaches ha e a
place in he c i e ion e alua ion p ocesses and will lead o mo e scien i ically consis en
c i e ion se s.
Compa a i e me hods use classi ica ion esea ch me hodology o conside lis s o c i e ia
al eady in use and a e especially use ul in iden i ying edundancies and inc easing
s anda diza ion (Deming & Swa ield, 2011). I can also help iden i y gaps in knowledge. As
men ioned abo e, i is impo an ha bo h c i e ia and indica o de elopmen pass h ough
phases o c ea ion and s anda diza ion so ha o ganiza ions and egions can wo k oge he
o compa e da a on managemen p ocesses. Compa a i e me hods will be an impo an pa
o he ea ly s age o mos e alua ions and i is his me hod ha is unde ocus he e.
An in e es ing a enue o c i e ia e alua ion a e hose ha use a ype o amewo k analysis
o de elop indica o se s. In his app oach, he indica o se is he key aspec and no he
quali y o he indi idual indica o (Niemeije , 2008). O as Dahl (2012) s a es: “Simply
compiling many sepa a e indica o s o sus ainabili y canno p o ide an adequa e measu e o
he o e all sus ainabili y o he sys em.” Again, he e a e a wide ange o di e en
amewo ks used o help in he selec ion and modi ica ion o indica o s. His o ically,
a ia ions o he DPSIR (D i ing Fo ce– P essu e–S a e– Impac -Response) o he cause-
e ec app oach has been a p incipal sys em used o s uc u e he selec ion p ocess and
sugges cohesi e indica o se s (WWAP, 2003). The DPSIR model like mos amewo ks
help o sp ead he indica o s o e a ange o human and na u al p ocesses, in heo y, gi ing
he o e all e alua ion mo e balance.
Niemeije (2008) sugges s ha causal-e ec amewo ks can be imp o ed o ha e mo e
e ec on he o ma ion o he indica o se s ins ead o being used as jus an o ganiza ional
me hod. The causal-e ec amewo k is also conside ed as a ype o sys ems app oach which
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
conside s he analysis o sys em in lows, s ocks, and ou pu s, acco ding o he concep o
sys em dynamics (WWAP, 2003).
O he common amewo ks include he compa men app oach in which he c i e ion co e
a ange o compa men s such as wa e , ai , ea h, and bio a, o simila ly a logical
amewo k whe e he amewo k is s uc u ed by goals which lead o ac i i ies. Fo
example, he goal o po e y educ ion is ealized by capaci y building in ag icul u al
sys ems and he e ec i eness o he capaci y building p ocess will need o be measu ed by
indica o s. Ku z e al., (2001) ake a sligh ly di e en app oach o e ing a amewo k ha
p esen s c i e ia in a hie a chical o ganiza ion ha conside ou p incipal g oups o c i e ia
o indica o selec ion and de elopmen . These a e:
 Concep ual ele ance
o Rele ance o he assessmen
o Rele ance o he ecological esou ce o unc ion a isk
 Feasibili y o implemen a ion
o Feasibili y o da a collec ion me hods
o Feasibili y o he logis ics
o Feasibili y o he in o ma ion managemen
o Feasibili y o he quali y assu ance
o Feasibili y o he mone a y cos s
 Response a iabili y
o Es ima ion o measu emen e o
o Wi hin-season empo al a iabili y
o Ac oss-yea a iabili y
o Spa ial a iabili y
 In e p e a ion and u ili y
o Disc imina o y abili y
o Da a quali y objec i es
o Assessmen h esholds ( o when o ake ac ion)
o Linkage o managemen ac ions
Wi h any amewo k, he need is o o e iding s uc u e, balance, and a means o
main aining a wholis ic app oach when conside ing he design o indica o se s.
Finally, bo om-up app oaches ha e he c i ical unc ion o in ol ing s akeholde s and
communi ies in he p ocess o de eloping his and all pa s o he decision-making p ocess.
A pa icipa i e app oach will no always lead o he o mula ion o clea c i e ia bu may
ins ead lead o he discussion o goals, objec i es, and impo an local issues which can lead
ei he o he di ec c ea ion o indica o s o he quali ies ha a e impo an o c i e ia. In
ac , in bo om-up app oaches he c i e ia may o en be o a mo e implici na u e. This
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
in o ma ion can be ansla ed in o c i e ia by specialis s in ol ed in he p ocess. In any case,
he design and iden i ica ion o a amewo k and he objec i es o he measu emen p ocess
ypically need local alida ion (Reed e al., 2005; F ase e al., 2006) and s akeholde
pa icipa ion can de e mine much o he concep ual s uc u e o he design o indica o
se s.
Impo an esea ch is needed in o he combina ion o hese me hods and as amewo k
models become inc easingly consis en , a ange o mul i ace ed me hods will eme ge. Once
mo e, i is impo an o s a e ha combina o ial esea ch is beyond he scope o his pape
which ocuses p incipally on he phase o compa ison and consolida ion. Consul a ion o
expe s is done obliquely h ough he quan i ica ion o li e a u e e e ences. Finally, while a
amewo k is no p esen ed a simple wo-pa (scien i ic o end-use ), p e- amewo k
s uc u e is used he e only as o m o p esen ing he esul s o he syn hesis o he c i e ia.
4. METHODOLOGY
Wi hin he discussion o c i e ia, his a icle deals p incipally wi h he use o classi ica ion
me hodology in o de o syn hesize and p opose some le el o s anda diza ion o c i e ia
wi hin he sus ainabili y/wa e esou ce use nexus. Wi h a e iew o he li e a u e, i was
possible o indica e some endencies in suppo o ce ain c i e ia as possible candida es o
s a us as “co e c i e ia.” These a e c i e ia ha ha e appea ed in a as a ay o s udies and
me hods as well as e lec he esul s o s udies om o he sec o s and con o m o he
Bellagio p inciples. As wi h he indica o s hemsel es, he numbe o c i e ia in use has
become nume ous and needs cons an e ision and consolida ion. Tha is he pu pose o his
s udy. The syn hesis p esen ed ep esen s an example o a p ocess o assessmen and
consolida ion wi hin he sus ainabili y and wa e esou ce use sec o and i is no mean o
be a comp ehensi e e ision o c i e ia o indica o s om all a eas connec ed o sus ainable
de elopmen . I was obse ed ha consolida ion is essen ial, and his assessmen would be a
aluable con ibu ion in main aining he o e all quali y o c i e ia in common use.
The p ocess o syn hesis s a ed wi h a bibliog aphical sea ch in o de o acqui e a
signi ican ly ep esen a i e selec ion o c i e ia ocusing on he domains o sus ainabili y and
wa e esou ce managemen . I included publica ions p esen ed by in e na ional ins i u ions
and na ional go e nmen s, as well as ones add essed by he scien i ic communi y in pee
e iewed in e na ional jou nals. In o al, 74 sou ces we e iden i ied in li e a u e e ision
and keywo d sea ches on Google Schola , Web o Science, and Scopus. These we e
examined and ound o con ain a o al o 346 men ions o c i e ia used o indica o
assessmen . These sou ces include publica ions om in e na ionally ecognized ins i u ions
ha a e enowned o hei eliable wo k wi h indica o s, such as he CBD (1999), EEA
(2005), FAO (1999), GRI (2002), IISD (2008), OECD (2003), UN (2007), UNEP (2006),
US EPA (2000), US GAO (2004), Wo ld Bank (2000), WHO (2002) and WWAP (2006).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
This s udy also examined a signi ican numbe o pee e iewed scien i ic pape s (A eline e
al., 2009; Bélange e al., 2012; B inghen i e al., 2011; Cloquell-Balles e e al., 2006;
Gudmundsson, 2010; Ku ka & Blackwood, 2013; Meul e al., 2009, Niemeije & de G oo
2008). Howe e , he majo i y o he publica ions analyzed by his esea ch p o ided
insu icien de ail abou he c i e ia selec ion p ocess, educing he possibili y o scien i ic
eplica ion (A eline e al., 2009).
Mos o he sou ces con ain a lis o c i e ia o he e alua ion o indica o s ela ed o wa e
esou ces, ishe y and ag icul u e bu o he sec o s we e conside ed including
anspo a ion, o es y, heal h, ene gy, biodi e si y and planning. S udies ha we e applied
on mul iple scales including local/na ional/in e na ional we e also conside ed. Fi e me a-
e iews o c i e ia o selec indica o s we e also analyzed, namely: WWAP (2006), UNEP
(2006), Niemeije & G oo (2008) and Ku ka and Blackwood (2013).
A ma ix o he 346 c i e ia iden i ied in his s udy was buil in o de o pe o m
classi ica ion analysis in o de o syn hesize and p opose possible s anda diza ion. The name
and de ini ion o he c i e ia we e ans e ed om he o iginal sou ces o he ma ix and
each c i e ion was examined in o de o demons a e he le els o o e lap, edundancy and
ambigui y. This syn he ic examina ion e ealed ha he 346 c i e ia we e in ac 60 unique
c i e ia. Redundancies we e p incipally ound o be ei he he use o he same name bu
wi h di e en de ini ions o he use o di e en names wi h de ini ions indica ing ha hey
we e, in ac , he same c i e ion. The scale o he educ ion om 364 o 60 demons a es
he signi ican le els o edundancy.
The objec i e o his s udy was o cla i y how c i e ion a e mul iplying and hus diminishing
e ec i eness due o high le els o edundancy. P oposals o s anda diza ion a e en a i e
and limi ed a his ime bu an essen ial i s s ep o c ea ing a mul i ace ed me hod o he
design o c i e ia se s o he sec o . The s udy p oposes a s anda d name and a desc ip ion
o each c i e ion, based on he ones p esen ed by he sou ces analyzed and he in insic
aspec s o each c i e ion. Fu he mo e, quan i ica ion based on he numbe o ci a ions
men ioned was ealized o suppo he ele ancy and s anda diza ion p ocess (Lu z & Hans-
Die e , 2008; Radicchi & Cas ellano, 2012). The numbe o sou ces ha conside each
c i e ion in ques ion as ele an was coun ed and he mo e imes a c i e ion was ci ed
indica ed ha hey we e s onge candida es o he unc ion o “co e c i e ia” o be u ilized
in u u e amewo k de elopmen .
In a u he e o o acili a e he use o he 60 indica o s wi h hei ele ance anking, he
c i e ia we e dis ibu ed in o wo p e- amewo k g oups one ocused on scien i ic ele ance
and accu acy and o mo e p obable in e es o op-down me hods and an end-use g oup
which inco po a es Ku z e al’s (2001) no ions o easibili y and u ili y and hese would
ypically be associa ed wi h bo om-up app oaches. This g ouping me hod is simila o ha
used by WWAP (2006). Bo h g oups p esen highly ci ed c i e ia ha indica e a ce ain
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
CBD – Con en ion on Biological Di e si y. (1999). De elopmen o Indica o s o Biological
Di e si y. Nai obi: Con en ion on Biological Di e si y, subsidia y Body on Scien i ic,
Technical and Technological Ad ice. Repo No. UNEP/CBD/SBSTTA/5/12, 14 pp
Cla k, W., & Dickson, N. (1999). The global en i onmen al assessmen p ojec : Lea ning om
e o s o link science and policy in an in e dependen wo ld. Acclima ions, 8, 6–7. Re ie ed
om h p://www.hks.ha a d.edu/gea/pubs/99a _wc_geaacc.pd
Cloquell-Balles e , V. A., Mon e de-Díaz, R., & San ama ina-Siu ana, M. C. (2006). Indica o s
alida ion o he imp o emen o en i onmen al and social impac quan i a i e assessmen .
En i onmen al Impac Assessmen Re iew, 26(1), 79–105. doi:10.1016/j.eia .2005.06.002
Dahl, A.L. (2012). Achie emen s and gaps in indica o s o sus ainabili y. Ecological Indica o s 17
(2012) 14–19.
Deming, M.E. & Swa ield, S. (2011). Landscape A chi ec u e Resea ch: Inqui y, S a egy, Design.
Hoboken: John Wiley & Sons, Inc.
Doukas, H.C., And eas, B.M., & Psa as, J. E. (2007). Mul i-c i e ia decision aid o he
o mula ion o sus ainable echnological ene gy p io i ies using linguis ic a iables. Eu opean
Jou nal o Ope a ional Resea ch, 182:844–55.
EC - En i onmen al signals. (2003). Canada’s na ional en i onmen al indica o se ies 2003.
En i onmen Canada. Re ie ed om h p://www.ec.gc.ca/soe -
ee/English/Indica o _se ies/de- aul .c m#pic
EEA - Eu opean En i onmen Agency. (2003). Eu ope’s en i onmen : The hi d assessmen : En-
i onmen al assessmen epo no. 10. Copenhagen: Eu opean En i onmen Agency.
Re ie ed om h p:// epo s.eea.eu.in /en i onmen-
al_assessmen _ epo _2003_10/en/ ab_con en _RLR, on 2 Oc obe 2004.
EEA - Eu opean En i onmen Agency. (2005). EEA Co e Se o Indica o s—Guide. Eu opean
En i onmen Agency, Copenhagen. Repo No. 1/2005, 37 pp.
FAO - Food and Ag icul u e o ganiza ion o he Uni ed Na ions. (1999). Indica o s o Sus ainable
De elopmen o Ma ine Cap u e Fishe ies. Rome: Food and Ag icul u e O ganiga ion
(FAO). Re ie ed om p:// p. ao.o g/doc ep/ ao/004/x3307e/x3307e00.pd
F ase , E. D. G., Dougill, A. J., Mabee, W. E., Reed, M., & McAlpine, P. (2006). Bo om up and
op down: analysis o pa icipa o y p ocesses o sus ainabili y indica o iden i i- ca ion as a
pa hway o communi y empowe men and sus ainable en i onmen al managemen . Jou nal o
En i onmen al Managemen , 78:114–27.
G aymo e, M. L. M., Sipe, N. G., & Rickson, R. E. (2008). Regional sus ainabili y: How use ul
a e cu en ools o sus ainabili y assessmen a he egional scale? Ecological Economics, 67(3),
362-372. doi:10.1016/j.ecolecon.2008.06.002
G aymo e, M. L. M.,Wallis, A. M., & Richa ds, A. J. (2009). An index o egional sus ainabili y: a
GIS-based mul iple c i e ia analysis decision suppo sys em o p og essing sus ainabili y.
Ecological Complexi y, 6, 453–62.
GRI - Global Repo ing Ini ia i e. (2002). Sus ainabili y epo - ing guidelines 2002. Re ie ed
om h p://www. global epo ing.o g/guidelines/2002/con en s.asp, on 7 Ap il 2004
Gudmundsson, H. (2010). C i e ia and me hods o indica o assessmen and alida ion - a e iew o
gene al and sus ainable anspo ela ed indica o c i e ia and how o apply hem. Re ie ed om
h p://cos 356.in e s. /pub/ e e ence/ epo s/C356_2.2_ epo _c i e ia_HG_220410.p
d
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

Hák, T., Moldan, B., & Dahl, A. L. (Eds.). (2012). Sus ainabili y indica o s: a scien i ic assessmen
(Vol. 67). Island P ess.
Hák, T., Janouško á, S., & Moldan, B. (2016). Sus ainable De elopmen Goals: A need o
ele an indica o s. Ecological Indica o s, 60, 565-573.
Ha di, P., & Te ence Z. (1997). eds. Assessing sus ainable de elopmen : p inciples in p ac ice.
Winnipeg: In e na ional Ins i u e o Sus ainable De elopmen , Winnipeg, Saska chewan.
Re ie ed om h ps://www.iisd.o g/pd /bellagio.pd
Heink, U., & Kowa ik, I. (2010). Wha a e indica o s? On he de ini ion o indica o s in ecology
and en i onmen al planning. Ecological Indica o s, 10(3), 584–593.
h p://doi.o g/10.1016/j.ecolind.2009.09.009
IISD - In e na ional Ins i u e o Sus ainable De eloped. (2008). Bellagio STAMP: Sus ainabili y
Assessmen and Measu emen P inciples. Winnipeg: IISD. Re ie ed om
h p://www.iisd.o g/pd /2009/b ochu e_bellagios amp.pd
ITFM - In e go e nmen al Task Fo ce on Moni o ing Wa e Quali y. (1995) Wa e -quali y
moni o ing in he Uni ed S a es. U.S. Geological Su ey, Wa e In o ma ion Coo dina ion
P og am, Washing on. h p://acwi.go /appendixes/index.h ml
James, C. A., Ke shne , J., O’Neill, S., & Le in, P. S. (2012). A me hodology o e alua ing and
anking wa e quan i y indica o s in suppo o ecosys em-based managemen . En i onmen al
Managemen , 49(3), 703–19. doi:10.1007/s00267-012-9808-7
Juwana, I., Mu il, N., & Pe e a, B. J. C. (2012). Indica o -based wa e sus ainabili y assessmen -
a e iew. The Science o he To al En i onmen , 438, 357–71.
doi:10.1016/j.sci o en .2012.08.093
Khadka, C., & Vacik, H. (2012). Compa ing a op-down and bo om-up app oach in he
iden i ica ion o c i e ia and indica o s o sus ainable communi y o es managemen in
Nepal., Fo es y: An In e na ional Jou nal o Fo es Resea ch, 85(1), 145-158.
Klug, H., & Kmoch, A. (2014). Ope a ionalizing en i onmen al indica o s o eal ime mul i-
pu pose decision making and ac ion suppo . Ecological Modelling, 295, 66–74.
doi:10.1016/j.ecolmodel.2014.04.009
Ku ka, T., & Blackwood, D. (2013). Pa icipa o y selec ion o sus ainabili y c i e ia and indica o s
o bioene gy de elopmen s. Renewable and Sus ainable Ene gy Re iews, 24, 92–102.
doi:10.1016/j. se .2013.03.062
Ku z, J.C., Jackson, L.E., & Fishe , W.S. (2001). S a egies o E alua ing Indica o s Based on
Guidelines om he En i onmen al P o ec ion Agency’s O ice o Resea ch and
De elopmen . Ecological Indica o s, 1(1), 49–60.
La imo e, B., Smi h, C. T., Ti us, B. D., S upak, I., & Egnell, G. (2009). En i onmen al ac o s
in wood uel p oduc ion: oppo uni ies, isks, and c i e ia and indica o s o sus ainable
p ac ices. Biomass & Bioene gy, 33, 1321–42.
Lu z, B., & Hans-Die e , D. (2008). Wha do ci a ion coun s measu e? A e iew o s udies on
ci ing beha io , Jou nal o Documen a ion, 64(1),45-80.
Lyy imäki, J., Tapio, P., Va ho, V., & Söde man, T. (2013). The use, non-use and misuse o
indica o s in sus ainabili y assessmen and communica ion. In e na ional Jou nal o Sus ainable
De elopmen & Wo ld Ecology, 20(5), 385–393.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
Mendoza, G. A., & P abhu, R. (2000). De elopmen o a me hodology o selec ing c i e ia and
indica o s o sus ainable o es managemen : a case s udy on pa icipa o y assessmen .
En i onmen al managemen , 26(6), 659-673.
Masca enhas, A., Nunes, L. M., & Ramos, T. B. (2015). Selec ion o sus ainabili y indica o s o
planning combining s akeholde s' pa icipa ion and da a educ ion echniques. Jou nal o
Cleane P oduc ion, 92, 295-307.
McCool, S. F., & S ankey, G. H. (2004). Indica o s o sus ainabili y: Challenges and oppo uni ies
a he in e ace o science and policy. En i onmen al Managemen , 33(3), 294–305.
doi:10.1007/s00267-003-0084-4
Meul, M., Ne ens, F., & Reheul, D. (2009). Valida ing sus ainabili y indica o s: Focus on
ecological aspec s o Flemish dai y a ms. Ecological Indica o s, 9(2), 284–295.
doi:10.1016/j.ecolind.2008.05.007
Mi chell, G., May, A., & McDonald, A. (1995). PICABUE: a me hodological amewo k o he
de elopmen o indica o s o sus ainable de elopmen . The In e na ional Jou nal o
Sus ainable De elopmen & Wo ld Ecology, 2(2), 104-123.
Nicholson, E., Collen, B., Ba ausse, A., Blancha d, J. L., Cos elloe, B. T., Sulli an, K. M. E., …
Milne -Gulland, E. J. (2012). Making obus policy decisions using global biodi e si y
indica o s. PLoS ONE, 7(7). doi:10.1371/jou nal.,pone.0041128
Niemeije , D., & de G oo , R. S. (2008). A concep ual amewo k o selec ing en i onmen al
indica o se s. Ecological Indica o s, 8(1), 14–25. doi:10.1016/j.ecolind.2006.11.012
NCHOD 2005. Compendium o Clinical and Heal h Indica o s Use Guide. Na ional Cen e o
Heal h Ou comes De elopmen (NCHOD), London si e - London School o Hygiene and
T opical Medicine.
NRC - Na ional Resea ch Council. (2000). Ecological Indica o s o he Na ion. Washing on, DC:
Na ional Academy P ess. Re ie ed om h p://books.nap.edu/
books/0309068452/h ml/1.h ml.
OECD - O ganiza ion o Economic Co-ope a ion and De elopmen . (2001). En i onmen al
Indica o s: Towa ds Sus ainable De elopmen . Pa is: OECD, 155 pp.
OECD - O ganiza ion o Economic Co-ope a ion and De elopmen . (2003). OECD
en i onmen al indica o s: De elopmen , measu emen and use, Re e ence pape . Pa is:
OECD. Re ie ed o m h p://www. oecd.o g/da aoecd/7/47/24993546.pd .
OECD - O ganiza ion o Economic Coope a ion and De elopmen . (1994). En i onmen al
indica o s: OECD co e se . Pa is
Ols hoo n, X., Ty eca, D., Weh meye , W., & Wagne , M. (2001). En i onmen al indica o s o
business: a e iew o he li e a u e and s anda disa ion me hods. Jou nal o Cleane P oduc ion,
9, 453–63.
Pa is, T. M., & Ka es, R. W. (2003). Cha ac e izing and Measu ing Sus ainable De elopmen .
Annual Re iew o En i onmen and Resou ces, 28(1), 559–586.
doi:10.1146/annu e .ene gy.28.050302.105551
Pas ille Conso ium (The). (2002). Indica o s in o ac ion: Local sus ainabili y indica o se s in hei
con ex . Final epo . London: London School o Economics h p://www.lse.
ac.uk/Dep s/geog aphy/Pas ille/FinalRepo Web.pd .
P esco -Allen, R. (2001). The wellbeing o na ions. Washing on DC: Island P ess.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
Radicchi, F., & Cas ellano, C. (2012). Tes ing he ai ness o ci a ion indica o s o compa ison
ac oss scien i ic domains: The case o ac ional ci a ion coun s. Jou nal o In o me ics, 6(1),
121–130. doi:10.1016/j.joi.2011.09.002
Reed, M., F ase , E. D., Mo se, S., & Dougill, A. J. (2005). In eg a ing me hods o de eloping
sus ainabili y indica o s o acili a e lea ning and ac ion. Ecology and socie y, 10(1).
Ro e e, E. L. L., Soa es, J. B., Oli ei a, L. B., & Lau ia, T. (2010). Sus ainable expansion o
elec ici y sec o : sus ainabili y indica o s as an ins umen o suppo decision making.
Renewable & Sus ainable Ene gy Re iews, 14, 422–9.
Segnes am, L. (2002). Indica o s o en i onmen and sus ainable de elopmen : heo ies and
p ac ical expe ience. Washing on: Wo ld Bank. Re ie ed om
h p://si e esou ces.wo ldbank.o g/INTEEI/936217-
1115801208804/20486265/Indica o so En i onmen andSus ainableDe elopmen 2003
Shmele , S. E., & Rod íguez-Labajos, B. (2009). Dynamic mul idimensional assessmen o
sus ainabili y a he mac o le el: he case o Aus ia. Ecological Economics, 68, 2560–73.
Singh, A., Bed ich M., & Thomas L. (2002). Making science o sus ainable de elopmen mo e
policy ele an : New ools o analysis. ICSU Se ies on Science o Sus ainable De elopmen ,
no. 8. In e na ional Council o Science Re ie ed om
h p://g id2.c .usgs.go /publica ions/new ools.pd .
Singh, R. K., Mu y, H. R., Gup a, S. K., & Dikshi , A. K. (2009). An o e iew o sus ainabili y
assessmen me hodologies. Ecological Indica o s, 9(2), 189-212.
doi:10.1016/j.ecolind.2008.05.011
SNZ - S a is ics New Zealand. (2002) Socio-economic indica o s o he en i onmen .
En i onmen al S a is ics Team, Ch is chu ch.
h p://www2.s a s.go .nz/domino/ex e nal/web/p od_se .ns /
092edeb76ed5aa6bcc256a e0081d84e/94b7 3198c9d9111cc256c1500171ea6?OpenDocume
n
Spangenbe g, J. H. (2008). Second o de go e nance: Lea ning p ocesses o iden i y indica o s.
Co po a e Social Responsibili y and En i onmen al Managemen , 15(3), 125–139.
h p://doi.o g/10.1002/cs .137
Sp eng, D. & Wils, A. (2000) Indica o s o Sus ainabili y: Indica o s in Va ious Scien i ic
Disciplines, Alliance o Global Sus ainabili y. AGS Repo .
Tanguay, G. A., Rajaonson, J., & The ien, M. C. (2013). Sus ainable ou ism indica o s: Selec ion
c i e ia o policy implemen a ion and scien i ic ecogni ion. Jou nal o Sus ainable Tou ism,
21(6), 862-879.
UN - Uni ed Na ions. (2007). Indica o s o Sus ainable De elopmen  : Guidelines and Me hodologies. New
Yo k: Uni ed Na ions. Re ie ed om
h p://www.un.o g/esa/sus de /na lin o/indica o s/guidelines.pd
UNEP - Uni ed Na ions En i onmen P og amme. (2006). En i onmen al Indica o s o No h Ame ica.
Nai obi: UNEP. Re ie ed om h p://www.unep.o g/pd /NA_Indica o s_FullVe sion.pd
US EPA – Uni ed S a es En i onmen al P o ec ion Agency. (2000). E alua ion Guidelines Fo
Ecological Indica o s. Washing on. Re ie ed om
h p://www.epa.go /emap/h ml/pubs/docs/ esdocs/ecol_ind.pd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
US GAO - Uni ed S a es Go e nmen Accoun abili y O ice. (2004). En i onmen al indica o s:
Be e coo dina ion is needed o de elop en i onmen al indica- o s se s ha in o m
decisions. Repo o Cong essional eques e s. US Go e nmen Accoun abili y O ice.
Ve a, I., & Langlois, L. (2007). Ene gy indica o s o sus ainable de elopmen . Ene gy, 32, 875–82.
Wang, J., Jing, Y., Zhang, C., & Zhao, J. (2009). Re iew on mul i-c i e ia decision analysis aid in
sus ainable ene gy decision-making. Renewable & Sus ainable Ene gy Re iews, 13,2263–78.
WHO - Wo ld Heal h O ganiza ion. (2002). Heal h in Sus ainable De elopmen Planning: The Role o
Indica o s (p. 42). Gene a.
Wo ld Bank. (2000). De eloping Indica o s: Lessons Lea ned om Cen al Ame ica (p. 22). Washing on.
Re ie ed om h p://si e esou ces.wo ldbank.o g/INTEEI/811099-
1115809852605/20486445/De elopingIndica o sLessonsLea nedF omCen alAme ica2001p
a 1.pd
WWAP - Wo ld Wa e Assessmen P og amme. (2003). The Uni ed Na ions Wo ld Wa e
De elopmen Repo : Wa e o People Wa e o li e. Pa is: UNESCO, and London:
Ea hscan.
WWAP - Wo ld Wa e Assessmen P og amme. (2006). The Uni ed Na ions Wo ld Wa e
De elopmen Repo 2: Wa e - A Sha ed Responsibili y. Pa is: UNESCO, and London:
Ea hscan.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65