Measu es and Me a-Measu es o he
Supe ised E alua ion o Image Segmen a ion
Jo di Pon -Tuse and Fe an Ma ques
Uni e si a Poli `
ecnica de Ca alunya Ba celonaTech∗
h p://ima ge.upc.edu
Abs ac
This pape ackles he supe ised e alua ion o image
segmen a ion algo i hms. Fi s , i su eys and s uc u es
he measu es used o compa e he segmen a ion esul s wi h
a g ound u h da abase; and p oposes a new measu e:
he p ecision- ecall o objec s and pa s. To compa e he
goodness o hese measu es, i de ines h ee quan i a i e
me a-measu es in ol ing six s a e o he a segmen a ion
me hods. The me a-measu es consis in assuming some
plausible hypo heses abou he esul s and assessing how
well each measu e e lec s hese hypo heses. As a con-
clusion, his pape p oposes he p ecision- ecall cu es o
bounda ies and o objec s-and-pa s as he ool o choice
o he supe ised e alua ion o image segmen a ion. We
make he da ase s and code o all he measu es publicly
a ailable.
1. In oduc ion
Since he ad en o sliding window objec de ec o s [32],
much e o has been pu in o p o iding be e spa ial delin-
ea ion beyond sliding windows [16]. Seman ic segmen a-
ion is he inal objec i e, whe e de ec ion and segmen a ion
mee , bu i is s ill a om being sol ed [8].
In his scena io, bo om-up segmen a ion me hods o en
play an impo an ole in he p oposed algo i hms [1,5],
and hus imp o ing segmen a ion echniques would en ail
imp o emen s owa ds be e seman ic segmen a ion [18].
In such a challenge, p o iding benchma ks ha help e-
sea che s unde s and he weak and s ong poin s o hei
algo i hms is o pa amoun impo ance.
In his di ec ion, in he ield o objec de ec ion assess-
men , Hoiem e al. [11] s ess ha he esul s should be
e alua ed beyond pe o mance summa y measu es in o de
o “help unde s and how one me hod could be imp o ed.”
∗This wo k has been pa ially suppo ed by he Spanish Minis e io
de Ciencia e Inno aci´
on, unde p ojec TEC2010-18094 and FPU g an
AP2008-01164.
(a) (b) (c)
Figu e 1. Examples o he me a-measu e p inciples: How good a e
he e alua ion measu es a dis inguishing hese pai s o pa i ions?
In o he wo ds, esea che s need be e eedback om he
e alua ion han a single numbe .
Back o segmen a ion assessmen , he p ecision- ecall
cu es o bounda ies [20] a e good examples o ools ha
p o ide iche eedback han he F-measu e used as sum-
ma y. Mo eo e , as poin ed ou by [2], in addi ion o
bounda y-based measu es, egion-o ien ed measu es should
be conside ed when assessing segmen a ions. Howe e , he
cu en ones a e limi ed o summa y measu es [30,22,20,
2,15,13,7,4,25,24].
Ou i s con ibu ion is a egion-based p ecision- ecall
en i onmen o he assessmen o image segmen a ion. In-
spi ed by [12,11] and by he ac ha pa s o objec s
a e impo an clues o objec de ec ion [9], we p esen he
p ecision- ecall o objec s and pa s, which is based on
classi ying he egions in o objec and pa s candida es.
Summa y measu es also play a ole in pe o mance com-
pa ison, hus he ques ion ha now a ises is how o compa e
he goodness o an e alua ion measu e. In o he wo ds, we
should de ine a me a-measu e o compa e he e alua ion
measu es. The p inciple o a me a-measu e is o assume a
plausible hypo hesis abou he segmen a ion e alua ion and
analyze how well measu es ma ch his hypo hesis.
Some p e ious wo ks based hei claims on quali a i e
me a-measu es, ha is, showing he beha io o he mea-
su es on pa icula quali a i e examples [4,30]. Ex ensi e
quan i a i e me a-measu es, howe e , a e desi able.
1
The i s app oach o an ex ensi e quan i a i e me a-
measu e was p oposed in [19]. The hypo hesis in his wo k
was ha measu es should be able o disc imina e be ween
wo pai s o human-ma ked pa i ions coming om di e -
en images ( o ins ances, he wo pa i ions in Figu e 1.a).
In an anno a ed da abase wi h mul iple pa i ions pe image,
he quan i a i e me a-measu e was de ined as he numbe o
same-image pa i ion pai s ha he measu e judges as less
simila han o he pai s o pa i ions coming om di e en
images. [14] p esen ed a compa ison o some measu es in
e ms o his me a-measu e.
The second con ibu ion o his pape is o p esen wo
new me a-measu es. Ins ead o basing ou hypo heses on
human-made pa i ions, we ex end he analysis o pa i ions
om six S a e-o - he-A (SoA) segmen a ion algo i hms.
The i s assump ion is ha measu es should be capable
o dis inguishing such pa i ions om hose ob ained wi h-
ou aking in o accoun he con en o he image. In ou
case, ollowing he p oposal in [2], we use a quad ee, i.e.,
a hie a chical homogeneous ec angula g id. The me a-
measu e is hen de ined as he numbe o esul s om SoA
algo i hms ha a e judged wo se han he quad ee. As a
quali a i e example, we assess how well a measu e dis in-
guishes be ween pa i ions like Figu e 1.b.
As a second app oach, we assume ha any measu e
should be able o dis inguish a pa i ion ob ained by a SoA
me hod on an image om a pa i ion ob ained by he same
me hod bu on a di e en image, as he wo pa i ions shown
in Figu e 1.c. The me a-measu e in his case is de ined as
he numbe o cases in which he measu e co ec ly judges
he same-image pa i ion as be e .
The hi d con ibu ion is o su ey and s uc u e a wide
se o e alua ion measu es and he newly-p oposed one and
compa e hem using he h ee p e iously discussed me a-
measu es. We show ha he wo p ecision- ecall measu es
(bounda y- and objec s-and-pa s-based) ha e ou s anding
esul s as summa y measu es wi h espec o he es o mea-
su es, while p o iding iche in o ma ion o esea che s
o in e p e he esul s. We u he in e p e hese wo
p ecision- ecall en i onmen s by compa ing six SoA seg-
men a ion algo i hms.
We make he code o compu e all he measu es publicly
a ailable in [28], as well as all he segmen a ion esul s o
make ou esea ch ep oducible and o make i e o less o
esea che s o assess hei segmen a ion me hods.
The emainde o he pape is o ganized as ollows. Sec-
ion 2 e iews and s uc u es he main segmen a ion mea-
su es a ailable in he li e a u e. Sec ion 3mo i a es and de-
sc ibes he newly p oposed measu e. Sec ion 4p esen s he
wo new me a-measu es and he al eady a ailable one used
o compa e he e alua ion measu es. Sec ion 5p esen s
he expe imen al compa ison o he measu es using he
h ee me a-measu es. I also shows he applicabili y o he
bounda y-based and he newly p oposed p ecision- ecall
cu es o objec s and pa s in he compa ison o six SoA
segmen a ion echniques. Sec ion 6concludes he pape .
2. Measu e Re iew and S uc u e
The s a e-o - he-a measu es can be classi ied depend-
ing on he image pa i ion in e p e a ion on which hey a e
based. The mos common in e p e a ion is as a clus e ing o
he pixel se in o a numbe o subse s o egions, A pa i ion
can also be in e p e ed as a wo-class clus e ing o he se o
pai s o pixels, wi h some pai s linking pixels om he same
egion and o he s linking pixels om di e en egions. Fi-
nally, a pa i ion can be ep esen ed as a wo-class clus e ing
o he pixel con ou s in o bounda ies and non-bounda ies.
The ollowing sec ions e iew he main measu es ound
unde each o hese in e p e a ions, keeping he no a ion
om he o iginal pape s whe e possible. Table 1shows an
o e iew o he s udied measu es.
2.1. Pixel-Se Clus e ing
The di ec ional Hamming dis ance om one pa i ion
S o ano he S0[15,13] is de ined as:
DH(S⇒S0) = n−X
R0∈S0
max
R∈S|R0∩R|(1)
whe e Rand R0a e egions in Sand S0, espec i ely, and
nis he numbe o pixels in he image. In [4] his same
measu e was coined as asymme ic pa i ion dis ance. I
is equi alen o he achie able segmen a ion accu acy [23]
used in supe pixel assessmen .
A symme ic e sion o his measu e was p esen ed
in [7] as he an Dongen dis ance:
d D (S, S0) = DH(S0⇒S) + DH(S⇒S0)(2)
The segmen a ion co e ing o a pa i ion Sby a pa i-
ion S0was de ined in [2] as:
C(S0→S) = 1
nX
R∈S
|R| · max
R0∈S0
|R∩R0|
|R∪R0|(3)
The in ui i e s ep u he is o measu e he maximum
o e lap when pe o ming a bijec i e ma ching be ween he
egions o he wo pa i ions. This idea was p esen ed in [4]
as symme ic pa i ion-dis ance, in [14] as bipa i e-g aph-
ma ching (BGM) dis ance, and in he con ex o clus e ing
compa ison, in [22] as classi ica ion e o dis ance. I is
shown in [4] ha i is equi alen o he minimum numbe
o pixels ha mus no be aken in o accoun o he wo
pa i ions o be iden ical.
In [19], he consis ency o he BSDS300 human pa i-
ions is analyzed by means o wo measu es GCE,LCE,
aiming a being obus agains di e en g anula i ies o he
Pa i ion In e p e a ion Measu e Rep esen a i e Re e ences No a ion
Pixel-se clus e ing
Di ec ional Hamming dis ance [13,4]DH
an Dongen dis ance [7]d D
Segmen a ion co e ing [2]C
Bipa i e g aph ma ching [14,4]BGM
Bidi ec ional consis ency e o [19]BCE
Va ia ion o in o ma ion [22]VoI
Pai s-o -pixels classi ica ion P obabilis ic Rand index [26,30]PRI
P ecision-Recall o egions [19]P ,R
Bounda y map P ecision-Recall o bounda ies [17,19]Pb,Rb
Table 1. Measu e s uc u e o e iew o he h ee in e p e a ions o an image pa i ion
scene in e p e a ion. As he au ho poin s ou , hese mea-
su es a e no sui able o gene al-pu pose image segmen a-
ion e alua ion. The same wo k p oposes a measu e ha is
no anspa en o o e segmen a ion: he bidi ec ional con-
sis ency e o (BCE), which can be ew i en as:
BCE(S,S0
)=1−1
nX
R∈S
R0∈S0
|R∩R0|min|R∩R0|
|R|,|R∩R0|
|R0|
(4)
The wo k in [22] in oduced a new poin o iew o he
measu es o clus e ing assessmen based on in o ma ion-
heo e ic esul s. The au ho de ines a disc e e andom a i-
able aking N alues ha consis s in andomly picking any
pixel in he pa i ion S={R1, . . . , RN}and obse ing he
egion i belongs o. Assuming all he pixels equally p ob-
able o pick, he en opy H(S)associa ed wi h a pa i ion
is de ined as he en opy o such andom a iable. The mu-
ual in o ma ion I(S,S0
)be ween wo pa i ions is de ined
equi alen ly. The a ia ion o in o ma ion is hen:
VoI (S, S0) = H(S) + H(S0)−2I(S, S0)(5)
I can be no malized by log N, i s maximum possible alue.
2.2. Pai s-o -Pixels Classi ica ion
An image pa i ion can be iewed as a classi ica ion o all
he pai s o pixels in o wo classes: pai s o pixels belong-
ing o he same egion, and pai s o pixels om di e en
egions. Fo mally, le I={p1, . . . , pn}be he se o pix-
els o he image and conside he se o all pai s o pixels
P={(pi, pj)∈I×I|i<j}. Gi en wo pa i ions Sand
S0, we di ide Pin o ou di e en se s, depending on whe e
a pai (pi, pj)o pixels all [22]:
P11: in he same egion bo h in Sand S0,
P10: in he same egion in Sbu di e en in S0,
P01: in he same egion in S0bu di e en in S,
P00: in di e en egions bo h in Sand S0.
The Rand index, o iginally de ined in [26] as a clus e -
ing e alua ion measu e, a ises na u ally in his con ex :
RI (S,S0
) = |P00 |+|P11|
|P| . I coun s he pai s o pixels ha
ha e cohe en labels o he wo pa i ions being compa ed,
wi h espec o he numbe o possible pai s o pixels.
In he con ex o image segmen a ion and ha ing a se
{Gi}o g ound- u h pa i ions o he same image, he
P obabilis ic Rand Index [30] is compu ed as:
PRI (S, {Gi}) = X
i
RI (S,Gi
)(6)
In his same con ex , he p ecision- ecall o e-
gions [19] is de ined as:
P =|P11|
|P11|+|P10|R =|P11|
|P11|+|P01|(7)
As a summa y measu e, he F measu e F is used.
This pai o measu es would be a candida e in ou ques
o a non-bounda y-based p ecision- ecall measu e. As i
will be shown in he expe imen s, howe e , his measu e
does no p o ide good me a-e alua ion sco es.
2.3. Bounda y Map
All measu es abo e could be applied o any clus e ing
algo i hm, no ma e he na u e o he elemen s being clas-
si ied. In ac , he majo i y o he indices p esen ed come
om he applica ion o gene al-clus e ing assessmen mea-
su es o image segmen a ion.
Image pixels, howe e , a e spa ially dis ibu ed in he im-
age plane, and so he concep o neighbo hood a ises na u-
ally. The e o e, an image pa i ion wi h connec ed com-
ponen s can be unambiguously de ined by hei bounda ies,
i.e., a bijec ion could be made be ween all possible image
pa i ions and all possible closed bounda ies maps.
Recalling he de ini ion o Pas he se o pai s o pixels
in he image, le us de ine he se o pai s o neighbo ing
pixels as N ⊂ P. One can de ine a bijec ion be ween he
se o bounda y segmen s Band Nlinking each segmen o
he pai o pixels a each o i s sides. Using his no a ion,
bounda y de ec ion can be unde s ood as a wo-class clus-
e ing o B, di iding he segmen s in o hose being bound-
a ies and hose no . This way, compa ing wo pa i ions can
be ansla ed in o compa ing wo clus e ing o B.
To be obus o unno iceable shi s o bounda y local-
iza ion, [17] p oposes o compu e he op imal ma ching be-
ween he segmen s o bounda ies o he wo pa i ions as
a maximum-weigh bipa i e-g aph ma ching. The algo-
i hm is imp o ed in [19,20] leading o he well-known
p ecision- ecall o bounda ies (Pb,Rb, and Fb).
3. Measu e P oposal
In he con ex o image segmen a ion e alua ion,
p ecision- ecall cu es o bounda ies [19,20] a e a boon
o esea che s. They s a is ically e lec , o ins ance, ha
an algo i hm is p o iding oo coa se segmen a ions (low e-
call, high p ecision) o ins ead i s esul s a e oo agmen ed
(low p ecision, high ecall).
As poin ed ou by [2], howe e , egion benchma ks a e
also needed apa om he bounda y benchma ks when as-
sessing image segmen a ion. Region benchma ks, howe e ,
a e cu en ly limi ed o summa y measu es as he ones e-
iewed in Sec ion 2.
This sec ion p esen s a new egion benchma k ha goes
beyond he summa y measu es: he p ecision- ecall o ob-
jec s and pa s. Mo i a ed by he ac ha image segmen-
a ion is inc easingly being used as a p elimina y s ep o
objec de ec ion [18,1], we p opose o assess segmen a ion
unde his pe spec i e, ha is, we in e p e egions in a pa -
i ion as po en ial objec candida es, and classi y hem as
co ec o no . Simila ly, we in e p e egions in an o e seg-
men a ion as pa s o objec s, i me ged oge he can o m
an objec o he g ound u h (inspi ed by [12] in ange im-
age segmen a ion e alua ion).
P ecision and ecall a e hen compu ed as he ac ion
o weigh ed candida es wi h espec o he o al numbe o
egions, ha is, pa candida es a e only pa ially coun ed.
Fo mally, le S={R1, . . . , RN}be an image pa i ion
and {Gk}a se o g ound- u h pa i ions o he same image.
We conside he se G={R0
1, . . . , R0
M}o all he egions
in {Gk}. Fo each pai o egions Ri∈S,R0
j∈Gwe
compu e he ela i e o e laps as:
Oij
S=|Ri∩R0
j|
|Ri|Oij
G=|Ri∩R0
j|
|R0
j|
We de ine an objec h eshold γoand a pa h eshold
γp< γoand classi y he egions in bo h pa i ions as de-
sc ibed in Algo i hm 1, whe e “←” means ha a egion is
classi ied only i i p e iously did no ha e a mo e a o able
classi ica ion.
Le oc and oc0be he numbe o objec candida es in S
and G, espec i ely (no e ha hey can di e , gi en ha G
Algo i hm 1 Region candida es classi ica ion
1: o all Ri∈S,R0
j∈Gdo
2: i Oij
S>γoand Oij
G>γo hen
3: Ri, R0
j←Objec candida es
4: else i Oij
S>γpand Oij
G>γo hen
5: Ri←F agmen a ion candida e
6: R0
j←Pa candida e
7: else i Oij
S>γoand Oij
G>γp hen
8: Ri←Pa candida e
9: R0
j←F agmen a ion candida e
10: else
11: Ri, R0
j←Noise
12: end i
13: end o
can be o med by mo e han one pa i ion and hus a egion
in Scan be ma ched as objec wi h mo e han one egion in
G), and pc and pc0 he numbe o pa candida es. Rega d-
ing he agmen a ion candida es, we compu e he pe cen -
age o he objec ha could be o med om he ma ched
pa s. Fo mally, we de ine he amoun o agmen a ion
(Ri)o a egion Ri∈Sas he addi ion o he ela i e
o e laps o he pa candida es ma ched o Ri:
(Ri) = X
jnOij
Gs. . Oij
S> γoo(8)
and 0(R0
j)is de ined equi alen ly o G. The global ag-
men a ion and 0is compu ed adding he amoun o ag-
men a ion among all he agmen a ion candida es o Sand
G, espec i ely. Figu e 2shows a oy example o illus a e
he p oposed classi ica ion and measu es.
We hen de ined he p ecision- ecall o objec s and
pa s as ollows:
Pop =oc + +βpc
|S|Rop =oc0+ 0+βpc0
|G|(9)
Pa i ion G ound T u h
Figu e 2. Classi ica ion o he egions in o objec and pa can-
dida es. The ec angles a e classi ied as objec candida es, despi e
no ully o e lapping. The pa i ion ci cle is a agmen a ion can-
dida e wi h a agmen a ion o 1 (pa s co e i o ally), and he
g ound- u h hal -ci cles a e pa s candida es. The opposi e holds
o he iangles, bu in his case he agmen a ion is 0.9. Bo h
pen agons a e classi ied as noise.
In ui i ely, in a comple ely o e segmen ed esul , he
ecall would be high bu he p ecision e y low. Con-
e sely, a comple ely unde segmen ed esul (one single e-
gion) would en ail a high p ecision bu e y low ecall. As
a summa y measu e, we p opose o use he F measu e (Fop)
be ween Pop and Rop.
4. Me a-Measu es
This sec ion is abou how o compa e he goodness o
he segmen a ion e alua ion measu es. The objec i e o his
sec ion is he e o e no o ell which segmen a ion algo i hm
o use, bu which e alua ion measu es be e summa ize he
quali y o hese algo i hms. To dis inguish hese wo anal-
yses, we will e e o he quan i a i e me ics o compa e
segmen a ion measu es as me a-measu es.
A me a-measu e analysis mus ely on accep ed hypo he-
ses abou he segmen a ion esul s and assess how cohe en
he measu es a e wi h such hypo heses. As examples, an
accep ed hypo hesis can be he human judgmen o quali y
o some pa icula examples. The me a-measu e is hen de-
ined as a quan iza ion o how cohe en he e alua ion mea-
su es a e wi h his judgmen [30,4].
To p o ide s a is ically signi ican esul s, howe e , one
mus go beyond a hand ul o examples and p o ide a quan-
i a i e analysis on an anno a ed da abase. The emain-
de o his sec ion explains one me a-measu e al eady pub-
lished in he li e a u e (Sec. 4.1) and p esen s wo new me a-
measu es (Sec. 4.2 and 4.3).
4.1. Swapped-Image Human Disc imina ion
Gi en an image, he e is no unique alid segmen a ion,
since i depends on he pe cep ion o he scene, he le el o
de ails, e c. In o de o cope wi h his a iabili y, he Be ke-
ley segmen a ion da ase (BSDS300 [21] and BSDS500 [2])
consis s o a se o images each o hem manually seg-
men ed by mo e han one indi idual.
The hypo hesis behind he i s me a-measu e is ha an
e alua ion measu e should be able o ell apa he g ound-
u h pa i ions coming om wo di e en images. In
o he wo ds, gi en a pai o g ound- u h pa i ions om
BSDS500, a measu e should be able o ell whe he hey
come om he same image ( hus di e ences a e an accep -
able e inemen ) o di e en images (unaccep able disc ep-
ancies).
As i s p oposed by [19] o e alua e he cohe ence o
BSDS300, gi en an e alua ion measu e m, we compu e
he P obabili y Densi y Func ion (PDF) o he alues o m
o all he pai s o pa i ions in BSDS500, g ouped in wo
classes: hose coming om di e en images and hose om
he same one. Figu e 3shows he PDFs o hese wo ypes
o pai s o pa i ions using he Fbmeasu e.
A simple classi ie was hen de ined se ing a h eshold
on he measu e o disc imina e he wo ypes o pai s. The
0.2 0.4 0.6 0.8
0
2
4
6
8
Fb
F equency
Fb=0.15 Fb=0.75
Fb=0.28 Fb=0.33
Figu e 3. Dis ibu ion o Fb o he same-image pai s o pa i ions
( ) and di e en -image pai s ( ). In g ay ec angles, ou
ep esen a i e pai s o pa i ions: a pai o co ec ly classi ied as
di e en image (up-le ) and as same image (up- igh ); and a pai
inco ec ly classi ied as di e en image (down-le ) and as same
image (down- igh ).
Swapped-Image Human Disc imina ion (SIHD) me a-
measu e is de ined as he pe cen age o co ec classi i-
ca ions o ha classi ie , ha is, he sum o he a ea un-
de he cu e abo e and below he h eshold o he same-
image and di e en -image pai s, espec i ely. (In he o igi-
nal wo k, he au ho s epo ed he Bayes Risk.)
As quali a i e examples, Figu e 3depic s ou pai s o
pa i ions as ep esen a i es o he ype o mis akes and co -
ec classi ica ions using Fb.
4.2. SoA-Baseline Disc imina ion
One o he easons why SIHD can be c i icized is he
ac ha i is based only on human-made pa i ions, ha
is, i does no show how measu es handle he eal-wo ld
disc epancies ound be ween SoA segmen a ion me hods.
This subsec ion and he ollowing a e de o ed o p esen
wo me a-mesu es based on SoA segmen a ion esul s.
The hypo hesis on which we base he me a-measu e p e-
sen ed in his sec ion is ha e alua ion measu es should be
able o dis inguish be ween (i) pa i ions ob ained by any
SoA segmen a ion me hod on a gi en image and (ii) pa -
i ions ob ained ega dless o he image, ha is, pa i ions
ha a e c ea ed wi hou aking in o accoun he con en o
he image. These pa i ions a e in e p e ed as a baseline,
ha is, he esul s ha could be ob ained by chance.
As in [2], we use a quad ee as baseline. In pa icula ,
we build he hie a chical pa i ions s a ing om he whole

image and i e a i ely di iding he egions in o ou equal
ec angles. Figu e 1.b shows an example o pa i ion ob-
ained by a SoA me hod and by a quad ee.
Fo each o he echniques conside ed as SoA segmen-
a ion me hods, we compu e he numbe o images in he
da ase in which an e alua ion measu e co ec ly judges
ha he baseline esul is wo se han he SoA gene a ed
pa i ion. We e e o he esul ing me a-measu e as SoA-
Baseline Disc imina ion (SABD), and i is de ined as he
global pe cen age o co ec judgmen s o a gi en measu e.
4.3. Swapped-Image SoA Disc imina ion
Segmen a ion e alua ion measu es a e o en used o ad-
jus he pa ame e s o a segmen a ion echnique. They a e
he e o e used o compa e di e en pa i ions c ea ed by he
same algo i hm. To inco po a e his ype o compa isons o
he me a-measu es, we compa e (i) he esul s c ea ed by a
SoA segmen a ion echnique wi h (ii) he esul s c ea ed by
ha same algo i hm bu on a di e en image.
In o he wo ds, we compa e he g ound- u h o a ce ain
image wi h wo esul s ob ained using he same algo i hm
and pa ame e iza ion: (i) one segmen a ion o ha same im-
age and (ii) one o a di e en image. The hypo hesis in his
case is ha he e alua ion measu es should judge ha he
same-image esul is be e han he di e en -image one. In
he example o Figu e 1.c, he measu e should judge ha he
i s pa i ion is be e han he second one compa ed bo h
wi h he g ound- u h o he o me . In his me a-measu e,
e alua ion measu es ha e o ackle he po en ial bias o he
SoA me hods owa ds hei speci ic ype o esul s.
Fo each SoA segmen a ion echnique, we compu e he
numbe o images in he da ase in which an e alua ion
measu e co ec ly judges ha he same-image SoA esul
is be e han he di e en -image one. We de ine he me a-
measu e Swapped-Image SoA Disc imina ion as he pe -
cen age o esul s in he da abase, o all he SoA me hods,
ha he measu es co ec ly disc imina es.
5. Expe imen al Valida ion
The s a e o he a o segmen a ion is ep esen ed in
his pape by he ollowing six me hods: he Ul ame -
ic Con ou Maps on he gPb con ou de ec o (gPb-OWT-
UCM) [2], he E icien G aph-Based (EGB) image seg-
men a ion algo i hm [10], he Mean Shi (MShi ) algo-
i hm [6], he No malized Cu s (NCu s) algo i hm [29], and
wo ypes o Bina y Pa i ion T ees [27]: he No malized
Weigh ed Euclidean dis ance be ween Models wi h Con-
ou complexi y (NWMC) ee [31], and he Independen
Iden ically Dis ibu ed - Kullback Leible (IID-KL) ee [3].
The exac pa ame e iza ions o each algo i hm is de ailed
a [28], whe e we also publish he code o all measu es and
me a-measu es used in his wo k. All me hods a e assessed
Measu e Global Me a-Measu e
Me a-Meas. SIHD SABD SISD
Fb98.4 99.5 95.6 100.0
Fop 96.7 98.4 94.2 97.5
VoI 94.0 96.9 87.5 97.7
C(S→{Gi})91.5 93.1 86.0 95.3
d D 90.7 95.1 86.9 90.1
DH(S⇒{Gi})89.5 78.5 91.3 98.8
BCE 89.2 93.3 78.9 95.4
BGM 88.1 90.7 81.6 92.0
PRI 86.7 77.7 88.8 93.7
C({Gi}→S)86.3 91.3 77.4 90.1
F 86.1 77.0 84.2 97.1
DH({Gi}⇒S)80.5 73.0 92.1 76.5
Table 2. Measu e compa ison in e ms o quan i a i e me a-
measu es. Values e e o pe cen ages o co ec esul s
a he Op imal Da ase Scale (ODS) [2] wi h espec o each
e alua ion measu e.
The pa ame e alues o he newly p oposed measu e
a e: γo= 0.95,γp= 0.25, and β= 0.1. They ha e
been ained on he aining se o BSDS500 [2], op imizing
he global me a-measu e desc ibed in he ollowing sec ion
(See Table 2). No e ha his op imiza ion would no ha e
been easible wi hou such quan i a i e me a-measu es.
Me a-Measu es Resul s: Table 2shows he h ee me a-
measu e esul s o he es se o BSDS500, as well as
a global summa y me a-measu e. Gi en ha each me a-
measu e ep esen s a pe cen age o co ec esul s, we de-
ine he global me a-measu e as he global pe cen age o
co ec esul s.
In global e ms, Fband Fop a e he wo op- anked sum-
ma y measu es. On op o ha , hey bo h p o ide much
iche in o ma ion in o m o p ecision- ecall cu es, hus
we p opose he pai Fb-Fop as he measu es o choice.
Rega ding he compu a ional cos o he measu es,
he mean ime o image o compu e he dis ances
o he mul iple-pa i ion g ound u h o BSDS500 is
3.79 ±2.06 s o Fband a leas one o de o magni ude
lowe o he es o measu es. In pa icula , Fop akes
0.078 ±0.020 s.
In scena ios whe e he ime limi a ions a e igh , he au-
ho s belie e ha Fop would be he ool o choice. To p o-
ide an in-dep h analysis o he inal esul s, he andem o
p ecision- ecall cu es o bounda ies and o objec s-and-
pa s would be he mos adequa e op ion. The ollowing
sec ion p o ides a ho ough analysis o bo h amewo ks on
he six SoA me hods used in his pape .
P ecision-Recall F amewo ks: Figu e 4shows he
bounda y and objec s-and-pa s p ecision- ecall cu es o
Bounda ies
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
P ecision
Human [0.81-0.21] MShi [0.60]
gPb-OWT-UCM [0.73] IID-KL [0.57]
NCu s [0.63] NWMC [0.55]
EGB [0.61] Quad ee [0.41]
Objec s and Pa s
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
P ecision
Human [0.56-0.05]
gPb-OWT-UCM [0.35]
MShi [0.23]
NWMC [0.22]
NCu s [0.21]
IID-KL [0.19]
EGB [0.16]
Quad ee [0.06]
Figu e 4. P ecision-Recall cu es o bounda ies (le ) and o objec s and pa s ( igh ). The solid cu es ep esen he six SoA segmen a ion
me hods and he quad ee (see legends). In dashed lines wi h he same colo , he SoA echniques assessed on a swapped image. The ma ke
on each cu e is placed on he Op imal Da ase Scale (ODS). The isola ed ed as e isks e e o he human pe o mance assessed on he
same image and on a swapped image. In he legend, he F measu e o he ma ked poin on each cu e is p esen ed in b acke s.
he six SoA segmen a ion me hods s udied and he human
pe o mance. P io o he assessmen o segmen a ion ech-
niques, le us ocus on he compa ison o he wo e alua ion
amewo ks.
I is no iceable ha he human baseline pe o mance (hu-
man assessed on a di e en image) o Fbis 0.21, which
could be in e p e ed as Fbbeing oo lax. In his same di ec-
ion, he baseline bounda y p ecision o Fbis be ween 0.2
and 0.3, ha is, any esul , no ma e how w ong i is, will
be judged as p o iding a leas a 0.2 p ecision.
While in he case o Fop he human baseline is co ec ly
downg aded o 0.05 (as well as he swapped-image esul s),
hen he su p ising ac is ha human pe o mance is as low
as 0.56 (0.81 in Fb), which could en ail ha Fop is oo s ic .
Al hough he dynamic ange is a li le highe in Fb(0.60
e sus 0.51), he gap be ween he bes me hod and humans
is much highe in Fop (0.08 e sus 0.21). In o he wo ds,
Fop gi es mo e esolu ion a he places whe e imp o e-
men s o e he SoA would be placed.
Rega ding he compa ison among segmen a ion ech-
niques, bo h amewo ks con i m ha he gPb-OWT-UCM
echnique has ou s anding esul s wi h espec o he es .
The ad an ages o going beyond he summa y measu es
a e also clea on hese plo s. Fo ins ance, he summa y
Fbmeasu e o quad ee (0.41) judges his echnique close
o NWMC (0.55), bu in he p ecision- ecall cu es i is
clea ha quad ee is much wo se. Simila ly, judging by
Fb, NWMC would be clea ly disca ded bu i we a e in e -
es ed in low ecall a es i could be o in e es (apa om
gPb-OWT-UCM).
As common poin s be ween he wo measu es, NCu s is
judged as being much be e a high ecall a es han a low
ones and con e sely, NWMC is much be e a high p eci-
sion a es. The measu es a e cohe en also in he ac ha
human esul s ha e a be e p ecision han ecall.
As one o he main disc epan poin s, howe e , EGB is
judged as he hi d bes echnique by Fbwhile being he
wo se o Fop. To u he analyze his beha io , Figu e 5
shows an image (a), an EGB esul (b), and he associa ed
g ound u h (c). The EGB esul consis s o hin long e-
gions ha su ound he objec bu do no close. The assess-
men alue o his esul is Fb= 0.62 and Fop = 0.05.
F om a egion-based poin o iew, his ype o esul s is
co ec ly penalized by Fop and no by Fb, since as a con-
ou de ec o he esul is co ec .
To sum up, bo h measu es a e complemen a y hus we
p opose hem in andem as he ool o choice o image seg-
men a ion e alua ion.
(a) (b) (c)
Figu e 5. EGB esul co ec ly penalized by Fop bu no by Fb
6. Conclusions
This pape e iews an ex ensi e se o segmen a ion e al-
ua ion measu es and p esen s he new p ecision- ecall mea-
su e o objec s and pa s. Th ee me a-measu es a e used
( wo newly p oposed) o quan i a i ely compa e he good-
ness o he e alua ion measu es. The esul s show ha
he andem bounda y and objec s-and-pa s p ecision- ecall
cu es is a good candida e o benchma king segmen a-
ion algo i hms; since apa om ob aining he bes me a-
measu e esul s, hei p ecision- ecall cu es p o ide ich
knowledge abou he esul s. By making ou code and
da ase s publicly a ailable we allow esea che s o easily
assess hei esul s and gain deepe unde s anding o hei
algo i hms.
Re e ences
[1] P. A bel´
aez, B. Ha iha an, C. Gu, S. Gup a, L. Bou de , and
J. Malik. Seman ic segmen a ion using egions and pa s. In
CVPR, 2012. 1,4
[2] P. A bel´
aez, M. Mai e, C. C. Fowlkes, and J. Malik. Con-
ou de ec ion and hie a chical image segmen a ion. IEEE
TPAMI, 33(5):898–916, 2011. 1,2,3,4,5,6
[3] F. Calde e o and F. Ma ques. Region me ging echniques
using in o ma ion heo y s a is ical measu es. IEEE TIP,
19(6):1567–1586, 2010. 6
[4] J. S. Ca doso and L. Co e-Real. Towa d a gene ic e alua ion
o image segmen a ion. IEEE TIP, 14(11):1773–1782, 2005.
1,2,3,5
[5] J. Ca ei a and C. Sminchisescu. Cons ained pa ame ic
min-cu s o au oma ic objec segmen a ion. In CVPR, 2010.
1
[6] D. Comaniciu and P. Mee . Mean shi : a obus app oach
owa d ea u e space analysis. IEEE TPAMI, 24(5):603 –619,
2002. 6
[7] S. Dongen. Pe o mance c i e ia o g aph clus e ing and
ma ko clus e expe imen s. Technical Repo INS-R0012,
Cen um oo Wiskunde en In o ma ica (CWI), Ams e dam,
The Nede lands, 2000. 1,2,3
[8] M. E e ingham, L. Van Gool, C. K. I. Williams, J. Winn, and
A. Zisse man. The PASCAL Visual Objec Classes Chal-
lenge 2012 (VOC2012) Resul s. www.pascal-ne wo k.o g/
challenges/VOC/ oc2012/wo kshop/index.h ml. 1
[9] P. Felzenszwalb, R. Gi shick, D. McAlles e , and D. Ra-
manan. Objec de ec ion wi h disc imina i ely ained pa -
based models. IEEE TPAMI, 32(9):1627–1645, 2010. 1
[10] P. F. Felzenszwalb and D. P. Hu enloche . E icien g aph-
based image segmen a ion. IJCV, 59:2004, 2004. 6
[11] D. Hoiem, Y. Chodpa humwan, and Q. Dai. Diagnosing e o
in objec de ec o s. In ECCV, 2012. 1
[12] A. Hoo e , G. Jean-Bap is e, X. Jiang, P. Flynn, H. Bunke,
D. Goldgo , K. Bowye , D. Egge , A. Fi zgibbon, and
R. Fishe . An expe imen al compa ison o ange image seg-
men a ion algo i hms. IEEE TPAMI, 18:673–689, 1996. 1,
4
[13] Q. Huang and B. Dom. Quan i a i e me hods o e alua ing
image segmen a ion. In ICIP, 1995. 1,2,3
[14] X. Jiang, C. Ma i, C. I nige , and H. Bunke. Dis ance mea-
su es o image segmen a ion e alua ion. EURASIP J. Appl.
Signal P ocess., 2006:1–10, 2006. 2,3
[15] T. Kanungo, B. Dom, W. Niblack, and D. S eele. A as
algo i hm o MDL-based mul i-band image segmen a ion.
Technical epo , IBM Reasea ch Di ision, RJ 9754 (84640),
1994. 1,2
[16] C. Lampe , M. Blaschko, and T. Ho mann. Beyond sliding
windows: Objec localiza ion by e icien subwindow sea ch.
In CVPR, 2008. 1
[17] G. Liu and R. Ha alick. Assignmen p oblem in edge de ec-
ion pe o mance e alua ion. In CVPR, 2000. 3,4
[18] T. Malisiewicz and A. A. E os. Imp o ing spa ial suppo
o objec s ia mul iple segmen a ions. In BMVC, 2007. 1,4
[19] D. Ma in. An Empi ical App oach o G ouping and Segmen-
a ion. PhD hesis, EECS Depa men , Uni e si y o Cali o -
nia, Be keley, Aug 2003. 2,3,4,5
[20] D. Ma in, C. Fowlkes, and J. Malik. Lea ning o de ec na -
u al image bounda ies using local b igh ness, colo , and ex-
u e cues. IEEE TPAMI, 26(5):530–549, 2004. 1,4
[21] D. Ma in, C. Fowlkes, D. Tal, and J. Malik. A da abase
o human segmen ed na u al images and i s applica ion o
e alua ing segmen a ion algo i hms and measu ing ecologi-
cal s a is ics. In ICCV, 2001. 5
[22] M. Meil˘
a. Compa ing clus e ings: an axioma ic iew. In
ICML, 2005. 1,2,3
[23] S. Nowozin, P. Gehle , and C. Lampe . On pa ame e lea n-
ing in c -based app oaches o objec class image segmen a-
ion. In ECCV, 2010. 2
[24] B. Peng and L. Zhang. E alua ion o image segmen a ion
quali y by adap i e g ound u h composi ion. In ECCV,
2012. 1
[25] J. Pon -Tuse and F. Ma ques. Supe ised assessmen o seg-
men a ion hie a chies. In ECCV, 2012. 1
[26] W. Rand. Objec i e c i e ia o he e alua ion o clus e ing
me hods. Jou nal o he Ame ican S a is ical Associa ion,
66(336):846–850, 1971. 3
[27] P. Salembie and L. Ga ido. Bina y pa i ion ee as an e i-
cien ep esen a ion o image p ocessing, segmen a ion, and
in o ma ion e ie al. IEEE TIP, 9(4):561–576, 2000. 6
[28] Segmen a ion E alua ion Code. h ps://ima ge.upc.edu/web
/ esou ces/supe ised-e alua ion-image-segmen a ion. 2,6
[29] J. Shi and J. Malik. No malized cu s and image segmen a-
ion. IEEE TPAMI, 22(8):888–905, 2000. 6
[30] R. Unnik ishnan, C. Pan o a u, and M. Hebe . Towa d ob-
jec i e e alua ion o image segmen a ion algo i hms. IEEE
TPAMI, 29(6):929–944, 2007. 1,3,5
[31] V. Vilaplana, F. Ma ques, and P. Salembie . Bina y pa i-
ion ees o objec de ec ion. IEEE TIP, 17(11):2201–2216,
2008. 6
[32] P. Viola and M. Jones. Rapid objec de ec ion using a boos ed
cascade o simple ea u es. In CVPR, 2001. 1