Measures and meta-measures for the supervised evaluation of image segmentation

Measu es and Me a-Measu es o he Supe ised E alua ion o Image Segmen a ion Jo di Pon -Tuse and Fe an Ma ques Uni e si a Poli ` ecnica de Ca alunya Ba celonaTech∗ h p://ima ge.upc.edu Abs ac This pape ackles he supe ised e alua ion o image segmen a ion algo i hms. Fi s , i su eys and s uc u es he measu es used o compa e he segmen a ion esul s wi h a g ound u h da abase; and p oposes a new measu e: he p ecision- ecall o objec s and pa s. To compa e he goodness o hese measu es, i de ines h ee quan i a i e me a-measu es in ol ing six s a e o he a segmen a ion me hods. The me a-measu es consis in assuming some plausible hypo heses abou he esul s and assessing how well each measu e e lec s hese hypo heses. As a con- clusion, his pape p oposes he p ecision- ecall cu es o bounda ies and o objec s-and-pa s as he ool o choice o he supe ised e alua ion o image segmen a ion. We make he da ase s and code o all he measu es publicly a ailable. 1. In oduc ion Since he ad en o sliding window objec de ec o s [32], much e o has been pu in o p o iding be e spa ial delin- ea ion beyond sliding windows [16]. Seman ic segmen a- ion is he inal objec i e, whe e de ec ion and segmen a ion mee , bu i is s ill a om being sol ed [8]. In his scena io, bo om-up segmen a ion me hods o en play an impo an ole in he p oposed algo i hms [1,5], and hus imp o ing segmen a ion echniques would en ail imp o emen s owa ds be e seman ic segmen a ion [18]. In such a challenge, p o iding benchma ks ha help e- sea che s unde s and he weak and s ong poin s o hei algo i hms is o pa amoun impo ance. In his di ec ion, in he ield o objec de ec ion assess- men , Hoiem e al. [11] s ess ha he esul s should be e alua ed beyond pe o mance summa y measu es in o de o “help unde s and how one me hod could be imp o ed.” ∗This wo k has been pa ially suppo ed by he Spanish Minis e io de Ciencia e Inno aci´ on, unde p ojec TEC2010-18094 and FPU g an AP2008-01164. (a) (b) (c) Figu e 1. Examples o he me a-measu e p inciples: How good a e he e alua ion measu es a dis inguishing hese pai s o pa i ions? In o he wo ds, esea che s need be e eedback om he e alua ion han a single numbe . Back o segmen a ion assessmen , he p ecision- ecall cu es o bounda ies [20] a e good examples o ools ha p o ide iche eedback han he F-measu e used as sum- ma y. Mo eo e , as poin ed ou by [2], in addi ion o bounda y-based measu es, egion-o ien ed measu es should be conside ed when assessing segmen a ions. Howe e , he cu en ones a e limi ed o summa y measu es [30,22,20, 2,15,13,7,4,25,24]. Ou i s con ibu ion is a egion-based p ecision- ecall en i onmen o he assessmen o image segmen a ion. In- spi ed by [12,11] and by he ac ha pa s o objec s a e impo an clues o objec de ec ion [9], we p esen he p ecision- ecall o objec s and pa s, which is based on classi ying he egions in o objec and pa s candida es. Summa y measu es also play a ole in pe o mance com- pa ison, hus he ques ion ha now a ises is how o compa e he goodness o an e alua ion measu e. In o he wo ds, we should de ine a me a-measu e o compa e he e alua ion measu es. The p inciple o a me a-measu e is o assume a plausible hypo hesis abou he segmen a ion e alua ion and analyze how well measu es ma ch his hypo hesis. Some p e ious wo ks based hei claims on quali a i e me a-measu es, ha is, showing he beha io o he mea- su es on pa icula quali a i e examples [4,30]. Ex ensi e quan i a i e me a-measu es, howe e , a e desi able. 1 The i s app oach o an ex ensi e quan i a i e me a- measu e was p oposed in [19]. The hypo hesis in his wo k was ha measu es should be able o disc imina e be ween wo pai s o human-ma ked pa i ions coming om di e - en images ( o ins ances, he wo pa i ions in Figu e 1.a). In an anno a ed da abase wi h mul iple pa i ions pe image, he quan i a i e me a-measu e was de ined as he numbe o same-image pa i ion pai s ha he measu e judges as less simila han o he pai s o pa i ions coming om di e en images. [14] p esen ed a compa ison o some measu es in e ms o his me a-measu e. The second con ibu ion o his pape is o p esen wo new me a-measu es. Ins ead o basing ou hypo heses on human-made pa i ions, we ex end he analysis o pa i ions om six S a e-o - he-A (SoA) segmen a ion algo i hms. The i s assump ion is ha measu es should be capable o dis inguishing such pa i ions om hose ob ained wi h- ou aking in o accoun he con en o he image. In ou case, ollowing he p oposal in [2], we use a quad ee, i.e., a hie a chical homogeneous ec angula g id. The me a- measu e is hen de ined as he numbe o esul s om SoA algo i hms ha a e judged wo se han he quad ee. As a quali a i e example, we assess how well a measu e dis in- guishes be ween pa i ions like Figu e 1.b. As a second app oach, we assume ha any measu e should be able o dis inguish a pa i ion ob ained by a SoA me hod on an image om a pa i ion ob ained by he same me hod bu on a di e en image, as he wo pa i ions shown in Figu e 1.c. The me a-measu e in his case is de ined as he numbe o cases in which he measu e co ec ly judges he same-image pa i ion as be e . The hi d con ibu ion is o su ey and s uc u e a wide se o e alua ion measu es and he newly-p oposed one and compa e hem using he h ee p e iously discussed me a- measu es. We show ha he wo p ecision- ecall measu es (bounda y- and objec s-and-pa s-based) ha e ou s anding esul s as summa y measu es wi h espec o he es o mea- su es, while p o iding iche in o ma ion o esea che s o in e p e he esul s. We u he in e p e hese wo p ecision- ecall en i onmen s by compa ing six SoA seg- men a ion algo i hms. We make he code o compu e all he measu es publicly a ailable in [28], as well as all he segmen a ion esul s o make ou esea ch ep oducible and o make i e o less o esea che s o assess hei segmen a ion me hods. The emainde o he pape is o ganized as ollows. Sec- ion 2 e iews and s uc u es he main segmen a ion mea- su es a ailable in he li e a u e. Sec ion 3mo i a es and de- sc ibes he newly p oposed measu e. Sec ion 4p esen s he wo new me a-measu es and he al eady a ailable one used o compa e he e alua ion measu es. Sec ion 5p esen s he expe imen al compa ison o he measu es using he h ee me a-measu es. I also shows he applicabili y o he bounda y-based and he newly p oposed p ecision- ecall cu es o objec s and pa s in he compa ison o six SoA segmen a ion echniques. Sec ion 6concludes he pape . 2. Measu e Re iew and S uc u e The s a e-o - he-a measu es can be classi ied depend- ing on he image pa i ion in e p e a ion on which hey a e based. The mos common in e p e a ion is as a clus e ing o he pixel se in o a numbe o subse s o egions, A pa i ion can also be in e p e ed as a wo-class clus e ing o he se o pai s o pixels, wi h some pai s linking pixels om he same egion and o he s linking pixels om di e en egions. Fi- nally, a pa i ion can be ep esen ed as a wo-class clus e ing o he pixel con ou s in o bounda ies and non-bounda ies. The ollowing sec ions e iew he main measu es ound unde each o hese in e p e a ions, keeping he no a ion om he o iginal pape s whe e possible. Table 1shows an o e iew o he s udied measu es. 2.1. Pixel-Se Clus e ing The di ec ional Hamming dis ance om one pa i ion S o ano he S0[15,13] is de ined as: DH(S⇒S0) = n−X R0∈S0 max R∈S|R0∩R|(1) whe e Rand R0a e egions in Sand S0, espec i ely, and nis he numbe o pixels in he image. In [4] his same measu e was coined as asymme ic pa i ion dis ance. I is equi alen o he achie able segmen a ion accu acy [23] used in supe pixel assessmen . A symme ic e sion o his measu e was p esen ed in [7] as he an Dongen dis ance: d D (S, S0) = DH(S0⇒S) + DH(S⇒S0)(2) The segmen a ion co e ing o a pa i ion Sby a pa i- ion S0was de ined in [2] as: C(S0→S) = 1 nX R∈S |R| · max R0∈S0 |R∩R0| |R∪R0|(3) The in ui i e s ep u he is o measu e he maximum o e lap when pe o ming a bijec i e ma ching be ween he egions o he wo pa i ions. This idea was p esen ed in [4] as symme ic pa i ion-dis ance, in [14] as bipa i e-g aph- ma ching (BGM) dis ance, and in he con ex o clus e ing compa ison, in [22] as classi ica ion e o dis ance. I is shown in [4] ha i is equi alen o he minimum numbe o pixels ha mus no be aken in o accoun o he wo pa i ions o be iden ical. In [19], he consis ency o he BSDS300 human pa i- ions is analyzed by means o wo measu es GCE,LCE, aiming a being obus agains di e en g anula i ies o he Pa i ion In e p e a ion Measu e Rep esen a i e Re e ences No a ion Pixel-se clus e ing Di ec ional Hamming dis ance [13,4]DH an Dongen dis ance [7]d D Segmen a ion co e ing [2]C Bipa i e g aph ma ching [14,4]BGM Bidi ec ional consis ency e o [19]BCE Va ia ion o in o ma ion [22]VoI Pai s-o -pixels classi ica ion P obabilis ic Rand index [26,30]PRI P ecision-Recall o egions [19]P ,R Bounda y map P ecision-Recall o bounda ies [17,19]Pb,Rb Table 1. Measu e s uc u e o e iew o he h ee in e p e a ions o an image pa i ion scene in e p e a ion. As he au ho poin s ou , hese mea- su es a e no sui able o gene al-pu pose image segmen a- ion e alua ion. The same wo k p oposes a measu e ha is no anspa en o o e segmen a ion: he bidi ec ional con- sis ency e o (BCE), which can be ew i en as: BCE(S,S0 )=1−1 nX R∈S R0∈S0 |R∩R0|min|R∩R0| |R|,|R∩R0| |R0| (4) The wo k in [22] in oduced a new poin o iew o he measu es o clus e ing assessmen based on in o ma ion- heo e ic esul s. The au ho de ines a disc e e andom a i- able aking N alues ha consis s in andomly picking any pixel in he pa i ion S={R1, . . . , RN}and obse ing he egion i belongs o. Assuming all he pixels equally p ob- able o pick, he en opy H(S)associa ed wi h a pa i ion is de ined as he en opy o such andom a iable. The mu- ual in o ma ion I(S,S0 )be ween wo pa i ions is de ined equi alen ly. The a ia ion o in o ma ion is hen: VoI (S, S0) = H(S) + H(S0)−2I(S, S0)(5) I can be no malized by log N, i s maximum possible alue. 2.2. Pai s-o -Pixels Classi ica ion An image pa i ion can be iewed as a classi ica ion o all he pai s o pixels in o wo classes: pai s o pixels belong- ing o he same egion, and pai s o pixels om di e en egions. Fo mally, le I={p1, . . . , pn}be he se o pix- els o he image and conside he se o all pai s o pixels P={(pi, pj)∈I×I|i<j}. Gi en wo pa i ions Sand S0, we di ide Pin o ou di e en se s, depending on whe e a pai (pi, pj)o pixels all [22]: P11: in he same egion bo h in Sand S0, P10: in he same egion in Sbu di e en in S0, P01: in he same egion in S0bu di e en in S, P00: in di e en egions bo h in Sand S0. The Rand index, o iginally de ined in [26] as a clus e - ing e alua ion measu e, a ises na u ally in his con ex : RI (S,S0 ) = |P00 |+|P11| |P| . I coun s he pai s o pixels ha ha e cohe en labels o he wo pa i ions being compa ed, wi h espec o he numbe o possible pai s o pixels. In he con ex o image segmen a ion and ha ing a se {Gi}o g ound- u h pa i ions o he same image, he P obabilis ic Rand Index [30] is compu ed as: PRI (S, {Gi}) = X i RI (S,Gi )(6) In his same con ex , he p ecision- ecall o e- gions [19] is de ined as: P =|P11| |P11|+|P10|R =|P11| |P11|+|P01|(7) As a summa y measu e, he F measu e F is used. This pai o measu es would be a candida e in ou ques o a non-bounda y-based p ecision- ecall measu e. As i will be shown in he expe imen s, howe e , his measu e does no p o ide good me a-e alua ion sco es. 2.3. Bounda y Map All measu es abo e could be applied o any clus e ing algo i hm, no ma e he na u e o he elemen s being clas- si ied. In ac , he majo i y o he indices p esen ed come om he applica ion o gene al-clus e ing assessmen mea- su es o image segmen a ion. Image pixels, howe e , a e spa ially dis ibu ed in he im- age plane, and so he concep o neighbo hood a ises na u- ally. The e o e, an image pa i ion wi h connec ed com- ponen s can be unambiguously de ined by hei bounda ies, i.e., a bijec ion could be made be ween all possible image pa i ions and all possible closed bounda ies maps. Recalling he de ini ion o Pas he se o pai s o pixels in he image, le us de ine he se o pai s o neighbo ing pixels as N ⊂ P. One can de ine a bijec ion be ween he se o bounda y segmen s Band Nlinking each segmen o he pai o pixels a each o i s sides. Using his no a ion, bounda y de ec ion can be unde s ood as a wo-class clus- e ing o B, di iding he segmen s in o hose being bound- a ies and hose no . This way, compa ing wo pa i ions can be ansla ed in o compa ing wo clus e ing o B. To be obus o unno iceable shi s o bounda y local- iza ion, [17] p oposes o compu e he op imal ma ching be- ween he segmen s o bounda ies o he wo pa i ions as a maximum-weigh bipa i e-g aph ma ching. The algo- i hm is imp o ed in [19,20] leading o he well-known p ecision- ecall o bounda ies (Pb,Rb, and Fb). 3. Measu e P oposal In he con ex o image segmen a ion e alua ion, p ecision- ecall cu es o bounda ies [19,20] a e a boon o esea che s. They s a is ically e lec , o ins ance, ha an algo i hm is p o iding oo coa se segmen a ions (low e- call, high p ecision) o ins ead i s esul s a e oo agmen ed (low p ecision, high ecall). As poin ed ou by [2], howe e , egion benchma ks a e also needed apa om he bounda y benchma ks when as- sessing image segmen a ion. Region benchma ks, howe e , a e cu en ly limi ed o summa y measu es as he ones e- iewed in Sec ion 2. This sec ion p esen s a new egion benchma k ha goes beyond he summa y measu es: he p ecision- ecall o ob- jec s and pa s. Mo i a ed by he ac ha image segmen- a ion is inc easingly being used as a p elimina y s ep o objec de ec ion [18,1], we p opose o assess segmen a ion unde his pe spec i e, ha is, we in e p e egions in a pa - i ion as po en ial objec candida es, and classi y hem as co ec o no . Simila ly, we in e p e egions in an o e seg- men a ion as pa s o objec s, i me ged oge he can o m an objec o he g ound u h (inspi ed by [12] in ange im- age segmen a ion e alua ion). P ecision and ecall a e hen compu ed as he ac ion o weigh ed candida es wi h espec o he o al numbe o egions, ha is, pa candida es a e only pa ially coun ed. Fo mally, le S={R1, . . . , RN}be an image pa i ion and {Gk}a se o g ound- u h pa i ions o he same image. We conside he se G={R0 1, . . . , R0 M}o all he egions in {Gk}. Fo each pai o egions Ri∈S,R0 j∈Gwe compu e he ela i e o e laps as: Oij S=|Ri∩R0 j| |Ri|Oij G=|Ri∩R0 j| |R0 j| We de ine an objec h eshold γoand a pa h eshold γp< γoand classi y he egions in bo h pa i ions as de- sc ibed in Algo i hm 1, whe e “←” means ha a egion is classi ied only i i p e iously did no ha e a mo e a o able classi ica ion. Le oc and oc0be he numbe o objec candida es in S and G, espec i ely (no e ha hey can di e , gi en ha G Algo i hm 1 Region candida es classi ica ion 1: o all Ri∈S,R0 j∈Gdo 2: i Oij S>γoand Oij G>γo hen 3: Ri, R0 j←Objec candida es 4: else i Oij S>γpand Oij G>γo hen 5: Ri←F agmen a ion candida e 6: R0 j←Pa candida e 7: else i Oij S>γoand Oij G>γp hen 8: Ri←Pa candida e 9: R0 j←F agmen a ion candida e 10: else 11: Ri, R0 j←Noise 12: end i 13: end o can be o med by mo e han one pa i ion and hus a egion in Scan be ma ched as objec wi h mo e han one egion in G), and pc and pc0 he numbe o pa candida es. Rega d- ing he agmen a ion candida es, we compu e he pe cen - age o he objec ha could be o med om he ma ched pa s. Fo mally, we de ine he amoun o agmen a ion (Ri)o a egion Ri∈Sas he addi ion o he ela i e o e laps o he pa candida es ma ched o Ri: (Ri) = X jnOij Gs. . Oij S> γoo(8) and 0(R0 j)is de ined equi alen ly o G. The global ag- men a ion and 0is compu ed adding he amoun o ag- men a ion among all he agmen a ion candida es o Sand G, espec i ely. Figu e 2shows a oy example o illus a e he p oposed classi ica ion and measu es. We hen de ined he p ecision- ecall o objec s and pa s as ollows: Pop =oc + +βpc |S|Rop =oc0+ 0+βpc0 |G|(9) Pa i ion G ound T u h Figu e 2. Classi ica ion o he egions in o objec and pa can- dida es. The ec angles a e classi ied as objec candida es, despi e no ully o e lapping. The pa i ion ci cle is a agmen a ion can- dida e wi h a agmen a ion o 1 (pa s co e i o ally), and he g ound- u h hal -ci cles a e pa s candida es. The opposi e holds o he iangles, bu in his case he agmen a ion is 0.9. Bo h pen agons a e classi ied as noise. In ui i ely, in a comple ely o e segmen ed esul , he ecall would be high bu he p ecision e y low. Con- e sely, a comple ely unde segmen ed esul (one single e- gion) would en ail a high p ecision bu e y low ecall. As a summa y measu e, we p opose o use he F measu e (Fop) be ween Pop and Rop. 4. Me a-Measu es This sec ion is abou how o compa e he goodness o he segmen a ion e alua ion measu es. The objec i e o his sec ion is he e o e no o ell which segmen a ion algo i hm o use, bu which e alua ion measu es be e summa ize he quali y o hese algo i hms. To dis inguish hese wo anal- yses, we will e e o he quan i a i e me ics o compa e segmen a ion measu es as me a-measu es. A me a-measu e analysis mus ely on accep ed hypo he- ses abou he segmen a ion esul s and assess how cohe en he measu es a e wi h such hypo heses. As examples, an accep ed hypo hesis can be he human judgmen o quali y o some pa icula examples. The me a-measu e is hen de- ined as a quan iza ion o how cohe en he e alua ion mea- su es a e wi h his judgmen [30,4]. To p o ide s a is ically signi ican esul s, howe e , one mus go beyond a hand ul o examples and p o ide a quan- i a i e analysis on an anno a ed da abase. The emain- de o his sec ion explains one me a-measu e al eady pub- lished in he li e a u e (Sec. 4.1) and p esen s wo new me a- measu es (Sec. 4.2 and 4.3). 4.1. Swapped-Image Human Disc imina ion Gi en an image, he e is no unique alid segmen a ion, since i depends on he pe cep ion o he scene, he le el o de ails, e c. In o de o cope wi h his a iabili y, he Be ke- ley segmen a ion da ase (BSDS300 [21] and BSDS500 [2]) consis s o a se o images each o hem manually seg- men ed by mo e han one indi idual. The hypo hesis behind he i s me a-measu e is ha an e alua ion measu e should be able o ell apa he g ound- u h pa i ions coming om wo di e en images. In o he wo ds, gi en a pai o g ound- u h pa i ions om BSDS500, a measu e should be able o ell whe he hey come om he same image ( hus di e ences a e an accep - able e inemen ) o di e en images (unaccep able disc ep- ancies). As i s p oposed by [19] o e alua e he cohe ence o BSDS300, gi en an e alua ion measu e m, we compu e he P obabili y Densi y Func ion (PDF) o he alues o m o all he pai s o pa i ions in BSDS500, g ouped in wo classes: hose coming om di e en images and hose om he same one. Figu e 3shows he PDFs o hese wo ypes o pai s o pa i ions using he Fbmeasu e. A simple classi ie was hen de ined se ing a h eshold on he measu e o disc imina e he wo ypes o pai s. The 0.2 0.4 0.6 0.8 0 2 4 6 8 Fb F equency Fb=0.15 Fb=0.75 Fb=0.28 Fb=0.33 Figu e 3. Dis ibu ion o Fb o he same-image pai s o pa i ions ( ) and di e en -image pai s ( ). In g ay ec angles, ou ep esen a i e pai s o pa i ions: a pai o co ec ly classi ied as di e en image (up-le ) and as same image (up- igh ); and a pai inco ec ly classi ied as di e en image (down-le ) and as same image (down- igh ). Swapped-Image Human Disc imina ion (SIHD) me a- measu e is de ined as he pe cen age o co ec classi i- ca ions o ha classi ie , ha is, he sum o he a ea un- de he cu e abo e and below he h eshold o he same- image and di e en -image pai s, espec i ely. (In he o igi- nal wo k, he au ho s epo ed he Bayes Risk.) As quali a i e examples, Figu e 3depic s ou pai s o pa i ions as ep esen a i es o he ype o mis akes and co - ec classi ica ions using Fb. 4.2. SoA-Baseline Disc imina ion One o he easons why SIHD can be c i icized is he ac ha i is based only on human-made pa i ions, ha is, i does no show how measu es handle he eal-wo ld disc epancies ound be ween SoA segmen a ion me hods. This subsec ion and he ollowing a e de o ed o p esen wo me a-mesu es based on SoA segmen a ion esul s. The hypo hesis on which we base he me a-measu e p e- sen ed in his sec ion is ha e alua ion measu es should be able o dis inguish be ween (i) pa i ions ob ained by any SoA segmen a ion me hod on a gi en image and (ii) pa - i ions ob ained ega dless o he image, ha is, pa i ions ha a e c ea ed wi hou aking in o accoun he con en o he image. These pa i ions a e in e p e ed as a baseline, ha is, he esul s ha could be ob ained by chance. As in [2], we use a quad ee as baseline. In pa icula , we build he hie a chical pa i ions s a ing om he whole image and i e a i ely di iding he egions in o ou equal ec angles. Figu e 1.b shows an example o pa i ion ob- ained by a SoA me hod and by a quad ee. Fo each o he echniques conside ed as SoA segmen- a ion me hods, we compu e he numbe o images in he da ase in which an e alua ion measu e co ec ly judges ha he baseline esul is wo se han he SoA gene a ed pa i ion. We e e o he esul ing me a-measu e as SoA- Baseline Disc imina ion (SABD), and i is de ined as he global pe cen age o co ec judgmen s o a gi en measu e. 4.3. Swapped-Image SoA Disc imina ion Segmen a ion e alua ion measu es a e o en used o ad- jus he pa ame e s o a segmen a ion echnique. They a e he e o e used o compa e di e en pa i ions c ea ed by he same algo i hm. To inco po a e his ype o compa isons o he me a-measu es, we compa e (i) he esul s c ea ed by a SoA segmen a ion echnique wi h (ii) he esul s c ea ed by ha same algo i hm bu on a di e en image. In o he wo ds, we compa e he g ound- u h o a ce ain image wi h wo esul s ob ained using he same algo i hm and pa ame e iza ion: (i) one segmen a ion o ha same im- age and (ii) one o a di e en image. The hypo hesis in his case is ha he e alua ion measu es should judge ha he same-image esul is be e han he di e en -image one. In he example o Figu e 1.c, he measu e should judge ha he i s pa i ion is be e han he second one compa ed bo h wi h he g ound- u h o he o me . In his me a-measu e, e alua ion measu es ha e o ackle he po en ial bias o he SoA me hods owa ds hei speci ic ype o esul s. Fo each SoA segmen a ion echnique, we compu e he numbe o images in he da ase in which an e alua ion measu e co ec ly judges ha he same-image SoA esul is be e han he di e en -image one. We de ine he me a- measu e Swapped-Image SoA Disc imina ion as he pe - cen age o esul s in he da abase, o all he SoA me hods, ha he measu es co ec ly disc imina es. 5. Expe imen al Valida ion The s a e o he a o segmen a ion is ep esen ed in his pape by he ollowing six me hods: he Ul ame - ic Con ou Maps on he gPb con ou de ec o (gPb-OWT- UCM) [2], he E icien G aph-Based (EGB) image seg- men a ion algo i hm [10], he Mean Shi (MShi ) algo- i hm [6], he No malized Cu s (NCu s) algo i hm [29], and wo ypes o Bina y Pa i ion T ees [27]: he No malized Weigh ed Euclidean dis ance be ween Models wi h Con- ou complexi y (NWMC) ee [31], and he Independen Iden ically Dis ibu ed - Kullback Leible (IID-KL) ee [3]. The exac pa ame e iza ions o each algo i hm is de ailed a [28], whe e we also publish he code o all measu es and me a-measu es used in his wo k. All me hods a e assessed Measu e Global Me a-Measu e Me a-Meas. SIHD SABD SISD Fb98.4 99.5 95.6 100.0 Fop 96.7 98.4 94.2 97.5 VoI 94.0 96.9 87.5 97.7 C(S→{Gi})91.5 93.1 86.0 95.3 d D 90.7 95.1 86.9 90.1 DH(S⇒{Gi})89.5 78.5 91.3 98.8 BCE 89.2 93.3 78.9 95.4 BGM 88.1 90.7 81.6 92.0 PRI 86.7 77.7 88.8 93.7 C({Gi}→S)86.3 91.3 77.4 90.1 F 86.1 77.0 84.2 97.1 DH({Gi}⇒S)80.5 73.0 92.1 76.5 Table 2. Measu e compa ison in e ms o quan i a i e me a- measu es. Values e e o pe cen ages o co ec esul s a he Op imal Da ase Scale (ODS) [2] wi h espec o each e alua ion measu e. The pa ame e alues o he newly p oposed measu e a e: γo= 0.95,γp= 0.25, and β= 0.1. They ha e been ained on he aining se o BSDS500 [2], op imizing he global me a-measu e desc ibed in he ollowing sec ion (See Table 2). No e ha his op imiza ion would no ha e been easible wi hou such quan i a i e me a-measu es. Me a-Measu es Resul s: Table 2shows he h ee me a- measu e esul s o he es se o BSDS500, as well as a global summa y me a-measu e. Gi en ha each me a- measu e ep esen s a pe cen age o co ec esul s, we de- ine he global me a-measu e as he global pe cen age o co ec esul s. In global e ms, Fband Fop a e he wo op- anked sum- ma y measu es. On op o ha , hey bo h p o ide much iche in o ma ion in o m o p ecision- ecall cu es, hus we p opose he pai Fb-Fop as he measu es o choice. Rega ding he compu a ional cos o he measu es, he mean ime o image o compu e he dis ances o he mul iple-pa i ion g ound u h o BSDS500 is 3.79 ±2.06 s o Fband a leas one o de o magni ude lowe o he es o measu es. In pa icula , Fop akes 0.078 ±0.020 s. In scena ios whe e he ime limi a ions a e igh , he au- ho s belie e ha Fop would be he ool o choice. To p o- ide an in-dep h analysis o he inal esul s, he andem o p ecision- ecall cu es o bounda ies and o objec s-and- pa s would be he mos adequa e op ion. The ollowing sec ion p o ides a ho ough analysis o bo h amewo ks on he six SoA me hods used in his pape . P ecision-Recall F amewo ks: Figu e 4shows he bounda y and objec s-and-pa s p ecision- ecall cu es o Bounda ies 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Recall P ecision Human [0.81-0.21] MShi [0.60] gPb-OWT-UCM [0.73] IID-KL [0.57] NCu s [0.63] NWMC [0.55] EGB [0.61] Quad ee [0.41] Objec s and Pa s 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Recall P ecision Human [0.56-0.05] gPb-OWT-UCM [0.35] MShi [0.23] NWMC [0.22] NCu s [0.21] IID-KL [0.19] EGB [0.16] Quad ee [0.06] Figu e 4. P ecision-Recall cu es o bounda ies (le ) and o objec s and pa s ( igh ). The solid cu es ep esen he six SoA segmen a ion me hods and he quad ee (see legends). In dashed lines wi h he same colo , he SoA echniques assessed on a swapped image. The ma ke on each cu e is placed on he Op imal Da ase Scale (ODS). The isola ed ed as e isks e e o he human pe o mance assessed on he same image and on a swapped image. In he legend, he F measu e o he ma ked poin on each cu e is p esen ed in b acke s. he six SoA segmen a ion me hods s udied and he human pe o mance. P io o he assessmen o segmen a ion ech- niques, le us ocus on he compa ison o he wo e alua ion amewo ks. I is no iceable ha he human baseline pe o mance (hu- man assessed on a di e en image) o Fbis 0.21, which could be in e p e ed as Fbbeing oo lax. In his same di ec- ion, he baseline bounda y p ecision o Fbis be ween 0.2 and 0.3, ha is, any esul , no ma e how w ong i is, will be judged as p o iding a leas a 0.2 p ecision. While in he case o Fop he human baseline is co ec ly downg aded o 0.05 (as well as he swapped-image esul s), hen he su p ising ac is ha human pe o mance is as low as 0.56 (0.81 in Fb), which could en ail ha Fop is oo s ic . Al hough he dynamic ange is a li le highe in Fb(0.60 e sus 0.51), he gap be ween he bes me hod and humans is much highe in Fop (0.08 e sus 0.21). In o he wo ds, Fop gi es mo e esolu ion a he places whe e imp o e- men s o e he SoA would be placed. Rega ding he compa ison among segmen a ion ech- niques, bo h amewo ks con i m ha he gPb-OWT-UCM echnique has ou s anding esul s wi h espec o he es . The ad an ages o going beyond he summa y measu es a e also clea on hese plo s. Fo ins ance, he summa y Fbmeasu e o quad ee (0.41) judges his echnique close o NWMC (0.55), bu in he p ecision- ecall cu es i is clea ha quad ee is much wo se. Simila ly, judging by Fb, NWMC would be clea ly disca ded bu i we a e in e - es ed in low ecall a es i could be o in e es (apa om gPb-OWT-UCM). As common poin s be ween he wo measu es, NCu s is judged as being much be e a high ecall a es han a low ones and con e sely, NWMC is much be e a high p eci- sion a es. The measu es a e cohe en also in he ac ha human esul s ha e a be e p ecision han ecall. As one o he main disc epan poin s, howe e , EGB is judged as he hi d bes echnique by Fbwhile being he wo se o Fop. To u he analyze his beha io , Figu e 5 shows an image (a), an EGB esul (b), and he associa ed g ound u h (c). The EGB esul consis s o hin long e- gions ha su ound he objec bu do no close. The assess- men alue o his esul is Fb= 0.62 and Fop = 0.05. F om a egion-based poin o iew, his ype o esul s is co ec ly penalized by Fop and no by Fb, since as a con- ou de ec o he esul is co ec . To sum up, bo h measu es a e complemen a y hus we p opose hem in andem as he ool o choice o image seg- men a ion e alua ion. (a) (b) (c) Figu e 5. EGB esul co ec ly penalized by Fop bu no by Fb 6. Conclusions This pape e iews an ex ensi e se o segmen a ion e al- ua ion measu es and p esen s he new p ecision- ecall mea- su e o objec s and pa s. Th ee me a-measu es a e used ( wo newly p oposed) o quan i a i ely compa e he good- ness o he e alua ion measu es. The esul s show ha he andem bounda y and objec s-and-pa s p ecision- ecall cu es is a good candida e o benchma king segmen a- ion algo i hms; since apa om ob aining he bes me a- measu e esul s, hei p ecision- ecall cu es p o ide ich knowledge abou he esul s. By making ou code and da ase s publicly a ailable we allow esea che s o easily assess hei esul s and gain deepe unde s anding o hei algo i hms. Re e ences [1] P. A bel´ aez, B. Ha iha an, C. Gu, S. Gup a, L. Bou de , and J. Malik. Seman ic segmen a ion using egions and pa s. In CVPR, 2012. 1,4 [2] P. A bel´ aez, M. Mai e, C. C. Fowlkes, and J. Malik. Con- ou de ec ion and hie a chical image segmen a ion. IEEE TPAMI, 33(5):898–916, 2011. 1,2,3,4,5,6 [3] F. Calde e o and F. Ma ques. Region me ging echniques using in o ma ion heo y s a is ical measu es. IEEE TIP, 19(6):1567–1586, 2010. 6 [4] J. S. Ca doso and L. Co e-Real. Towa d a gene ic e alua ion o image segmen a ion. IEEE TIP, 14(11):1773–1782, 2005. 1,2,3,5 [5] J. Ca ei a and C. Sminchisescu. Cons ained pa ame ic min-cu s o au oma ic objec segmen a ion. In CVPR, 2010. 1 [6] D. Comaniciu and P. Mee . Mean shi : a obus app oach owa d ea u e space analysis. IEEE TPAMI, 24(5):603 –619, 2002. 6 [7] S. Dongen. Pe o mance c i e ia o g aph clus e ing and ma ko clus e expe imen s. Technical Repo INS-R0012, Cen um oo Wiskunde en In o ma ica (CWI), Ams e dam, The Nede lands, 2000. 1,2,3 [8] M. E e ingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisse man. The PASCAL Visual Objec Classes Chal- lenge 2012 (VOC2012) Resul s. www.pascal-ne wo k.o g/ challenges/VOC/ oc2012/wo kshop/index.h ml. 1 [9] P. Felzenszwalb, R. Gi shick, D. McAlles e , and D. Ra- manan. Objec de ec ion wi h disc imina i ely ained pa - based models. IEEE TPAMI, 32(9):1627–1645, 2010. 1 [10] P. F. Felzenszwalb and D. P. Hu enloche . E icien g aph- based image segmen a ion. IJCV, 59:2004, 2004. 6 [11] D. Hoiem, Y. Chodpa humwan, and Q. Dai. Diagnosing e o in objec de ec o s. In ECCV, 2012. 1 [12] A. Hoo e , G. Jean-Bap is e, X. Jiang, P. Flynn, H. Bunke, D. Goldgo , K. Bowye , D. Egge , A. Fi zgibbon, and R. Fishe . An expe imen al compa ison o ange image seg- men a ion algo i hms. IEEE TPAMI, 18:673–689, 1996. 1, 4 [13] Q. Huang and B. Dom. Quan i a i e me hods o e alua ing image segmen a ion. In ICIP, 1995. 1,2,3 [14] X. Jiang, C. Ma i, C. I nige , and H. Bunke. Dis ance mea- su es o image segmen a ion e alua ion. EURASIP J. Appl. Signal P ocess., 2006:1–10, 2006. 2,3 [15] T. Kanungo, B. Dom, W. Niblack, and D. S eele. A as algo i hm o MDL-based mul i-band image segmen a ion. Technical epo , IBM Reasea ch Di ision, RJ 9754 (84640), 1994. 1,2 [16] C. Lampe , M. Blaschko, and T. Ho mann. Beyond sliding windows: Objec localiza ion by e icien subwindow sea ch. In CVPR, 2008. 1 [17] G. Liu and R. Ha alick. Assignmen p oblem in edge de ec- ion pe o mance e alua ion. In CVPR, 2000. 3,4 [18] T. Malisiewicz and A. A. E os. Imp o ing spa ial suppo o objec s ia mul iple segmen a ions. In BMVC, 2007. 1,4 [19] D. Ma in. An Empi ical App oach o G ouping and Segmen- a ion. PhD hesis, EECS Depa men , Uni e si y o Cali o - nia, Be keley, Aug 2003. 2,3,4,5 [20] D. Ma in, C. Fowlkes, and J. Malik. Lea ning o de ec na - u al image bounda ies using local b igh ness, colo , and ex- u e cues. IEEE TPAMI, 26(5):530–549, 2004. 1,4 [21] D. Ma in, C. Fowlkes, D. Tal, and J. Malik. A da abase o human segmen ed na u al images and i s applica ion o e alua ing segmen a ion algo i hms and measu ing ecologi- cal s a is ics. In ICCV, 2001. 5 [22] M. Meil˘ a. Compa ing clus e ings: an axioma ic iew. In ICML, 2005. 1,2,3 [23] S. Nowozin, P. Gehle , and C. Lampe . On pa ame e lea n- ing in c -based app oaches o objec class image segmen a- ion. In ECCV, 2010. 2 [24] B. Peng and L. Zhang. E alua ion o image segmen a ion quali y by adap i e g ound u h composi ion. In ECCV, 2012. 1 [25] J. Pon -Tuse and F. Ma ques. Supe ised assessmen o seg- men a ion hie a chies. In ECCV, 2012. 1 [26] W. Rand. Objec i e c i e ia o he e alua ion o clus e ing me hods. Jou nal o he Ame ican S a is ical Associa ion, 66(336):846–850, 1971. 3 [27] P. Salembie and L. Ga ido. Bina y pa i ion ee as an e i- cien ep esen a ion o image p ocessing, segmen a ion, and in o ma ion e ie al. IEEE TIP, 9(4):561–576, 2000. 6 [28] Segmen a ion E alua ion Code. h ps://ima ge.upc.edu/web / esou ces/supe ised-e alua ion-image-segmen a ion. 2,6 [29] J. Shi and J. Malik. No malized cu s and image segmen a- ion. IEEE TPAMI, 22(8):888–905, 2000. 6 [30] R. Unnik ishnan, C. Pan o a u, and M. Hebe . Towa d ob- jec i e e alua ion o image segmen a ion algo i hms. IEEE TPAMI, 29(6):929–944, 2007. 1,3,5 [31] V. Vilaplana, F. Ma ques, and P. Salembie . Bina y pa i- ion ees o objec de ec ion. IEEE TIP, 17(11):2201–2216, 2008. 6 [32] P. Viola and M. Jones. Rapid objec de ec ion using a boos ed cascade o simple ea u es. In CVPR, 2001. 1