Segment tracking via a spatiotemporal linking process including feedback stabilization in an n-d lattice model

Senso s 2009,9, 9355-9379; doi:10.3390/s91109355 OPEN ACCESS senso s ISSN 1424-8220 www.mdpi.com/jou nal/senso s A icle Segmen T acking ia a Spa io empo al Linking P ocess including Feedback S abiliza ion in an n-D La ice Model Babe e Dellen 1,2,?, E en E dal Aksoy 3and Flo en in W¨ o g¨ o e 3 1Be ns ein Cen e o Compu a ional Neu oscience G¨ o ingen, Max-Planck Ins i u e o Dynamics and Sel -O ganiza ion, Bunsens asse 10, 37073 G¨ o ingen, Ge many 2Ins i u de Rob` o ica i In o m` a ica Indus ial (CSIC-UPC), Llo ens i A igas 4-6, 08028 Ba celona, Spain 3Be ns ein Cen e o Compu a ional Neu oscience G¨ o ingen, Depa men o Compu a ional Neu oscience, III. Physikalisches Ins i u , Geo g-Augus Uni e si y G¨ o ingen - Biophysik, F ied ich-Hund Pla z 1, 37077 G¨ o ingen, Ge many; E-Mails: [email p o ec ed] (E.A.); wo [email p o ec ed] (F.W.) ?Au ho o whom co espondence should be add essed; E-Mail: [email p o ec ed]; Tel.: +49-551-391-0765; Fax: +49-551-397-720. Recei ed: 7 Augus 2009; in e ised o m: 5 No embe 2009 / Accep ed: 9 No embe 2009 / Published: 20 No embe 2009 Abs ac : Model- ee acking is impo an o sol ing asks such as mo ing-objec acking and ac ion ecogni ion in cases whe e no p io objec knowledge is a ailable. Fo his pu pose, we ex end he concep o spa ially synch onous dynamics in spin-la ice models o he spa io empo al domain o ack segmen s wi hin an image sequence. The me hod is ela ed o synch oniza ion p ocesses in neu al ne wo ks and based on supe pa amagne ic clus e ing o da a. Spin in e ac ions esul in he o ma ion o clus e s o co ela ed spins, p o iding an au oma ic labeling o co esponding image egions. The algo i hm obeys de ailed balance. This is an impo an p ope y as i allows o consis en spin- ans e ac oss subsequen ames, which can be used o segmen acking. The e o e, in he acking p ocess he co ec equilib ium will always be ound, which is an impo an ad ance as compa ed wi h o he mo e heu is ic acking p ocedu es. In he case o long image sequences, i.e., mo ies, he algo i hm is augmen ed wi h a eedback mechanism, u he s abilizing segmen acking. Keywo ds: model- ee segmen acking; image mo ion; image segmen a ion Senso s 2009,99356 1. In oduc ion How can me make sense ou o a complex isual scene ha ing no o only li le p io knowledge abou i s con en s and he objec s he ein? Such p oblems occu , o example, i we wish o lea n cause-e ec s in an hi he o unknown en i onmen . Vice e sa, many objec de ini ions a e only meaning ul wi hin he con ex o a gi en scena io and a se o possible ac ions. Objec acking, i.e., he assignmen o consis en labels o objec s in di e en ames o a ideo, is impo an o sol ing a ious asks in he ield o compu e ision, including au oma ic su eillance, human-compu e in e ac ion and a ic moni o ing [1]. Mos objec acking algo i hms equi e ha p ede ined objec s o in e es s a e de ec ed in he i s ame o in e e y ame o he mo ie. Howe e , in an unknown scena io, he acking o image segmen s, p esumably ep esen ing pa s o objec s, allows o pos pone objec de ini ion o a la e s ep o he isual-scene analysis. Se e al app oaches o segmen acking ha e been p oposed in he con ex o ideo segmen a ion [2–9]. Some app oaches ely on segmen ing each ame independen ly, e.g., by classi ying pixels in o egions based on simila i y in he ea u e space, ollowed by a segmen ma ching s ep based on hei low-le el ea u es [2–5]. O he me hods use mo ion p ojec ion o link segmen s, i.e., he posi ion o a segmen in a u u e ame is es ima ed om i s cu en posi ion and mo ion ea u es [6–9]. C eme s (2003) acked mo ion segmen s by join ly sol ing mo ion segmen a ion and mo ion es ima ion by minimizing a single ene gy unc ional [10]. In he wo ks desc ibed abo e, assump ions o he na u e o objec s being acked o he da a i sel is being made ei he in he acking p ocedu e i sel (ma ching assump ions) o al eady in he segmen a ion s ep, o example by assuming a p io i a da a model o some kind. In ou wo k, we aim o educe a p io i assump ions on he da a by choosing a da a-d i en me hod, i.e., supe pa amagne ic clus e ing o da a, o he segmen a ion p ocedu e. Supe pa amagne ic clus e ing inds he equilib ium s a es o a e omagne ic Po s model de ined by an ene gy unc ion in he supe pa amagne ic phase [11–17]. The Po s model [11] is a gene aliza ion o he Ising model [18] which desc ibes a sys em o in e ac ing g anula e omagne s o spins, ha can be in qdi e en s a es, cha ac e izing he poin ing di ec ion o he espec i e spin ec o s. Depending on he empe a u e, i.e., diso de in oduced o he sys em, he spin sys em can be in he pa amagne ic, he supe pa amagne ic, o he e omagne ic phase. In he e omagne ic phase, all spins a e aligned, while in he pa amagne ic phase he sys em is in a s a e o comple e diso de . In he supe pa amagne ic phase egions o aligned spins coexis s. Bla e al. (1998) applied he Po s model o he image segmen a ion p oblems in a way ha in he supe pa amagne ic phase egions o aligned spins co espond o a na u al pa i ion o he image da a [15]. Finding he image pa i ion co esponds o he compu a ion o he equilib ium s a es o he Po s model. Impo an ly, he ene gy unc ion used he e only consis s o in e ac ion e ms and does no con ain a da a e m, hence no p io da a model needs o be assumed, e.g. o al numbe o segmen s o a p io i de ined pixel-label assignmen s. In his sense, he me hod is model ee. In con as , me hods which ind solu ions by compu ing he minimum o an ene gy unc ion usually equi e a da a e m—o he wise only i ial solu ions a e ob ained. Hence, he equilib ium-s a e app oach o he image segmen a ion p oblem has o be conside ed as undamen ally di e en om app oaches inding he minimum ene gy con igu a ion o ene gy unc ions in Ma ko andom ields [19,20]. As a Senso s 2009,99357 consequence, he me hod is non-pa ame ic and hus mo e gene ic, only equi ing a pixel-pixel simila i y c i e ium, e.g., g ay alue, colo o ex u e, o be de ined wi hou making any u he p io assump ions abou he da a. The equilib ium s a es o he Po s model ha e been app oxima ed in he pas using he Me opolis-Has ings algo i hm wi h annealing [21] and he me hods on clus e upda ing, which a e known o accele a e he equilib a ion o he sys em by sho ening he co ela ion imes be ween dis an spins. P ominen algo i hms a e Swendsen-Wang [22], Wol [23], and ene gy-based clus e upda ing (ECU) [16,17]. In addi ion, he ECU algo i hm has been shown o p o ide s able segmen a ion esul s o e a la ge empe a u e ange (which is he main con ol pa ame e ) han Swendsen-Wang [16,17]. All o hese me hods obey de ailed balance, ensu ing con e gence o he sys em o he equilib ium s a e. Howe e , con e gence has only been shown o be polynomial o special cases o he Po s model. The Po s model is known o be NP ha d in he gene al case. The elaxa ion p ocesses which can be emula ed in spin-la ice models a e o en being used as app oxima ions o synch oniza ion p ocesses wi hin neu al assemblies [12–17]. In ou me hod, acking o segmen s is accomplished h ough simul aneous segmen a ion o adjacen ames which a e linked using local co espondence in o ma ion, e.g., compu ed ia s anda d algo i hms o op ic low [24]. By pe o ming mo ion p ojec ion and image segmen a ion wi hin a single p ocess we do no equi e any explici ma ching o image segmen s, hus u he educing a p io i assump ions ha ha e o be made abou he da a. Synch onizing he segmen a ion p ocess o adjacen ames has u he he ad an age ha pa i ioning ins abili ies can be educed. Usually, he segmen a ion (o pa i ion) o an image is sensi i e o global and local changes o he image, i.e., small changes in illumina ion, he appea ance/disappea ance o objec s pa s, causing he pa i ion o change om one ame o he nex . To u he s abilize segmen acking in he case o long image sequences, we de eloped a eedback con ol mechanism, which allows segmen a ion ins abili ies, e.g., sudden disappea ances o segmen s, o be de ec ed and emo ed by adjus ing a con ol pa ame e o he segmen a ion algo i hm. Thus, he main con ibu ion he e is he de elopmen o a da a-d i en, model- ee acking algo i hm, i.e., no p io objec knowledge is equi ed in o de o ack image segmen s om ame o ame. Po en ial applica ions include mo ing objec de ec ion and acking, and ac ion ecogni ion and classi ica ion om mo ies [25]. The pape is s uc u ed as ollows: In Sec ion 2, we ex end he me hod o supe pa amagne ic clus e ing in spin models o he empo al dimension and in oduce he con olle algo i hm. We also discuss he me hod in on o he backg ound o ene gy minimiza ion me hods in Ma ko andom ields. In Sec ion 3, we i s e i y he co e algo i hm using sho image sequences because hese a e mo e sui able o in oduce and es he me hod. We u he in es iga e he sensi i i y o he algo i hm o sys em pa ame e s and noise. Then, we demons a e ha segmen acking can be achie ed o eal mo ies. The pe o mance o he me hod is quan i ied in e ms o pa i ioning consis ency along an a i icial image sequence. In Sec ion 4, he esul s a e discussed. 2. Algo i hmic F amewo k Segmen acking can be oughly di ided in o he ollowing sub asks: (i) image segmen a ion, (ii) linking ( acking), and (iii) s abiliza ion ( acking). The hi d poin acknowledges ha segmen s, unlike Senso s 2009,99358 objec s, a e no pe se s able en i ies, bu a e sensi i e o changes in he isual scene. Sub asks (i–ii) will be sol ed using a conjoin spin- elaxa ion p ocess emula ed in an n-dimensional (n-D) la ice, which de ines he co e algo i hm (Sec ion 2.1). Local co espondence in o ma ion o linking is ob ained using s anda d algo i hms o ei he s e eo o op ic low [24,26]. The conjoin segmen a ion app oach has he ad an age ha he spin- elaxa ion p ocesses o adjacen images synch onize, educing pa i ioning ins abili ies. Since simul aneous segmen a ion o long image sequences is p ac ically impossible due o he high compu a ional cos s, we usually spli he image sequence in o a sequence o pai s. Fo example, he subsequen ames 0, 1and 2a e spli in o wo pai s { 0, 1}and { 1, 2}, whe e he las ame o p e ious pai is iden ical o he i s ame o he nex pai . I a segmen o he las ame o { 0, 1} and a segmen o he i s ame o { 1, 2}occupy he same image egion, we can assign he same segmen label o bo h segmen s. This way segmen s can be acked h ough he en i e sequence. Since he algo i hm p ese es de ailed balance (Sec ion 2.1), spins can be ans e ed om one ame o he nex , g ea ly educing he numbe o i e a ions equi ed o achie e a s able segmen a ion. We u he s abilize segmen acking by in oducing a eedback con olle (Sec ion 2.2). In long image sequences, pa i ioning ins abili ies a e likely o a ise a some poin du ing he acking p ocess. Thus, segmen s may be los due o me ging o spli ing o segmen s. The eedback con olle de ec s hese kind o ins abili ies and adjus s a con ol pa ame e o he co e algo i hm o eco e he o iginal segmen s. 2.1. Co e Algo i hm The me hod o supe pa amagne ic clus e ing has been p e iously used o segmen single images [15–17]. Applying his amewo k o image sequences equi es spin in e ac ions o ake place ac oss ames. Due came a and objec mo ion he images unde go changes du ing he cou se o ime. To connec di e en ames, he mapping om one ame o he nex needs o be known a leas in some app oxima ion. We sol e his p oblem in he ollowing way: Poin co espondences, de i ed using algo i hms o dispa i y o op ic- low compu a ion, can be inco po a ed in o he Po s model [11] by allowing spins belonging o di e en ames o he image sequence o in e ac i he espec i e pixels belong o locally co esponding image poin s. Then, spins belonging o he di e en ames o he sequences a e elaxed simul aneously, esul ing in a synch onized segmen a ion o he images o he sequence. The in e - ame spin in e ac ions cause he spins o co esponding image egions o align, and, hus, hey will be assigned o he same segmen . Since he o ma ion o segmen s is a collec i e p ocess, he poin co espondences do no ha e o be e y accu a e no does he algo i hm equi e poin co espondences o each pixel. I is usually su icien i he a ailable co espondences cap u e he cha ac e is ics o he scene only oughly. The aim o his wo k is o ind co esponding image egions in image sequences, i.e., s e eo pai s and mo ion sequences. The segmen acking ask is o mula ed as ollows. Gi en an image sequence S con aining poin s p(x, y, )wi h coo dina es (x, y, )as elemen s, whe e xand ylabel he posi ion wi hin each image, while labels he ame numbe , hen we wan o ind a pa i ioning P= P 1, .., P Mo S in Mg oups such ha Senso s 2009,99359 (i) P i∩ P j= 0 and P i6= Ø o all g oups (ii) i poin p∈ P i, hen s(p, P i)> s(p, P j), whe e sis a unc ion measu ing he a e age dis ance o a poin o he elemen s o a g oup and i6=j (iii) i p(xi, yi, i)∈ P , hen p(xi+4xi, yi+4yi, i+ 1) ∈ P , whe e 4xiand 4yia e he shi s o poin p(xi, yi, i)along he xand yaxes, espec i ely, om ame i o ame i+ 1. To pe o m his ask, we assign a spin a iable σi(o label) o each pixel (o si e) io he image sequence. To inco po a e cons ain s in he o m o local co espondence in o ma ion, we dis inguish be ween neighbo s wi hin a single ame (2D bonds) and neighbo s ac oss ames (n-D bonds). We c ea e a 2D bond (i, k)2Dbe ween wo pixels wi hin he same ame wi h coo dina es (xi, yi, i)and (xk, yk, k)i |(xi−xk)| ≤ ε2D(1) |(yi−yk)| ≤ ε2D(2) i= k(3) whe e ε2Dis he 2D-in e ac ion ange o he spins, a pa ame e o he sys em. Ac oss ames, we c ea e a n-D bond (i, j)nD be ween wo spins iand ji |(xi+dx ij −xj)| ≤ εnD (4) |(yi+dy ij −yj)| ≤ εnD (5) i6= j(6) aij > τ (7) whe e εnD is he n-D in e ac ion ange. The alues dx ij and dy ij a e he shi s o he pixels be ween ames iand jalong he axis xand y, espec i ely, ob ained om he op ic- low map o dispa i y map. The pa ame e s aij ∈[0,1] a e he espec i e ampli udes (o con idences) o he op ic- low o s e eo algo i hm. A la ge ampli ude sugges s a la ge con idence in he compu ed local co espondence. Pa ame e τis a h eshold, emo ing all local co espondences ha ing a small ampli ude. We de ine o e e y bond on he la ice he dis ance 4ij =|gi−gj|(8) whe e giand gja e he ea u e ec o s, e.g., g ay alue, colo , o ex u e, o he pixels iand j, espec i ely. The mean dis ance ¯ 4is ob ained by a e aging o e all bonds. We u he de ine an in e ac ion s eng h Jij = 1 − 4ij /¯ 4(9) Senso s 2009,99360 The spin model is now implemen ed such a way ha neighbo ing spins wi h simila colo ha e he endency o align. We use a q-s a e Po s model [11] wi h he Hamil onian H=−X hiki2D Jikδ(σi−σk)−X hijinD Jijδ(σi−σj)(10) He e, hiki2Dand hijinD deno e ha i, k and i, j a e connec ed by bonds (i, k)2Dand (i, j)nD, espec i ely. The K onecke δ unc ion is de ined as δ(a) = 1 i a= 0 and ze o i o he wise. The segmen a ion p oblem is hen sol ed by inding clus e s o co ela ed spins in he low empe a u e equilib ium s a es o he Hamil onian H. The empe a u e pa ame e de e mines he amoun o diso de in oduced o he sys em. The spin s a es ha e o be obse ed o e se e al i e a ions o iden i y clus e s as g oups o co ela ed spins. The o al numbe Mo segmen s is hen de e mined by coun ing he compu ed segmen s. The spin s a es σican ake in ege alues be ween 1and q, whe e qis a pa ame e o he algo i hm. The numbe o segmen s is no cons ained by he pa ame e q. Please no e ha segmen labels a e no iden ical o spin s a es. Spins which belong o he same segmen a e always in he same spin s a e, howe e , he e e se is no necessa ily ue. No e ha he local co espondences used in he algo i hm o c ea e n-D bonds a e p ecompu ed and a e no al e ed o op imized when compu ing he equilib ium s a e. The compu a ion o local co espondences is no he aim o his pape . We ind he equilib ium spin con igu a ion using a clus e ing algo i hm. In a i s s ep, “sa is ied” bonds, i.e., bonds connec ing spins ha ing iden ical spin alues σi=σj, a e iden i ied. Then, in a second s ep, he sa is ied bonds a e “ ozen” wi h a some p obabili y Pij. Pixels connec ed by ozen bonds de ine a clus e , which a e upda ed by assigning o all spins inside he same clus e s he same new alue [22]. In he me hod o supe pa amagne ic clus e ing p oposed by Bla e al. (1996) [15] his is done independen ly o each clus e . In his pape , we will employ he me hod o ene gy-based clus e upda ing (ECU), whe e new alues a e assigned in conside a ion o he ene gy gain calcula ed o a neighbo hood o he ega ded clus e [16,17]. A schema ic o he spin sys em o an image sequence is depic ed in Figu e 1A. The ECU algo i hm compu ing he equilib ium o Hconsis s o he ollowing s eps: 1. Ini ializa ion: A spin alue σibe ween 1 and qis assigned andomly o each spin i. Each spin ep esen s a pixel o he image sequence. 2. Compu ing bond eezing p obabili ies: I wo spins iand ja e connec ed by a bond and a e in he same spin s a e σi=σj, hen he bond is ozen wi h a p obabili y Pij = 1 −exp(−0.5Jij/T)(11) Nega i e p obabili ies a e se o ze o. 3. Clus e iden i ica ion: Pixels which a e connec ed by ozen bonds de ine a clus e . A pixel belonging o a clus e uhas by de ini ion no ozen bond o a pixel belonging o a di e en clus e . Senso s 2009,99361 4. Clus e upda ing: We pe o m a Me opolis upda e [22,27] ha upda es all spins o each clus e simul aneously o a common new spin alue. The new spin alue o a clus e cis compu ed conside ing he ene gy gain ob ained om a clus e upda e o a new spin alue wk, whe e he index kdeno es he possible spin alues be ween 0and q, espec i ely. No e ha his new spin alue is, once chosen, a cons an and has o be dis inguished om he spin a iables σi. Upda ing he espec i e clus e o he new alue esul s in a new spin con igu a ion Wc k. Figu e 1. A The spin s a es (upwa d and downwa d poin ing a ows) o pixels i,k, and ja e shown be o e and a e a spin upda e o wo adjacen ames and +1 o an image sequence. The whi e and black ci cles indica e pixels o small and la ge g ay alues, espec i ely. Pixel iin e ac s wi h pixels kand jin i s 2D and 3D neighbo hood (shaded a eas), espec i ely, which a e in he same spin s a e. BPai wise (3D) segmen a ion o mo ies. A eedback con olle de ec s segmen a ion ins abili ies and adjus s he con ol pa ame e To he co e algo i hm (3D segmen a ion) o eco e los segmen s. d F ame F ame +1 Be o e Upda e A e Upda e ij i i j k k 3D 2D A B j Con olle 3D Segmen a ion F ame numbe T The p obabili y o he choosing he new spin alue wk o clus e c is compu ed by conside ing he in e ac ions o all spins in he clus e cwi h hose ou side he clus e , assuming ha all spins o he clus e a e upda ed o he new spin alue wk, gi ing an ene gy E(Wc k) = X i∈c£−X hiji2D ck6=cj ηJijδ(σi−σj)−X hijinD ck6=cj ηaijJij δ(σi−σj)¤ (12) Senso s 2009,99362 whe e hiki2D, ck6=cjand hijinD, ck6=cja e he nonclus e neighbo hoods o spin i. He e, Nis he o al numbe o pixels o he image sequence. The cons an ηis chosen o be 0.5. Simila o a Gibbs sample , he selec ing p obabili y P(Wc k) o choosing he new spin alue o be wkis gi en by P(Wc k) = exp(E(Wc k))/ q X l=1 exp(E(Wc l)) (13) 5. I e a ion: The new spin s a es a e e u ned o s ep 2 o he algo i hm, and s eps 2–5 a e epea ed, un il he o al numbe o clus e s s abilizes, i.e., he change o he numbe o clus e s is small compa ed o he change o he numbe clus e s du ing he i s en i e a ions (see also Figu e 5F). In p ac ice, we usually use a ixed numbe o i e a ions. 6. Segmen s a e de ined as g oups o co ela ed spins and can be ex ac ed using a h esholding p ocedu e. All pai s o pixels connec ed by a bond (i, j)wi h c(σi, σj)> θ a e conside ed o be iends, i.e., hey mus belong o he same segmen . The unc ion ccompu es he co ela ion o he spin s a es o iand jo e se e al i e a ions. Then, all mu ual iends a e assigned o he same segmen . Finally, Mis de e mined by coun ing he o al numbe o segmen s. In p ac ice, we ind i su icien o ake he clus e s ound in he las i e a ion as segmen s. In an ea lie s udy we had p o ided e idence ha his algo i hm obeys de ailed balance. The ull p oo shall no be epea ed he e and can be ound in [16]. De ailed balance assu es ha he p oposed algo i hm compu es an equilib ium spin con igu a ion o he ene gy unc ion and ha his is Bol zmann dis ibu ed. Howe e , he equilib ium migh no be ound in polynomial ime (see Sec ion 2.3). The consequence o de ailed balance is ha spin s a es can be ans e ed ac oss image pai s, whe e spins a e being calcula ed o one pai ( he i s pai ) and hen pixels in he nex wo ames ( he second pai ) a e jus assigned hese spins om whe e on a new elaxa ion p ocess s a s (see Figu e 5 o an example). Hence, he elaxa ion p ocess o he second pai (and all o ollow) is much as e when using spin ans e and he sys em will always a i e a he he modynamic equilib ium making spin- ans e based segmen a ion conco dan ac oss ames. No e, his p ope y allows consis en segmen acking ac oss many ames wi hou addi ional assump ions (see Figu e 5), which equi es mo e e o wi h o he me hods. The ollowing should be no ed. In his me hod, bonds be ween adjacen ames a e c ea ed om he p ecompu ed op ic- low o dispa i y maps and ozen wi h a p obabili y ha depends on he ea u e simila i y o he espec i e co esponding pixels. Whe he hese bonds a e ozen in he inal con igu a ion (i.e., he espec i e p oposed poin co espondences a e accep ed), and hus p omo e he o ma ion o a ce ain segmen co espondence, is decided inhe en ly by inding he equilib ium s a e o Equa ion 10. This p ocedu e makes he me hod mo e obus agains e o s in he op ic- low o dispa i y map. 2.2. Feedback Con ol Segmen a ion ins abili ies a ising du ing he acking p ocess can be pa ly emo ed by adjus ing he empe a u e pa ame e o he co e algo i hm. The empe a u e choice a ec s he o ma ion o segmen s, Senso s 2009,99363 hence, a segmen which has been los in a p e ious ame can some imes be eco e ed by inc easing he empe a u e o a ce ain pe iod. The eedback con olle acks he size o he segmen s and eac s i he size o a segmen changes suddenly. The i s con olle unc ion Pj C( ) =    1i 4Sj( )< τ1, exp [−4Sj( )/α]/β o he wise,(14) measu es he p obabili y o change o segmen j, whe e Sj( )is he size o segmen ja ame and 4Sj( ) = Sj( )−Sj( −1), and α,β,τ1a e cons an pa ame e s. The his o y o segmen jin e ms o occu ence is cap u ed by he second con olle unc ion Pj H( ) = 0.4Hj( −1) + 0.3Hj( −2) + 0.2Hj( −3) + 0.1Hj( −4) (15) wi h Hj( ) = 1 i Sj( )>0and ze o o he wise. Segmen a ion ins abili ies may cause a segmen o be los , o example h ough segmen me ging o spli ing. We de ine wo h eshold pa ame e s τ2and τ3. An unexpec ed segmen loss is de ec ed by he con olle i he condi ions Sj= 0 (16) PC< τ2(17) and PH> τ3(18) a e ul illed. An unexpec ed segmen appea ance is de ec ed by he con olle i he condi ions PC< τ2(19) and PH< τ3(20) a e ul illed. The iden i ies o he a ec ed segmen s a e s o ed by he con olle . The empe a u e o he co e algo i hm is a ied using p ede ined empe a u e s eps 4T. The segmen a ion is epea ed a he new empe a u e T+4Tonly o he a ec ed segmen s o he ames. I he los segmen s can be eco e ed a one o hese empe a u es, he a ec ed segmen s a e elabeled acco dingly. Fu he mo e, he new empe a u e T+4T alues o he a ec ed segmen s a e inhe i ed o he nex ame in o de o p e en he same segmen a ion e o . No e ha he empe a u e alue can be inc eased o a p ede ined maximum alue, o he wise undesi ed a i ac s migh be obse ed. A schema ic o he en i e sys em, i.e., co e algo i hm wi h eedback con ol, is p esen ed in Figu e 1B. 2.3. Compa ison wi h Ene gy-Minimiza ion-Based Me hods in Ma ko Random Fields The me hod desc ibed in his pape is compa ed wi h ene gy minimiza ion in Ma ko andom ields. Simila i ies and di e ences o he app oaches a e analyzed and summa ized in his sec ion. Pa icula a en ion is gi en o combina o ial g aph cu s me hods, which ha e p o ided powe ul compu e ision algo i hms o s e eo, mo ion, image es o a ion, and image segmen a ion in ecen yea s [19,20]. Senso s 2009,99370 Figu e 5. A Th ee ames o a walking sequence, labeled 0, 1, and 2, espec i ely. B Op ic- low ec o s coding he mapping om ame 0 o 1, and 1 o 2.CSpin s a es a e 100 i e a ions. DSpin s a es o a sequence consis ing o ames 0and 1.ESpin s a es o a sequence consis ing o 1and 2.FThe numbe o clus e s as a unc ion o he i e a ion numbe o he i s sequence con aining ame 0and 1(dashed line) and o he second sequence con aining ame 1and 2(solid line). GEnla ged plo o he second sequence. 0 1 2 A 0 20 40 60 80 0.5 1 1.5 2 2.5 3x104 I e a ion 0 20 40 60 80 0.49 0.5 0.51 0.52 x10 Numbe o clus e s B C D E FG 0 1 4 Numbe o clus e s I e a ion When analyzing long mo ion sequences, i is ine icien o apply he algo i hm o all ames a once because he compu a ional cos s inc ease wi h he numbe o pixels. Hence, we spli he sequences in Senso s 2009,99371 pai s o wo ames a a ime, whe e he las ame o he p e ious sequence is iden ical wi h he i s ame o he nex sequence. Then, we ini ialize he spin s a es o each sequence wi h he inal spin s a es o he p e ious sequence. The spin s a es o he i s sequence con aining ame 0and 1a e 100 i e a ions a e shown in Figu e 5D. Then, he algo i hm is applied o he nex pai , con aining ame 1and 2, whe e he spin s a es o bo h ame ha e been ini ialized o he inal spin s a es o ame 1 o he p e ious sequence. The spin s a es a e 13 i e a ions a e shown in Figu e 5E, demons a ing ha he numbe o i e a ions equi ed o achie e a sa is ying segmen a ion esul is g ea ly educed by his echnique. The numbe o clus e s o he i s sequence and he second sequence a e displayed as a unc ion o he i e a ion numbe in Figu e 5F, dashed and solid line, espec i ely. The numbe o clus e s o he second sequence is plo ed as a unc ion o he i e a ion numbe a a di e en scale (Figu e 5G). Ini ially, he numbe o clus e s dec eases sligh ly and hen app oaches a s able s a e. In a mo ion sequence, he numbe o clus e s is expec ed no o change much om one ame o he nex . Mainly he bounda ies o he clus e s eo ganize du ing he i s i e a ions. The segmen s o adjacen image pai s a e connec ed as ollows. Two segmen s belonging o he segmen a ion o ame 1o pai { 0, 1}and ame 1o pai { 1, 2}, espec i ely, a e assigned he same label i hey occupy he same egion in image ame 1. Fo his pu pose, we compu e he pe cen age o he clus e a eas which o e lap in he image space. I he o e lap is la ge han a ixed h eshold, he clus e s a e assigned he same label. This way we can ack he segmen h ough he whole sequence. 3.2. Segmen T acking wi h Feedback S abiliza ion We add eedback con ol (see Sec ion 2.2) wi h pa ame e s α= 200 pixels, β= 0.8,τ1= 50 pixels, τ2= 0.9, and τ3= 0.6 o he co e algo i hm wi h empe a u e T= 0.05 and apply he algo i hm o long image sequences. The i s mo ie shows a hand aking a ed apple om a pla e wi h se e al ui s. A ew ames o he mo ie a e depic ed in he uppe panel o Figu e 6A. I he co e algo i hm is applied a cons an empe a u e wi hou eedback con ol, he ed segmen and he ligh pink segmen , ep esen ing he espec i e pa s o he ed apple and he o ange, a e los a ame numbe 45 due a segmen a ion ins abili y: The ed segmen and he ligh pink segmen me ge and o m a new segmen , colo ed in ligh blue (see Figu e 6A, middle panel). I eedback con ol is included, his segmen a ion ins abili y is de ec ed and he o iginal segmen s can be eco e ed by inc easing he empe a u e in s eps o 4T= 0.15. As a consequence, he segmen s can be con inuously acked, as shown in Figu e 6A (lowe panel). The segmen s ep esen ing he cup could be eco e ed using he same mechanism. The wo k o he eedback con olle is u he illus a ed in Figu e 6B, whe e he segmen size is plo ed as a unc ion o he ame numbe o he segmen s ep esen ing he ed apple and he o ange wi hou and wi h eedback con ol, depic ed as ed, blue, b own and g een lines, espec i ely. A ame numbe 45 he segmen sizes o he ed apple and he o ange d op unexpec edly o ze o ( ed and blue lines), hus indica ing a segmen a ion ins abili y (see Sec ion 2.2). As a consequence, he eedback con olle is ac i a ed and he empe a u e o he co e algo i hm is inc eased un il he o iginal segmen s a e eco e ed (b own and g een lines). The esul s o he whole mo ie a e shown in Figu e 7A. We u he applied he algo i hm o ano he mo ie, showing he illing o a cup wi h suga (Figu e 7B). The mo ie is challenging because i con ains ligh e lexions and changing shadows. Howe e , he algo i hm is capable o acking he main segmen s o he mo ie, i.e., he wo cups and he hand. Senso s 2009,99372 Figu e 6. Feedback con ol o segmen a ion s abiliza ion. AA ew ames o a mo ie showing a hand aking a ed apple om a pla e a e shown oge he wi h he esul s o he co e algo i hm wi hou and wi h eedback con ol (uppe , middle, and lowe panel, espec i ely). BThe segmen size is plo ed as a unc ion o he ame numbe o he segmen s ep esen ing he ed apple and he o ange wi hou and wi h eedback con ol, depic ed as ed, blue, b own and g een lines, espec i ely. A ame numbe 45 he segmen sizes o he ed apple and he o ange d op unexpec edly o ze o ( ed and blue lines), and he eedback con ol is ac i a ed, inc easing he empe a u e Tun il he o iginal segmen s a e eco e ed (b own and g een lines). 0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 0 100 200 300 400 500 600 700 800 F ame Numbe Segmen size [numbe o pixels] Feedback con ol ac i a ed Red apple (no eedback) Red apple (wi h eedback) O ange (no eedback) O ange (wi h eedback) A B We ob ained simila segmen - acking esul s o o he eal mo ies, e.g., Mo ing Objec , Making Sandwich, Opening a Book. Resul s can be ound a h p://www.nld.ds.mpg.de/∼e en/Mo ies.h ml. Senso s 2009,99373 Figu e 7. Segmen acking o eal mo ies. AThe algo i hm (wi h eedback con ol) is applied o a mo ie showing a hand aking an apple om a pla e (uppe panel). The co esponding segmen - acking esul s a e depic ed below. BThe esul s o he algo i hm (wi h eedback con ol) o a mo ie showing he illing o a cup a e shown. Segmen - acking esul s o o he eal mo ies can be ound a h p://www.nld.ds.mpg.de/∼e en/Mo ies.h ml/. A B We use an a i icial image sequence o demons a e ha bo h 3D linking and eedback con ol imp o e consis ency o he pa i ioning in o segmen s o adjacen ames. The o iginal image consis s o 4×4 uni o mly- alued squa es. By adding Gaussian noise o he image, we c ea e an image sequence o 40 ames (see Figu e 8A). We apply he algo i hm a T= 0.1(i) wi hou 3D linking and wi hou eedback, (ii) wi h 3D linking and wi hou eedback, and (iii) wi h 3D linking and wi h eedback. Example segmen a ion esul s a e p esen ed in Figu e 8B. Fo case (i), he pa i ioning is o en changing om ame o ame. The labels o adjacen ames only co espond o each o he by chance, since no linking in 3D is employed (independen segmen a ions). Fo case (ii), he pa i ioning is mo e s able, bu b eaks Senso s 2009,99374 s ill occu equen ly. These b eaks can be la gely p e en ed by eedback con ol. We quan i y he s abili y o adjacen image pa i ions by compu ing he spa ial g adien o each segmen ed ame. We ind a pa i ion bounda y i he g adien is la ge han ze o. The pa i ion-bounda y images o adjacen ames a e hen sub ac ed, he absolu e alue is aken, and he sum is compu ed o e all pixels, which we call he e he pa i ion e o . In Figu e 8C, he his og ams o he pa i ion e o o e he whole sequence a e shown o all h ee cases (i-iii) wi h mean pa i ion e o s (a e aged o e all ames) o 35.72,19.44, and 2.59, espec i ely. Figu e 8. Imp o ing pa i ioning consis ency using 3D linking and eedback con ol. A F om he o iginal image consis ing o 4×4uni o mly- alued squa es, a s a ic image sequence is c ea ed by adding Gaussian noise o each ame. BSegmen a ion esul s o ou algo i hm wi hou 3D linking (no acking), wi h 3D linking, and wi h bo h 3D linking and eedback con ol. CRespec i e his og ams showing he numbe o ames as a unc ion o he pa i ion e o s (see ex ). 0 20 40 60 80 0 5 10 15 20 25 30 35 0 20 40 60 80 0 5 10 15 20 25 30 35 0 20 40 60 80 0 5 10 15 20 25 30 35 A B C 2D 3D 3D + eedback 18 19 20 21 22 15 16 17 F ame 3D + eedback 3D 2D Pa i ion e o [pixels] Numbe Numbe Numbe Senso s 2009,99375 4. Discussion We p esen ed an algo i hm o model- ee segmen acking based on a no el, conjoin amewo k, combining local co espondences and image segmen a ion o synch onize he segmen a ion o adjacen images. The algo i hm p o ides a pa i ioning o he image sequence in segmen s, such ha poin s in a segmen a e mo e simila o each o he han o poin s in ano he segmen , and such ha co esponding image poin s belong o he same segmen . We es ed he me hod on a ious syn he ic and eal image sequences, and showed s able and eliable esul s o e all, hus ul illing he mos impo an equi emen o segmen a ion algo i hms. The me hod leads o he o ma ion o s able egion co espondences despi e la gely incomple e dispa i y o op ic- low maps. Simila algo i hms o he ex ac ion o egion co espondences could po en ially be cons uc ed using o he image segmen a ion algo i hms, i.e., me hods based on agglome a i e clus e ing [36,37]. We decided o use physics-based model o i s concep ual simplici y which allowed us o in eg a e local co espondence in o ma ion in a s aigh o wa d way. I u he has he ad an age ha he in e ac ing pa s a e inhe en ly con e ging o he equilib ium s a e and hus a e no being apped in local ex ema (de ailed balance). As a consequence, he esul is independen o he ini ial condi ions, allowing us o apply he algo i hm o long image sequences ia spin-s a es ans e . This allows o consis en segmen acking ac oss many ames wi hou addi ional assump ions, which is mos o he ime no immedia ely possible wi h o he me hods. In addi ion, no assump ions abou he unde lying da a a e equi ed, e.g. he numbe o segmen s, leading o a model- ee segmen a ion. This has he consequence ha a single pixel o dis inc g ay alue (compa ed o i s neighbo s) migh de ine a single segmen . In algo i hms, which pa i ion he image in o a ixed and usually small numbe o segmen s, his phenomenon does no occu . This, howe e , is a p oblem as in all ealis ic si ua ions one ne e knows how many segmen s exis and sel -adjus men o he o al numbe o segmen s is, hus, usually desi ed as compa ed o a p e-de ined maximal numbe . We u he in oduced a eedback con olle which allows o de ec segmen a ion ins abili ies, i.e., me ging and spli ing o segmen s. The eedback con olle adjus s he con ol pa ame e o he co e algo i hm in o de o eco e he o iginal segmen s. This allows keeping ack o segmen e en in long mo ies. Segmen acking has been pe o med p e iously in he con ex o ideo segmen a ion [2–9]. Ou me hod di e s om hese app oaches in he choice o he segmen a ion algo i hm, he way linking is achie ed, and he addi ion o a eedback con olle which de ec s segmen a ion ins abili ies. Supe pa amagne ic clus e ing allows a model- ee unsupe ised segmen a ion o he image sequences, including a sel -adjus men o he o al numbe o segmen s. Linking is in oduced ough local co espondence in o ma ion which synch onizes he spin- elaxa ion p ocess o adjacen images. This app oach has he ad an age ha he pa i ions o adjacen images a e less p une o pa i ioning ins abili ies (see also Figu e 8). Fu he , ou me hod does no equi e co esponding egions o ul ill any segmen simila i y c i e ium. Finally, eedback con ol allows segmen a ion ins abili ies occu ing in long sequences o be emo ed by assuming ha “good” segmen s change hei size in a con inuous “p edic able” manne . Senso s 2009,99376 The e ha e been a ew o he app oaches combining image segmen a ion wi h co espondence in o ma ion. The wo k by Toshe e al. [38] uses a join -image g aph con aining edges ep esen ing in a-image simila i ies and in e -image ea u e ma ches o compu e ma ching egions. Join segmen a ion has also been employed by Ro he e al. [39] using his og am ma ching. The con olle employed in his model se es he de ec ion and emo al o segmen a ion ins abili ies. No assump ions abou he objec s gi ing ise o he measu emen s, i.e., segmen s, a e made he e, excep om ha hey canno appea and disappea all o a sudden (objec cons ancy) and a e hus in some way p edic able. P edic able beha io o objec s p o ides he basis o acking me hods using op imal il e s such as he Kalman il e [40], in e ac ing mul iple models [41], o pa icle il e s [42]. In hese cases howe e s onge assump ions abou he na u e o he objec s a e being made. Bo h some knowledge o he dynamics o objec s (as hen e lec ed by he measu emen s) and hei appea ance is assumed [43]. In he con ex o speci ic applica ions as o example he acking o mo ing objec s, Kalman- il e -based mo ion could po en ially be combined wi h ou me hod. Kuma e al. (2006) p oposed a me hod o mul i a ge acking in which blobs, i.e., connec ed egions ob ained om segmen ing mo ing o eg ound objec s ia a backg ound subs ac ion me hod, a e acked wi h a Kalman il e while handling spli s and me ges be ween blobs. Ce ain elemen s o his app oach may p o ide a means o u he ad ancing he segmen - acking p ocedu e desc ibed in his pape . The me hod desc ibed he e is ela ed o ene gy minimiza ion in Ma ko andom ields which has been used o sol e ision p oblems many imes be o e [19,29,44–47] (see also Sec ion 2.3). While he algo i hmic p ocedu es used o ind he ene gy minimum sha e ea u es wi h he ones employed by ou me hod, undamen al di e ences exis be ween he me hods. Supe pa amagne ic clus e ing aims a inding he equilib ium s a es o a Po s model wi hou ex e nal ield o da a ene gy e m a a ce ain empe a u e, and no a global ene gy minimum. Fo mula ing ision p oblems in e ms o ene gy minimiza ion equi es a da a penal y e m (o ex e nal ield), necessi a ing some p io knowledge abou he da a ha is being modeled. Hence, he solu ions ha e o be conside ed less gene ic and qui e di e en o ou s by na u e. The algo i hm has po en ial applica ions in model- ee mo ing objec de ec ion and acking by me ging cohe en ly mo ing segmen s (Ges al law o common a e). The me hod is u he applicable o ac ion- ecogni ion asks, whe e ce ain cha ac e is ic ac ion pa e ns a e in e ed om he spa io empo al ela ionships o segmen s. Fi s esul s o his p oblem a e epo ed in [25]. In he u u e, ex u e cues may be inco po a ed in o he algo i hm o allow acking o segmen s de ined by ex u e. Cu en ly, he algo i hm equi es ≈4-5s pe ame o images o size 160×140 pixels and ≈43 s pe ame o images o size 360 ×240 pixels (Taking-an-apple sequence) on an In el Dual Co e CPU wi h 3.16 GHz RAM ( o each co e). Since ou goal is he de elopmen o a ision- on end o eal- ime ideo segmen acking on op o which o he algo i hms, i.e., mo ing-objec de ec ion/ acking and ac ion ecogni ion, can be applied, we a e cu en ly in es iga ing ways o imp o e compu a ion ime. Fas e p ocessing on a CPU could be achie ed by imp o ing he clus e iden i ica ion s ep using a as e algo i hm [48] and by applying he algo i hm no o he pixels i sel , bu o a omic egions ob ained om Canny edge de ec ion ollowed by edge acing and con ou closing. The la e echnique has been used be o e o imp o e speed in ene gy minimiza ion [29]. Fu he mo e we a e cu en ly de eloping a pa allel implemen a ion on GPUs. So a , we eached ame a es be ween ≈10 −23 ames/s o images o Senso s 2009,99377 size 360 ×240 pixels (ce ain algo i hmic p ocedu es o he me hod ha e been eplaced by al e na i e compu a ion schemes o ma ch he equi emen s o he pa allel a chi ec u e). Acknowledgmen We hank Sinan Kalkan o aluable discussion. The wo k has ecei ed suppo om he Ge man Minis y o Educa ion and Resea ch (BMBF) ia he Be ns ein Cen e o Compu a ional Neu oscience (BCCN) G¨ o ingen unde G an No. 01GQ0430 and he EU P ojec PACO-PLUS unde Con ac No. 027657. Re e ences 1. Yilmaz, A.; Ja ed, O.; Shah, M. Objec acking: a su ey. ACM Compu . Su . 2006,38, 1–45. 2. Choi, J.G.; Lee, S.W.; Kim, S.D. Spa io- empo al ideo segmen a ion using a join simila i y measu e. IEEE T ans. Ci cui s Sys . Video Technol. 1997,7, 279–286. 3. Salembie , P.; Ma ques, F. Region-based ep esen a ions o image and ideo: Segmen a ion ools o mul imedia se ices. IEEE T ans. Ci cui s Sys . Video Technol. 1999,9, 1147–1169. 4. Tuncel, E.; Onu al, L. U iliza ion o he ecu si e sho es spanning ee algo i hm o ideo-objec segmen a ion by 2-D a ine mo ion modeling. IEEE T ans. Ci cui s Sys . Video Technol. 2000, 10, 776–781. 5. Deng, Y.; Manjuna h, B.S. Unsupe ised segmen a ion o colo - ex u e egions in images and ideo. IEEE T ans. Pa e n Anal. Machine In ell. 2001,23, 800–810. 6. Pa as, L.; Hend iks, E.A.; Lagendijk, R.L. Video segmen a ion by MAP labeling o wa e shed segmen s. IEEE T ans. Pa e n Anal. Machine In ell. 2001,23, 326–332. 7. Yokoyama, Y.; Miyamo o, Y.; Oh a, M. Ve y low bi a e ideo coding using a bi a ily shaped egion-based mo ion compensa ion. IEEE T ans. Ci cui s Sys . Video Technol. 1995,5, 500–507. 8. Wang, D. Unsupe ised ideo segmen a ion based on wa e heds and empo al acking. IEEE T ans. Ci cui s Sys . Video Technol. 1998,8, 539–546. 9. Meza is, V.; Kompa sia is, I.; S in zis, M.G. Video objec segmen a ion using Bayes-based empo al acking and ajec o y-based egion me ging. IEEE T ans. Ci cui s Sys . Video Technol. 2004,14, 782–795. 10. C eme s, D. A a ia ional amewo k o image segmen a ion combining mo ion es ima ion and shape egula iza ion. In P oceedings o IEEE Compu e Socie y Con e ence on Compu e Vision and Pa e n Recogni ion, Madison, WI, USA, 2003; pp. 53–58. 11. Po s, R.B. Some gene alized o de -diso de ans o ma ions. P oc. Camb idge Philos. Soc. 1952, 48, 106–109. 12. Geman, D.; Geman, S.; G a igne, C.; Dong, P. Bounda y de ec ion by cons ained op imiza ion. IEEE T ans. Pa . Anal. Mach. In . 1990,12, 609–628. 13. Vo b ¨ uggen, J.C. Zwei Modelle zu da enge iebenen Segmen ie ung isuelle Da en. Ha i Deu sch: F ank u am Main, Ge many, 1995. Senso s 2009,99378 14. Eckes, C.; Vo b ¨ uggen, J.C. Combining da a-d i en and model-based cues o segmen a ion o ideo sequences. In P oceedings o WCNN Wo ld Con e ence on Neu al Ne wo ks, San Diego, CA, USA, 1996. 15. Bla , M.; Wiseman, S.; Domany, E. Supe pa ame ic clus e ing o da a. Phys. Re . Le . 1996, 76, 3251–3254. 16. Opa a, R.; W¨ o g¨ o e , F. A as and obus clus e upda e algo i hm o image segmen a ion in spin-la ice models wi hou annealing – isual la encies e isi ed. Neu al Compu . 1998, 10, 1547–1566. 17. on Fe be , C.; W¨ o g¨ o e , F. Clus e upda e algo i hm and ecogni ion. Phys. Re . E 2000, 62, 1461–1664. 18. Ising, E. Bei ag zu Theo ie des Fe omagne ismus. Zei sch i ¨ u Physik 1925,31, 253–258. 19. Boyko , Y.; Veksle , O.; Zabih, R. Fas App oxima e Ene gy Minimiza ion ia G aph Cu s. IEEE T ans. Pa . Anal. Mach. In . 1999,23, 2001. 20. Boyko , Y.; Kolmogo o , V. An expe imen al compa ison o min-cu /max- low algo i hms o ene gy minmiza ion in ision. IEEE T ans. PAMI 2004,26, 1124–1137. 21. Geman, D.; Geman, S. S ochas ic elaxa ion, Gibbs dis ibu ions, and he Bayesian es o a ion o images. IEEE T ans. Pa . Anal. Mach. In . 1984,6, 721–741. 22. Swendsen, R.; Wang, S. Nonuni e sal c i ical dynamics in Mon e Ca lo simula ions. Phys. Re . Le . 1987,76, 86–88. 23. Wol , U. Collec i e Mon e Ca lo upda ing o spin sys ems. Phys. Re . Le . 1989,62, 361–364. 24. Ba on, J.L.; Flee , D.J.; Beauchemin, S.; Bu ki , T. Pe o mance o op ical low echniques. In . J. Compu . Vis. 1994,12, 43–77. 25. Aksoy, E.E.; W¨ o g¨ o e , F.; Dellen, B. Recognizing objec -ac ion ela ions om seman ic scene g aphs. 2009, submi ed. 26. Saba ini, S.P.; Gas aldi, G.; Sola i, F.; Diaz, J.; Ros, E.; Pauwels, K.; Hulle, K.M.M.V.; Pugeaul , N.; K ¨ uge , N. Compac and Accu a e Ea ly Vision P ocessing in he Ha monic Space. In P oceedings o In e na ional Con e ence on Compu e Vision Theo y and Applica ions (VISAPP), Ba celona, Spain, 2007. 27. Me opolis, N.; Rosenblu h, A.W.; Rosenblu h, A.H.T.; Telle , E. Equa ions o s a e calcula ions by as compu ing machines. J. Chem. Phys. 1953,21, 1087–1091. 28. Felzenszwalb, P.; Hu enloche , D.P. E icien Belie P opaga ion o Ea ly Vision. In P oceedings o IEEE Compu e Socie y Con e ence on Compu e Vision and Pa e n Recogni ion, Washing on, DC, USA, 2004; pp. 261–268. 29. Ba bu, A.; Zhu, S.C. Gene alizing Swendson-Wang o sampling a bi a y pos e io p obabili ies. T ans. Pa e n Anal. Mach. In ell. 2005,27, 1239–1253. 30. Wol , U. Collec i e mon e ca lo upda ing o spin sys ems. Phys. Re . Le . 1989,62, 361–364. 31. Coope , C.; F ieze, A.M. Mixing p ope ies o he Swendsen-Wang p ocess on ce ain classes o g aphs. Random S uc . Algo i hm. 1999,15, 241–261. 32. Go e, V.K.; Je um, M.R. The Swendsen-Wang P ocess Does No Always Mix Rapidly. ACM: El Paso, TX, USA, 1996; pp. 674–681. Senso s 2009,99379 33. Onsage , L. C ys al S a is ic. I. Two-dimensional model wi h an o de -diso de ansi ion. Phys. Re . 1944,64, 117–149. 34. Lucas, B.D.; Kanade, T. An i e a i e image egis a ion echnique wi h an applica ion o s e eo ision. In P oceedings o DARPA IU Wo kshop, Pi sbu gh, PA, USA, 1981, pp. 121–130. 35. Dellen, B.; W¨ o g¨ o e , F. A local algo i hm o he compu a ion o op ic low ia cons uc i e in e e ence o global Fou ie componen s. In P oceedings o B i ish Machine Vision Con e ence, Leeds, UK, 2008, Vol. 2, pp. 795–804. 36. Wa d, J.H. Hie a chical g ouping o op imize an objec i e unc ion. J. Am. S a is ical Assoc. 1963, 58, 236–244. 37. F ¨ an i, P.; Vi majoki, O.; Hau ama¨ aki, V. Fas agglome a i e clus e ing using a k-nea es neighbo g aph. IEEE T ans. Pa e n Anal. Mach. In ell. 2006,28, 1875–1881. 38. Toshe , A.; Shi, J.; Daniilidis, K. Image ma ching ia saliency egion co espondences. In P oceedings o IEEE Con e ence on Compu e Vision and Pa e n Recogni ion, Minneapolis, MN, USA, 2007. 39. Ro he , C.; Minka, T.; Blake, A.; Kolmogo o , V. Cosegmen a ion o Image Pai s by His og am Ma ching - Inco po a ing a Global Cons ain in o MRFs. In P oceedings o IEEE Con e ence on Compu e Vision and Pa e n Recogni ion, Washing on, DC, USA, 2006; pp. 993–1000. 40. Kalman, R.E. A new app oach o linea il e ing and p edic ion p oblems. J. Basic Eng. 1960, 82, 35–45. 41. Mazo , E.; A e buch, A.; Ba -Shalom, Y.; Dayan, J. In e ac ing mul iple model me hods in a ge acking: A su ey. IEEE T ans. Ae osp. Elec on. Sys . 1998,34, 103–123. 42. A ulampalam, M.; Maskell, S.; Go don, N.; Clapp, T. A u o ial on pa icle il e s o online nonlinea /nonGaussian bayesian acking. IEEE T ans. Signal P ocess. 2002,50, 174–188. 43. Kuma , P.; Rangana h, S.; Sengup a, K.; Weimin, H. Coope a i e Mul i a ge T acking Wi h E icien Spli and Me ge Handling. IEEE T ans. Ci c. Sys . Video. T. 2006,16, 1477–1490. 44. G eig, D.; Po eous, B.; Seheul , A. Exac maximum a pos e io i es ima ion o bina y images. Appl. S a .-J. Roy. S a . Soc. B 1989,31, 271–279. 45. Fe a i, P.; F igessi, A.; de Sa, P. Fas app oxima e maximum a pos e io i es o a ion o mul icolou images. Appl. S a .-J. Roy. S a . Soc. B 1995,57, 485–500. 46. Roy, S.; Cox, I. A maximum- low o mula ion o he n-came a s e eo co espondence p oblem. In P oceedings o In e na ional Con e ence on Compu e Vision, Bombay, India, 1998. 47. Ishikawa, H.; Geige , D. Occlusions, discon inui ies, and epipoloa lines in s e eo. In P oceedings o Eu opean Con e ence on Compu e Vision, F eibu g, Ge many, 1998; pp. 232–248. 48. He, L.; Chao, Y.; Suzuki, K.; Wu, K. Fas connec ed-componen labeling. Pa e n Recogni . 2009, 42, 1977–1987. c °2009 by he au ho s; licensee Molecula Di e si y P ese a ion In e na ional, Basel, Swi ze land. This a icle is an open-access a icle dis ibu ed unde he e ms and condi ions o he C ea i e Commons A ibu ion license h p://c ea i ecommons.o g/licenses/by/3.0/.