Senso s 2009,9, 9355-9379; doi:10.3390/s91109355
OPEN ACCESS
senso s
ISSN 1424-8220
www.mdpi.com/jou nal/senso s
A icle
Segmen T acking ia a Spa io empo al Linking P ocess
including Feedback S abiliza ion in an n-D La ice Model
Babe e Dellen 1,2,?, E en E dal Aksoy 3and Flo en in W¨
o g¨
o e 3
1Be ns ein Cen e o Compu a ional Neu oscience G¨
o ingen, Max-Planck Ins i u e o Dynamics
and Sel -O ganiza ion, Bunsens asse 10, 37073 G¨
o ingen, Ge many
2Ins i u de Rob`
o ica i In o m`
a ica Indus ial (CSIC-UPC), Llo ens i A igas 4-6, 08028 Ba celona,
Spain
3Be ns ein Cen e o Compu a ional Neu oscience G¨
o ingen, Depa men o Compu a ional
Neu oscience, III. Physikalisches Ins i u , Geo g-Augus Uni e si y G¨
o ingen - Biophysik,
F ied ich-Hund Pla z 1, 37077 G¨
o ingen, Ge many; E-Mails: [email p o ec ed] (E.A.);
wo [email p o ec ed] (F.W.)
?Au ho o whom co espondence should be add essed; E-Mail: [email p o ec ed];
Tel.: +49-551-391-0765; Fax: +49-551-397-720.
Recei ed: 7 Augus 2009; in e ised o m: 5 No embe 2009 / Accep ed: 9 No embe 2009 /
Published: 20 No embe 2009
Abs ac : Model- ee acking is impo an o sol ing asks such as mo ing-objec acking
and ac ion ecogni ion in cases whe e no p io objec knowledge is a ailable. Fo his
pu pose, we ex end he concep o spa ially synch onous dynamics in spin-la ice models o
he spa io empo al domain o ack segmen s wi hin an image sequence. The me hod is ela ed
o synch oniza ion p ocesses in neu al ne wo ks and based on supe pa amagne ic clus e ing
o da a. Spin in e ac ions esul in he o ma ion o clus e s o co ela ed spins, p o iding an
au oma ic labeling o co esponding image egions. The algo i hm obeys de ailed balance.
This is an impo an p ope y as i allows o consis en spin- ans e ac oss subsequen
ames, which can be used o segmen acking. The e o e, in he acking p ocess he co ec
equilib ium will always be ound, which is an impo an ad ance as compa ed wi h o he mo e
heu is ic acking p ocedu es. In he case o long image sequences, i.e., mo ies, he algo i hm
is augmen ed wi h a eedback mechanism, u he s abilizing segmen acking.
Keywo ds: model- ee segmen acking; image mo ion; image segmen a ion
Senso s 2009,99356
1. In oduc ion
How can me make sense ou o a complex isual scene ha ing no o only li le p io knowledge abou
i s con en s and he objec s he ein? Such p oblems occu , o example, i we wish o lea n cause-e ec s
in an hi he o unknown en i onmen . Vice e sa, many objec de ini ions a e only meaning ul wi hin he
con ex o a gi en scena io and a se o possible ac ions.
Objec acking, i.e., he assignmen o consis en labels o objec s in di e en ames o a ideo, is
impo an o sol ing a ious asks in he ield o compu e ision, including au oma ic su eillance,
human-compu e in e ac ion and a ic moni o ing [1]. Mos objec acking algo i hms equi e ha
p ede ined objec s o in e es s a e de ec ed in he i s ame o in e e y ame o he mo ie. Howe e ,
in an unknown scena io, he acking o image segmen s, p esumably ep esen ing pa s o objec s,
allows o pos pone objec de ini ion o a la e s ep o he isual-scene analysis. Se e al app oaches o
segmen acking ha e been p oposed in he con ex o ideo segmen a ion [2–9]. Some app oaches ely
on segmen ing each ame independen ly, e.g., by classi ying pixels in o egions based on simila i y
in he ea u e space, ollowed by a segmen ma ching s ep based on hei low-le el ea u es [2–5].
O he me hods use mo ion p ojec ion o link segmen s, i.e., he posi ion o a segmen in a u u e
ame is es ima ed om i s cu en posi ion and mo ion ea u es [6–9]. C eme s (2003) acked mo ion
segmen s by join ly sol ing mo ion segmen a ion and mo ion es ima ion by minimizing a single ene gy
unc ional [10].
In he wo ks desc ibed abo e, assump ions o he na u e o objec s being acked o he da a i sel is
being made ei he in he acking p ocedu e i sel (ma ching assump ions) o al eady in he segmen a ion
s ep, o example by assuming a p io i a da a model o some kind. In ou wo k, we aim o educe a p io i
assump ions on he da a by choosing a da a-d i en me hod, i.e., supe pa amagne ic clus e ing o da a,
o he segmen a ion p ocedu e.
Supe pa amagne ic clus e ing inds he equilib ium s a es o a e omagne ic Po s model de ined by
an ene gy unc ion in he supe pa amagne ic phase [11–17]. The Po s model [11] is a gene aliza ion o
he Ising model [18] which desc ibes a sys em o in e ac ing g anula e omagne s o spins, ha can be
in qdi e en s a es, cha ac e izing he poin ing di ec ion o he espec i e spin ec o s. Depending on
he empe a u e, i.e., diso de in oduced o he sys em, he spin sys em can be in he pa amagne ic, he
supe pa amagne ic, o he e omagne ic phase. In he e omagne ic phase, all spins a e aligned, while
in he pa amagne ic phase he sys em is in a s a e o comple e diso de . In he supe pa amagne ic phase
egions o aligned spins coexis s. Bla e al. (1998) applied he Po s model o he image segmen a ion
p oblems in a way ha in he supe pa amagne ic phase egions o aligned spins co espond o a na u al
pa i ion o he image da a [15]. Finding he image pa i ion co esponds o he compu a ion o he
equilib ium s a es o he Po s model. Impo an ly, he ene gy unc ion used he e only consis s o
in e ac ion e ms and does no con ain a da a e m, hence no p io da a model needs o be assumed, e.g.
o al numbe o segmen s o a p io i de ined pixel-label assignmen s. In his sense, he me hod is model
ee. In con as , me hods which ind solu ions by compu ing he minimum o an ene gy unc ion usually
equi e a da a e m—o he wise only i ial solu ions a e ob ained. Hence, he equilib ium-s a e app oach
o he image segmen a ion p oblem has o be conside ed as undamen ally di e en om app oaches
inding he minimum ene gy con igu a ion o ene gy unc ions in Ma ko andom ields [19,20]. As a
Senso s 2009,99357
consequence, he me hod is non-pa ame ic and hus mo e gene ic, only equi ing a pixel-pixel simila i y
c i e ium, e.g., g ay alue, colo o ex u e, o be de ined wi hou making any u he p io assump ions
abou he da a.
The equilib ium s a es o he Po s model ha e been app oxima ed in he pas using he
Me opolis-Has ings algo i hm wi h annealing [21] and he me hods on clus e upda ing, which a e
known o accele a e he equilib a ion o he sys em by sho ening he co ela ion imes be ween dis an
spins. P ominen algo i hms a e Swendsen-Wang [22], Wol [23], and ene gy-based clus e upda ing
(ECU) [16,17]. In addi ion, he ECU algo i hm has been shown o p o ide s able segmen a ion esul s
o e a la ge empe a u e ange (which is he main con ol pa ame e ) han Swendsen-Wang [16,17].
All o hese me hods obey de ailed balance, ensu ing con e gence o he sys em o he equilib ium s a e.
Howe e , con e gence has only been shown o be polynomial o special cases o he Po s model. The
Po s model is known o be NP ha d in he gene al case. The elaxa ion p ocesses which can be emula ed
in spin-la ice models a e o en being used as app oxima ions o synch oniza ion p ocesses wi hin neu al
assemblies [12–17].
In ou me hod, acking o segmen s is accomplished h ough simul aneous segmen a ion o adjacen
ames which a e linked using local co espondence in o ma ion, e.g., compu ed ia s anda d algo i hms
o op ic low [24]. By pe o ming mo ion p ojec ion and image segmen a ion wi hin a single p ocess we
do no equi e any explici ma ching o image segmen s, hus u he educing a p io i assump ions ha
ha e o be made abou he da a. Synch onizing he segmen a ion p ocess o adjacen ames has u he
he ad an age ha pa i ioning ins abili ies can be educed. Usually, he segmen a ion (o pa i ion) o
an image is sensi i e o global and local changes o he image, i.e., small changes in illumina ion, he
appea ance/disappea ance o objec s pa s, causing he pa i ion o change om one ame o he nex .
To u he s abilize segmen acking in he case o long image sequences, we de eloped a eedback
con ol mechanism, which allows segmen a ion ins abili ies, e.g., sudden disappea ances o segmen s, o
be de ec ed and emo ed by adjus ing a con ol pa ame e o he segmen a ion algo i hm.
Thus, he main con ibu ion he e is he de elopmen o a da a-d i en, model- ee acking algo i hm,
i.e., no p io objec knowledge is equi ed in o de o ack image segmen s om ame o ame.
Po en ial applica ions include mo ing objec de ec ion and acking, and ac ion ecogni ion and
classi ica ion om mo ies [25].
The pape is s uc u ed as ollows: In Sec ion 2, we ex end he me hod o supe pa amagne ic
clus e ing in spin models o he empo al dimension and in oduce he con olle algo i hm. We also
discuss he me hod in on o he backg ound o ene gy minimiza ion me hods in Ma ko andom
ields. In Sec ion 3, we i s e i y he co e algo i hm using sho image sequences because hese a e
mo e sui able o in oduce and es he me hod. We u he in es iga e he sensi i i y o he algo i hm
o sys em pa ame e s and noise. Then, we demons a e ha segmen acking can be achie ed o eal
mo ies. The pe o mance o he me hod is quan i ied in e ms o pa i ioning consis ency along an
a i icial image sequence. In Sec ion 4, he esul s a e discussed.
2. Algo i hmic F amewo k
Segmen acking can be oughly di ided in o he ollowing sub asks: (i) image segmen a ion, (ii)
linking ( acking), and (iii) s abiliza ion ( acking). The hi d poin acknowledges ha segmen s, unlike
Senso s 2009,99358
objec s, a e no pe se s able en i ies, bu a e sensi i e o changes in he isual scene. Sub asks (i–ii)
will be sol ed using a conjoin spin- elaxa ion p ocess emula ed in an n-dimensional (n-D) la ice,
which de ines he co e algo i hm (Sec ion 2.1). Local co espondence in o ma ion o linking is
ob ained using s anda d algo i hms o ei he s e eo o op ic low [24,26]. The conjoin segmen a ion
app oach has he ad an age ha he spin- elaxa ion p ocesses o adjacen images synch onize, educing
pa i ioning ins abili ies.
Since simul aneous segmen a ion o long image sequences is p ac ically impossible due o he high
compu a ional cos s, we usually spli he image sequence in o a sequence o pai s. Fo example, he
subsequen ames 0, 1and 2a e spli in o wo pai s { 0, 1}and { 1, 2}, whe e he las ame o
p e ious pai is iden ical o he i s ame o he nex pai . I a segmen o he las ame o { 0, 1}
and a segmen o he i s ame o { 1, 2}occupy he same image egion, we can assign he same
segmen label o bo h segmen s. This way segmen s can be acked h ough he en i e sequence. Since
he algo i hm p ese es de ailed balance (Sec ion 2.1), spins can be ans e ed om one ame o he
nex , g ea ly educing he numbe o i e a ions equi ed o achie e a s able segmen a ion.
We u he s abilize segmen acking by in oducing a eedback con olle (Sec ion 2.2). In
long image sequences, pa i ioning ins abili ies a e likely o a ise a some poin du ing he acking
p ocess. Thus, segmen s may be los due o me ging o spli ing o segmen s. The eedback con olle
de ec s hese kind o ins abili ies and adjus s a con ol pa ame e o he co e algo i hm o eco e he
o iginal segmen s.
2.1. Co e Algo i hm
The me hod o supe pa amagne ic clus e ing has been p e iously used o segmen single
images [15–17]. Applying his amewo k o image sequences equi es spin in e ac ions o ake place
ac oss ames. Due came a and objec mo ion he images unde go changes du ing he cou se o ime.
To connec di e en ames, he mapping om one ame o he nex needs o be known a leas in
some app oxima ion. We sol e his p oblem in he ollowing way: Poin co espondences, de i ed using
algo i hms o dispa i y o op ic- low compu a ion, can be inco po a ed in o he Po s model [11] by
allowing spins belonging o di e en ames o he image sequence o in e ac i he espec i e pixels
belong o locally co esponding image poin s. Then, spins belonging o he di e en ames o he
sequences a e elaxed simul aneously, esul ing in a synch onized segmen a ion o he images o he
sequence. The in e - ame spin in e ac ions cause he spins o co esponding image egions o align,
and, hus, hey will be assigned o he same segmen . Since he o ma ion o segmen s is a collec i e
p ocess, he poin co espondences do no ha e o be e y accu a e no does he algo i hm equi e poin
co espondences o each pixel. I is usually su icien i he a ailable co espondences cap u e he
cha ac e is ics o he scene only oughly.
The aim o his wo k is o ind co esponding image egions in image sequences, i.e., s e eo pai s
and mo ion sequences. The segmen acking ask is o mula ed as ollows. Gi en an image sequence
S
con aining poin s p(x, y, )wi h coo dina es (x, y, )as elemen s, whe e xand ylabel he posi ion wi hin
each image, while labels he ame numbe , hen we wan o ind a pa i ioning P=
P
1, ..,
P
Mo
S
in
Mg oups such ha
Senso s 2009,99359
(i)
P
i∩
P
j= 0 and
P
i6= Ø o all g oups
(ii) i poin p∈
P
i, hen s(p,
P
i)> s(p,
P
j), whe e sis a unc ion measu ing he a e age dis ance o a
poin o he elemen s o a g oup and i6=j
(iii) i p(xi, yi, i)∈
P
, hen p(xi+4xi, yi+4yi, i+ 1) ∈
P
, whe e 4xiand 4yia e he shi s o
poin p(xi, yi, i)along he xand yaxes, espec i ely, om ame i o ame i+ 1.
To pe o m his ask, we assign a spin a iable σi(o label) o each pixel (o si e) io he image
sequence. To inco po a e cons ain s in he o m o local co espondence in o ma ion, we dis inguish
be ween neighbo s wi hin a single ame (2D bonds) and neighbo s ac oss ames (n-D bonds). We
c ea e a 2D bond (i, k)2Dbe ween wo pixels wi hin he same ame wi h coo dina es (xi, yi, i)and
(xk, yk, k)i
|(xi−xk)| ≤ ε2D(1)
|(yi−yk)| ≤ ε2D(2)
i= k(3)
whe e ε2Dis he 2D-in e ac ion ange o he spins, a pa ame e o he sys em. Ac oss ames, we c ea e
a n-D bond (i, j)nD be ween wo spins iand ji
|(xi+dx
ij −xj)| ≤ εnD (4)
|(yi+dy
ij −yj)| ≤ εnD (5)
i6= j(6)
aij > τ (7)
whe e εnD is he n-D in e ac ion ange. The alues dx
ij and dy
ij a e he shi s o he pixels be ween
ames iand jalong he axis xand y, espec i ely, ob ained om he op ic- low map o dispa i y
map. The pa ame e s aij ∈[0,1] a e he espec i e ampli udes (o con idences) o he op ic- low o
s e eo algo i hm. A la ge ampli ude sugges s a la ge con idence in he compu ed local co espondence.
Pa ame e τis a h eshold, emo ing all local co espondences ha ing a small ampli ude.
We de ine o e e y bond on he la ice he dis ance
4ij =|gi−gj|(8)
whe e giand gja e he ea u e ec o s, e.g., g ay alue, colo , o ex u e, o he pixels iand j,
espec i ely. The mean dis ance ¯
4is ob ained by a e aging o e all bonds. We u he de ine an
in e ac ion s eng h
Jij = 1 − 4ij /¯
4(9)

Senso s 2009,99360
The spin model is now implemen ed such a way ha neighbo ing spins wi h simila colo ha e he
endency o align. We use a q-s a e Po s model [11] wi h he Hamil onian
H=−X
hiki2D
Jikδ(σi−σk)−X
hijinD
Jijδ(σi−σj)(10)
He e, hiki2Dand hijinD deno e ha i, k and i, j a e connec ed by bonds (i, k)2Dand (i, j)nD,
espec i ely. The K onecke δ unc ion is de ined as δ(a) = 1 i a= 0 and ze o i o he wise. The
segmen a ion p oblem is hen sol ed by inding clus e s o co ela ed spins in he low empe a u e
equilib ium s a es o he Hamil onian H. The empe a u e pa ame e de e mines he amoun o diso de
in oduced o he sys em. The spin s a es ha e o be obse ed o e se e al i e a ions o iden i y clus e s
as g oups o co ela ed spins. The o al numbe Mo segmen s is hen de e mined by coun ing he
compu ed segmen s. The spin s a es σican ake in ege alues be ween 1and q, whe e qis a pa ame e
o he algo i hm. The numbe o segmen s is no cons ained by he pa ame e q. Please no e ha segmen
labels a e no iden ical o spin s a es. Spins which belong o he same segmen a e always in he same
spin s a e, howe e , he e e se is no necessa ily ue. No e ha he local co espondences used in he
algo i hm o c ea e n-D bonds a e p ecompu ed and a e no al e ed o op imized when compu ing he
equilib ium s a e. The compu a ion o local co espondences is no he aim o his pape .
We ind he equilib ium spin con igu a ion using a clus e ing algo i hm. In a i s s ep, “sa is ied”
bonds, i.e., bonds connec ing spins ha ing iden ical spin alues σi=σj, a e iden i ied. Then, in a
second s ep, he sa is ied bonds a e “ ozen” wi h a some p obabili y Pij. Pixels connec ed by ozen
bonds de ine a clus e , which a e upda ed by assigning o all spins inside he same clus e s he same
new alue [22]. In he me hod o supe pa amagne ic clus e ing p oposed by Bla e al. (1996) [15]
his is done independen ly o each clus e . In his pape , we will employ he me hod o ene gy-based
clus e upda ing (ECU), whe e new alues a e assigned in conside a ion o he ene gy gain calcula ed o
a neighbo hood o he ega ded clus e [16,17]. A schema ic o he spin sys em o an image sequence is
depic ed in Figu e 1A.
The ECU algo i hm compu ing he equilib ium o Hconsis s o he ollowing s eps:
1. Ini ializa ion: A spin alue σibe ween 1 and qis assigned andomly o each spin i. Each spin
ep esen s a pixel o he image sequence.
2. Compu ing bond eezing p obabili ies: I wo spins iand ja e connec ed by a bond and a e in he
same spin s a e σi=σj, hen he bond is ozen wi h a p obabili y
Pij = 1 −exp(−0.5Jij/T)(11)
Nega i e p obabili ies a e se o ze o.
3. Clus e iden i ica ion: Pixels which a e connec ed by ozen bonds de ine a clus e . A pixel
belonging o a clus e uhas by de ini ion no ozen bond o a pixel belonging o a di e en
clus e .
Senso s 2009,99361
4. Clus e upda ing: We pe o m a Me opolis upda e [22,27] ha upda es all spins o each clus e
simul aneously o a common new spin alue. The new spin alue o a clus e cis compu ed
conside ing he ene gy gain ob ained om a clus e upda e o a new spin alue wk, whe e he
index kdeno es he possible spin alues be ween 0and q, espec i ely. No e ha his new spin
alue is, once chosen, a cons an and has o be dis inguished om he spin a iables σi. Upda ing
he espec i e clus e o he new alue esul s in a new spin con igu a ion Wc
k.
Figu e 1. A The spin s a es (upwa d and downwa d poin ing a ows) o pixels i,k, and ja e
shown be o e and a e a spin upda e o wo adjacen ames and +1 o an image sequence.
The whi e and black ci cles indica e pixels o small and la ge g ay alues, espec i ely. Pixel
iin e ac s wi h pixels kand jin i s 2D and 3D neighbo hood (shaded a eas), espec i ely,
which a e in he same spin s a e. BPai wise (3D) segmen a ion o mo ies. A eedback
con olle de ec s segmen a ion ins abili ies and adjus s he con ol pa ame e To he co e
algo i hm (3D segmen a ion) o eco e los segmen s.
d
F ame F ame +1
Be o e
Upda e
A e
Upda e
ij
i
i
j
k
k
3D
2D
A
B
j
Con olle
3D Segmen a ion
F ame numbe
T
The p obabili y o he choosing he new spin alue wk o clus e c is compu ed by conside ing
he in e ac ions o all spins in he clus e cwi h hose ou side he clus e , assuming ha all spins
o he clus e a e upda ed o he new spin alue wk, gi ing an ene gy
E(Wc
k) = X
i∈c£−X
hiji2D
ck6=cj
ηJijδ(σi−σj)−X
hijinD
ck6=cj
ηaijJij δ(σi−σj)¤
(12)
Senso s 2009,99362
whe e hiki2D, ck6=cjand hijinD, ck6=cja e he nonclus e neighbo hoods o spin i. He e, Nis
he o al numbe o pixels o he image sequence. The cons an ηis chosen o be 0.5.
Simila o a Gibbs sample , he selec ing p obabili y P(Wc
k) o choosing he new spin alue o be
wkis gi en by
P(Wc
k) = exp(E(Wc
k))/
q
X
l=1
exp(E(Wc
l)) (13)
5. I e a ion: The new spin s a es a e e u ned o s ep 2 o he algo i hm, and s eps 2–5 a e epea ed,
un il he o al numbe o clus e s s abilizes, i.e., he change o he numbe o clus e s is small
compa ed o he change o he numbe clus e s du ing he i s en i e a ions (see also Figu e 5F).
In p ac ice, we usually use a ixed numbe o i e a ions.
6. Segmen s a e de ined as g oups o co ela ed spins and can be ex ac ed using a h esholding
p ocedu e. All pai s o pixels connec ed by a bond (i, j)wi h c(σi, σj)> θ a e conside ed o be
iends, i.e., hey mus belong o he same segmen . The unc ion ccompu es he co ela ion o
he spin s a es o iand jo e se e al i e a ions. Then, all mu ual iends a e assigned o he same
segmen . Finally, Mis de e mined by coun ing he o al numbe o segmen s. In p ac ice, we ind
i su icien o ake he clus e s ound in he las i e a ion as segmen s.
In an ea lie s udy we had p o ided e idence ha his algo i hm obeys de ailed balance. The ull
p oo shall no be epea ed he e and can be ound in [16]. De ailed balance assu es ha he p oposed
algo i hm compu es an equilib ium spin con igu a ion o he ene gy unc ion and ha his is Bol zmann
dis ibu ed. Howe e , he equilib ium migh no be ound in polynomial ime (see Sec ion 2.3).
The consequence o de ailed balance is ha spin s a es can be ans e ed ac oss image pai s, whe e
spins a e being calcula ed o one pai ( he i s pai ) and hen pixels in he nex wo ames ( he second
pai ) a e jus assigned hese spins om whe e on a new elaxa ion p ocess s a s (see Figu e 5 o an
example). Hence, he elaxa ion p ocess o he second pai (and all o ollow) is much as e when using
spin ans e and he sys em will always a i e a he he modynamic equilib ium making spin- ans e
based segmen a ion conco dan ac oss ames. No e, his p ope y allows consis en segmen acking
ac oss many ames wi hou addi ional assump ions (see Figu e 5), which equi es mo e e o wi h
o he me hods.
The ollowing should be no ed. In his me hod, bonds be ween adjacen ames a e c ea ed om
he p ecompu ed op ic- low o dispa i y maps and ozen wi h a p obabili y ha depends on he
ea u e simila i y o he espec i e co esponding pixels. Whe he hese bonds a e ozen in he inal
con igu a ion (i.e., he espec i e p oposed poin co espondences a e accep ed), and hus p omo e he
o ma ion o a ce ain segmen co espondence, is decided inhe en ly by inding he equilib ium s a e
o Equa ion 10. This p ocedu e makes he me hod mo e obus agains e o s in he op ic- low o
dispa i y map.
2.2. Feedback Con ol
Segmen a ion ins abili ies a ising du ing he acking p ocess can be pa ly emo ed by adjus ing he
empe a u e pa ame e o he co e algo i hm. The empe a u e choice a ec s he o ma ion o segmen s,
Senso s 2009,99363
hence, a segmen which has been los in a p e ious ame can some imes be eco e ed by inc easing he
empe a u e o a ce ain pe iod.
The eedback con olle acks he size o he segmen s and eac s i he size o a segmen changes
suddenly. The i s con olle unc ion
Pj
C( ) = 


1i 4Sj( )< τ1,
exp [−4Sj( )/α]/β o he wise,(14)
measu es he p obabili y o change o segmen j, whe e Sj( )is he size o segmen ja ame and
4Sj( ) = Sj( )−Sj( −1), and α,β,τ1a e cons an pa ame e s. The his o y o segmen jin e ms o
occu ence is cap u ed by he second con olle unc ion
Pj
H( ) = 0.4Hj( −1) + 0.3Hj( −2) + 0.2Hj( −3) + 0.1Hj( −4) (15)
wi h Hj( ) = 1 i Sj( )>0and ze o o he wise.
Segmen a ion ins abili ies may cause a segmen o be los , o example h ough segmen me ging o
spli ing. We de ine wo h eshold pa ame e s τ2and τ3. An unexpec ed segmen loss is de ec ed by he
con olle i he condi ions
Sj= 0 (16)
PC< τ2(17)
and PH> τ3(18)
a e ul illed. An unexpec ed segmen appea ance is de ec ed by he con olle i he condi ions
PC< τ2(19)
and PH< τ3(20)
a e ul illed. The iden i ies o he a ec ed segmen s a e s o ed by he con olle . The empe a u e o
he co e algo i hm is a ied using p ede ined empe a u e s eps 4T. The segmen a ion is epea ed a
he new empe a u e T+4Tonly o he a ec ed segmen s o he ames. I he los segmen s can be
eco e ed a one o hese empe a u es, he a ec ed segmen s a e elabeled acco dingly. Fu he mo e,
he new empe a u e T+4T alues o he a ec ed segmen s a e inhe i ed o he nex ame in o de o
p e en he same segmen a ion e o . No e ha he empe a u e alue can be inc eased o a p ede ined
maximum alue, o he wise undesi ed a i ac s migh be obse ed.
A schema ic o he en i e sys em, i.e., co e algo i hm wi h eedback con ol, is p esen ed in Figu e 1B.
2.3. Compa ison wi h Ene gy-Minimiza ion-Based Me hods in Ma ko Random Fields
The me hod desc ibed in his pape is compa ed wi h ene gy minimiza ion in Ma ko andom ields.
Simila i ies and di e ences o he app oaches a e analyzed and summa ized in his sec ion. Pa icula
a en ion is gi en o combina o ial g aph cu s me hods, which ha e p o ided powe ul compu e ision
algo i hms o s e eo, mo ion, image es o a ion, and image segmen a ion in ecen yea s [19,20].
Senso s 2009,99370
Figu e 5. A Th ee ames o a walking sequence, labeled 0, 1, and 2, espec i ely. B
Op ic- low ec o s coding he mapping om ame 0 o 1, and 1 o 2.CSpin s a es a e
100 i e a ions. DSpin s a es o a sequence consis ing o ames 0and 1.ESpin s a es o
a sequence consis ing o 1and 2.FThe numbe o clus e s as a unc ion o he i e a ion
numbe o he i s sequence con aining ame 0and 1(dashed line) and o he second
sequence con aining ame 1and 2(solid line). GEnla ged plo o he second sequence.
0
1
2
A
0 20 40 60 80
0.5
1
1.5
2
2.5
3x104
I e a ion
0 20 40 60 80
0.49
0.5
0.51
0.52
x10
Numbe o clus e s
B
C
D
E
FG
0
1
4
Numbe o clus e s
I e a ion
When analyzing long mo ion sequences, i is ine icien o apply he algo i hm o all ames a once
because he compu a ional cos s inc ease wi h he numbe o pixels. Hence, we spli he sequences in

Senso s 2009,99371
pai s o wo ames a a ime, whe e he las ame o he p e ious sequence is iden ical wi h he i s
ame o he nex sequence. Then, we ini ialize he spin s a es o each sequence wi h he inal spin
s a es o he p e ious sequence. The spin s a es o he i s sequence con aining ame 0and 1a e
100 i e a ions a e shown in Figu e 5D. Then, he algo i hm is applied o he nex pai , con aining ame
1and 2, whe e he spin s a es o bo h ame ha e been ini ialized o he inal spin s a es o ame 1
o he p e ious sequence. The spin s a es a e 13 i e a ions a e shown in Figu e 5E, demons a ing
ha he numbe o i e a ions equi ed o achie e a sa is ying segmen a ion esul is g ea ly educed by
his echnique. The numbe o clus e s o he i s sequence and he second sequence a e displayed
as a unc ion o he i e a ion numbe in Figu e 5F, dashed and solid line, espec i ely. The numbe
o clus e s o he second sequence is plo ed as a unc ion o he i e a ion numbe a a di e en scale
(Figu e 5G). Ini ially, he numbe o clus e s dec eases sligh ly and hen app oaches a s able s a e. In
a mo ion sequence, he numbe o clus e s is expec ed no o change much om one ame o he nex .
Mainly he bounda ies o he clus e s eo ganize du ing he i s i e a ions.
The segmen s o adjacen image pai s a e connec ed as ollows. Two segmen s belonging o he
segmen a ion o ame 1o pai { 0, 1}and ame 1o pai { 1, 2}, espec i ely, a e assigned he same
label i hey occupy he same egion in image ame 1. Fo his pu pose, we compu e he pe cen age
o he clus e a eas which o e lap in he image space. I he o e lap is la ge han a ixed h eshold, he
clus e s a e assigned he same label. This way we can ack he segmen h ough he whole sequence.
3.2. Segmen T acking wi h Feedback S abiliza ion
We add eedback con ol (see Sec ion 2.2) wi h pa ame e s α= 200 pixels, β= 0.8,τ1= 50 pixels,
τ2= 0.9, and τ3= 0.6 o he co e algo i hm wi h empe a u e T= 0.05 and apply he algo i hm
o long image sequences. The i s mo ie shows a hand aking a ed apple om a pla e wi h se e al
ui s. A ew ames o he mo ie a e depic ed in he uppe panel o Figu e 6A. I he co e algo i hm is
applied a cons an empe a u e wi hou eedback con ol, he ed segmen and he ligh pink segmen ,
ep esen ing he espec i e pa s o he ed apple and he o ange, a e los a ame numbe 45 due a
segmen a ion ins abili y: The ed segmen and he ligh pink segmen me ge and o m a new segmen ,
colo ed in ligh blue (see Figu e 6A, middle panel). I eedback con ol is included, his segmen a ion
ins abili y is de ec ed and he o iginal segmen s can be eco e ed by inc easing he empe a u e in s eps
o 4T= 0.15. As a consequence, he segmen s can be con inuously acked, as shown in Figu e 6A
(lowe panel). The segmen s ep esen ing he cup could be eco e ed using he same mechanism.
The wo k o he eedback con olle is u he illus a ed in Figu e 6B, whe e he segmen size is
plo ed as a unc ion o he ame numbe o he segmen s ep esen ing he ed apple and he o ange
wi hou and wi h eedback con ol, depic ed as ed, blue, b own and g een lines, espec i ely. A ame
numbe 45 he segmen sizes o he ed apple and he o ange d op unexpec edly o ze o ( ed and blue
lines), hus indica ing a segmen a ion ins abili y (see Sec ion 2.2). As a consequence, he eedback
con olle is ac i a ed and he empe a u e o he co e algo i hm is inc eased un il he o iginal segmen s
a e eco e ed (b own and g een lines). The esul s o he whole mo ie a e shown in Figu e 7A.
We u he applied he algo i hm o ano he mo ie, showing he illing o a cup wi h suga (Figu e 7B).
The mo ie is challenging because i con ains ligh e lexions and changing shadows. Howe e , he
algo i hm is capable o acking he main segmen s o he mo ie, i.e., he wo cups and he hand.
Senso s 2009,99372
Figu e 6. Feedback con ol o segmen a ion s abiliza ion. AA ew ames o a mo ie
showing a hand aking a ed apple om a pla e a e shown oge he wi h he esul s o he co e
algo i hm wi hou and wi h eedback con ol (uppe , middle, and lowe panel, espec i ely).
BThe segmen size is plo ed as a unc ion o he ame numbe o he segmen s ep esen ing
he ed apple and he o ange wi hou and wi h eedback con ol, depic ed as ed, blue, b own
and g een lines, espec i ely. A ame numbe 45 he segmen sizes o he ed apple and he
o ange d op unexpec edly o ze o ( ed and blue lines), and he eedback con ol is ac i a ed,
inc easing he empe a u e Tun il he o iginal segmen s a e eco e ed (b own and g een
lines).
0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88
0
100
200
300
400
500
600
700
800
F ame Numbe
Segmen size [numbe o pixels]
Feedback con ol ac i a ed
Red apple (no eedback)
Red apple (wi h eedback)
O ange (no eedback)
O ange (wi h eedback)
A
B
We ob ained simila segmen - acking esul s o o he eal mo ies, e.g., Mo ing Objec , Making
Sandwich, Opening a Book. Resul s can be ound a h p://www.nld.ds.mpg.de/∼e en/Mo ies.h ml.
Senso s 2009,99373
Figu e 7. Segmen acking o eal mo ies. AThe algo i hm (wi h eedback con ol)
is applied o a mo ie showing a hand aking an apple om a pla e (uppe panel). The
co esponding segmen - acking esul s a e depic ed below. BThe esul s o he algo i hm
(wi h eedback con ol) o a mo ie showing he illing o a cup a e shown. Segmen - acking
esul s o o he eal mo ies can be ound a h p://www.nld.ds.mpg.de/∼e en/Mo ies.h ml/.
A
B
We use an a i icial image sequence o demons a e ha bo h 3D linking and eedback con ol imp o e
consis ency o he pa i ioning in o segmen s o adjacen ames. The o iginal image consis s o 4×4
uni o mly- alued squa es. By adding Gaussian noise o he image, we c ea e an image sequence o 40
ames (see Figu e 8A). We apply he algo i hm a T= 0.1(i) wi hou 3D linking and wi hou eedback,
(ii) wi h 3D linking and wi hou eedback, and (iii) wi h 3D linking and wi h eedback. Example
segmen a ion esul s a e p esen ed in Figu e 8B. Fo case (i), he pa i ioning is o en changing om
ame o ame. The labels o adjacen ames only co espond o each o he by chance, since no linking
in 3D is employed (independen segmen a ions). Fo case (ii), he pa i ioning is mo e s able, bu b eaks
Senso s 2009,99374
s ill occu equen ly. These b eaks can be la gely p e en ed by eedback con ol. We quan i y he
s abili y o adjacen image pa i ions by compu ing he spa ial g adien o each segmen ed ame. We
ind a pa i ion bounda y i he g adien is la ge han ze o. The pa i ion-bounda y images o adjacen
ames a e hen sub ac ed, he absolu e alue is aken, and he sum is compu ed o e all pixels, which we
call he e he pa i ion e o . In Figu e 8C, he his og ams o he pa i ion e o o e he whole sequence
a e shown o all h ee cases (i-iii) wi h mean pa i ion e o s (a e aged o e all ames) o 35.72,19.44,
and 2.59, espec i ely.
Figu e 8. Imp o ing pa i ioning consis ency using 3D linking and eedback con ol. A
F om he o iginal image consis ing o 4×4uni o mly- alued squa es, a s a ic image
sequence is c ea ed by adding Gaussian noise o each ame. BSegmen a ion esul s o
ou algo i hm wi hou 3D linking (no acking), wi h 3D linking, and wi h bo h 3D linking
and eedback con ol. CRespec i e his og ams showing he numbe o ames as a unc ion
o he pa i ion e o s (see ex ).
0 20 40 60 80
0
5
10
15
20
25
30
35
0 20 40 60 80
0
5
10
15
20
25
30
35
0 20 40 60 80
0
5
10
15
20
25
30
35
A
B
C
2D 3D 3D +
eedback
18
19
20
21
22
15
16
17
F ame
3D +
eedback
3D
2D
Pa i ion e o [pixels]
Numbe Numbe Numbe
Senso s 2009,99375
4. Discussion
We p esen ed an algo i hm o model- ee segmen acking based on a no el, conjoin amewo k,
combining local co espondences and image segmen a ion o synch onize he segmen a ion o adjacen
images. The algo i hm p o ides a pa i ioning o he image sequence in segmen s, such ha poin s in a
segmen a e mo e simila o each o he han o poin s in ano he segmen , and such ha co esponding
image poin s belong o he same segmen . We es ed he me hod on a ious syn he ic and eal image
sequences, and showed s able and eliable esul s o e all, hus ul illing he mos impo an equi emen
o segmen a ion algo i hms. The me hod leads o he o ma ion o s able egion co espondences
despi e la gely incomple e dispa i y o op ic- low maps. Simila algo i hms o he ex ac ion o
egion co espondences could po en ially be cons uc ed using o he image segmen a ion algo i hms,
i.e., me hods based on agglome a i e clus e ing [36,37]. We decided o use physics-based model
o i s concep ual simplici y which allowed us o in eg a e local co espondence in o ma ion in a
s aigh o wa d way. I u he has he ad an age ha he in e ac ing pa s a e inhe en ly con e ging
o he equilib ium s a e and hus a e no being apped in local ex ema (de ailed balance). As a
consequence, he esul is independen o he ini ial condi ions, allowing us o apply he algo i hm
o long image sequences ia spin-s a es ans e . This allows o consis en segmen acking ac oss
many ames wi hou addi ional assump ions, which is mos o he ime no immedia ely possible wi h
o he me hods. In addi ion, no assump ions abou he unde lying da a a e equi ed, e.g. he numbe
o segmen s, leading o a model- ee segmen a ion. This has he consequence ha a single pixel o
dis inc g ay alue (compa ed o i s neighbo s) migh de ine a single segmen . In algo i hms, which
pa i ion he image in o a ixed and usually small numbe o segmen s, his phenomenon does no occu .
This, howe e , is a p oblem as in all ealis ic si ua ions one ne e knows how many segmen s exis and
sel -adjus men o he o al numbe o segmen s is, hus, usually desi ed as compa ed o a p e-de ined
maximal numbe .
We u he in oduced a eedback con olle which allows o de ec segmen a ion ins abili ies, i.e.,
me ging and spli ing o segmen s. The eedback con olle adjus s he con ol pa ame e o he co e
algo i hm in o de o eco e he o iginal segmen s. This allows keeping ack o segmen e en in
long mo ies.
Segmen acking has been pe o med p e iously in he con ex o ideo segmen a ion [2–9].
Ou me hod di e s om hese app oaches in he choice o he segmen a ion algo i hm, he way
linking is achie ed, and he addi ion o a eedback con olle which de ec s segmen a ion ins abili ies.
Supe pa amagne ic clus e ing allows a model- ee unsupe ised segmen a ion o he image sequences,
including a sel -adjus men o he o al numbe o segmen s. Linking is in oduced ough local
co espondence in o ma ion which synch onizes he spin- elaxa ion p ocess o adjacen images. This
app oach has he ad an age ha he pa i ions o adjacen images a e less p une o pa i ioning
ins abili ies (see also Figu e 8). Fu he , ou me hod does no equi e co esponding egions o ul ill
any segmen simila i y c i e ium. Finally, eedback con ol allows segmen a ion ins abili ies occu ing
in long sequences o be emo ed by assuming ha “good” segmen s change hei size in a con inuous
“p edic able” manne .

Senso s 2009,99376
The e ha e been a ew o he app oaches combining image segmen a ion wi h co espondence
in o ma ion. The wo k by Toshe e al. [38] uses a join -image g aph con aining edges ep esen ing
in a-image simila i ies and in e -image ea u e ma ches o compu e ma ching egions. Join
segmen a ion has also been employed by Ro he e al. [39] using his og am ma ching.
The con olle employed in his model se es he de ec ion and emo al o segmen a ion ins abili ies.
No assump ions abou he objec s gi ing ise o he measu emen s, i.e., segmen s, a e made he e, excep
om ha hey canno appea and disappea all o a sudden (objec cons ancy) and a e hus in some way
p edic able. P edic able beha io o objec s p o ides he basis o acking me hods using op imal il e s
such as he Kalman il e [40], in e ac ing mul iple models [41], o pa icle il e s [42]. In hese cases
howe e s onge assump ions abou he na u e o he objec s a e being made. Bo h some knowledge o
he dynamics o objec s (as hen e lec ed by he measu emen s) and hei appea ance is assumed [43]. In
he con ex o speci ic applica ions as o example he acking o mo ing objec s, Kalman- il e -based
mo ion could po en ially be combined wi h ou me hod. Kuma e al. (2006) p oposed a me hod o
mul i a ge acking in which blobs, i.e., connec ed egions ob ained om segmen ing mo ing o eg ound
objec s ia a backg ound subs ac ion me hod, a e acked wi h a Kalman il e while handling spli s and
me ges be ween blobs. Ce ain elemen s o his app oach may p o ide a means o u he ad ancing he
segmen - acking p ocedu e desc ibed in his pape .
The me hod desc ibed he e is ela ed o ene gy minimiza ion in Ma ko andom ields which has
been used o sol e ision p oblems many imes be o e [19,29,44–47] (see also Sec ion 2.3). While
he algo i hmic p ocedu es used o ind he ene gy minimum sha e ea u es wi h he ones employed
by ou me hod, undamen al di e ences exis be ween he me hods. Supe pa amagne ic clus e ing
aims a inding he equilib ium s a es o a Po s model wi hou ex e nal ield o da a ene gy e m a a
ce ain empe a u e, and no a global ene gy minimum. Fo mula ing ision p oblems in e ms o ene gy
minimiza ion equi es a da a penal y e m (o ex e nal ield), necessi a ing some p io knowledge abou
he da a ha is being modeled. Hence, he solu ions ha e o be conside ed less gene ic and qui e di e en
o ou s by na u e.
The algo i hm has po en ial applica ions in model- ee mo ing objec de ec ion and acking by
me ging cohe en ly mo ing segmen s (Ges al law o common a e). The me hod is u he applicable o
ac ion- ecogni ion asks, whe e ce ain cha ac e is ic ac ion pa e ns a e in e ed om he spa io empo al
ela ionships o segmen s. Fi s esul s o his p oblem a e epo ed in [25]. In he u u e, ex u e cues
may be inco po a ed in o he algo i hm o allow acking o segmen s de ined by ex u e.
Cu en ly, he algo i hm equi es ≈4-5s pe ame o images o size 160×140 pixels and ≈43 s pe
ame o images o size 360 ×240 pixels (Taking-an-apple sequence) on an In el Dual Co e CPU wi h
3.16 GHz RAM ( o each co e). Since ou goal is he de elopmen o a ision- on end o eal- ime
ideo segmen acking on op o which o he algo i hms, i.e., mo ing-objec de ec ion/ acking and
ac ion ecogni ion, can be applied, we a e cu en ly in es iga ing ways o imp o e compu a ion ime.
Fas e p ocessing on a CPU could be achie ed by imp o ing he clus e iden i ica ion s ep using a as e
algo i hm [48] and by applying he algo i hm no o he pixels i sel , bu o a omic egions ob ained om
Canny edge de ec ion ollowed by edge acing and con ou closing. The la e echnique has been used
be o e o imp o e speed in ene gy minimiza ion [29]. Fu he mo e we a e cu en ly de eloping a pa allel
implemen a ion on GPUs. So a , we eached ame a es be ween ≈10 −23 ames/s o images o
Senso s 2009,99377
size 360 ×240 pixels (ce ain algo i hmic p ocedu es o he me hod ha e been eplaced by al e na i e
compu a ion schemes o ma ch he equi emen s o he pa allel a chi ec u e).
Acknowledgmen
We hank Sinan Kalkan o aluable discussion. The wo k has ecei ed suppo om he Ge man
Minis y o Educa ion and Resea ch (BMBF) ia he Be ns ein Cen e o Compu a ional Neu oscience
(BCCN) G¨
o ingen unde G an No. 01GQ0430 and he EU P ojec PACO-PLUS unde Con ac
No. 027657.
Re e ences
1. Yilmaz, A.; Ja ed, O.; Shah, M. Objec acking: a su ey. ACM Compu . Su . 2006,38, 1–45.
2. Choi, J.G.; Lee, S.W.; Kim, S.D. Spa io- empo al ideo segmen a ion using a join simila i y
measu e. IEEE T ans. Ci cui s Sys . Video Technol. 1997,7, 279–286.
3. Salembie , P.; Ma ques, F. Region-based ep esen a ions o image and ideo: Segmen a ion ools
o mul imedia se ices. IEEE T ans. Ci cui s Sys . Video Technol. 1999,9, 1147–1169.
4. Tuncel, E.; Onu al, L. U iliza ion o he ecu si e sho es spanning ee algo i hm o ideo-objec
segmen a ion by 2-D a ine mo ion modeling. IEEE T ans. Ci cui s Sys . Video Technol. 2000,
10, 776–781.
5. Deng, Y.; Manjuna h, B.S. Unsupe ised segmen a ion o colo - ex u e egions in images and ideo.
IEEE T ans. Pa e n Anal. Machine In ell. 2001,23, 800–810.
6. Pa as, L.; Hend iks, E.A.; Lagendijk, R.L. Video segmen a ion by MAP labeling o wa e shed
segmen s. IEEE T ans. Pa e n Anal. Machine In ell. 2001,23, 326–332.
7. Yokoyama, Y.; Miyamo o, Y.; Oh a, M. Ve y low bi a e ideo coding using a bi a ily shaped
egion-based mo ion compensa ion. IEEE T ans. Ci cui s Sys . Video Technol. 1995,5, 500–507.
8. Wang, D. Unsupe ised ideo segmen a ion based on wa e heds and empo al acking. IEEE
T ans. Ci cui s Sys . Video Technol. 1998,8, 539–546.
9. Meza is, V.; Kompa sia is, I.; S in zis, M.G. Video objec segmen a ion using Bayes-based
empo al acking and ajec o y-based egion me ging. IEEE T ans. Ci cui s Sys . Video Technol.
2004,14, 782–795.
10. C eme s, D. A a ia ional amewo k o image segmen a ion combining mo ion es ima ion and
shape egula iza ion. In P oceedings o IEEE Compu e Socie y Con e ence on Compu e Vision
and Pa e n Recogni ion, Madison, WI, USA, 2003; pp. 53–58.
11. Po s, R.B. Some gene alized o de -diso de ans o ma ions. P oc. Camb idge Philos. Soc. 1952,
48, 106–109.
12. Geman, D.; Geman, S.; G a igne, C.; Dong, P. Bounda y de ec ion by cons ained op imiza ion.
IEEE T ans. Pa . Anal. Mach. In . 1990,12, 609–628.
13. Vo b ¨
uggen, J.C. Zwei Modelle zu da enge iebenen Segmen ie ung isuelle Da en. Ha i
Deu sch: F ank u am Main, Ge many, 1995.
Senso s 2009,99378
14. Eckes, C.; Vo b ¨
uggen, J.C. Combining da a-d i en and model-based cues o segmen a ion o
ideo sequences. In P oceedings o WCNN Wo ld Con e ence on Neu al Ne wo ks, San Diego, CA,
USA, 1996.
15. Bla , M.; Wiseman, S.; Domany, E. Supe pa ame ic clus e ing o da a. Phys. Re . Le . 1996,
76, 3251–3254.
16. Opa a, R.; W¨
o g¨
o e , F. A as and obus clus e upda e algo i hm o image segmen a ion
in spin-la ice models wi hou annealing – isual la encies e isi ed. Neu al Compu . 1998,
10, 1547–1566.
17. on Fe be , C.; W¨
o g¨
o e , F. Clus e upda e algo i hm and ecogni ion. Phys. Re . E 2000,
62, 1461–1664.
18. Ising, E. Bei ag zu Theo ie des Fe omagne ismus. Zei sch i ¨
u Physik 1925,31, 253–258.
19. Boyko , Y.; Veksle , O.; Zabih, R. Fas App oxima e Ene gy Minimiza ion ia G aph Cu s. IEEE
T ans. Pa . Anal. Mach. In . 1999,23, 2001.
20. Boyko , Y.; Kolmogo o , V. An expe imen al compa ison o min-cu /max- low algo i hms o
ene gy minmiza ion in ision. IEEE T ans. PAMI 2004,26, 1124–1137.
21. Geman, D.; Geman, S. S ochas ic elaxa ion, Gibbs dis ibu ions, and he Bayesian es o a ion o
images. IEEE T ans. Pa . Anal. Mach. In . 1984,6, 721–741.
22. Swendsen, R.; Wang, S. Nonuni e sal c i ical dynamics in Mon e Ca lo simula ions. Phys. Re .
Le . 1987,76, 86–88.
23. Wol , U. Collec i e Mon e Ca lo upda ing o spin sys ems. Phys. Re . Le . 1989,62, 361–364.
24. Ba on, J.L.; Flee , D.J.; Beauchemin, S.; Bu ki , T. Pe o mance o op ical low echniques. In .
J. Compu . Vis. 1994,12, 43–77.
25. Aksoy, E.E.; W¨
o g¨
o e , F.; Dellen, B. Recognizing objec -ac ion ela ions om seman ic scene
g aphs. 2009, submi ed.
26. Saba ini, S.P.; Gas aldi, G.; Sola i, F.; Diaz, J.; Ros, E.; Pauwels, K.; Hulle, K.M.M.V.; Pugeaul ,
N.; K ¨
uge , N. Compac and Accu a e Ea ly Vision P ocessing in he Ha monic Space. In
P oceedings o In e na ional Con e ence on Compu e Vision Theo y and Applica ions (VISAPP),
Ba celona, Spain, 2007.
27. Me opolis, N.; Rosenblu h, A.W.; Rosenblu h, A.H.T.; Telle , E. Equa ions o s a e calcula ions
by as compu ing machines. J. Chem. Phys. 1953,21, 1087–1091.
28. Felzenszwalb, P.; Hu enloche , D.P. E icien Belie P opaga ion o Ea ly Vision. In P oceedings
o IEEE Compu e Socie y Con e ence on Compu e Vision and Pa e n Recogni ion, Washing on,
DC, USA, 2004; pp. 261–268.
29. Ba bu, A.; Zhu, S.C. Gene alizing Swendson-Wang o sampling a bi a y pos e io p obabili ies.
T ans. Pa e n Anal. Mach. In ell. 2005,27, 1239–1253.
30. Wol , U. Collec i e mon e ca lo upda ing o spin sys ems. Phys. Re . Le . 1989,62, 361–364.
31. Coope , C.; F ieze, A.M. Mixing p ope ies o he Swendsen-Wang p ocess on ce ain classes o
g aphs. Random S uc . Algo i hm. 1999,15, 241–261.
32. Go e, V.K.; Je um, M.R. The Swendsen-Wang P ocess Does No Always Mix Rapidly. ACM: El
Paso, TX, USA, 1996; pp. 674–681.
Senso s 2009,99379
33. Onsage , L. C ys al S a is ic. I. Two-dimensional model wi h an o de -diso de ansi ion. Phys.
Re . 1944,64, 117–149.
34. Lucas, B.D.; Kanade, T. An i e a i e image egis a ion echnique wi h an applica ion o s e eo
ision. In P oceedings o DARPA IU Wo kshop, Pi sbu gh, PA, USA, 1981, pp. 121–130.
35. Dellen, B.; W¨
o g¨
o e , F. A local algo i hm o he compu a ion o op ic low ia cons uc i e
in e e ence o global Fou ie componen s. In P oceedings o B i ish Machine Vision Con e ence,
Leeds, UK, 2008, Vol. 2, pp. 795–804.
36. Wa d, J.H. Hie a chical g ouping o op imize an objec i e unc ion. J. Am. S a is ical Assoc. 1963,
58, 236–244.
37. F ¨
an i, P.; Vi majoki, O.; Hau ama¨
aki, V. Fas agglome a i e clus e ing using a k-nea es neighbo
g aph. IEEE T ans. Pa e n Anal. Mach. In ell. 2006,28, 1875–1881.
38. Toshe , A.; Shi, J.; Daniilidis, K. Image ma ching ia saliency egion co espondences. In
P oceedings o IEEE Con e ence on Compu e Vision and Pa e n Recogni ion, Minneapolis, MN,
USA, 2007.
39. Ro he , C.; Minka, T.; Blake, A.; Kolmogo o , V. Cosegmen a ion o Image Pai s by His og am
Ma ching - Inco po a ing a Global Cons ain in o MRFs. In P oceedings o IEEE Con e ence on
Compu e Vision and Pa e n Recogni ion, Washing on, DC, USA, 2006; pp. 993–1000.
40. Kalman, R.E. A new app oach o linea il e ing and p edic ion p oblems. J. Basic Eng. 1960,
82, 35–45.
41. Mazo , E.; A e buch, A.; Ba -Shalom, Y.; Dayan, J. In e ac ing mul iple model me hods in a ge
acking: A su ey. IEEE T ans. Ae osp. Elec on. Sys . 1998,34, 103–123.
42. A ulampalam, M.; Maskell, S.; Go don, N.; Clapp, T. A u o ial on pa icle il e s o online
nonlinea /nonGaussian bayesian acking. IEEE T ans. Signal P ocess. 2002,50, 174–188.
43. Kuma , P.; Rangana h, S.; Sengup a, K.; Weimin, H. Coope a i e Mul i a ge T acking Wi h
E icien Spli and Me ge Handling. IEEE T ans. Ci c. Sys . Video. T. 2006,16, 1477–1490.
44. G eig, D.; Po eous, B.; Seheul , A. Exac maximum a pos e io i es ima ion o bina y images.
Appl. S a .-J. Roy. S a . Soc. B 1989,31, 271–279.
45. Fe a i, P.; F igessi, A.; de Sa, P. Fas app oxima e maximum a pos e io i es o a ion o mul icolou
images. Appl. S a .-J. Roy. S a . Soc. B 1995,57, 485–500.
46. Roy, S.; Cox, I. A maximum- low o mula ion o he n-came a s e eo co espondence p oblem. In
P oceedings o In e na ional Con e ence on Compu e Vision, Bombay, India, 1998.
47. Ishikawa, H.; Geige , D. Occlusions, discon inui ies, and epipoloa lines in s e eo. In P oceedings
o Eu opean Con e ence on Compu e Vision, F eibu g, Ge many, 1998; pp. 232–248.
48. He, L.; Chao, Y.; Suzuki, K.; Wu, K. Fas connec ed-componen labeling. Pa e n Recogni . 2009,
42, 1977–1987.
c
°2009 by he au ho s; licensee Molecula Di e si y P ese a ion In e na ional, Basel, Swi ze land.
This a icle is an open-access a icle dis ibu ed unde he e ms and condi ions o he C ea i e Commons
A ibu ion license h p://c ea i ecommons.o g/licenses/by/3.0/.