Mo ion Rep esen a ion using Composi e Ene gy Fea u es
Raquel Dosil
Co esponding Au ho
Dep. de Elec ónica e Compu ación, Uni . de San iago de Compos ela,
Campus Uni e si a io Su , s/n, 15782, San iago de Compos ela, Spain
e-mail add ess: d[email p o ec ed]
Phone: +34 981 563 100
Fax: +34 981 528 012
Xosé R. Fdez-Vidal
Escola Poli écnica Supe io , Uni . de San iago de Compos ela,
Campus Uni e si a io, s/n, 27002, Lugo, Spain
E-mail add ess: [email protected]
Xosé M. Pa do
Dep. de Elec ónica e Compu ación, Uni . de San iago de Compos ela,
Campus Uni e si a io Su , s/n, 15782, San iago de Compos ela, Spain
E-mail add ess: pa [email protected]
Abs ac .
This wo k ackles he segmen a ion o appa en -mo ion om a bo om-up pe spec i e. When no
in o ma ion is a ailable o build p io high-le el models, he only al e na i e a e bo om-up
echniques. Hence, he whole segmen a ion p ocess elies on he sui abili y o he low-le el
ea u es selec ed o desc ibe mo ion. A wide a ie y o low-le el spa io- empo al ea u es ha e
been p oposed so a . Howe e , all o hem su e om di e se d awbacks. He e, we p opose
he use o composi e ene gy ea u es in bo om-up mo ion segmen a ion o sol e se e al o hese
p oblems.
1
Composi e ene gy ea u es a e clus e s o ene gy il e s –pai s o band-pass il e s in
quad a u e–, each one sensi i e o a di e en se o scale, o ien a ion, di ec ion o mo ion and
speed. They a e g ouped in o de o econs uc independen mo ion pa e ns in a ideo
sequence. A composi e ene gy ea u e, his is, he esponse o one o hese clus e s o il e s, can
be buil as a combina ion o he esponses o he indi idual il e s. The e o e, i inhe i s he
desi able p ope ies o ene gy il e s bu p o iding a mo e comple e ep esen a ion o mo ion
pa e ns.
In his pape , we will p esen ou app oach o in eg a ion o composi e ea u es based on he
concep o Phase Cong uence. We will show some esul s ha illus a e he capabili ies o his
low-le el mo ion ep esen a ion and i s use ulness in bo om-up mo ion segmen a ion and
acking.
Keywo ds: Spa io- empo al ene gy il e ing; Fea u e in eg a ion; Composi e ene gy ea u es;
Appa en -mo ion segmen a ion and acking.
1. In oduc ion
Many ecen mo ion segmen a ion echniques ha e been e y success ul due o he use o op-
down app oaches [1][2]. These echniques can be applied as long as he e is some a ailable
p io model o empla e o he a ge objec . I his is no he case, he pe o mance o
segmen a ion depends on he ules applied o in e high le el ep esen a ions om low le el
da a –bo om-up p ocess–, bu also on he quali y o he low le el ep esen a ion i sel . A wide
a ie y o low-le el spa io- empo al ea u es ha e been p oposed in li e a u e so a [3].
Howe e , all o hem su e om di e ese d awbacks. They can be classi ied in o mo ion
de ec ion and mo ion es ima ion echniques.
Mo ion de ec ion me hods classi y poin s in o s a ic o mobile. As a consequence, hey a e only
alid o s a ic backg ound scenes, and can no classi y mo ion pa e ns acco ding o hei
eloci y and di ec ion o mo ion. A e y popula class o app oaches o mo ion de ec ion is
backg ound sub ac ion. Mo ing objec s a e iden i ied as g oups o connec ed pixels whose
2
ea u es di e om hose o he backg ound model. Backg ound modelling is usually
accomplished using Gaussian mix u e models [4]. Al hough hese echniques a e he mos
widely used, hey a e no ee o limi a ions. They a e e y sensi i e o scene changes in
illumina ion and mo ion. Ano he solu ion o backg ound modelling is PCA analysis wi hin
spa io- empo al pixel blocks, which educes unce ain y and p o ided mo e s abili y o noise
and illumina ion changes [5]. Finally, empo al di e encing app oaches, de ec mo ing pixels
by di e encing be ween wo o h ee adjacen ames [6]. They a e highly adap i e o scene
changes bu ail o ex ac ull o eg ound egions when o eg ound objec s ha e uni o m
ex u e o mo e.
On he o he hand, mo ion es ima ion compu es he pa ame e s o some mo ion model based on
some low-le el mo ion ea u e. The mos popula low-le el mo ion ea u e is op ical low.
Many echniques ha e been p oposed o he es ima ion o op ical low, bu hey all p esen
di e se kinds o p oblems [7][8]. Allowed mo ions a e usually es ic ed o some speci ic
model, such as ansla ional o a ine mo ion. Mo eo e , mo ion in homogeneous egions is no
de ec ed by op ical low. In gene al, mos op ical low es ima ion echniques assume b igh ness
cons ancy along ames, which in eal si ua ions does no always hold. Pa icula ly, di e en ial
me hods o op ical low es ima ion ha a e consis en wi h he b igh ness cons ancy
assump ion a e no e y obus o noise and aliasing. Finally, mos op ical low es ima ion
echniques p esen s ong limi a ions in he allowed displacemen s o he objec s om one ame
o he nex one. Di e en ial me hods y o ind he posi ion o a pixel in he nex ame, bu
sea ch is es ic ed o a small neighbou hood. This limi a ion can be o e come by coa se- o- ine
analysis o by imposing smoo hness cons ain s [7]. S ill, la ge displacemen s a e usually
p oblema ic, as well as occlusions.
Sa o and Agga wal [9] ha e p oposed an o iginal app oach o mo ion es ima ion based on he
ep esen a ion o linea mo ion pa e ns using he so-called Tempo al Spa io-Veloci y (TSV)
ans o m. I consis s o a Hough ans o m e alua ed o e windowed spa io- empo al images.
Segmen a ion is accomplished by simple h esholding o he TSV image. Each esul ing blob
ep esen s a local linea mo ion pa e n. Con e sely, he TSV ans o m has p o ed o be e y
3
obus o noise and o easily deal wi h la ge displacemen s and occlusions. I s main d awback is
ha i is limi ed o ansla ional mo ion wi h cons an eloci y.
He e, we ha e paid a en ion o an ea ly app oach o low le el mo ion ep esen a ion based on
he concep o ene gy il e ing [10][11][12][13][14]. I consis s in he es ima ion o mo ion om
he ampli ude o he esponses o spa io- empo al il e pai s in quad a u e, uned o di e en
scales and o ien a ions. Spa io- empo al o ien a ion sensi i i y is ansla ed in o sensi i i y o
spa ial o ien a ion, speed, and di ec ion o mo ion. These echniques a e known o be obus o
noise and aliasing, o gi e con iden measu emen s o eloci y, and o allow an easy ea men
o he ape u e p oblem, i.e., he eliable es ima ion o he di ec ion o mo ion. Howe e , o he
bes o ou knowledge he e is no mo ion segmen a ion me hod based on ene gy il e ing.
Ene gy il e ing pe o ms a de ec ion o plana s uc u es in a spa io- empo al domain. Hence, i
is s aigh o wa d o co ela e pixels o a mo ing objec om di e en ames, a oiding
p oblems wi h occlusions and la ge displacemen s. Mo eo e , ene gy il e ing is obus o noise
and aliasing. Besides, ene gy ea u es a e sui able o ex u e desc ip ion in MPEG-7 ideo
sequences o con en -based e ie al applica ions [15]. The main limi a ion o ene gy ea u es
o mo ion ep esen a ion is being es ic ed o ansla ional mo ion wi h cons an eloci y. In
his pape , we p opose he use o composi e ene gy ea u es in bo om-up mo ion segmen a ion
o cope wi h his incon enience.
Composi e ene gy ea u es a e clus e s o ene gy ea u es, his is, he esponses o band-pass
pai s o il e s in quad a u e, each o hem sensi i e o a di e en se o scale, o ien a ion,
di ec ion o mo ion, and speed. Ene gy ea u es a e g ouped oge he in o de o iden i y and
econs uc independen mo ion pa e ns in a ideo sequence. This means ha he elemen s o be
clus e ed a e whole ea u es, no pixels. A composi e ene gy ea u e is buil as a combina ion
indi idual ene gy ea u es in a clus e . The e o e, i inhe i s he desi able p ope ies o ene gy
ea u es bu p o iding a mo e comple e ep esen a ion o mo ion pa e ns. Composi e ene gy
ea u es ha e p o ed o be a powe ul ool o he ep esen a ion o isually independen spa ial
pa e ns in 2D da a [16], olume ic da a [17][18], and ideo sequences [19].
To iden i y ele an composi e ea u es in a sequence, i is necessa y o de ine an in eg a ion
4
c i e ion able o ela e band-pass ene gy ea u es con ibu ing o he same mo ion pa e n.
Ea lie app oaches [16][19] use complex s a is ical measu es wi h high compu a ional cos . In
p e ious wo ks [17][18][20], we ha e in oduced an in eg a ion c i e ion ha imp o es
compu a ional cos and pe o mance. Ins ead o applying a bi a y s a is ical ules, ou c i e ion
is inspi ed in biological ision. I is based on he hypo hesis o Mo one and Owens [21] ha he
Human Visual Sys em (HVS) pe cei es ea u es a poin s o locally maximal Phase Cong uence
(PC). PC is he measu e o he local deg ee o alignmen o he local phase o Fou ie
componen s o a signal. The sensi i i y o he HVS o PC has been s udied by o he au ho s as
well [14][22][23][24]. He e, we ha e ex ended his concep o spa io- empo al signals o de ine
ou c i e ion o clus e ing o spa io- empo al ene gy ea u es.
We will show ha composi e ea u es clus e ed unde his c i e ion ep esen isually
independen mo ion pa e ns in a ideo sequence, wi h di e en eloci y, di ec ion, and/o scale
con en , and ha i pe o ms mo ion es ima ion wi hou he imposi ion o any mo ion model. We
will see ha , unlike many o he p e iously ci ed me hod, his ep esen a ion is obus o noise
and illumina ion changes. Besides, i na u ally co ela es in o ma ion om di e en ames, so
ha i easily deals wi h occlusions and la ge in e - ame displacemen s, while common low
le el ea u es need he aid o acking echniques o ind he co espondence o he posi ion o
an objec om di e en ames.
We ha e applied he composi e ea u e ep esen a ion as he basis o a me hod o segmen a ion
and acking. Typical bo om-up me hods o mo ion acking include ac i e models [25][26],
Bayesian egion classi ica ion [27][28], and Kalman il e ing [29]. A comp ehensi e su ey can
be ound in [3]. We ha e chosen a geodesic ac i e model [25] o acking. We apply ou
ep esen a ion o bo h he ini ializa ion o he model a each ame and as a low le el ea u e o
guide he e olu ion o he model.
The ou line o his chap e is as ollows. In sec ion 2, we desc ibe in de ail he composi e ea u e
de ec ion p ocess. Sec ion 3 explains how he geodesic ac i e model uses he low le el mo ion
ep esen a ion o guide segmen a ion. In sec ion 4, we illus a e he beha iou o he composi e
ea u e ep esen a ion in di e en p oblema ic si ua ions, including some s anda d ideo
5
sequences. We will also show he esul s o segmen a ion and acking. In sec ion 5, we
expound he conclusions o he wo k.
2. Composi e-Fea u e De ec o Syn hesis
As a o emen ioned, composi e ene gy ea u es a e clus e s o ene gy ea u es. Then, in he i s
place, we mus pe o m a mul i esolu ion decomposi ion o ob ain he indi idual ene gy
ea u es. To his end, we apply a bank o non-causal spa io- empo al ene gy il e s o he ideo
sequence. The complex- alued olume gene a ed as he esponse o a spa io- empo al ene gy
il e o a gi en ideo sequence is he e called a band-pass ene gy ea u e. We will call
composi e ene gy ea u es o mo ion pa e ns wi h mul iple speed, di ec ion and scale con en s
gene a ed as he combina ion o band-pass ene gy ea u es in a clus e . The se o il e s
associa ed o an ene gy ea u e clus e a e e e ed o as composi e- ea u e de ec o .
Fea u e g ouping is accomplished by applying clus e analysis o he se o band-pass ea u es o
he ideo sequence. We pe o m clus e ing o he ea u es as a whole, no poin wise, since a
poin in space can be occupied by se e al isual pa e ns wi h di e en equency con en .
Hence, mo e ha one composi e ea u e can ha e la ge esponse in he same loca ion in space
and ime. This is no allowed by c isp clus e ing o pixels. Besides, pixel clus e ing is mo e
sensi i e o local a ia ions and noise.
Clus e ing is accomplished using a hie a chical algo i hm, which is based on a dissimila i y
ma ix e lec ing he dis ances among band-pass ea u es. The dissimila i y measu e is de ined
in o de o iden i y band-pass ea u es con ibu ing o he same local maxima o Phase
Cong uence (PC). We will see ha he co ela ion coe icien can p o ide a good global
measu e o he PC be ween band-pass ea u es [17][20].
Finally, each composi e ea u e is econs uc ed as a combina ion o he esponses o he il e s
in a gi en clus e . Ins ead o a simple linea combina ion, we p opose a mo e sophis ica ed
pooling exp ession, in o de o enhance he ep esen a ion. The nex subsec ions p esen he
de ails o he p ocess.
6
2.1 Bank o Spa io-Tempo al Fil e s
The basis unc ion o he bank o spa io- empo al il e s applied he e [17][20] is an ex ension o
3D o he log Gabo unc ion [30]. The il e is designed in he equency domain, since i has
no analy ical exp ession in he spa ial domain. Fil e ing is ealized as he inne p oduc be ween
he ans e unc ion o he il e and he Fou ie ans o m o he sequence. Fil e ing in he
Fou ie domain is e y as when using Fas Fou ie T ans o m and In e se Fas Fou ie
T ans o m algo i hms.
The il e s’ ans e unc ion T is designed in sphe ical equency coo dina es as he p oduc o
sepa able ac o s, R and S, in he adial and angula componen s espec i ely, such ha T=R·S.
The adial e m R is gi en by he log Gabo unc ion [30]
,
(1)
whe e
i is he s anda d de ia ion and
i is he cen al adial equency o he il e .
The angula componen is designed o achie e o ien a ion selec i i y in bo h he azimu hal
componen
i o he il e , which e lec s he spa ial o ien a ion o he pa e n in a ame and he
di ec ion o mo emen , and he ele a ion componen
i, ela ed o he speed and di ec ion o
mo ion. Fo s a ic pa e ns,
i=0. To achie e o a ional symme y, S is de ined as a Gaussian on
he angula dis ance
be ween he posi ion ec o o a gi en poin in he spec al domain and
he di ec ion o he il e =(cos
i·cos
i,cos
i·sin
i,sin
i) [31]
, wi h , (2)
whe e is exp essed in Ca esian coo dina es and
i is he angula s anda d de ia ion.
Ac i e il e s a e selec ed om a p ede ined band pa i ioning o he 3D spec al domain.
F equency bands a e desc ibed by he cen al equency (
i,
i,
i) o he il e s and hei wid h
pa ame e s (
i,
i). The selec ion o hese pa ame e s de e mines he kind o ene gy ea u es
yield by he mul i esolu ion decomposi ion. In his applica ion we desi e o ge wide co e age o
7
he scale space, uni o m sampling o he o ien a ion space, and high o ien a ion sensi i i y. To
his end, he pa ame e s o he bank a e es ablished as ollows.
F equency is sampled so ha
i={1/2, 1/4, 1/8, 1/16}, in pixels1. Pa ame e
i is de e mined
o each band in o de o ob ain a 2 oc a e bandwid h.
i is sampled uni o mly while he numbe
o
i samples dec eases wi h ele a ion.
i a e de e mined in o de o keep a cons an “densi y” o
il e s, by main aining equal a c-leng h be ween adjacen
i samples o e he uni adius sphe e.
i is se o 25º o all o ien a ions. Following his c i e ion, he il e bank has been designed
using 23 o ien a ions, i.e. (
i,
i) pai s. In o al, we ha e 23 o ien a ion 4 scales = 92 bands
wi h a ce ain o e lapping be ween neighbou ing bands, yielding a edundan decomposi ion
and a wide co e age o he spec um.
2.2 Selec ion o Ac i e Bands
To achie e imp o ed pe o mance, i is con enien o educe he numbe o bands in ol ed in
clus e analysis. The exclusion o equency channels ha a e no likely o con ibu e o mo ion
pa e ns acili a es he iden i ica ion o clus e s associa ed o composi e mo ion ea u es.
Fu he mo e, i educes compu a ional cos . Fea u es selec ed o clus e ing a e called ac i e. A
p e ious solu ion o ac i e band selec ion, applied in [17], consis ed in he de ec ion o bands
enclosing spec al ampli ude alues o e a maximum noise le el. This implies he se ing o a
h eshold pa ame e and he use o a adial median il e , which is highly compu a ionally
expensi e. He e, we ha e in oduced a channel selec ion s age based on a s a is ical analysis o
he ampli ude esponses o he band-pass ea u es. This solu ion ou pe o ms ha in [17] in bo h
compu a ional cos and e iciency.
Ou me hod o he selec ion o ac i e channels is based on he wo ks o Field [32] and Nes a es
e al. [33]. Field has s udied he s a is ics o he esponses o a mul i esolu ion decomposi ion
based on log-Gabo wa ele s ha esembles he coding in he isual sys em o mammalians. He
has obse ed ha he his og ams o he il e esponses a e no Gaussian, bu lep oku ic
dis ibu ions –poin ed dis ibu ions wi h long ails–, e ealing he spa se na u e o bo h he
8
senso y coding and he ea u es om na u al images. Acco ding o Field, when he pa ame e s
o he wa ele codi ica ion i hose in he mammalian isual sys em, he his og am o he
esponses is highly lep oku ic. This is e lec ed in he ou h cumulan o he dis ibu ion.
Namely, he uses he ku osis o cha ac e ize he spa seness o he esponse.
Rega ding spa io- empo al analysis, Nes a es e al. [33] applied channel selec ion o a bank o
hi d o de Gaussian de i a i e il e s based on he s a is ics o il e s esponses. They ha e
obse ed ha ea u es co esponding o mobile a ge s p esen spa se esponses han hose
associa ed o backg ound –wea he s a ic o mo ing. This ac is illus a ed in Fig. 1. They
measu e di e en s a is ical magni udes e lec ing spa seness o he ampli ude esponse, ealize
a anking o he channels based on such measu es, and pe o m channel selec ion by aking he
n i s channels in he anking, whe e n is a p e ixed numbe .
Based on hese wo wo ks, we ha e designed ou il e selec ion me hod. The s a is ical measu e
employed o cha ac e ize each channel is he ku osis excess
2
,(3)
whe e k4 and k2 a e espec i ely he ou h and second cumulan s o a his og am. I he ku osis
excess akes a posi i e alue, he dis ibu ion is called lep oku ic and p esen s a na ow peak
and long ails. I i is nega i e, he dis ibu ion is called pla yku ic and p esen s a b oad cen al
lobe and sho ails. Dis ibu ions wi h ze o ku osis excess, like he Gaussian dis ibu ion, a e
called mesoku ic.
We measu e γ2 o bo h he eal and imagina y componen s o each ea u e
i and hen we
compose a single measu e
(4)
Ins ead o selec ing a ixed numbe o channels wi h he la ges alues o
, we pe o m clus e
analysis o iden i y wo clus e s, one o ac i e channels, wi h la ge alues o
, and ano he o
non ac i e channels. He e, we ha e applied a k-means algo i hm. The clus e o ac i e channels
is iden i ied as he one wi h highe
on a e age.
9
emapped o he in e al [–1,1]. The ze o-le el o he esul ing image is he ini ial s a e o he
con ou .
(13)
When he objec emains s a ic du ing a numbe o ames he isual pa e n has a null esponse.
Fo his eason, he ini ial model is de ined as he weigh ed sum o wo e ms, espec i ely
associa ed o he cu en and p e ious ames. The con ibu ion om he p e ious ame mus be
e y small.
(14)
wi h wk and wk– 1 being posi i e eal cons an s ha e i y wk+wk– 1 =1. In he expe imen s
p esen ed in nex sec ion, wk=0.9, wk – 1=0.1, K=20 and 0=0.1.
4 Resul s
In his sec ion, some esul s a e p esen ed o show he beha iou o he me hod in p oblema ic
si ua ions. The comple e ideo sequences wi h he o iginal da a and he segmen a ion esul s a e
a ailable a h p://www-g a.dec.usc.es/~ dosil/mo ion_segmen a ion_examples.h m.They a e
summa ized in he nex subsec ions.
Example #1
The ollowing example shows he abili y o he me hod o deal wi h complex mo ion pa e ns
wi h a ia ions in illumina ion, non linea mo ion and de o ma ions. In pa icula , he ollowing
sequence, a agmen o 27 ames o he s anda d mo ie know as “silen ” – op ow o Fig. 7–,
p esen s a mobile objec , wi h a iable shape, speed and di ec ion. As can be app ecia ed, he
mo ion pa e n o he hand can no be p ope ly desc ibed by an a ine ans o ma ion. Mo eo e ,
he b igh ness cons ancy assump ion is no e i ied in his case.
The middle ow o Fig. 7 shows one o he wo iden i ied composi e pa e ns, ep esen ing he
16
mo ing hand. I can be seen ha , despi e he complexi y o he image, he composi e- ea u e
ep esen a ion model is able o isola e he hand and p ope ly ep esen i s changing shape in
di e en ames. Images in he bo om ow o Fig. 7 p esen he segmen a ion esul s.
Example #2
In his example, we use a agmen o 27 ames o he well-known sequence “ lowe ga den” –
see Fig. 9. I is a s a ic scene eco ded by a mo ing came a, so ha all he pixels in he scene a e
mo ing. The composi e ene gy ea u e ep esen a ion iden i ies wo mo ion pa e ns, one o he
mo ing ee and one o he mo ing backg ound. This is possible because he ee is placed in a
o eg ound plane, so ha i s speed is la ge . When isualizing a cu in he x- plane on bo h he
image and he mo ion pa e ns –Fig. 8–, i can be seen ha di e en speeds a e ansla ed in o
di e en o ien a ions in he spa io- empo al domain. The composi e ene gy ea u e
ep esen a ion has been able o clus e ene gy ea u es wi h simila equencies –speeds,
o ien a ions and sizes.
The esul o segmen a ion using one o he de ec ed mo ion pa e ns is shown in Fig. 9. The
image po en ial in ha example is compu ed om he composi e ene gy ea u e co esponding
o he ee. Hence, he e a e no deep minima o he po en ial caused by mo ing backg ound
con ou s, which leads o a co ec segmen a ion.
Example #3
In his example we use a agmen o 22 ames o he well-known “ able ennis” ideo
sequence, shown in i s ow o Fig. 10, which p esen s non linea mo ion pa e ns and la ge
in e - ame displacemen s, due o a small sampling a e. This makes i di icul o ind he
co espondence be ween he posi ions o he objec in wo consecu i e ames.
The composi e ea u e ep esen a ion is able o iden i y he ene gy componen s o he mo ion
pa e n, and o gene a e a composi e ea u e in which he mo ion pa e n is isola ed om he
s a ic backg ound –see Fig. 10, middle ow. The esul o segmen a ion is shown in bo om ow
o Fig. 10. Since he ac i e model is ini ialized om he mo ion pa e n a each ame, he la ge
17
in e - ame displacemen s do no in luence acking.
Example #4
Occlusions gi e ise o he same p oblem as wi h as objec s. In he nex example, in Fig. 11, a
sequence o 31 ames is showing a cylinde olling behind ano he objec , comple ely
disappea ing om he scene du ing some ames. The middle ow in Fig. 11 shows one o he
composi e ea u es iden i ied by ou ep esen a ion me hod, co esponding o he mo ing
cylinde , in i s ampli ude ep esen a ion. The o he composi e ea u e de ec ed, no shown he e,
co esponds he s a ic objec s.
The esul o segmen a ion using his composi e ea u e is shown in las ow o Fig. 11. As can
be seen, ini ializa ion wi h he composi e ene gy ea u e leads o a co ec segmen a ion, e en
when he objec disappea s om he scene du ing se e al ames. The model collapses when he
cylinde disappea s behind a s a ic objec and is eini ialized au oma ically when i eappea s,
wi hou he need o a p io model.
Example #5
The nex example, in Fig. 12, wi h 39 ames, p esen s occlusions as well, bu now he
occluding objec is also mo ing, which complica es acking. The scene shows wo people
walking in opposi e di ec ions. The second and hi d ows in Fig. 12 show wo o he composi e
ea u es iden i ied by ou ep esen a ion me hod, co esponding o he wo mo ing pe sons. The
emainde composi e ea u es de ec ed, no shown he e, co espond o s a ic objec s. Again, he
ep esen a ion model is capable o decomposing he scene in o isually independen mo ion
pa e ns by in eg a ing band-pass ene gy ea u es.
Two ac i e models ha e been op imized, each guided by one o hese composi e ea u es.
Segmen a ion esul s a e p esen ed in he ou h and i h ows o Fig. 12. The use o composi e
ea u es o bo h ini ializa ion and de ini ion o image po en ial a oids in e e ences be ween
di e en mo ion pa e ns du ing model op imiza ion. Besides, acking is s aigh o wa d since
he e is only one model o each pa e n in each ame, al hough i may be spli .
18
Example #6
The ollowing example is a agmen o he s anda d ideo sequence “coas gua d”, comp ising
48 ames. This sequence shows wo mobile objec s, one o hem being ollowed by he came a.
The esul s o his sequence a e p esen ed in Fig. 13. The model classi ies he il e s
co esponding o he mo ing backg ound as non ac i e, excluding hem om he subsequen
analysis. The model is able o iden i y wo mo ion pa e ns, one o each o he mo ing objec s.
Segmen a ion o he s a ic objec is no in e e ed by backg ound ex u e o backg ound mobile
con ou s.
Example #7
The las example, in Fig. 14.a and e, is a sequence o 126 ames showing wo mo ion pa e ns
wi h di e en scales, speeds and di ec ions o mo ion. Bo h pa e ns a e occluded in di e en
pa s o he sequence. The backg ound is no comple ely s a ic, bu he e a e also local mo ion
pa e ns –b anches in he ees a e mo ing due o he wind. I is a low esolu ion ideo, wi h a
high noise le el.
The p oposed model classi ies he il e s con ibu ing o he backg ound as non ac i e, including
local mo ion in he b anches o he ees –see Fig. 15.b– and ac i e il e s gi e place o wo
mo ion pa e ns, one o each mo ing objec –Fig. 15.c, d, and g. Segmen a ion esul s o bo h
composi e ea u es a e shown in Fig. 15. Ini ializa ion wi h composi e ea u es sol es he
p oblem o occlusions in bo h si ua ions. Image po en ials om composi e ea u es a oid
in e e ences be ween he wo mo ion pa e ns and in e e ences wi h mo ion in backg ound
pixels.
5 Discussion and conclusions
Resul s show ha he p oposed model o he ep esen a ion o mo ion is able o g oup band-
pass ea u es associa ed o di e en mo ion pa e ns in a scene wi hou he use o p io
19
in o ma ion, and o isola e isually independen mo ion pa e ns. The s udied examples co e
some o he mos common p oblems in mo ion es ima ion, namely, p esence o noise, mo ing
backg ound, a ia ions in illumina ion, non a ine mo ion pa e ns, scenes wi h mul iple mo ion
pa e ns, occlusions, and la ge displacemen s be ween neighbou ing ames. As summa ized in
he in oduc ion, o he common low le el mo ion ep esen a ions p esen di icul ies in some o
hese si ua ions.
The key cha ac e is ic o he composi e ea u e ep esen a ion is ha in eg a ion is accomplished
by clus e ing o spa io- empo al ene gy ea u es as a whole, no by poin -wise egion
classi ica ion. This ac yields a ep esen a ion ha in insically co ela es in o ma ion om
di e en ames. This p ope y is esponsible o he obus ness o pa ial and o al occlusions
and la ge in e - ame displacemen s o de o ma ions. This p ope y allows he di ec use o his
ep esen a ion in guiding a high le el segmen a ion echnique like he ac i e model, wi hou any
in e media e s ep o egion classi ica ion o objec sea ch om one ame o he nex .
Segmen a ion esul s p o e he success o he combina ion o composi e ea u e ep esen a ion
and high le el modelling by a le el-se app oach. The p oblem o he ini ializa ion o ac i e
models is easily sol ed using he ampli ude ep esen a ion o composi e ea u es. Fu he mo e,
con e gence o he model o he a ge objec is ensu ed by building he image po en ial om
ea u es ha disc imina e he a ge objec om o he s uc u es and mo ing objec s in he
scene. In u u e, we plan o ackle mo e complex cases by applying he composi e ea u e
ep esen a ion o aid op-down mo ion analysis.
6. Acknowledgemen s
This wo k has been inancially suppo ed by he Minis y o Educa ion and Science o he
Spanish Go e nmen , h ough he esea ch p ojec TIN2006-08447.
7. Re e ences
[1]. H.T. Nguyen, A.W.M. Smeulde s, Fas Occluded Objec T acking by a Robus
20
Appea ance Fil e , IEEE T ans. Pa e n Anal. Mach. In ell., 26 (2004) 1099-1104.
[2]. C. Ke ann, F. Hei z, A Hie a chical Ma ko Modeling App oach o he Segmen a ion
and T acking o De o mable Shapes, G aphical Models and Image P ocessing, 60 (1995)
173-195.
[3]. W. Hu, T. Tan, L. Wang, S. Maybank. A Su ey on Visual Su eillance o Objec Mo ion
and Beha io s, IEEE T ans on Sys ems, Man and Cybe ne ics –Pa C: Applica ions and
Re iews, 34 (2004) 334-351.
[4]. C. S au e , W. G imson, Adap i e Backg ound Mix u e Models o Real-Time T acking,
in P oc. IEEE Con . Compu e Vision and Pa e n Recogni ion, Vol 2., 1999, 246-254.
[5]. L.J. La ecki, V. Megalooikonomou, R. Miezianko, D. Pok ajac, Using Spa io empo al
Blocks o Reduce he Unce ain y in De ec ing and T acking Mo ing Objec s in Video,
In elligen Sys ems Technologies and Applica ions, 1 (2006) 376-392.
[6]. A.J. Lip on, H. Fujiyoshi, R.S. Pa il, Mo ing Ta ge Classi ica ion and T acking om
Real-Time Video, in P oc. IEEE Wo kshop Applica ions o Compu e Vision, 1998, 8-14.
[7]. J.L. Ba on, D.J. Flee , S.S. Beauchemin, Pe o mance o Op ical Flow Techniques, In J
Compu Vis, 12 (1994) 43-77.
[8]. C. S ille , J. Kon ad, Es ima ing Mo ion in Image Sequences: A Tu o ial on Modeling and
Compu a ion o 2D Mo ion, IEEE Signal P ocessing Magazine, 16 (1999) July 71-91.
[9]. K. Sa o, J.K. Agga wal, Tempo al Spa io-Veloci y T ans o m and i s Applica ion o
T acking and In e ac ion, Compu e Vision and Image Unde s anding, 96 (2004) 100-128.
[10]. D.J. Heege , Model o he Ex ac ion o Image Flow, J. Op . Soc. Am. A, 4 (1987) 1555-
1471.
[11]. E.P. Simoncelli, E.H. Adelson, Compu ing Op ical Flow Dis ibu ions using Spa io-
Tempo al Fil e s, MIT Media Lab. Vision and Modeling, Tech. Repo 165, 1991. URL:
h p://web.mi .edu/pe sci/people/adelson/pub_pd s/simoncelli_compu .pd
[12]. A.B. Wa son, A.J. Ahumada J ., Model o Human Visual-Mo ion Sensing, J. Op . Soc.
Am. A, 2 (1985) 322-342.
[13]. E.H. Adelson, J.R. Be gen, Spa io empo al Ene gy Models o he Pe cep ion o Mo ion,
21
J. Op . Soc. Am. A,2 (1985) 284-299.
[14]. D. Flee , Measu emen o Image Veloci y, Kluwe Academic Publishe s, Massachuse s,
1992.
[15]. Y.M. Ro, M. Kim, H.K. Kang, B.S. Manjuna h, J. Kim, MPEG-7 Homogeneous Tex u e
Desc ip o , ETRI Jou nal, 23 (2001) 41-51.
[16]. R. Rod íguez-Sánchez, J.A. Ga cía, J. Fdez-Valdi ia, X.R. Fdez-Vidal, The RGFF
Rep esen a ional Model: A Sys em o he Au oma ically Lea ned Pa i ion o “Visual
Pa e ns” in Digi al Images, IEEE T ans. Pa e n Anal. Mach. In ell, 21(1999) 1044-1073.
[17]. R. Dosil, Da a D i en De ec ion o Composi e Fea u e De ec o s o 3D Image Analysis,
PhD Thesis, Uni e sidade de San iago de Compos ela (Spain), 2005. URL:
h p://www-g a.dec.usc.es/~ dosil/ ichei os/ hesis_dosil.pd
[18]. R. Dosil, X.M. Pa do, X.R. Fdez-Vidal, Decomposi ion o 3D Medical Images in o Visual
Pa e ns, IEEE T ans. on Biomedical Enginee ing, 52 (2005) 2115-2118.
[19]. J. Chamo o-Ma ínez, J. Fdez-Valdi ia, J.A. Ga cía, J. Ma ínez-Baena, A F equency
Domain App oach o he Ex ac ion o Mo ion Pa e ns, IEEE In . Con . on Acous ics,
Speech and Signal P ocessing, Vol. 3, Hong Kong, 2003, pp. 165-168.
[20]. R. Dosil, X.R. Fdez-Vidal, X.M. Pa do, Dissimila i y Measu es o Visual Pa e n
Pa i ioning, in: J. Ma ques, N. Pé ez de la Blanca (Eds.), LNCS: Pa e n Recogni ion and
Image Analysis, Vol. 3523, Sp inge -Ve lag Be lin Heidelbe g, 2005, pp. 287-294.
[21]. M.C. Mo one, R.A. Owens, Fea u e De ec ion om Local Ene gy, Pa e n Recogni ion
Le e s, 6 (1987) 303-313.
[22]. A. Oppenheim, J. Lim, The Impo ance o Phase in Signals, P oc. o he IEEE, 69 (1981)
529–541.
[23]. J. Ross, M.C. Mo one, D. Bu , The Condi ions unde which Mach Bands a e Visible,
Vision Resea ch, 29 (1989) 699–715.
[24]. J. du Bu , Ramp Edges, Mach Bands and he Func ional Signi icance o he Simple Cell
Assembly, Biological Cybe ne ics, 70 (1994) 449–461.
[25]. N. Pa agios, R. De iche, Geodesic Ac i e Con ou s and Le el Se s o he De ec ion and
22
T acking o Mo ing Objec s, IEEE T ans. Pa e n Anal. Mach. In ell., 22 (2000) 266-279.
[26]. A.-R. Mansou i, J. Kon ad, Mul iple Mo ion Segmen a ion wi h Le el Se s, IEEE T ans.
on Image P ocessing, 12 (2003) 201-220.
[27]. M.M. Chang, A.M. Tekalp, M.I. Sezan, Simul aneous Mo ion Es ima ion and
Segmen a ion, IEEE T ans. on Image P ocessing, 6 (1997) 1326-1333.
[28]. R. Mon oliu, F. Pla, An I e a i e Region-G owing Algo i hm o Mo ion Segmen a ion
and Es ima ion, In . Jou nal o In elligen Sys ems, 20 (2005) 577-590.
[29]. Y. Boyko , D.P. Hu enloche , Adap i e Bayesian Recogni ion in T acking Rigid Objec s,
IEEE In . Con . on Compu e Vision and Pa e n Recogni ion (CVPR), Vol. 2, 2000, pp.
697-704.
[30]. D.J. Field, Wha is he Goal o Senso y Coding. Neu al Compu a ion, 6 (1994) 559-601.
[31]. F.G.A. Faas, L.J. an Vlie , 3D-O ien a ion Space; Fil e s and Sampling, in: J. Bigun, T.
Gus a sson (Eds.), LNCS: Scandina ian Con e ence on Image Analysis, Vol. 2749,
Sp inge -Ve lag Be lin Heidelbe g, 2003, pp.36-42.
[32]. D.J. Field, Scale–In a iance and Sel -Simila “Wa ele ” T ans o ms: An Analysis o
Na u al Scenes and Mammalian Visual Sys ems, in: M. Fa ge, J.C.R. Hun , J.C.
Vassilicos, (Eds.), Wa ele s, ac als and Fou ie T ans o ms, Cla endon P ess, Ox o d,
1993, pp. 151-193.
[33]. O. Nes a es, C. Mi a e , J. San ama ia, R. Na a o, Au oma ic Enhancemen o Noisy
Image Sequences Th ough Local Spa io empo al Spec um Analysis, Op ical Enginee ing,
39 (2000) 1457-1469.
[34]. S. Venka esh, R. Owens, On he Classi ica ion o Image Fea u es, Pa e n Recogni ion
Le e s, 11 (1990) 339-349.
[35]. A. Jain, R. Dubes, Algo i hms o Clus e ing Da a, P en ice Hall, New Je sey, 1988.
[36]. N.R. Pal, J. Biswas, Clus e Valida ion Using g aph Theo e ic Concep s, Pa e n
Recogni ion, 30 (1996) 847-857.
[37]. M. Kass, A. Wi kin, D. Te zopoulos, Snakes: Ac i e Con ou Models, In . Jou nal o
Compu e Vision, 55 (1988) 321-331.
23
[38]. V. Caselles, R. Kimmel, G. Sapi o. Geodesic Ac i e Con ou s, In J Compu Vis, 22,
(1997) 61-79.
[39]. J. Weicke , G. Kühne, Fas Me hods o Implici Ac i e Con ou Models, in: S. Oshe , N.
Pa agios (Eds.), Geome ic Le el Se Me hods in Imaging, Vision and G aphics, Sp inge ,
New Yo k, 2003, pp. 43-58.
[40]. R. Dosil, X.M. Pa do, Gene alized Ellipsoids and Aniso opic Fil e ing o Segmen a ion
Imp o emen in 3D Medical Imaging, Image and Vision Compu ing, 21 (2003) 325-343.
24
(a)
(b) (c)
(d) (e)
Fig. 1. (a) A ame o he “Silen ” s anda d sequence, showing a mo ing hand. (b) and (d) A
ame o he eal pa o wo band-pass ea u es o he Silen ideo sequence. (c) and (e)
His og ams co esponding o band-pass ea u es in (b) and (d) espec i ely.
25
Fig. 10. Top: Le and cen e images co espond o wo consecu i e ames o he “ able
ennis” ideo sequence. The image a he le is a a e sal cu in he y- plane. Fo ames on
op ow, Middle Row: amp o he selec ed composi e- ea u e, and Bo om: Segmen a ion
ob ained wi h one o he de ec ed composi e- ea u es.
32
Fig. 11. Top: Th ee ames o a ideo sequence whe e a mo ing objec is o ally occluded
du ing se e al ames. Middle Row: Ini ializa ion o he ames using he amp ep esen a ion
o one o he de ec ed composi e- ea u e. Bo om: Segmen a ion using ini ializa ion wi h he
composi e ea u e
33
Fig. 12. Th ee ames o a sequence showing wo occluding mo ion pa e ns. 1s ow: Inpu
da a. 2nd and 3 d ows: e en o wo o he ob ained composi e- ea u es, co esponding o he
wo mo ion pa e s. 4 h and 5 h ows: Segmen a ions p oduced using composi e- ea u es om
ows 2nd and 3 d espec i ely.
34
Fig. 13. Top and bo om ow co espond o wo ames o he “cos gua d” ideo sequence: 1s
column: Inpu da a. 2nd and 3 d columns: abs o he wo mo ion pa e ns isola ed by he
composi e- ea u e ep esen a ion mode and 4 h column: segmen a ions p oduced using he
composi e ea u e in 3 d column.
35
(a) (b)
(c) (d)
(e) ( ) (g)
Fig. 14. (a) A ame o he inpu sequence. (b) eal om he g oup o non ac i e il e s. (c) and
(d). eal om he composi e ea u es de ec ed by he model. (e) T a e sal cu o he inpu
sequence – ame index inc eases om op o bo om. ( ) and (g) T a e sal cu o abs om he
wo composi e ea u es de ec ed. Occlusions a e be e app ecia ed in his iew.
36
Fig. 15. Le column: Se e al ames o he inpu sequence. Cen e and Le columns:
Co esponding segmen a ions p oduced using each o he wo de ec ed composi e ea u es.
37
