Optical flow estimation with consistent spatio-temporal coherence models

Op ical Flow Es ima ion wi h Consis en Spa io- empo al Cohe ence Models Ja ie S´ anchez, Agus ´ ın Salgado and Nelson Monz´ on Cen o de Tecnolog´ ıas de la Imagen (CTIM) Depa amen o de In o m´ a ica y Sis emas Uni e si y o Las Palmas de G an Cana ia, Spain {jsanchez, asalgado}@dis.ulpgc.es, [email p o ec ed] Keywo ds: Op ical Flow, Va ia ional Me hods, PDE, Tempo al Cohe ence. Abs ac : In his wo k we p opose a new a ia ional model o he consis en es ima ion o mo ion ields. The aim o his wo k is o de elop app op ia e spa io- empo al cohe ence models. In his sense, we p opose wo main con ibu ions: a nonlinea low cons ancy assump ion, simila in spi i o he nonlinea b igh ness cons ancy assump ion, which con enien ly ela es low ields a di e en ime ins an s; and a nonlinea empo al egula - iza ion scheme, which complemen s he spa ial egula iza ion and can cope wi h piecewise con inuous mo ion ields. These con ibu ions pose a cong uen a ia ional model since all he ene gy e ms, excep he spa ial egula iza ion, a e based on nonlinea wa pings o he low ield. This model is mo e gene al han i s spa ial coun e pa , p o ides mo e accu a e solu ions and p ese es he con inui y o op ical lows in ime. In he expe imen al esul s, we show ha he me hod a ains be e esul s and, in pa icula , i conside ably imp o es he accu acy in he p esence o la ge displacemen s. 1 INTRODUCTION The es ima ion o mo ion ields is a key p oblem in compu e ision. I se es as a basis o many appli- ca ions, such us s e eoscopic ision and 3D scene e- cons uc ion, medical image analysis, s uc u e om mo ion, objec acking and o he s. I we a e gi en a ideo sequence, and we wan o ind he mo ion o he objec s in he images, ou me hod should p o ide a solu ion ha is consis en h ough he sequence. In his wo k, we add ess he p oblem o empo al co- he ence in op ical low me hods. The aim is o de ise new me hods ha allow inding con inuous low ields in ime. Op ical low me hods can be u he imp o ed i empo al in o ma ion is p ope ly managed (Weicke and Schn¨ o , 2001). In his wo k, he au ho s p opose a me hod ha is a s aigh ex apola ion o he spa ial cohe ence model o he empo al dimension, based on a con inuous spa io- empo al egula iza ion scheme. Mo e ecen ly, some au ho s ha e gene alized he use o he low empo al de i a i e. Typically, he em- po al in o ma ion is coupled wi h he spa ial g adien in he o m o a non-quad a ic 3D smoo hing ope a- o . Howe e , in (S´ anchez e al., 2012), he au ho s analyze he beha io o a con inuous empo al egu- la ize and show se e al expe imen s whe e i ails. Black (Black, 1994) uses obus unc ionals o deal wi h ou lie s and in oduces a empo al con inu- i y s a egy o accoun o he empo al cohe ence o he sequence. This empo al con inui y is based on a p edic ion s ep and an a achmen o he low o he p edic ed alue. I wa ps he low ield o es ima e i s alue in he ollowing ame. This is in e es ing, be- cause he wa ping allows inding he co ec low co - espondences. Mo e ecen ly, he e has been se e al wo ks dealing wi h empo al cohe ence in di e en ways: o ins ance, in (Sun e al., 2010) he empo al consis ency is es ablished easoning on he segmen a- ion on laye s. We p opose se e al con ibu ions: on he one hand, we in oduce a nonlinea low cons ancy as- sump ion ha i s wi h he nonlinea da a assump- ion; on he o he hand, we p opose a no el non- linea low egula iza ion scheme ha can deal wi h non-con inuous op ical lows. Ano he con ibu ion is a new anis opic di usion ope a o based on he Nagel-Enkelmann ope a o . This new ope a o allows espec ing he objec bounda ies du ing he di usion p ocess, a he same ime ha i a oids o e segmen a- ion in ex u e egions. The o me con ibu ion was mo i a ed by he e- sul s p esen ed in (Salgado and S´ anchez, 2006). The expe imen al esul s showed ha he use o a nonlin- ea empo al o mula ion o he low ield p o ided e y good esul s. Tha was he i s ime ha such a nonlinea low assump ion was in oduced. Fo he second con ibu ion, we in oduce a non-con inuous low egula iza ion scheme a he PDE le el. This is a pu e egula iza ion app oach ha eplaces he adi- ional con inuous empo al smoo hing. In Sec ion 2 we examine he new ene gy model and explain he no el empo al cohe ence s a egy. The minimiza ion o he ene gy model and some nu- me ical de ails a e explained in Sec ion 3. In he ex- pe imen al esul s – Sec ion 4 – we es ou me hod using a syn he ic sequence. Finally he conclusions in Sec ion 5. 2 NONLINEAR VARIATIONAL MODEL I we ha e a se o images Ij(x), wi h j=1, .., N, N he numbe o ames and x= (x,y), he aim is o ind a se o op ical low unc ions, {hi(x)}, wi h i=1, .., N−1. We decompose ou ene gy unc ional in wo sepa a e pa s: E({hi(x)}) = ES({hi(x)})+ET({hi(x)}).(1) The i s e m on he igh , ES, s ands o he spa- ial ene gy model and he second e m, ET, is he en- e gy model co esponding o he empo al cohe ence s a egy. The spa ial model eads as ollows: ES=ZN−1 ∑ i=1 Ψ(Ii(x)−Ii+1(x+hi(x)))2dx +γZN−1 ∑ i=1 Ψk∇Ii(x)−∇Ii+1(x+hi(x))k2dx +αZN−1 ∑ i=1 Ψ(N(∇Ii,∇hi))dx,(2) wi h Ψs2=√s2+ε2(εa p e ixed small con- s an , e.g. 0.01). This kind o unc ion mi iga es he e ec o ou lie s and beha es like TV egu- la iza ion app oaches when used in he smoo hness e m. The ad an age o a TV smoo hing scheme is ha i p ese es discon inui ies o he low. We use he aniso opic di usion ope a o , N(∇Ii,∇hi) = ace∇hT i(x)D(∇Ii)∇hi(x), p oposed in (Nagel and Enkelmann, 1986), which p ese es discon inu- i ies o he images in he low ield, D(.)de ined as: D(∇I) = ∇I⊥T∇I⊥+λ2Id k∇Ik2+2λ2, wi h Id he iden i y ma ix. λde e mines he g adien alue om which he aniso opy is ac i a ed. This pa ame e can be compu ed om he mo e in ui i e iso opic ac ion, 0 ≤s≤1, in oduced in ( ´ Al a ez e al., 2000). Fo he empo al ene gy model, we ollow he ideas p esen ed in (Salgado and S´ anchez, 2006). Gi en ha an objec in he sequence may unde go la ge displacemen s, we ha e o deal wi h in o ma- ion ha is wa ped h ough he lows. In ac , gi en a low hi(x), a ins an i, i s co esponding low in he ollowing ime ins an is hi+1(x+hi(x)). I hi(x)is la ge, hen he empo al de i a i e canno be com- pu ed, bu he p e ious co espondence s ill holds. Thus, one way o ela e mo ion ields a di e en ime ins an s is h ough he low cons ancy assump- ion (FCA), hi(x) = hi+1(x+hi(x)). The e o e, he empo al cohe ence model, ET, can be o mula ed as, ET=βZN−2 ∑ i=1 Φkhi(x)−hi+1(x+hi(x))k2dx, (3) wi h Φs2=e−k∇Ikκ√s2+ε2, wi h κ=0.8 and ε= 0.01. This e m is cong uen wi h he b igh ness and g adien cons ancy e ms. In he p esence o la ge displacemen s, his empo al model is cohe en wi h he spa ial o mula ion and ela es alues a he co - ec posi ions. No e ha when objec displacemen s a e e y small, his e m can be seen as an app oxi- ma ion o he empo al de i a i e o he low, which has shown o be e ec i e in a con inuous se ing (e.g., (Weicke and Schn¨ o , 2001) o (Papenbe g e al., 2006)). 3 MINIMIZING THE ENERGY MODEL In his sec ion we de i e he Eule -Lag ange equa- ions o (2) and (3). Then, we in oduce a nonlinea egula iza ion scheme a he PDE, which closely e- sembles a con inuous empo al smoo hing app oach. The Eule -Lag ange equa ions o he spa ial en- e gy model (2) a e: 0=Ψ0(Ii(x)−Ii+1(x+hi(x)))2 ·(Ii(x)−Ii+1(x+hi(x)))·∇Ii+1(x+hi(x)) +γ Ψ0k∇Ii(x)−∇Ii+1(x+hi(x))k2 ·(∇Ii(x)−∇Ii+1(x+hi(x)))·HIi+1(x+hi(x)) +αdi Ψ0(N(∇Ii,∇hi))·D(∇Ii)·∇hi,(4) whe e HIi+1is he Hessian ma ix. The empo al en- e gy model (3) yields he ollowing Eule -Lag ange equa ions: 0=β Φ0khi(x)−hi+1(x+hi(x))k2 ·(hi(x)−hi+1(x+hi(x)))T ·Id −∇hT i+1(x+hi(x)) +β Φ0 hi(x)−hi−1(x+h∗ i−1(x))  2 ·hi(x)−hi−1(x+h∗ i−1(x))·|J(x)|,(5) whe e |J(x)|s ands o he absolu e alue o he Jaco- bian ma ix, wi h J(x) = 1+u∗ i−1,x1+ ∗ i−1,y− u∗ i−1,y ∗ i−1,x.h∗ i−1=u∗ i−1, ∗ i−1Tis he backwa d low om ame Ii o Ii−1. In o de o de i e (hi(x)−hi+1(x+hi(x))) wi h espec o hi+1(x), we can use he change o a iables z=x+hi−1(x). This change allows us o emo e he nonlinea i y inside he low. The backwa d low, h∗ i−1, na u ally appea s due o his change o a iables. We use a g adien descen app oach o ind he so- lu ion o he abo e PDE. The nonlinea e ms, e.g. Ii+1(x+hi(x)), a e linea ized using i s o de Tay- lo expansions. In he empo al cohe en amewo k, we use Di ichle bounda y condi ions o he las and i s ames, whe eas Neumann bounda y condi ions a e used in he spa ial domain. We use a s anda d coa se- o- ine s a egy o deal wi h la ge displace- men s, based on a py amidal s uc u e. The sys em o equa ions is spa se, so i can be e icien ly sol ed by means o he Gauss-Seidel o SOR me hod in each scale. We in oduce a nonlinea empo al smoo hing scheme. I s o mula ion is in ui i ely de i ed om he second o de empo al de i a i e o he low ield, u ≈ui,j,k+1−2ui,j,k+ui,j,k−1. In he PDE, his sec- ond o de de i a i e has a con inuous empo al egu- la izing e ec ha is consis en i he low ield a ies smoo hly ac oss he image sequence. We p opose a new solu ion, which is simila in spi i o his nume - ical app oxima ion, and is sui able o dealing wi h non-con inuous displacemen s. This is a nonlinea o mula ion ha pu s in o co espondence he co ec low alues in di e en ames. I is no e iden how o abs ac his idea a he ene gy le el in Equa ion (3). As be o e, we also use L1 unc ions o u n he me hod mo e obus agains ou lie s, in he ollowing way: TS=δ Φ0 hi−1(x+h∗ i−1(x))−hi+1(x+hi(x))  2 ·hi−1(x+h∗ i−1(x))−2hi(x)+hi+1(x+hi(x)) (6) This e m p o ides a new scheme a he PDE le el and has o be combined wi h he p e ious PDE equa- ions (4) and (5). In he expe imen s, we show ha his nonlinea smoo hing p o ides e y good esul s: i has a simila gain as in he con inuous case, bu i co ec ly handles la ge discon inui ies in he mo ion ield. 4 EXPERIMENTAL RESULTS Nex we examine he beha io o he empo al mod- els in oduced in equa ions (1) and (6). Fo his, we use a simple sequence o a squa e ansla ing o e a ex u ed backg ound. The squa e is mo ing 15 pixels pe ame, while he backg ound mo es 3 pixels in he same di ec ion. In he i s ow o Fig. 1, we show he hi d ame o he squa e sequence, i s g ound u h, and he bes spa ial solu ion ound. In he second ow, we show h ee empo al solu ions: he i s o he nonlinea empo al a achmen de ined in (3); he sec- ond, o he nonlinea empo al smoo hnes app oach de ined in (6); and, inally, using bo h empo al e ms. The colo , in he mo ion ield, ep esen s he di ec ion and, he in ensi y, i s magni ude. Figu e 1: Squa e sequence. Fi s ow: one o he images o he Squa e sequence, he g ound u h and he bes spa ial solu ion ound. Second ow: h ee empo al solu ions wi h β=8, δ=25 and (β=1,δ=25), espec i ely. The imp o emen o he empo al me hods wi h espec o he spa ial solu ion is impo an . As ex- pec ed, he spa ial me hod p oduces highe e o s a he mo ion discon inui ies and, mo e signi ican ly, a he occlusions. Table 1 shows he a e age End-poin (EPE) and Angula (AAE) e o s o hese esul s. The i s empo al esul , co esponding o he i s im- age in he second ow o Fig. 1, p o ides an impo an imp o emen on he EPE and, mo e no iceable, on he AAE. The imp o emen in accu acy is s ill mo e im- po an i we use he nonlinea empo al smoo hing scheme (Equa ion (6)) o a combina ion o bo h. We obse e ha he nonlinea empo al smoo h- ing scheme (6) beha es be e han he empo al a - achmen , e en a he mo ion bounda ies. The g aph- ics in Fig. 2 show he EPE o e e y ame on he Table 1: EPE and AAE o he Squa e sequence. Me hod EPE AAE Spa ial 0.071 0.629o Tempo al 1 (β=8) 0.049 0.204o Tempo al 2 (δ=25) 0.036 0.134o Tempo al 3 (β=1,δ=25) 0.035 0.138o squa e sequence. F ame by ame, he op ical lows a e mo e accu a e in he empo al me hods. We also obse e ha he esul s a e e y s able, especially in he middle o he ’Tempo al 2 (δ)’ line. Reasonably, he ames a he beginning and end o he sequence p esen highe e o s, due o he Di ichle bounda y condi ions. Figu e 2: EPE in each op ical low o he Squa e sequence. 5 CONCLUSIONS In his pape we ha e p esen ed a new spa io- empo al cohe ence model o he consis en es ima ion o op- ical lows. We ha e ocused on di e en nonlinea low assump ions ha a e mo e con iden in he es- ima ion o mo ion ields han p e ious app oaches. These nonlinea assump ions co ec ly i wi h he s anda d nonlinea b igh ness and g adien cons ancy e ms, can cope wi h gene al image sequences and p o ide be e solu ions. In pa icula , we ha e p o- posed wo main con ibu ions: on he one hand, we ha e in oduced he nonlinea low cons ancy assump- ion (FCA) in he ene gy model. This e m ela es low ields a di e en ime ins an s and is consis- en wi h he es o he ene gy e ms. On he o he hand, we ha e p oposed a nonlinea empo al di u- sion scheme a he PDE le el, which p oduces con- inuous lows in ime. We ha e seen ha his new scheme is mo e gene al han using he con inuous empo al egula iza ion o he low, wi h he ad an- age ha i con enien ly deals wi h con inuous and non-con inuous eloci ies. In ac , i he mo ion is e y small, his e m app oxima es a con inuous em- po al smoo hing scheme. In he expe imen al esul s, we ha e shown ha he me hod p o ides impo an accu acy imp o emen s, specially in he p esence o la ge displacemen s. The esul s a e p omising in bo h cases, al hough we obse e a be e pe o mance o he nonlinea empo al smoo hing scheme in gene al. Ano he in e es ing esul o he empo al cohe ence schemes is ha he backg ound mo ion oscilla ions end o disappe . These oscilla ions clea ly appea in he spa ial me hod, in egions whe e he e is no appa - en mo ion. ACKNOWLEDGEMENTS This wo k has been pa ly ounded by he Spanish Minis y o Science and Inno a ion h ough he e- sea ch p ojec TIN2011-25488. REFERENCES ´ Al a ez, L., Weicke , J., and S´ anchez, J. (2000). Reli- able es ima ion o dense op ical low ields wi h la ge displacemen s. In e na ional Jou nal o Compu e Vi- sion, 39(1):41–56. Black, M. J. (1994). Recu si e non-linea es ima ion o dis- con inuous low ields. In P oceedings o he hi d Eu- opean con e ence on Compu e ision ( ol. 1), ECCV ’94, pages 138–145, Secaucus, NJ, USA. Sp inge - Ve lag New Yo k, Inc. Nagel, H. H. and Enkelmann, W. (1986). An in es iga ion o smoo hness cons ain s o he es ima ion o dis- placemen ec o ields om image sequences. IEEE T ansanc ions on Pa e n Analysis and Machine In el- ligence, 8:565–593. Papenbe g, N., B uhn, A., B ox, T., Didas, S., and Weick- e , J. (2006). Highly Accu a e Op ic Flow Compu- a ion wi h Theo e ically Jus i ied Wa ping. In e na- ional Jou nal o Compu e Vision, 67(2):141–158. Salgado, A. and S´ anchez, J. (2006). A empo al egula ize o la ge op ical low es ima ion. In IEEE In e na- ional Con e ence on Image P ocessing ICIP, pages 1233–1236. S´ anchez, J., Monz´ on, N., and Salgado, A. (2012). Robus op ical low es ima ion. IPOL: Image P ocessing On- line, P ep in :1–18. Sun, D., Sudde h, E., and Black, M. J. (2010). Lay- e ed Image Mo ion wi h Explici Occlusions, Tempo- al Consis ency, and Dep h O de ing. In La e y, J., Williams, C. K. I., Shawe-Taylo , J., Zemel, R., and Culo a, A., edi o s, Ad ances in Neu al In o ma ion P ocessing Sys ems 23, olume 23, pages 2226–2234. Weicke , J. and Schn¨ o , C. (2001). Va ia ional Op ic Flow Compu a ion wi h a Spa io-Tempo al Smoo h- ness Cons ain . Jou nal o Ma hema ical Imaging and Vision, 14(3):245–255.