Di ec Es ima ion o he Backwa d Flow
Ja ie S´
anchez, Agus ´
ın Salgado and Nelson Monz´
on
Cen o de Tecnolog´
ıas de la Imagen (CTIM)
Depa amen o de In o m´
a ica y Sis emas
Uni e si y o Las Palmas de G an Cana ia, Spain
{jsanchez, asalgado}@dis.ulpgc.es, [email p o ec ed]
Keywo ds: Backwa d Flow, In e se Op ical Flow, Back Regis a ion, In e se Mapping, Op ical Flow, Image Regis a ion,
Occlusions, Disocclusions.
Abs ac : The aim o his wo k is o p opose a new me hod o es ima ing he backwa d low di ec ly om he op ical
low. We assume ha he op ical low has al eady been compu ed and we need o es ima e he in e se mapping.
This mapping is no bijec i e due o he p esence o occlusions and disocclusions, he e o e i is no possible o
es ima e he in e se unc ion in he whole domain. Values in hese egions has o be guessed om he a ailable
in o ma ion. We p opose an accu a e algo i hm o calcula e he backwa d low uniquely om he op ical low,
using a simple ela ion. Occlusions a e illed by selec ing he maximum mo ion and disocclusions a e illed
wi h wo di e en s a egies: a min- ill s a egy, which ills each disoccluded egion wi h he minimum alue
a ound he egion; and a es ic ed min- ill app oach ha selec s he minimum alue in a close neighbo hood.
In he expe imen al esul s, we show he accu acy o he me hod and compa e he esul s using hese wo
s a egies.
1 INTRODUCTION
In his a icle we add ess he p oblem o es ima ing
he backwa d low. The op ical low is calcula ed
om he sou ce o he a ge image and he back-
wa d low is he in e se mapping om he a ge o
he sou ce image. I we know he o wa d low, hen
i is possible o es ima e he backwa d co espondence
wi h some limi a ions.
This is impo an in p oblems whe e i is necessa y
o ind he co espondences back in ime. This is he
case, o ins ance, in symme ic op ical low me hods,
e.g. (Ch is ensen and Johnson, 2001), ( ´
Al a ez e al.,
2007b), (Ashbu ne , 2007) o (Yang e al., 2008).
These me hods in oduce he in e se op ical low o
imp o e he cohe ence be ween he o wa d and back-
wa d lows.
In (Ch is ensen and Johnson, 2001), o ins ance,
he au ho s p esen a me hod o compu ing he image
egis a ion o medical images, which elies explici ly
in he compu a ion o he in e se mapping. I wo ks
o di eomo phic ans o ma ions, whe e he ela ion
is bijec i e and di e en iable. The backwa d low is
calcula ed using an i e a i e algo i hm, ne e heless,
his i e a i e p ocess may slow down he me hod and
i does no wo k in occluded o disoccluded egions.
In he symme ic me hod p esen ed in (Cachie
and Rey, 2000), he in e se op ical low is compu ed
using a New on scheme. This solu ion is simila o
he p e ious i e a i e p ocess, so i p esen s he same
d awbacks, p o iding poo esul s in occluded and
disoccluded egions. Ano he in e es ing symme ic
model is p oposed in ( ´
Al a ez e al., 2007a). In his
case, he op ical low is compu ed as a unc ion in he
middle posi ion be ween wo ames, so i does no
compu e he in e se op ical low explici ly.
The backwa d low has also been used in spa io-
empo al op ical low me hods, e.g. (Salgado and
S´
anchez, 2006) o (S´
anchez e al., 2013). The objec-
i e is o p ese e he empo al cohe ence o he op-
ical lows wi h he p e ious es ima ed lows: he in-
e se op ical low is used o ind he co espondences
back in ime and impose some kind o empo al con-
inui y.
Ano he example o he use o he backwa d low
is gi en in (Lieb e al., 2005) and (Lookingbill e al.,
2007). In his case, he au ho s p opose a me hod ha
elies on he e e se op ical low o au oma ically ol-
low he oad in au onomous ca s. I acks ea u es
om he cu en posi ion o a pas posi ion, so he
obo may iden i y simila shapes a di e en scales.
We p opose a new me hod o es ima ing he back-
wa d low di ec ly om he op ical low. We a e gi en
he o wa d low and we a e in e es ed in compu ing
he in e se op ical low wi h high p ecision. No -
mally, his is no an in e se unc ion because he e a e
egions, like occlusions and disocclusions, whe e i is
no possible o es ablish a co espondence.
The p oposed algo i hm is simple, as and ac-
cu a e. I au oma ically handles occlusions, whe eas
disocclusions a e illed using wo dis inc app oaches:
on he one hand, we use a min- ill s a egy, ha asso-
cia es he minimum alue a ound he egion; on he
o he hand, we p opose a es ic ed min- ill app oach,
ha ills disocclusions wi h he minimum low alue
a a gi en dis ance.
In he expe imen al esul s, we s udy he p e-
cision o ou me hod using syn he ic s anda d se-
quences om he Middlebu y benchma k da abase
(Bake e al., 2007). The esul s show ha he accu-
acy o his me hod is high. We compa e be ween he
solu ions ob ained by he wo illing s a egies. The
es ic ed min- ill app oach p o ides be e le el o
accu acies when disoccluded egions a e la ge.
In Sec ion 2, we explain he basis o es ima -
ing he backwa d low. The algo i hm is designed in
Sec ion 3 and he s a egies o ill disocclusions a e
explained in Sec ion 4. In he expe imen al esul s,
Sec ion 5, we e alua e he algo i hm using se e al
sequences om he Middlebu y benchma k da ase s.
Finally, he conclusions in Sec ion 6.
2 ESTIMATING THE BACKWARD
FLOW
The op ical low is he appa en mo ion o he objec s
in a sequence o images. This is gi en by a ec o
ield, h(x)=(u(x), (x)), ha pu s in co espondence
he pixels o a sou ce and a a ge image. The back-
wa d low, h∗(x) = (u∗(x), ∗(x)), is he in e se map-
ping om he a ge o he sou ce image. The ela ion
be ween he backwa d and o wa d op ical lows is
gi en by
h(x) = −h∗(x+h(x)).(1)
This ela ion can be in ui i ely de i ed om he
g aphic depic ed in Fig. 1: i we ollow he pa h o
he o wa d and backwa d lows, we should a i e o
he ini ial posi ion. This is ue excep in occluded
and disoccluded egions.
As depic ed in Fig. 2, disocclusions appea in he
back side o he mo ing objec s, p oduced by emp y
egions whe e no co espondences can be es ablished.
Occlusions appea in he on side o he mo ing
objec s, whe e se e al co espondences a i e o he
same posi ion. These wo p oblems a e easy o de ec
Figu e 1: Backwa d low.
bu hei solu ion ha e o be o e come om di e en
pe spec i es.
Figu e 2: Occlusions and disocclusions. When he blue
squa e mo es ho izon ally, i c ea es a disocclusion and oc-
clusion be o e and a e he squa e, espec i ely.
The disocclusion p oblem can be add essed as a
illing p ocedu e, since he in o ma ion is only a ail-
able om he neighbo s. Occlusions can be ega ded
as a selec ion p oblem, whe e se e al alues a e as-
signed o he same posi ion, and we need o selec
he app op ia e one. Occlusions occu because an ob-
jec mo es in on o o behind o he objec s. In his
wo k we deal wi h occlusions due o objec s mo ing
in on o o he objec s. This is a simple case ha
only depends on he ec o ield. The case, in whe e
an objec mo es behind o he objec s, is no so simple
and should ely on he image in ensi ies as well.
3 ALGORITHM
Algo i hm 1 shows he s eps o compu e he backwa d
low. This algo i hm is simple and e y as : only
one pass o e he image is necessa y o compu e he
low. The inpu is he o wa d low. A he beginning,
we ini ialize he backwa d low o a high numbe . In
each pixel, x= (x,y), we compu e he co esponding
posi ion in he o he image, x+h(x).
We ob ain he ou neighbo s a ound his posi ion,
gi en by (x+/−,y+/−), and he in e pola ion weigh s,
w1,w2,w3,w4. These a iables a e shown in Fig. 3.
These weigh s ep esen he p opo ional a ea o he
pixel ha co esponds o each neighbo . Then, we
es ima e he magni ude o he o wa d low and com-
pa e i wi h he magni ude o he backwa d lows in
he neighbo hood.
Algo i hm 1: Backwa d low es ima ion
Inpu :u,
Ou pu :u∗, ∗
Ini ialize u∗, ∗ o a big numbe
o i←1 o sizeydo
o j←1 o sizexdo
x←j+u(j,i)
y←i+ (j,i)
Find he ou neighbo s o
(x,y):(x+/−,y+/−)
Compu e he bilinea in e pola ion
weigh s: w1,w2,w3,w4
d←u(j,i)2+ (j,i)2
d1←u∗(x−,y−)2+ ∗(x−,y−)2
d2←u∗(x+,y−)2+ ∗(x+,y−)2
d3←u∗(x−,y+)2+ ∗(x−,y+)2
d4←u∗(x+,y+)2+ ∗(x+,y+)2
i w1≥0,25 and d≥d1 hen
u∗(x−,y−)← −u(j,i)
∗(x−,y−)← − (j,i)
end
i w2≥0,25 and d≥d2 hen
u∗(x+,y−)← −u(j,i)
∗(x+,y−)← − (j,i)
end
i w3≥0,25 and d≥d3 hen
u∗(x−,y+)← −u(j,i)
∗(x−,y+)← − (j,i)
end
i w4≥0,25 and d≥d4 hen
u∗(x+,y+)← −u(j,i)
∗(x+,y+)← − (j,i)
end
end
end
Fill disocclusions
I he alue o he o wa d magni ude is bigge
han he p e ious s o ed backwa d low, and he co e-
sponding weigh is bigge han a gi en h eshold, hen
we keep he nega i e alue o he low a ha posi ion.
This h eshold has been se o 0,25 because i ep e-
sen s he si ua ion whe e he co espondence alls in
he middle o he pixel.
In his way, he occlusions a e au oma ically han-
dled by he algo i hm: i he e a e collisions in one
posi ion, we e ain he low wi h highe magni ude.
This is sui able when he occlusions a e p oduced by
he as es mo ing objec s.
Figu e 3: No a ion.
Al hough his is no he mos gene al case, i is in-
e es ing because he e a e many sequences in which
his assump ion holds. Also no e ha his algo i hm
is e y simple and as , and he s o age equi emen s
a e e y low, since all he in o ma ion is s o ed in he
same ou pu a ays. In he las s ep, he algo i hm
ills disocclusions. These occu whe e no co espon-
dences ha e been ound, hus i is he opposi e si ua-
ion o he occlusions.
4 FILLING DISOCCLUSIONS
In o de o ill disocclusions, we ha e o look o he
alues a ound he egion. No mally, disocclusions
happen because a mo ing objec le us see he back-
g ound. The mo ion in his egion canno be dis-
co e ed, unless we make some simple assump ions.
The minimum low assump ion es ablishes ha a good
guess is he minimum alue a ound he disocclusion.
In his sense, we p opose wo s a egies.
4.1 Min- ill s a egy
This s a egy ies o ill disocclusions wi h he min-
imum alue a ound he egions. This can be accom-
plished in he ollowing s eps: i s , disocclusions a e
g ouped in egions by means o a connec ed compo-
nen labeling p ocess (Suzuki e al., 2003).
The secod s ep consis s in associa ing a minimum
alue o each label: we go o e he image and, any
ime we ind a disocclusion, we sea ch o he mini-
mum alue in i s neigbo hood and compa e wi h he
accumula ed minimum o i s co esponding label.
Once we ha e ob ained he minimum alue o
each label, we go o e he image again and assign he
co esponding alue o each disocclusion. This s a -
egy is simple and as . I wo ks co ec ly i he size o
he egions a e small.
4.2 Res ic ed min- ill s a egy
A simple s a egy ha wo ks p ope ly when egions
a e e y la ge, is he es ic ed min- ill s a egy. The
idea is o ind he minimum alue ha is nea he cu -
en posi ion. Fo each disocclusion, we selec he
minimum alue in a window.
This p ocess is ca ied ou i e a i ely un il e e y
disocclusion is illed: we go o e he image and y
o ill disocclusions using a ix-sized window. The
size o he window may no be la ge enough o a ain
alues ou side he egion. In his case, he p ocess
is un again o ill he emaining disocclusions. The
de aul window adius is 5 in he expe imen al esul s.
5 EXPERIMENTAL RESULTS
In he i s expe imen , Fig. 4, we use he Yosemi e
sequence. Fig. 5 shows he colo scheme used o
ep esen he o ien a ion and magni ude o he ec o
ields.
The a e age unning imes o he expe imen s is
abou 0.020 seconds in a PC In el Co e i5 wi h wo
p ocesso s and 8,00 GB RAM. These a e age imes
include he ime o ead he inpu da a and w i e he
ou pu o disk.
Figu e 5: Colo scheme.
In o de o e alua e he me hod, we compu e he
backwa d low wice, (h∗)∗, so ha we a i e o
he o iginal g ound u h. Then, we compa e wi h
he o iginal g ound u h using he a e age end-poin
(EPE) and angula (AAE) e o s (Bake e al., 2007),
o bo h s a egies. No e ha he es ima ed e o mea-
su e may be di ided by wo o accoun o he ac ual
Min- ill Res . min- ill
Sequence EPE AAE EPE AAE
Yosemi e 0.084 0.529o0.012 0.307o
Table 1: EPE and AAE o he Yosemi e sequence.
Min- ill Res . min- ill
Sequence EPE AAE EPE AAE
G o e2 0.035 0.560o0.026 0.417o
G o e3 0.295 3.258o0.170 1.466o
U ban2 0.294 1.547o0.086 0.345o
U ban3 0.371 2.465o0.151 1.068o
Venus 0.069 0.793o0.026 0.371o
Table 2: EPE and AAE o he Middlebu y sequences.
backwa d low e o . These esul s a e shown in Ta-
ble 1. We obse e ha he es ic ed min- ill s a egy
p o ides much be e esul s han he min- ill s a egy.
In Fig. 6, we show se e al esul s using he Mid-
dlebu y benchma k da abase. We ha e used se e al
es sequences o which he g ound u hs a e known.
In he hi d column o Fig. 6, we show he com-
pu ed backwa d low wi h disocclusions highligh ed
in whi e. The o h and i h columns con ain he solu-
ions o he min- ill and he es ic ed min- ill s a e-
gies, epec i ely.
We obse e ha disoccluded egions a e e y la ge
in some examples. Fo ins ance, in he las sequence,
he e is a la ge disocclusion in he bo de o he pape .
When his egion is illed wi h he minimum alue –
min- ill s a egy – we obse e a low mo ion (black
colo ), which is no consis en wi h he compu ed
backwa d low a ound he egion. The es ic ed min-
ill s a egy seems o p o ide be e esul s o his se-
quence.
In Table 2, we show he EPE and AAE o he
Middlebu y sequences. We obse e again ha he e-
s ic ed min- ill s a egy a ains be e esul s han he
basic min- ill s a egy.
6 CONCLUSIONS
In his wo k, we ha e p esen ed a e y accu a e
me hod o es ima ing he backwa d low. This
me hod only elies on he op ical low and can di ec ly
deal wi h occlusions. On he o he hand, in o de o
ill dissoclusions, we ha e p oposed wo al e na i es,
based on he minimum low.
The algo i hm is e y as , since i is only neces-
sa y o ca y ou one pass o e he image. This allows
us o each eal- ime p ocessing in cu en compu e s,
wi hou he need o in oduce any code pa alleliza ion.
In he expe imen al esul s, we ha e nume ically
e alua ed he new me hod and compa ed be ween he
wo illing app oaches. These esul s show ha he
me hod is e y accu a e and ha he es ic ed min-
ill app oach p o ides be e esul s.
The algo i hm does no ake in o accoun occlu-
sions due o he mo emen o objec s behind o he
s a ic objec s. In his case, he in o ma ion o he op-
ical low is no su icien and mo e in o ma ion om
he images is necessa y. This will be add essed in a
u u e wo k.
ACKNOWLEDGEMENTS
This wo k has been pa ly ounded by he Spanish
Minis y o Science and Inno a ion h ough he e-
sea ch p ojec TIN2011-25488.
REFERENCES
´
Al a ez, L., Cas a˜
no, C. A., K issian, K., Mazo a, L., Sal-
gado, A. J., and S´
anchez, J. (2007a). Symme ic Op-
ical Flow. In Mo eno D´
ıaz, R., Pichle , F., and Que-
sada A encibia, A., edi o s, Compu e Aided Sys ems
Theo y - EUROCAST 2007, olume 4739 o Lec u e
No es in Compu e Science, pages 676–683. Sp inge
Ve lag, Heidelbe g.
´
Al a ez, L., De iche, R., Papadopoulo, T., and S´
anchez, J.
(2007b). Symme ical dense op ical low es ima ion
wi h occlusions de ec ion. In e na ional Jou nal o
Compu e Vision, 75(3):371–385.
Ashbu ne , J. (2007). A as di eomo phic image egis a-
ion algo i hm. Neu oImage, 38(1):95–113.
Bake , S., Scha s ein, D., Lewis, J. P., Ro h, S., Black, M. J.,
and Szeliski, R. (2007). A da abase and e alua ion
me hodology o op ical low. In In e na ional Con-
e ence on Compu e Vision, pages 1–8.
Cachie , P. and Rey, D. (2000). Symme iza ion o he non-
igid egis a ion p oblem using in e sion-in a ian
ene gies: Applica ion o mul iple scle osis. In Delp,
S., DiGoia, A., and Ja amaz, B., edi o s, Medical Im-
age Compu ing and Compu e -Assis ed In e en ion
MICCAI 2000, olume 1935 o Lec u e No es in Com-
pu e Science, pages 472–481. Sp inge Be lin Hei-
delbe g.
Ch is ensen, G. and Johnson, H. (2001). Consis en image
egis a ion. IEEE T ansac ions on Medical Imaging,
20(7):568 –582.
Lieb, D., Lookingbill, A., and Th un, S. (2005). Adap i e
oad ollowing using sel -supe ised lea ning and e-
e se op ical low. In P oceedings o Robo ics: Sci-
ence and Sys ems, Camb idge, USA.
Lookingbill, A., Roge s, J., Lieb, D., Cu y, J., and Th un,
S. (2007). Re e se op ical low o sel -supe ised
adap i e au onomous obo na iga ion. In e na ional
Jou nal o Compu e Vision, 74:287–302.
Salgado, A. and S´
anchez, J. (2006). A empo al egula ize
o la ge op ical low es ima ion. In IEEE In e na-
ional Con e ence on Image P ocessing ICIP, pages
1233–1236.
S´
anchez, J., Salgado, A., and Monz´
on, N. (2013). Op-
ical low es ima ion wi h consis en spa io- empo al
cohe ence models. In 8 h In e na ional Con e ence on
Compu e Vision Theo y and Applica ions (VISAPP).
Suzuki, K., Ho iba, I., and Sugie, N. (2003). Linea - ime
connec ed-componen labeling based on sequen ial lo-
cal ope a ions. Compu e Vision and Image Unde -
s anding, 89(1):1–23.
Yang, D., Li, H., Low, D. A., Deasy, J. O., and Naqa,
I. E. (2008). A as in e se consis en de o mable im-
age egis a ion me hod based on symme ic op ical
low compu a ion. Physics in Medicine and Biology,
53(21):6143.

Sequence G ound u h Backwa d low Min- ill Res ic. min- ill
Figu e 4: Resul s o he Yosemi e sequence. Fi s column, he sou ce image; second column, he g ound u h; hi d column,
he compu ed backwa d low wi hou illing disocclusions (whi e colo ); ou h column, he backwa d low wi h he min- ill
s a egy; and, i h column, he backwa d low wi h he es ic ed min- ill s a egy.
Sequence G ound u h Backwa d low Min- ill Res ic. min- ill
Figu e 6: Resul s o he Middlebu y es sequences. Fi s column, he sou ce image; second column, he g ound u h op ical
low; hi d column, he compu ed backwa d low wi hou illing disocclusions; ou h column, he backwa d low wi h he
min- ill s a egy; and, i h column, he backwa d low wi h he es ic ed min- ill s a egy.