Pano amic Backg ound Modeling o PTZ Came as wi h Compe i i e Lea ning
Neu al Ne wo ks
Ka l Thu nho e -Hemsi∗, Ezequiel L´
opez-Rubio∗, En ique Dom´
ınguez∗,
Ra ael Ma cos Luque-Baena∗and Miguel A. Molina-Cabello∗
∗Depa men o Compu e Languages and Compu e Science
Uni e si y o M´
alaga, Bule a Louis Pas eu , 35, 29071 M´
alaga, Spain
Emails: {ka lkhade ,ezeql ,en iqued, mluque,miguelangel}@lcc.uma.es
Abs ac —The cons uc ion o a model o he backg ound o a
scene s ill emains as a challenging ask in ideo su eillance
sys ems, in pa icula o mo ing came as. This wo k p esen s
a no el app oach o cons uc ing a pano amic backg ound
model based on compe i i e lea ning neu al ne wo ks and a
subsequen piecewise linea in e pola ion by Delaunay iangu-
la ion. The app oach can handle a bi a y came a di ec ions
and zooms o a Pan-Til -Zoom (PTZ) came a-based su eil-
lance sys em. A e es ing he p oposed app oach on se e al
indoo sequences, he esul s demons a e ha he p oposed
me hod is e ec i e and sui able o use o eal- ime ideo
su eillance applica ions.
1. In oduc ion
Wi h hei high mobili y and zoom capabili y, pan- il -
zoom (PTZ) came as ha e become inc easingly popula
in moni o ing public a eas [1], [2], [3]. Omnidi ec ional
came as a e p omising candida es o moni o ing mul iple
la en ac i i ies in he a ea o in e es [4]. Howe e , hese
kind o came as ha e nonuni o m esolu ion and a e unable
o p o ide close obse a ions o pa icula a ge s. In hese
cases, whe e PTZ came as a e needed, he combina ion o
hese wo ypes o came as (omnidi ec ional and PTZ) is
p oposed in o de o acili a e a con inuous moni o ing o he
whole su eillance a ea and de ailed obse a ions o speci ic
a ge s simul aneously [5]. Ne e heless, his dual-came a
sys em may be s ill an expensi e and complex solu ion
in some scena ios. Fo his eason, in his pape we a e
ocusing on an ac i e sensing app oach o mul iple objec
de ec ion and acking using a single PTZ came a.
Mos came as used in su eillance a e s a ic, and he
scenes aken om his ype o came as only show one spe-
ci ic iew o he su eillance a ea. Fo hese images/ ideos,
he mos common and e icien app oach o mo ing objec
de ec ion is backg ound sub ac ion, ha consis s in main-
aining an up- o-da e model o he ixed backg ound and
de ec ing mo ing objec s as hose ha de ia e om such
model. Compa ed o o he app oaches, such as op ical low,
his app oach is compu a ionally a o dable o eal- ime
applica ions, is independen o mo ing objec eloci y, and
is no subjec o he o eg ound ape u e p oblem.
Howe e , adi ional backg ound sub ac ion algo i hms
assume he came as a e s a ic and his leads o alse de ec-
ion when he came a mo es [6], [7]. Due o his came a
mo emen , e en pixels belonging o s a ic objec s appea o
mo e in he came a ame (called ego-mo ion e ec ).
Ex ensi e esea ch has been ca ied ou ega ding objec
de ec ion o mo ing came as. Some p oposal a e based
on he op ical low clus e ing, ha consis s in calcula ing
dense o spa se op ical lows and clus e ing hem o iden i y
mo ing objec egions [8]. Ano he me hods a e based on he
es ima ion o he ans o ma ion pa ame e s be ween consec-
u i e ames [9]. Ou app oach is based on mosaicing he
backg ound [10], [11], ha consis in c ea ing a mosaiced o
pano amic backg ound image and hen using a backg ound
sub ac ion echnique o ex ac mo ing objec egions.
The p oblem o mo ing objec s de ec ion o PTZ cam-
e as is add essed in his pape , and we p opose a me hod
based on building a pano amic backg ound model using
a compe i i e neu al ne wo k. Apa o he adi ional and
equen ly ci ed seminal pape s ela ed o compe i i e lea n-
ing [12], [13], [14], ecen success ul applica ions in he
compu e ision ield can be ound in he li e a u e [15],
[16], [17], [18]. In ou app oach, a compe i i e neu al ne -
wo k is used o build a pano amic backg ound model o
objec de ec ion. Due o he huge inpu in o ma ion, a la ge
numbe o neu ons has been used and he neu on p o o ypes
ha e been o ganized in a quad- ee in o de o be quickly
e alua ed.
The es o he pape is o ganized as ollows. In sec ion
2, a mo e de ailed desc ip ion o he p oposed neu al model
is p esen ed. In sec ion 3, we p esen he esul s achie ed
wi h he implemen a ion o he p oposed app oach. Finally,
sec ion 4 includes some concluding ema ks.
2. The model
In his sec ion a compe i i e lea ning based sys em o
lea n he backg ound o a pano amic scene om he inpu
o a PTZ came a is p oposed. Fi s he da a acquisi ion
p ocedu e o ans o m he inpu ideo ames in o inpu
samples o he compe i i e lea ning ne wo k is conside ed
(Subsec ion 2.1). Then he compe i i e lea ning model is de-
sc ibed (Subsec ion 2.2). Finally, an in e pola ion p ocedu e
978-1-5090-6182-2/17/$31.00 ©2017 IEEE 396
is designed o es ima e he backg ound om he inal s a e
o he compe i i e lea ning ne wo k, which is based on a
Delaunay iangula ion (Subsec ion 2.3).
2.1. Da a acquisi ion
The da a om which we s a a e he acqui ed ideo
ames o he PTZ came a. These a e ideo ames wi h
ixed wid h and heigh , and e e y pixel has wo ame
coo dina es (x, y). To ans o m hese ame coo dina es o
ano he coo dina e sys em in he pano amic image, he i s
s ep is o ob ain he sphe ical coo dina es associa ed o he
ame coo dina es and hen ca y ou a scala ans o ma ion
o he dimensions o he pano amic image.
In o de o ind ou equi ed pola coo dina es (θ,φ) o
an a bi a y poin Abelonging o p ojec ion plane (x, y)
a pinhole came a model is used, consis ing o a sphe e
cen e ed on he coo dina e o igin and a p ojec ion plane
(see Fig. 1a), whose bounds a e he wid h and heigh o he
ame window in pixels: w,h. In his wo k he i ual PTZ
lib a y [19] is employed, so we mus ollow he coo dina e
sys em c i e ia used by ha lib a y. We s a wi h he case o
came a o ien a ion (pan, il ) = (0,0), which co esponds
o see he posi i e Z axis om he o igin. The gene al case,
when pan 6= 0, il 6= 0 is ob ained om he pa icula case
by means o a coo dina e sys em ans o ma ion.
The coo dina es o ec o OA in coo dina e sys em O
can be ound as p ojec ions on he coo dina e axes:
( x, y, z)=(x−w/2, h/2−y, )(1)
whe e z = , since poin s Aand Ca e loca ed on he same
plane. x and y is calcula ed aking in accoun ha Aon
he p ojec ion plane is based ela i ely o he le op co ne
o he ame. Radius is calcula ed using he e ical ield
o iew (F OV ), which is known. We only need o no ice
ha
FOC =
FOV /2, and F OC is a igh iangle, so
an(
FOV /2) = FC
⇔ =h/2
an(
FOV /2) (2)
This way he came a coo dina es a e compu ed. In o de
o ex end he pa icula case o he gene ic case and ob ain
he wo ld coo dina es we need o ecalcula e ec o OC
om he Ocoo dina e sys em in o he o a ed coo dina e
sys em O0. The coo dina es a e ound by means o mul ipli-
ca ion by he in e se o a double o a ion ma ix. This ma ix
can be ound by mul iplica ion o wo o a ion ma ices:
a ound Xand Zaxes by (pan, il ):
R=

1 0 0
0 cos( il )−sin( il )
0 sin( il ) cos( il )
·
·

cos(pan)−sin(pan) 0
sin(pan) cos(pan) 0
0 0 1 
(3)
A0= ( x0, y0, z0) = R−1·( x, y, z)(4)
Now, when he new coo dina es A0a e compu ed, we
only need he sphe ical coo dina es ans o ma ion o ob ain
θand φ:



x0=Rcos(φ) sin(θ)
y0=Rsin(φ) sin(θ)
z0=Rcos(θ)
(5)
By ea anging he o mulas:
(φ= a c an( y0
x0)
θ= a ccos( z0
R) = a ccos( z0
√ x02+ y02+ z02)(6)
The ob ained samples a e eal alued ec o s, whose wo
i s componen s a e he sphe ical coo dina es φand θo a
pixel, while he hi d las ones a e he obse ed RGB colo
alues a ha pixel:
x= (θ, φ, , g, b)=(x1, x2, x3, x4, x5)(7)
so ha x∈R5. A each ime ins an , a new episode S ⊂
R5is acqui ed which con ains one sample o each pixel o
he incoming ideo ame a ime . The e o e he ca dinal
o S is he numbe o pixels in he ideo ame.
2.2. Compe i i e lea ning ne wo k
Nex a compe i i e lea ning model o pano amic scenes
cap u ed by PTZ came as is de eloped. In o de o lea n
he de ails o he scene, a la ge numbe o neu ons N
is employed, which is lowe bu in he same o de o
magni ude as he numbe o pixels o he ull pano ama.
Following he s a egy in [20], he inpu ec o s x∈R5
a e di ided in o wo sec ions. The i s sec ion con ains
he posi ional in o ma ion in he ideo ame, while he
second sec ion con ains he colo ea u es. In ou case,
he i s sec ion comp ises he wo sphe ical coo dina es φ
and θ, while he second sec ion con ains he RGB colo
alues. All he componen s o he inpu ec o s a e used
o upda e he neu on p o o ypes wi∈R5, bu only he
wo i s ones pa icipa e in he compe i ion. This way, only
he posi ional in o ma ion is employed o de e mine which
neu on is he winne . The e o e, each neu on ep esen s he
a e age colo in a ecep i e ield which is one egion o a
Vo onoi esella ion o he ull pano ama. Tha is, he neu ons
specialize on small pixel neighbo hoods o he pano ama.
Since he posi ional componen s can ha e ac ional alues,
(x1, x2)∈R2, he ne wo k can lea n ac ional posi ional in-
o ma ion wi hou ha ing o ound o in ege pixel posi ions
in he pano ama, which a oids losing aluable in o ma ion.
Each neu on con ains a p o o ype wi∈R5and also
a Boolean lag bi∈ { ue, alse}. The lag is equi ed
o con ol he ini ializa ion o he neu on. Since he PTZ
came a does no co e he en i e pano ama a a ime, i is no
possible o ini ialize all he neu ons a he same ime. They
can only be ini ialized while hei posi ional componen s all
in o he cu en ield o iew. To his end, he Boolean lags
a e ini ialized o alse. The i s ime ha a neu on wins, i s
p o o ype is se o he inpu sample, and i s lag is se o ue.
397
Y
-X
Z
A=(x,y)
x
z =
y
C
F
V
O
θ
X
φ
h
w
X
Z
Y
z’
x’
y’
A’=( x’, y’, z’)
φ
θ
Figu e 1. Schemes o he PTZ came a model. a) Le : pinhole came a model wi h he o iginal o ien a ion. b) Righ : eo ien ed scheme o ob ain sphe ical
coo dina es.
F om ha poin on, he neu on will be upda ed acco ding
o he compe i i e lea ning ule.
The p oposed lea ning algo i hm is as ollows:
1) D aw a aining sample xa andom om om he
cu en episode S , which has no been conside ed
be o e, i.e. a andom sampling wi hou eplacemen
is done.
2) Find he nea es neu on qin e ms o Euclidean
dis ance acco ding o he i s wo ec o
componen s:
q= a g min
i∈{1,...,N}k(wi,1, wi,2)−(x1, x2)k(8)
3) I he lag bqis ue, hen go o s ep 4. O he wise,
se he p o o ype o he winning neu on q o he
aining sample and se i s lag o ue:
wq=x(9)
bq= ue (10)
Then go o s ep 5.
4) Upda e he winning neu on p o o ype acco ding o
he s anda d compe i i e lea ning ule:
∆wq=η( ) (x−wq)(11)
whe e η( )is a decaying lea ning a e ha a ies
depending on he ime s ep .
5) I all he samples o he cu en episode S ha e al-
eady been p ocessed, hen go o s ep 6. O he wise,
go o s ep 1.
6) I he las ime ins an has been eached, hen
s op. O he wise, inc emen he ime ins an coun e
, load he nex episode and go o s ep 1.
The e a e wo phases in he lea ning p ocess: i s he
o de ing phase whe e ηexpe iences a linea decay (ini ial
lea ning a e ηI); and hen he con e gence phase whe e η
emains cons an a a small alue (ηC). This is because he
o de ing phase is equi ed o he wa m-up o he algo i hm
only, and a e ha he sys em uns o an inde ini ely long
ime. The change o phase is moni o ed by a s ep pa ame e
n.
η( ) = ηI(1 − /n),i <n
ηC, o he wise (12)
A heo e ical analysis o he abo e algo i hm can be
ca ied ou . Since he compe i ion is done on he i s wo
componen s, he algo i hm seeks a local minimum o an
ene gy unc ion Ewhich only akes in o accoun hese
componen s:
E=X
i∈{1,...,N}X
x∈Fi
k(wi,1, wi,2)−(x1, x2)k2(13)
whe e Fiis he ecep i e ield o he i- h neu on:
Fq=x|q= a g min
i∈{1,...,N}k(wi,1, wi,2)−(x1, x2)k
(14)
On he o he hand, he p o o ype upda e is ca ied ou
on he h ee las componen s oo, so hose componen s
app oxima e he a e age colo o he ecep i e ield:
(wi,3, wi,4, wi,5)≈E[(x3, x4, x5)|Fi](15)
398
As a las ema k, i mus be poin ed ou ha he la ge
numbe o neu ons N o be used o his applica ion equi es
a conside able op imiza ion o he compe i ion equa ion
(8). This is accomplished by inse ing all he i s sec ions
(wi,1, wi,2)o he neu on p o o ypes in o a quad- ee [21].
This way (8) is e alua ed e y quickly, e en o alues o
Nin he millions.
2.3. Delaunay iangula ion based in e pola ion
Since he posi ions o he neu on p o o ypes a e gi en
by eal numbe s, he e is no di ec way o ob ain he es i-
ma ed colo o he in ege alued pixel coo dina es o he
pano ama. In o de o o e come his di icul y, we p opose
o build he Delaunay iangula ion [22] o he se o med by
he i s sec ions (wi,1, wi,2)o each neu on p o o ype. This
way, a iangula ion o he pano ama is ob ained. Then, o
each in ege alued pai o pixel coo dina es (y1, y2)∈N2,
he iangle which i belongs o is compu ed, along wi h i s
ba ycen ic coo dina es wi h espec o ha iangle:
y=λi(wi,1, wi,2)+λj(wj,1, wj,2)+λk(wk,1, wk,2)(16)
i, j, k ∈ {1, ..., N}(17)
λi, λj, λk≥0(18)
λi+λj+λk= 1 (19)
Then he es ima ed colo (y3, y4, y5)∈R3is ob ained
by linea in e pola ion wi h weigh s equal o he ba ycen ic
coo dina es:
(y3, y4, y5) = λi(wi,3, wi,4, wi,5) +
λj(wj,3, wj,4, wj,5) + λk(wk,3, wk,4, wk,5)(20)
The e o e a con inuous, piecewise linea unc ion is
employed o es ima e he colo o e he pano ama.
3. Expe imen al esul s
In his sec ion we epo he compu a ional expe imen s
we ha e ca ied ou and hei esul s. The so wa e and
ha dwa e ha ha e been used a e speci ied in Subsec ion
3.1. Then, he es ed ideo sequences a e desc ibed in 3.2.
The desc ip ions o he used pa ame e s a e in Subsec ion
3.3 and he desc ip ions o he compe i o s in 3.4. Finally,
he ob ained esul s om he expe imen s a e epo ed in
Subsec ion 3.5.
3.1. Me hods
The came a con ol module is based on he i ualp z
lib a y [19]. I simula es a PTZ came a om a pano amic
ideo sequence, and i is accessible om i s websi e 1. The
implemen a ion is w i en in C++ and i uses he OpenCV
and OpenGL lib a ies. This i ual came a has limi a ions
in i s e ical mo emen , going om 0 (up) o 180 (down)
deg ees.
To gene a e he inpu da a o he compe i i e neu al ne
and o he compe i o s, a exhaus i e scanning o he scene
has been ca ied ou . To simula e he eal beha io o a PTZ
came a in a p ac ical se ing, some limi a ions in he scan
ha e been imposed. S a ing a he op ( e ical 0 deg ees),
we u n 360 deg ees o he igh wi h a s ep o 10 deg ees,
and when we come back o he ini ial poin , we go down
10 deg ees. This p ocess is epea ed un il we a i e o he
bo om and we s a o go up again. Fu he mo e, a andom
zoom ( e ical ield o iew) is applied o each ame, bu
again, we y o simula e a eal si ua ion. To his end, we
gene a e a andom numbe in an in e al [F OV, FOV +
5] and mo e his in e al by s eps o 5 deg ees be ween a
minimum and maximum alue o he ield o iew, [70,140]
deg ees, in o de o a oid s ange images wi h i egula i ies.
A o al o 3000 came a ames ha e been sa ed in bina y
iles o ead hem synch onously by all he me hods, which
ha e been implemen ed in Ma lab R2015b. The epo ed
expe imen s ha e been ca ied ou on a 64-bi Pe sonal
Compu e wi h an eigh -co e In el i7 3.60GHz CPU, 32 GB
RAM and s anda d ha dwa e. The implemen a ion o ou
app oach does no use any GPU esou ces.
Pano amic g ound u h (GT) has been calcula ed doing
he median o he aw pano amic ideo ames. Fo each
me hod, we compa ed he pano amic image ob ained wi h
he g ound u h. Th ee quali y measu es we e used o
e alua e he p oposed app oach: he i s was he Mean
Squa ed E o (MSE) me ic (lowe is be e ), which is
commonly used in image p ocessing; he second was he
S uc u al Simila i y index (highe is be e ), which ocuses
on s uc u al simila i ies be ween images:
SSIM(x, y) = (2µxµy)(2σxy +c2)
(µ2
x+µ2
y+c1)(σ2
x+σ2
y+c2)(21)
whe e µxand µya e he mean alue o images xand y,
σxand σya e he s anda d de ia ion o images xand
y,σxy is he co a iance o xand y,c1= (k1L)2and
c2= (k2L)2(Lis he dynamic ange, k1= 0.01 and
k2= 0.03). The alues ob ained om (21) a e a e aged
o e he h ee RGB channels o ob ain he pe o mance o
colo images. The hi d quan i a i e pe o mance measu e is
he Bha acha ya coe icien BC [23], which measu es he
closeness o he wo disc e e pixel p obabili y dis ibu ions
Pand ˆ
Pco esponding o he GT and modeled images:
BC =
255
X
j=0
P(j)ˆ
P(j)(22)
1. h ps://bi bucke .o g/pie e luc s cha les/ i ualp z s andalone
399
TABLE 1. CONSIDERED PARAMETER VALUES FOR OUR COMPETITIVE
LEARNING METHOD
Pa ame e Value
Numbe o neu ons N(Numbe o pixels)/2
Ini ial lea ning a e ηI0.4
Con e gence lea ning a e ηC0.01
Numbe o s eps n1000
whe e BC ∈[0,1] and highe is be e .
In addi ion, CPU imes we e measu ed o line o each
inpu bina y ile o ob ain he eal unning ime o each
algo i hm pe ame, wi hou including he ex a wai ing
ime o acqui e he nex ame ha would be equi ed i
we did i online.
3.2. Sequences
Fo he expe imen s we used wo ideo da abases. One
o hem has h ee ideos which a e a ailable on he i -
ualp z websi e. They a e h ee indoo sequences, wo o
hem wi h he same scene, and hey a e named scena io3,
scena io4 and scena io5. Because he wo i s ones a e
e y simila , we ha e jus shown he image esul s o
he scena io3 (3500x1750 pixels and 566 ideo ames).
The ideo scena io5 (3500x1750 pixels and 1957 ames)
shows a oom wi h people mo ing on in and doing di e en
ac ions. On he o he hand, we used one ideo sequence o
he Li ls a web page2. We named his ideo as scena io6
(2880x1440 pixels and 1169 ames) and shows a beach
wi h people mo ing and playing beach oleyball. S a is ics
we e calcula ed using he ou sequences.
3.3. Pa ame e selec ion
A se o uned pa ame e s is needed o de ine he com-
pe i i e lea ning model. These ixed pa ame e s a e epo ed
in Table 1.
In addi ion, a s udy o he numbe o neu ons has been
done and he esul s a e displayed in Fig. 2. As seen, he
equilib ium s a e is eached quickly and excellen esul s a e
ob ained om only 612500 neu ons (10% o o al pixels).
3.4. Compe i o s
We ha e compa ed ou p oposed me hod wi h o he
h ee me hods. The compe ing me hods ead each ame,
hen posi ion i in he pano amic image ma ix and inally
compu e he mean o e all he ames.
Since da a inpu coo dina es a e almos always ac ional
numbe s, he pano amic in ege coo dina es whe e he in-
coming ame mus be placed we e calcula ed by ounding
he o iginal ac ional coo dina es. Besides, he coo dina e
pai s do no de ine a egula s ic ly mono onic g id, i.e.,
he poin s ha e no s uc u e o o de be ween hei ela i e
2. h ps://li ls a .com/
loca ions, so ha he usual in e pola ion me hods de ined o
ob ain he in e pola ed RGB alues in a pano ama canno be
used. The e o e, he sca e ed in e polan me hod om MAT-
LAB was employed o manage his si ua ion. I p o ides he
ollowing in e pola ion a ian s: ’nea es ’ (nea es -neighbo
in e pola ion), ’linea ’ (linea in e pola ion), and ’na u al’
(na u al-neighbo in e pola ion). The same inpu da a was
p o ided o all he compe i o s as well as ou me hod, i.e.
he 3000 came a ames sa ed in bina y iles.
3.5. Resul s
A compa ison o all he e alua ed me hods o one o
he selec ed ideos is epo ed in Fig. 3. This igu e shows
he compa a i e e olu ion o he me hods wi h espec o
he numbe o cap u ed PTZ came a ames, i.e. he ame
index in he ideo sequence. I can be obse ed ha when
a ce ain numbe o ames is eached ha co e s almos
all he pano amic image, which is a ound 500 ames, ou
me hod (in ed) a ains he bes pe o mance alues, as
compa ed o he compe ing me hods. In pa icula , MSE
enhances subs an ially, whe e lowe alues indica es ha
he ob ained RGB pixel alues a e mo e p ecise. When he
numbe o acqui ed ames inc eases o mo e han 2500,
all me hods end o pe o m simila ly, bu ou me hod s ill
emains as he bes one.
Some hing simila happens o he o he h ee ideos.
To summa ize i , we ha e calcula ed he mean and s anda d
de ia ion o he ou sequences o each me hod and o
each pe o mance measu e. These quali a i e esul s a e
shown in Table 2. As we can see, ou me hod ou pe o ms
he compe ing me hods, pa icula ly in he mean squa ed
e o . Highe alues o SSIM and BC con i m ha ou
compe i i e neu al model p oduces he bes app oxima ion
o he backg ound o he scene.
The CPU ime equi ed o p ocess one ame is a e y
impo an ea u e o be assessed. Table 2 shows he mean
equi ed ime o p ocess a bina y ile o a ame. Ou
me hod is a ound 85% as e han he in e pola ion me hods.
I compu es he winne neu ons and upda es he quad- ee
whe e hey a e s o ed in jus o e a second. I we conside
ha a mo emen o he i ual PTZ came a and he gene -
a ion o he bina y ile o he cu en ame akes be ween
one and h ee seconds, i u ns ou ha ou me hod is he
only one ha can be execu ed concu en ly wi h he PTZ
came a ame acquisi ion p ocess. The compe i o s iplica e
he equi ed ime and incu in a big ime delay o ob ain
he pano amic backg ound. The u iliza ion o a GPU would
imp o e be ween 25 and 50 imes he p ocessing a e.
In o de o ge a quali a i e poin o iew abou he
sui abili y o ou app oach, he pano amic backg ounds
gene a ed by each me hod ha e been compa ed. In Fig. 5
a window o 50x50 pixels o wo scena ios a e displayed,
wi h hei espec i e aw pano amic ideo ame. As i can
be obse ed, he quali a i e esul s o e ed by ou app oach
a e he mos simila o he g ound u h. This can be be e
app ecia ed on he op and he bo om o he pano amic
image, whe e he compe i o s p oduce black pixels ha
400

0246
Numbe o Neu ons
10
6
1.9
2
2.1
2.2
2.3
2.4
log10(MSE)
0246
Numbe o Neu ons
10
6
0.99
0.9
0
SSIM
0246
Numbe o Neu ons
10
6
0.999
0.99
0.9
BC
0246
Numbe o Neu ons
10
6
0.2
0.4
0.6
0.8
1
1.2
1.4
CPU ime (sec.)
Figu e 2. E olu ion o he MSE, SSIM, BC and CPU ime a ying he numbe o neu ons. scena io3 was used o he analysis. MSE has been displayed
in a loga i hmic scale (lowe is be e ), SSIM and BC in a e e se loga i hmic scale (highe is be e ).
0 1000 2000 3000
NumF ames
1.5
2
2.5
3
3.5
4
4.5
log10(MSE)
0 1000 2000 3000
NumF ames
0.99
0.9
0
SSIM
0 1000 2000 3000
NumF ames
0.999
0.99
0.9
0
BC
Ou s linea nea es na u al
Figu e 3. Compa a i e o he MSE, SSIM and BC o he ou me hods wi h scena io3. MSE has been displayed in a loga i hmic scale (lowe is be e ),
SSIM and BC in a e e se loga i hmic scale (highe is be e ).
should no be he e. The window o scena io6 is a clea
example o his. Also, in he le and he igh o he images
p oduced by he compe i o s he e a e mo e black pixels
han in ou app oach. Wi h espec o he mo ing people o
he ideo ame, all me hods emo e hem e icien ly and
no big di e ences a e no iced. An example o he esul o
ou me hod is shown in Fig. 4.
4. Conclusion
In his wo k, a me hodology o model he pano amic
backg ound o PTZ came as is p esen ed. I consis s o an
online lea ning me hod based on a compe i i e neu al ne -
wo k o ead each came a ame and p ocess i o gene a e a
pano amic image o he scene. Fou scenes ha e been es ed
o check he easibili y o he sys em, ob aining sui able
and success ul esul s. Also, i has been demons a ed ha
ou app oach ou pe o ms se e al compe ing me hods. I is
ema kable ha he p oposed model uses e y li le CPU
ime o p ocess each inpu ideo ame, which pe mi s a easy
in eg a ion in a eal ime PTZ came a ideo su eillance
sys em.
Acknowledgmen s
This wo k is pa ially suppo ed by he Minis y o Econ-
omy and Compe i i eness o Spain unde g an TIN2014-
53465-R, p ojec name Video su eillance by ac i e sea ch
o anomalous e en s. I is also pa ially suppo ed by
he Au onomous Go e nmen o Andalusia (Spain) unde
p ojec s TIC-6213, p ojec name De elopmen o Sel -
O ganizing Neu al Ne wo ks o In o ma ion Technologies;
and TIC-657, p ojec name Sel -o ganizing sys ems and
obus es ima o s o ideo su eillance. Finally, i is pa -
ially suppo ed by he Au onomous Go e nmen o Ex-
emadu a (Spain) unde he p ojec IB13113. All o hem
include unds om he Eu opean Regional De elopmen
Fund (ERDF). The au ho s hank ully he g an o he
Uni e si y o M´
alaga and acknowledge he compu e e-
sou ces, echnical expe ise and assis ance p o ided by he
SCBI (Supe compu ing and Bioin o ma ics) cen e o he
Uni e si y o M´
alaga. They also g a e ully acknowledge he
suppo o NVIDIA Co po a ion wi h he dona ion o he
Ti an X GPU. Ka l Thu nho e -Hemsi is unded by a PhD
schola ship om he Spanish Minis y o Educa ion, Cul u e
and Spo unde he FPU p og am.
Re e ences
[1] Chung-Chen Chen, Yi Yao, A. D i a, A. Koschan, and M. Abidi, “Co-
ope a i e mapping o mul iple PTZ came as in au oma ed su eillance
sys ems,” in 2009 IEEE Con e ence on Compu e Vision and Pa e n
Recogni ion. IEEE, 2009, pp. 1078–1084.
[2] C. Ding, B. Song, A. Mo ye, J. Fa ell, and A. Roy-Chowdhu y,
“Collabo a i e sensing in a dis ibu ed p z came a ne wo k,” IEEE
401
TABLE 2. MEAN (STANDARD DEVIATION)MEASURES FOR EACH METHOD ALONG THE FOUR SCENARIOS
MSE SSIM BC T ime(sec/ ame)
Ou s 106.87 (23.09) 0.9184 (0.0371) 0.9703 (0.0466) 1.1427 (0.1284)
Linea 233.92 (35.60) 0.9167 (0.0288) 0.9701 (0.0460) 7.6549 (0.8106)
Nea es neighbo 235.00 (35.12) 0.9176 (0.0282) 0.9701 (0.0463) 7.7251 (0.8171)
Na u al neighbo 234.00 (35.63) 0.9162 (0.0292) 0.9700 (0.0459) 9.1097 (1.0477)
F ame P oposed me hod
Figu e 4. Pano amic images o scena io5 o ame 153 and he p oposed me hod, esp.
T ansac ions on Image P ocessing, ol. 21, no. 7, pp. 3282–3295,
2012.
[3] K. Konda, N. Conci, and F. De Na ale, “Global co e age maximiza-
ion in p z-came a ne wo ks based on isual quali y assessmen ,”
IEEE Senso s Jou nal, ol. 16, no. 16, pp. 6317–6332, 2016.
[4] T. Boul , X. Gao, R. Micheals, and M. Eckmann, “Omni-di ec ional
isual su eillance,” Image and Vision Compu ing, ol. 22, no. 7, pp.
515–534, 2004.
[5] C.-H. Chen, Y. Yao, D. Page, B. Abidi, A. Koschan, and M. Abidi,
“He e ogeneous usion o omnidi ec ional and p z came as o mul i-
ple objec acking,” IEEE T ansac ions on Ci cui s and Sys ems o
Video Technology, ol. 18, no. 8, pp. 1052–1063, 2008.
[6] S. Kim, K. Yun, K. Yi, S. Kim, and J. Choi, “De ec ion o mo ing
objec s wi h a mo ing came a using non-pano amic backg ound
model,” Machine Vision and Applica ions, ol. 24, no. 5, pp. 1015–
1028, 2013.
[7] K. Xue, Y. Liu, G. Ogunmakin, J. Chen, and J. Zhang, “Pano amic
gaussian mix u e model and la ge-scale ange backg ound subs ac-
ion me hod o p z came a-based su eillance sys ems,” Machine
Vision and Applica ions, ol. 24, no. 3, pp. 477–492, 2013.
[8] P. D. Z. Va cheie and G. A. Bilodeau, “Adap i e uzzy pa icle
il e acke o a PTZ came a in an IP su eillance sys em,” IEEE
T ansac ions on Ins umen a ion and Measu emen , ol. 60, no. 2,
pp. 354–371, 2011.
[9] F. J. L´
opez-Rubio and E. L´
opez-Rubio, “Fo eg ound de ec ion o
mo ing came as wi h s ochas ic app oxima ion,” Pa e n Recogni ion
Le e s, ol. 68, pp. 161–168, 2015.
[10] P. Azza i and A. Be ilacqua, “Join spa ial and onal mosaic alignmen
o mo ion de ec ion wi h PTZ came a,” Lec u e No es in Compu e
Science, ol. 4142, pp. 764–775, 2006.
[11] A. Be ilacqua and P. Azza i, “High-Quali y Real Time Mo ion De-
ec ion Using PTZ Came as,” P oceedings o he IEEE In e na ional
Con e ence on Video and Signal Based Su eillance (AVSS 2006), pp.
1–6, 2006.
[12] S. Ahal , A. K ishnamu hy, P. Chen, and D. Mel on, “Compe i i e
lea ning algo i hms o ec o quan iza ion,” Neu al Ne wo ks, ol. 3,
no. 3, pp. 277–290, 1990.
[13] T. Uchiyama and M. A bib, “Colo image segmen a ion using compe -
i i e lea ning,” IEEE T ansac ions on Pa e n Analysis and Machine
In elligence, ol. 16, no. 12, pp. 1197–1206, 1994.
[14] E. Yai , “Compe i i e lea ning and so compe i ion o ec o quan-
ize design,” IEEE T ansac ions on Signal P ocessing, ol. 40, no. 2,
pp. 294–309, 1992.
[15] H.-P. Chen, X.-J. Shen, and J.-W. Long, “His og am-based colou im-
age uzzy clus e ing algo i hm,” Mul imedia Tools and Applica ions,
ol. 75, no. 18, pp. 11 417–11 432, 2016.
[16] E. Ozan, S. Ki anyaz, and M. Gabbouj, “Compe i i e quan iza ion
o app oxima e nea es neighbo sea ch,” IEEE T ansac ions on
Knowledge and Da a Enginee ing, ol. 28, no. 11, pp. 2884–2894,
2016.
[17] R. Valen e and T. Ab ao, “MIMO ansmi scheme based on mo -
phological pe cep on wi h compe i i e lea ning,” Neu al Ne wo ks,
ol. 80, pp. 9–18, 2016.
[18] H. Xie, X. Luo, C. Wang, S. Liu, X. Xu, and X. Tong, “Mul ispec al
emo e sensing image segmen a ion using i al penalized con olled
compe i i e lea ning and uzzy en opy,” So Compu ing, ol. 20,
no. 12, pp. 4709–4722, 2016.
[19] G. Chen, P. S -Cha les, W. Bouachi , T. Joeissein , G. Bilodeau,
and R. Be ge in, “Rep oducible e alua ion o pan- il -zoom
acking,” CoRR, ol. abs/1505.04502, 2015. [Online]. A ailable:
h p://a xi .o g/abs/1505.04502
[20] R. M. Luque-Baena, E. L´
opez-Rubio, E. Dom´
ınguez, E. J. Palomo,
and J. M. Je ez, “A sel -o ganizing map o imp o e ehicle de ec ion
in low moni o ing sys ems,” So Compu ing, ol. 19, no. 9, pp. 2499–
2509, 2015.
[21] S. F. F isken and R. N. Pe y, “Simple and e icien a e sal me hods
o quad ees and oc ees,” Jou nal o G aphics Tools, ol. 7, pp. 1–
11, 2002.
[22] D. Lee and B. Schach e , “Two algo i hms o cons uc ing a delaunay
iangula ion,” In e na ional Jou nal o Compu e & In o ma ion
Sciences, ol. 9, no. 3, pp. 219–242, 1980.
[23] A. Bha acha yya, “On a measu e o di e gence be ween wo
mul inomial popula ions,” Sankhy: The Indian Jou nal o S a is ics
(1933-1960), ol. 7, no. 4, pp. 401–406, 1946. [Online]. A ailable:
h p://www.js o .o g/s able/25047882
402
scena io3 scena io6
F ame
GT
P oposed
linea
nea es
na u al
Figu e 5. G aphical depic ion o he ope a ion o he p oposed me hod. Each ow show, om le o igh , up o down: a ame o a aw pano amic ideo
sequence, and sec ions ( ed squa e) o i s g ound u h, ou algo i hm, and he h ee compe i o s algo i hms. F ame 153 o scena io3 and ame 859 o
scena io6 a e shown.
403