i
In as uc u e o Machine Lea ning and
Compu e Vision
Dimi i Chikhladze
Disse aion submi ed in pa ial ul illmen
o he equi emen s o he deg ee o
Mas e o Science in In o ma ion Managemen
Thesis ad iso :
P o esso Mau o Cas elli
a NOVA IMS
iii
Machine lea ning in as uc u e o people de ec ion and acking.
Copy igh
©
Dimi i Chikhladze, Facul y o Sciences and Technology, NOVA Uni e -
si y Lisbon.
The Facul y o Sciences and Technology and he NOVA Uni e si y Lisbon ha e he
igh , pe pe ual and wi hou geog aphical bounda ies, o ile and publish his dis-
se a ion h ough p in ed copies ep oduced on pape o on digi al o m, o by any
o he means known o ha may be in en ed, and o dissemina e h ough scien i ic
eposi o ies and admi i s copying and dis ibu ion o non-comme cial, educa ional
o esea ch pu poses, as long as c edi is gi en o he au ho and edi o .
This documen was c ea ed using he (pd )L
A
T
EX p ocesso , based in he “no a hesis” empla e[1], de eloped a he Dep. In o má ica o FCT-NOVA [2].
[1] h ps://gi hub.com/joaomlou enco/no a hesis [2] h p://www.di. c .unl.p

Acknowledgemen s
Fo emos , I wan o hank P o esso Leona do Vanneschi who c ea ed he Mas e s in
Ad anced Analy ics, and made i in e es ing and enjoyable o us, and my supe iso
P o esso Mau o Cas elli who was suppo i e and p omp h oughou w i ing his
hesis. I hank he managemen , adminis a ion and all o he people a NOVA IMS o
all hei wo k and help. I wan o hank he eam o he Eu opean s a up whe e I did
he wo k on which his hesis is based. Co dially, I wan o hank all my cou sema es
om he Mas e s p og am o being cool cou sema es. Las bu no leas , I wan o
hank my wi e Pa icia o he help and suppo in oo many hings.
ii
Abs ac
The in as uc u e su ounding machine lea ning p ojec s is o u mos impo ance:
Machine lea ning p ojec s equi e da a acquisi ion mechanisms, so wa e o da a
p ocessing, as well as a benchma king pla o m o e alua ing pe o mance o machine
lea ning algo i hms o e ime. In his epo we desc ibe ou wo k aimed a de eloping
such in as uc u e o a Eu ope based compu e ision s a up specializing in human
beha iou acking. We discuss h ee p ojec s comp ising he wo k. One dedica ed
o c ea ing a machine lea ning da ase o human beha iou moni o ing, ano he o
de eloping a sc een-came a calib a ion ool, and hi d o se ing up a benchma king
pla o m. The p ojec s we e in eg a ed wi h he co e echnology o he s a up, and
will con inue o be applied in he u u e.
ix
CHAPTER 1. INTRODUCTION
me ics o he machine lea ning pipelines.
P ojec Desc ip ion
Da ase o Human
De ec ion and T acking
Da a collec ion/ Da a
p ocessing/ P ac ical 3D
compu e ision.
•
C ea ed he “Eu opean S a up Da ase ”
designed speci ically o human de ec ion
and body pose es ima ion.
•
O ganized collec ion o ideo da a om
mo e han 50 people.
•
Augmen ed he aw ideo da a wi h
sc een came a calib a ion pa ame e s.
•
Implemen ed he p og amming in e ace
o he esul ing da ase .
Sc een-Came a Calib a ion
Tool
So wa e de elopmen /
Compu e ision/ 3D
modeling.
•
De eloped a ool which inds sc een cam-
e a calib a ion pa ame e s which a e e-
qui ed o compu ing he g ound u h o
he da ase .
Benchma king Pla o m
P ac ical machine lea ning/
Da abase design/ Da a
enginee ing/ Dashboa d
design.
•
Buil a da abase o s o ing esul s om
ML pipelines.
•
Implemen ed da a mo emen modules be-
ween ML pipelines and he benchma k-
ing pla o m.
•
Designed a dashboa d o isualiza ion o
e alua ion me ics.
The es o he epo is o ganized in o ou chap e . The i s chap e gi es a e iew
o me hodologies and echnologies ele an o he wo k. While, he nex h ee chap e s
a e dedica ed o each o he p ojec s comp ising he wo k.
2

Chap e
2
Backg ound
In his chap e we gi e an o e iew o opics ele an o ou wo k. We discuss in-
us uc u e o machine lea ning. We o e iew compu e ision da ase s, and ouch
b ie ly 3D compu e ision and human de ec ion and acking.
2.1 Machine lea ning and i s in us uc u e
By a widely ci ed de ini ion, machine lea ning is a sub- ield o compu e science con-
ce ned wi h de eloping machines wi h he abili y o “lea n” wi hou being explici ly
p og ammed [12]. The impo ance o machine lea ning g ew wi h he inc ease o ca-
paci y and a ailabili y o compu a ional esou ces, leading in he las decades o some
e olu ionizing p ac ical applica ions. Today machine lea ning is a subjec o in en-
si e esea ch. The echnology gian s and uni e si ies a e ac i ely pu suing machine
lea ning esea ch and de elopmen . Nume ous s a -ups ha e eme ged de eloping
machine lea ning based p oduc s.
A machine lea ning algo i hm is de eloped o a p oblem ela ed o some da a
gene a ing p ocess. Cons uc ing a machine lea ning algo i hm in ol es speci ying
i s a chi ec u e and aining i on a da a ob ained om he da a gene a ing p ocess.
A ained algo i hm is hen able o make p edic ions o o he ypes o in elligen
decisions on a new da a om he same da a gene a ing p ocess. The abili y o pe o m
well on a new da a is called gene alizabili y, and is a c ucial p ope y o a machine
lea ning algo i hm. On p ac ical le el, alida ing he pe o mance o he algo i hm is
accomplished by es ing i on mo e da a.
Resea ch and de elopmen c ucially equi es exis ence o adequa e in as uc u e
su ounding i . Some o he impo an componen s o hese in as uc u e, ela ed o
he p ojec s in his epo , a e:
3
CHAPTER 2. BACKGROUND
• Da a acquisi ion/ da a collec ion amewo k.
• So wa e o da a p ocessing.
• Benchma king pla o m o algo i hms.
2.1.1 Da a in machine lea ning
Pe haps, he single mos impo an i em in he abo e lis is he a ailabili y o da a.
Da a is used o aining machine lea ning algo i hms as well as o es ing and quan-
i a i ely e alua ing he pe o mance o algo i hms o e ime. Indeed, he impo ance
o da a o machine lea ning can no be o e es ima ed. I is a gued ha da ase s a e
a leas as impo an o a i icial in elligence de elopmen as ad ances in algo i hms
hemsel es [20].
A ailabili y o adequa e da a is o u mos impo ance o companies which de elop
machine lea ning based p oduc s. This is ue because o wo easons: Fi s ly, machine
lea ning echnology equi es aining da a. Secondly, da a is equi ed o be able o
es he pe o mance o he echnology a any gi en ime, equen ly o a speci ic use
case. Tes ing and benchma king is impo an o wo pu poses: Fo he in e nal use, i
p o ides o de elope s and manage s in o ma ion abou he s a us o he p oduc , and
he abili y o see how algo i hms a e imp o ing wi h ime. Fo ex e nal use, company
is able o p o ide conc e e and e i iable in o ma ion o i s clien s, in es o s and o he
s akeholde s abou i s p oduc o speci ic applica ions.
Machine lea ning p oblems in ol e making some kind o judgemen om he da a.
A judgmen can be classi ying a da apoin in o a class o making some p edic ion om
i . This judgmen is he a ge o he lea ning. Lea ning can be di ided in o wo ypes,
supe ised lea ning and unsupe ised lea ning. In unsupe ised lea ning, one is
lea ning om da a ha does no include explici in o ma ion abou he a ge . While
in supe ised lea ning he unc ion ha maps he inpu da a poin s o he ou pu
a ge s is lea ned om a da ase o sample inpu -ou pu pai s. Fo unsupe ised
lea ning we only need samples o inpu da a. While, in he supe ised lea ning, we
need a aining da ase which consis s o inpu da a samples oge he wi h labels which
a e alues o he a ge aken as he g ound u h. The machine lea ning algo i hm
lea ns how o make co ec judgmen s om such aining da a. As an example, o
supe ised image classi ica ion ask, he aining da ase would consis o images each
o which is assigned a class label.
Building da ase s o supe ised lea ning comp ises a majo challenge. Labeling
la ge da ase s is a cos ly p ocess, diligence is needed o au oma ize i , and some imes
some deg ee o human in ol emen is necessa y. Me hodologies used o c ea e a
da ase a y. Da a may be collec ed in a lab in con olled expe imen s. Da a can be
sou ced om he in e ne , o hough c owdsou cing. Da a can be also ob ained by
syn hesizing i .
4
2.2. COMPUTER VISION DATASETS
Figu e 2.1: Example o images labeled wi h classes
2.1.2 So wa e o da a p ocessing
C ea ing machine lea ning da ase s besides human inpu equen ly equi es da a
p ocessing so wa e o a ious asks. The P ojec II in his epo is an example o de-
eloping such so wa e o he compu e ision ask o came a objec pose es ima ion.
2.1.3 Benchma king pla o ms
Fo machine lea ning companies i is a good p ac ice o ha e a benchma king ame-
wo k. De elopmen o machine lea ning solu ions o complex p oblems is a con-
s an ly e ol ing p ocess. In his si ua ion, exis ence o a p ope benchmak ing pla -
o m is impo an o quan i ying e o s, o knowing pe o mance me ics o he
cu en algo i hms, and o seeing how he models imp o e o e ime. In addi ion
in he indus y applica ion, he e migh be a ious use cases o which he algo i hms
apply. Then one needs o know wha a e he pe o mance me ics o each o he use
cases and how hey imp o e o e ime. To achie e his one needs p ope e sioning
sys em o models. One need o de ine use cases, and which da a is ele an o each o
hem. One needs a da abase o s o e model esul s. And, one needs a dashboa d o
isualiza ion o me ics.
2.2 Compu e ision da ase s
One o he a eas whe e machine lea ning has had a emendous in luence in ecen
yea s is compu e ision. In pa icula , he eeme gence o deep lea ning has e o-
lu ionized compu e ision in he las en yea s o so. Mode n con olu ional neu al
ne wo ks achie ed eno mous leaps in pa e n ecogni ion, pa e n de ec ion and o he
compu e ision asks compa ed o wha was p e iously possible wi h adi ional im-
age p ocessing me hods.
5
CHAPTER 2. BACKGROUND
In he case o compu e ision, he da a on which algo i hms ope a e a e images.
Supe ised aining and alida ion equi e image da a o be anno a ed, as o example,
o image classi ica ion each image is anno a ed by a label speci ying i s class. The up-
hea al o machine lea ning in compu e ision, besides de elopmen o new machine
lea ning a chi ec u es and imp o emen in compu ing capaci y, has been accompa-
nied wi h c ea ion o aining da ase s. A huge e o has been dedica ed o he ask
o cons uc ing machine lea ning da ase s o a ious compu e ision applica ions.
Some o he amous compu e ision da ase s include MNIST, ImageNe [5], CIFAR-10
[13], Mic oso COCO [14] and many o he s (see [19] o a long lis ).
Fo compu e ision applica ions, da a collec ion in ol es aking images o eco d-
ing ideos, which can be seen as sequences o images. In 3D compu e ision (see
below) in addi ion o RGB images we also ha e dep h images. Fo supe ised lea n-
ing, we need labels o images o ideo ames in ou da ase . F equen ly, he aw
da a is no anno a ed wi h aining labels di ec ly. Ra he we ha e anno a ions abou
he eco ding en i onmen , he wo ld. These anno a ions may include coo dina es
o wo ld objec s, coo dina e ans o ma ion pa ame e s e c. T aining labels o some
pa icula machine lea ning p oblem a e hen in e ed om hese pa ame e s. In ad-
di ion, we may need o eco d in o ma ion such as e.g. ligh ing condi ions and ime
o eco dings, as well as cha ac e is ics o wo ld objec s and/o pe sons, such as e.g.
mo ion speed o an objec and age o a pe son. The decision on wha wo ld in o ma ion
do we keep ack o du ing he eco dings is pa o he da a c ea ion p ocess.
One impo an cha ac e is ics o aining da ase is a iabili y. In p ac ice, aining
da ase is only a iny po ion o he eal da a o which he algo i hm is ying o
lea n he p oblem. To ain a good model he aining da a has o ep esen he whole
dis ibu ion o he eal da a which he model is ying o lea n. In compu e ision
applica ions, he image o ideo iles ha we collec should show ex ensi e deg ee o
a ia ion. Fi s we should ha e a iabili y in e ms o en i onmen , such as a iabili y
in ligh ing condi ions. Secondly, we should ha e he p oblem speci ic a iabili y: I
o example, ou model is ying o lea n dogs om images, in ou da ase we should
ha e images o dogs o a ied size, colou , appea ing in a ious body posi ion, showing
om di e en angles e c.
In p ac ice one also has o conside he a iabili y o use cases o he machine
lea ning applica ion. Fo example, i we a e s udying human a en ion sensing, we
will wan o conside use cases o d i ing, e ail shopping, gaming e c. Aside om
aining gene al models, i will be use ul o ain models o each pa icula use case.
Fo his again, adequa e da a is needed. The e o e, we should know which pa o ou
da a is ele an o which use case.
These a e some o he conside a ions ha should be aking in o accoun when
designing and building a compu e ision da ase s.
6
2.3. HUMAN DETECTION AND TRACKING
2.3 Human de ec ion and acking
The objec i e o acking is o es ima e loca ion and pose o an objec om a ideo.
The goal o human acking is o iden i y he pose o a ious body pa s (e.g. hands,
inge s, head e c.) a each ame, and ack i o e ime. In applica ions, e en ually he
objec o in e es may be beha iou ecogni ion, a en ion sensing, in en ion es ima ion,
emo ion ecogni ion e c. Compu a ionally howe e , es ima ing pose o body pa s
in ol es inding he loca ions o ce ain body landma ks, and es ima ing pa ame e s
de ining he pose o he body pa s. T acking hen in ol es es ima ing hese da a a
each ame o e ime.
Figu e 2.2: Human pose es ima ion and acking om he wo k [1]
Human de ec ion and acking da ase consis s o images o ideos con aining
people. The aining labels a e pa ame e s such as loca ions o human bodies and
human body landma ks and pa ame e s o a ious body poses such as head pose.
A numbe o machine lea ning da ase s exis o human de ec ion and acking,
many o hem specialized o mo e speci ic p oblems such as human pose es ima ion [2],
ace ecogni ion (see [6]), head pose es ima ion [8], [9] eye acking [7], [21] and so on.
They ha e been collec ed using di e en me hods and means. Some o hese we e c e-
a ed h ough con olled da a collec ions, o he s we e collec ed hough c owdsou cing
me hod. While, in o he app oaches image syn hesizing was used.
2.4 3D scanning and 3D compu e ision
3D scanning is a echnology which cap u es geome ic shape o objec s in he eal
wo ld and, possibly, analyses cha ac e is ics such as colo . The e a e numbe o 3D
7

CHAPTER 2. BACKGROUND
scanning me hods including lase scanning, ime-o - ligh and s uc u es-ligh echnolo-
gies ( o mo e de ails see [3]).
Popula mode n commodi y 3D scanning came as a e Realsence om In el and
Kinec om Mic oso . The o me is suppo ed by an open sou ce c oss pla o m SDK
[10]. These came as ha e bo h 3D scanning dep h s eam and an RGB s eam. F om
he dep h measu emen s and he came a pin-hole model i is possible o compu e poin
clouds, which a e se s o poin s in a 3D space. Poin clouds a e used o c ea e polygon
meshes. Besides, 3D scanning de ices calib a e be ween dep h and RGB s eams, and
gene a e ex u e mapping which de ines colo in o ma ion o he su ace scanned by
he dep h senso .
3D compu e g aphics s o e geome ic objec s in 3D space, such as meshes, o
isualizing hem in 2D o pe o ming compu a ional ope a ions on hem.
3D p o ides mo e in o ma ion han 2D images which can be exploi ed in compu -
ing. Machine lea ning, and in pa icula deep lea ning, has been applied o dep h and
colo da a o ecogni ion, de ec ion, acking e c.
8
Chap e
3
P ojec I: Machine Lea ning Da ase o
Human De ec ion and T acking
The Eu opean S a up uses da a o aining and benchma king i s machine lea ning
algo i hms. The objec i e o he p ojec was o c ea e a specialized machine lea ning
da ase o human de ec ion and acking. The da ase was o be comp ised o dep h
and RGB ideo eco dings, oge he wi h anno a ions abou he eco ding en i onmen .
The goal was o eco d mo e han 50 people o di e en appea ance wi hin a ying
ambien condi ions and con ex s. 1
3.1 Eu opean S a up Da ase
As a esul o he p ojec a Eu oean S a up Da ase was c ea ed. The da ase consis s
o a ew hund ed eco dings by dep h sensing came as. Each eco ding is a ound 2
minu es long. Each o hem has an RGB s eam and a dep h s eam. The eco dings
show a pe son doing simple body mo emen s ollowing he con en displayed on a
sc een placed in on o hem. The eco dings show g ea deg ee o di e si y. Pa -
icipan s o a ied age, gende , ace and appea ance whe e eco ded. The eco ding
ambien ligh ing condi ions a iabili y was maximized. Fu he mo e, eco dings we e
di e en ia ed by he sc een ype and he eco ding scena io. Th ee ypes o sc eens
we e used o he eco dings. The eco dings we e done in a numbe o scena ios, o
each o which speci ied ins uc ions we e gi en o he pa icipan s as o how o mo e
hei head o body du ing he expe imen .
Besides he ideos, he ollowing da a is included in he da ase . Fo each eco ding,
sc een came a pose pa ame e s a e sa ed. Fo each eco ding, pixel coo dina es o
1
Fo con iden iali y easons we can no e eal he exac na u e o he da ase , he e o e he desc ip ion
o he da ase and he da a collec ion p ocess a e gi en in gene al e ms.
9
CHAPTER 3. PROJECT I: MACHINE LEARNING DATASET FOR HUMAN
DETECTION AND TRACKING
elemen s o he sc een con en a e sa ed o abou 50 di e en ames a which i is
assumed ha he pa icipan ’s body posi ion was engaged wi h he sc een con en as
speci ied by he expe imen ules. Fo each eco ding, he e is a me ada a con aining
in o ma ion abou he sc een ype, he eco ding scena io, as well as eco ding ambien
condi ions and he pa icipan ’s cha ac e is ics. The ollowing ables summa ize he
Eu opean S a up Da ase :
Eu opean S a up Da ase
Mo e han 200 Reco dings
Reco ding
RGB ideo
Dep h ideo
F ame-wise pa ame e s o he sc een con en
Pa icipan
Scena io
Sc een ype
3.2 Da a collec ion
The da a was collec ed om mo e han 50 people o e h ee weeks pe iod. The da ase
was collec ed om olun ee pa icipan s.
Wi h each pa icipan a ew ideo eco ding expe imen s we e made. Du ing a
eco ding expe imen he pa icipan wa ches con en displayed on a sc een which
changes h ough ime. The pa icipan mo es hei body ollowing he shown con en ,
o example poin ing some body pa s a a ious elemen s o he sc een con en . A
dep h sensing came a is placed in on o he sc een acing he pa icipan , and eco ds
he pa icipan (see Figu e 3.1). F om ime o ime he pa icipan “clicks” on an inpu
de ice (such as a mouse o a space ba ) con i ming ha a ha momen hei body
was posi ioned ela i e o he sc een con en as speci ied by he ules. This is when
he pixel coo dina es o he sc een elemen s a e sa ed. In his way we ensu e ha a
he momen s o "clicks" he pa icipan was p ope ly engaged wi h he sc een con en .
Then, om he posi ion o he elemen s o he sc een con en in he wo ld i is possible
o in e he g ound u h body pose pa ame e s which a e he aining labels o ou
algo i hms.
10
3.2. DATA COLLECTION
Figu e 3.1: Reco ding scene. The pa icipan is asked o change he body posi ion
ollowing he con en shown on he sc een.
As a esul , he aw da a p oduced by he eco ding expe imen s consis ed o ame-
wise anno a ed ideo eco dings, which make up he co pus o he Eu opean S a up
Da ase .
The da a collec ion p ocess in ol ed a numbe o design and o ganiza ional chal-
lenges and decisions. One had o make su e ha he p ocess was ollowed co ec ly by
he pa icipan s and he o ganize s. Fo his pu pose a p ecise da a collec ion p o ocol
was designed. In pa icula he p o ocol de ined he ules o engagemen by pa ici-
pan s wi h he sc een con en . Also on he o ganiza ional side, since ou expe imen s
in ol ed people, da a p i acy issues had o be aken in o accoun . In pa icula we
ensu ed ha ou ac i i y was GDPR compa ible [18].
Logis ically, one had o secu e loca ions, olun ee pa icipan s and o he ma e ial
o he eco dings. These esou ces had o exhibi su icien di e si y. The eco ding
en i onmen s had o be di e se in illumina ion condi ion. As o pa icipan s, one had
o secu e people o di e se age, gende , e hnici y and o he cha ac e is ics. In addi ion,
we wan o eco d he pe son a di e en imes (o e ew days). The sc een on which
he engagemen con en was displayed had o be o di e en size and mobili y. We used
a LCD sc een, a p ojec o and a able which could be hand hold by he pa icipan .
Resou ce Di e si y Dimension
Loca ion Illumina ion condi ion
Pa icipan Age, Gende , E hnici y, Clo hing ype,
Time o eco ding
Sc een Size, Mobili y
11
CHAPTER 4. PROJECT II: SCREEN CAMERA CALIBRATION TOOL
The poin s whose coo dina es he calib a ion ool will collec a e he co ne s o a
checke boa d pa e n displayed on a sc een. The pixel coo dina es o he checke boa d
pa e n a e known. The p oblem is es ima ing hei coo dina es in WCS.
The WCS is ixed wi h espec o he eco ding came a. The e o e, we can es ima e
he WCS coo dina es o a poin which is isible by he eco ding came a. Howe e , he
eco ding came a is placed so ha i can no see he sc een. This means ha we can
no di ec ly es ima e coo dina es o a poin on a sc een by he eco ding came a. The
solu ion is o in oduce ano he came a in he calib a ion p ocess. The new came a,
which we will call an ex e nal came a is placed so ha i sees he sc een. A he same
ime we ge a e e ence objec , and place i so ha bo h he eco ding came a and ex e -
nal came a see i (see Figu e 4.3). Then, he ex e nal came a can ind coo dina es o a
poin on a sc een wi h espec o i s own coo dina e sys em. While since he e e ence
objec is isible by he bo h came as, bo h o hem can es ima e coo dina es o some
e e ence poin s in hei espec i e coo dina e sys ems. These e e ence coo dina es
hen a e used o ind he ans o ma ion om he ex e nal came a coo dina e sys em
o he eco ding came a coo dina e sys em.
Figu e 4.3: Calib a ion p ocess
The calib a ion ool was implemen ed wi h an online mode and a eco ding-based
mode. In he online mode he calib a ion ool uns on li e ideo da a om he wo
came as, and collec s poin co espondences on he ly. In he eco ded mode, he
calib a ion ool uns on sa ed eco dings.
In he online mode he calib a ion p ocess wo ks as ollows:
1.
The Sc een and he Reco ding Came a a e placed ixed in he posi ions in which
hey will be du ing he da a collec ion expe imen .
2. The Ex e nal Came a is posi ioned in a way ha i usually sees he Sc een.
3.
The Re e ence Objec is placed in a way ha i is usually isible by he bo h
came as.
18

4.3. CALIBRATION TOOL
4.
When he calib a ion ool s a s, a each li e ideo ame he ex e nal came a
es ima es coo dina es o he co ne s o he checke boa d pa e n, while bo h he
ex e nal came a and he eco ding came a es ima e coo dina es o some e e -
ence poin s on he e e ence objec (see Sec ion 4.5). Using he coo dina es o
he e e ence poin s es ima ed by bo h came as a ans o ma ion is ound be-
ween he coo dina e sys ems o he wo came as. Using his ans o ma ion he
checke boa d pa e n coo dina es a e ans o med o he WCS, and a e sa ed. A
he same ime he pixel coo dina es o he checke boa d pa e n a e sa ed. A e
ew ames, he poin s collec ion s ops. The collec ed poin co espondences a e
ed o he linea eg ession which inds he calib a ion pa ame e s.
In he eco ded mode he ool uns on eco ded ideos. Ano he di e ence o he
eco ded mode om he li e mode is ha , he e e ence poin coo dina es a e no
es ima ed by he eco ding came a ame-wise, bu o e he whole ideo. This means
ha , unlike in he li e mode, he e e ence objec has o s ay ixed a he posi ion.
Calib a ion in he eco ded mode is done as ollows:
1.
The Sc een and he Reco ding Came a a e placed ixed in he posi ions in which
hey will be du ing he da a collec ion expe imen .
2.
The Re e ence Objec is placed in a way ha i is isible o he Reco ding Came a,
and i s ays ixed.
3.
The Reco ding Came a makes a eco ding o he Re e ence Objec and a ideo is
sa ed, called In e nal Video.
4.
The Ex e nal Came a is posi ioned in a way ha i usually sees bo h he Re e ence
Objec and he Sc een, i makes a eco ding and sa es a ideo, called Ex e nal
Video.
5.
The calib a ion ool is un on he eco ded da a. Fi s i p ocesses he In e nal
Video om whe e i ge he coo dina es o he Re e ence Poin s on he Re e ence
Objec . Nex i p ocesses he Ex e nal Video. A each ame he coo dina es o
he co ne s o he Checke boa d pa e ns a e es ima ed. A he same ime he
coo dina es o he e e ence poin s a e es ima ed and using hem a ans o ma-
ion be ween he wo came as is ound. Then, using his ans o ma ion he WCS
coo dina es o he checke boa d co ne s a ha ame a e es ima ed and sa ed
oge he wi h he pixel coo dina es. A e he ideo da a is p ocessed and poin
co espondences a e collec ed he calib a ion pa ame e s a e ound by sol ing
he linea eg ession p oblem.
19
CHAPTER 4. PROJECT II: SCREEN CAMERA CALIBRATION TOOL
4.4 Es ima ing 3D coo dina es o poin s
As desc ibed abo e, he calib a ion p ocess in ol es es ima ing 3D coo dina es o
poin s, such as co ne s o a checke boa d pa e n, o e e ence poin s on a e e ence
objec .
A each ideo ame a checke boa d pa e n is de ec ed using OpenCV unc ions
[15]. This e u ns 2D coo dina es o he chessboa d co ne s on he image. Then, he
3D coo dina es a e es ima ed ei he by Pe spec i e-n-Poin me hod o by using dep h
ideo da a.
Pe spec i e-n-Poin me hod can es ima e 3D coo dina es o poin s om hei 2D
p ojec ions on he image. OpenCV unc ion Sol ePnP is he OpenCV implemen a ion
o his me hod which we used [15].
The al e na i e is o use 3D dep h da a, which b ings ex a p ecision o he es ima-
ion. In his app oach, i s using he dep h da a we es ima e he plane on which he
poin s lie (ou poin s always lie on a plane, ei he he sc een o he e e ence objec
which is plana ), by i ing a plane o he poin cloud o he dep h ame. Nex we ind
ays om he ocal poin o he came a going h ough he 2D p ojec ion o he poin s.
Finally we ind he 3D coo dina es as in e sec ion poin s o he es ima ed plane and
he ays.
4.5 T ans o ma ion be ween came a coo dina es
The calib a ion ool in ol es inding a ans o ma ion o m coo dina e sys em o one
came a o ano he one. This is done wi h help o a e e ence objec . Re e ence objec
is an objec which is seen by bo h came as. The wo came as es ima e ce ain poin s
on he e e ence objec , using which hen he ans o ma ion is ound.
Figu e 4.4: Examples o A uco ma ke s.
20
4.5. TRANSFORMATION BETWEEN CAMERA COORDINATES
The e e ence objec should ha e poin s which a e easily de ec able by a came a.
We used a ca dboa d o which we a ached A uco ma ke s [16], [17]. A uco ma ke is
a syn he ic squa e ma ke composed by a wide black bo de and a inne bina y ma ix
which de e mines i s iden i ie . A uco lib a y [17] con ains unc ions o de ec ing,
pose es ima ion and isualiza ion o A uco ma ke s, making A uco good ool o pose
es ima ion applica ions.
We placed se e al A uco ma ke s on ou e e ence objec , each A uco ma ke gi es
ou e e ence poin s, each o which is uniquely iden i iable by he A uco ma ke
iden i ie and he de e mined index among each ou on he same ma ke . In he
calib a ion ool, bo h came as de ec hese uniquely iden i ied e e ence poin s. As
a esul we ge 3D coo dina es o he e e ence poin s in he wo came a coo dina e
sys ems. The ans o ma ion be ween hese coo dina e sys em hen is ound wi h
Kabsch algo i hm [11] desc ibed below.
Figu e 4.5: T ans o ma ion be ween he came a coo dina e sys ems.
Gi en wo se s o co esponding poin s
A
and
B
(as o example, e e ence poin s
es ima ed by he wo came as in ou se ing abo e) he algo i hm o ind op imal
o a ion and ansla ion which aligns A o Bp oceeds as ollows:
1. Find cen oids o bo h se s:
cen oidA= mean(A),
cen oidB= mean(B).(4.9)
2. B ing bo h se s o he o igin:
Ao=A−cen oidA,
Bo=B−cen oidB.(4.10)
21
CHAPTER 4. PROJECT II: SCREEN CAMERA CALIBRATION TOOL
3. Find c oss-co a iance ma ix
H=AT
oBo.
4. Calcula e he SVD o he co a iance ma ix
H=USV T.
5. Find he o a ion ma ix by
R=V UT.
6. I de (R)<0, co ec Rby mul iplying i ’s hi d column by −1.
7. Find he ansla ion by
= cen oidB−R(cen oidA).
4.6 Valida ion o calib a ion pa ame e s
To alida e he calib a ion pa ame e s we implemen ed a isualiza ion ool. I uns
on a ideo made by he ex e nal came a ilming he eco ding came a and he sc een.
A each ame he sc een posi ion is es ima ed, hen using he ound sc een came a
calib a ion pa ame e s he posi ion o he came a is ound and a came a schema is
d awn on he ame. This allows us o make a isual compa ison o he es ima ed
came a posi ion and he eal came a posi ion.
22
4.6. VALIDATION OF CALIBRATION PARAMETERS
Figu e 4.6: Example o de ec ing a chessboa d pa e ns and A uco ma ke s. OpenCV
lib a y is used.
23

Chap e
5
P ojec III: Benchma king Pla o m
The Eu opean S a up de elops a cu ing-edge echnology which i s i es o cons an ly
imp o e. The Eu opean S a up needs a mechanism which will allow seeing how i s
echnology imp o es o e ime, bo h o in e nal and ex e nal use. The aim o his
p ojec was o kick s a a pla o m o his pu pose.
5.1 Benchma king pla o m
The benchma king pla o m consis s o a da abase o s o ing he ou pu o machine
lea ning pipelines and a dashboa d o isualiza ion o me ics, as well as da a mo e-
men modules o impo and expo o da a.
5.2 Me ics and use cases
The p ocessing pipeline equi es mul iple s eps, such as ace de ec ion, head pose es i-
ma ion, body pose es ima ion, e c. The me ics o in e es s a e quan i a i e measu es
o he pe o mance o he algo i hms, such as how many alse de ec ions a e made,
how many misses he sys em exhibi s, o how la ge a e he e o s. In he benchma king
pla o m we keep ack o hese me ics.
People de ec ion and acking pipelines change as algo i hms imp o e o e ime.
In he benchma king pla o m each pipeline should be uniquely iden i iable. This is
done using a gi ag.
In addi ion we designed se e al use cases, each o hem co esponding o applica-
ion use case such as beha iou acking while d i ing, o beha iou acking while
e ail shopping. A use case is iden i ied by i ele an da a. Fo example, o a e ail
25
CHAPTER 5. PROJECT III: BENCHMARKING PLATFORM
Figu e 5.1: Benchma king pla o m
shopping use case we migh use ideos which show people passing by he came a as
hey would in a e ail s o e aisles.
5.3 Da abase
We buil a da abase o ou benchma king pla o m using MySQL. Ou da abase
should s o es esul s o people de ec ion and acking expe imen s as well as ou
pipelines, use cases, pe sons (expe imen pa icipan s) and eco dings. The da abase
should also e lec he ela ionship among hese da a. Du ing he design p ocess, da a
o be s o ed was iden i ied and da a in e ela ionships de ined. We build a no malized
da abase wi h a ew ables. Some o he ables a e o da a om machine lea ning
pipelines, o example a able o body landma k coo dina es and a able o head pose
pa ame e s. O he ables a e o me ada a, o example a able o pipelines and a
able o eco dings.
We implemen ed da a expo module which expo s he esul s o unning a pipeline
o he da abase. We also implemen ed modules ha impo me ada a abou pe sons
and eco dings om he Eu opean S a up Da ase . Use case de inin ion mechanism
was also implemen ed.
26
5.4. DASHBOARD
5.4 Dashboa d
We designed a dashboa d wi h py hon Plo ly Dash lib a y [4]. The dashboa d isual-
izes he people de ec ion and acking me ics. The use o he dashboa d can speci y
pipeline and/o use case e c. Dash will load da a o ha speci ic eques , compu e
me ics and isualize hem.
Plo ly Dash de ines dashboa d page layou simila ly o HTML, and i comes wi h
a ious widge s o da a isualiza ion. Dash suppo s eac i e p og amming s yle
which we ollowed in ou implemen a ion. The inpu o he use is au oma ically
p opaga ed h oughou he dashboa d and me ics and isualiza ions a e upda ed.
27