Infrastructure for machine learning and computer vision

i In as uc u e o Machine Lea ning and Compu e Vision Dimi i Chikhladze Disse aion submi ed in pa ial ul illmen o he equi emen s o he deg ee o Mas e o Science in In o ma ion Managemen Thesis ad iso : P o esso Mau o Cas elli a NOVA IMS iii Machine lea ning in as uc u e o people de ec ion and acking. Copy igh © Dimi i Chikhladze, Facul y o Sciences and Technology, NOVA Uni e - si y Lisbon. The Facul y o Sciences and Technology and he NOVA Uni e si y Lisbon ha e he igh , pe pe ual and wi hou geog aphical bounda ies, o ile and publish his dis- se a ion h ough p in ed copies ep oduced on pape o on digi al o m, o by any o he means known o ha may be in en ed, and o dissemina e h ough scien i ic eposi o ies and admi i s copying and dis ibu ion o non-comme cial, educa ional o esea ch pu poses, as long as c edi is gi en o he au ho and edi o . This documen was c ea ed using he (pd )L A T EX p ocesso , based in he “no a hesis” empla e[1], de eloped a he Dep. In o má ica o FCT-NOVA [2]. [1] h ps://gi hub.com/joaomlou enco/no a hesis [2] h p://www.di. c .unl.p Acknowledgemen s Fo emos , I wan o hank P o esso Leona do Vanneschi who c ea ed he Mas e s in Ad anced Analy ics, and made i in e es ing and enjoyable o us, and my supe iso P o esso Mau o Cas elli who was suppo i e and p omp h oughou w i ing his hesis. I hank he managemen , adminis a ion and all o he people a NOVA IMS o all hei wo k and help. I wan o hank he eam o he Eu opean s a up whe e I did he wo k on which his hesis is based. Co dially, I wan o hank all my cou sema es om he Mas e s p og am o being cool cou sema es. Las bu no leas , I wan o hank my wi e Pa icia o he help and suppo in oo many hings. ii Abs ac The in as uc u e su ounding machine lea ning p ojec s is o u mos impo ance: Machine lea ning p ojec s equi e da a acquisi ion mechanisms, so wa e o da a p ocessing, as well as a benchma king pla o m o e alua ing pe o mance o machine lea ning algo i hms o e ime. In his epo we desc ibe ou wo k aimed a de eloping such in as uc u e o a Eu ope based compu e ision s a up specializing in human beha iou acking. We discuss h ee p ojec s comp ising he wo k. One dedica ed o c ea ing a machine lea ning da ase o human beha iou moni o ing, ano he o de eloping a sc een-came a calib a ion ool, and hi d o se ing up a benchma king pla o m. The p ojec s we e in eg a ed wi h he co e echnology o he s a up, and will con inue o be applied in he u u e. ix CHAPTER 1. INTRODUCTION me ics o he machine lea ning pipelines. P ojec Desc ip ion Da ase o Human De ec ion and T acking Da a collec ion/ Da a p ocessing/ P ac ical 3D compu e ision. • C ea ed he “Eu opean S a up Da ase ” designed speci ically o human de ec ion and body pose es ima ion. • O ganized collec ion o ideo da a om mo e han 50 people. • Augmen ed he aw ideo da a wi h sc een came a calib a ion pa ame e s. • Implemen ed he p og amming in e ace o he esul ing da ase . Sc een-Came a Calib a ion Tool So wa e de elopmen / Compu e ision/ 3D modeling. • De eloped a ool which inds sc een cam- e a calib a ion pa ame e s which a e e- qui ed o compu ing he g ound u h o he da ase . Benchma king Pla o m P ac ical machine lea ning/ Da abase design/ Da a enginee ing/ Dashboa d design. • Buil a da abase o s o ing esul s om ML pipelines. • Implemen ed da a mo emen modules be- ween ML pipelines and he benchma k- ing pla o m. • Designed a dashboa d o isualiza ion o e alua ion me ics. The es o he epo is o ganized in o ou chap e . The i s chap e gi es a e iew o me hodologies and echnologies ele an o he wo k. While, he nex h ee chap e s a e dedica ed o each o he p ojec s comp ising he wo k. 2 Chap e 2 Backg ound In his chap e we gi e an o e iew o opics ele an o ou wo k. We discuss in- us uc u e o machine lea ning. We o e iew compu e ision da ase s, and ouch b ie ly 3D compu e ision and human de ec ion and acking. 2.1 Machine lea ning and i s in us uc u e By a widely ci ed de ini ion, machine lea ning is a sub- ield o compu e science con- ce ned wi h de eloping machines wi h he abili y o “lea n” wi hou being explici ly p og ammed [12]. The impo ance o machine lea ning g ew wi h he inc ease o ca- paci y and a ailabili y o compu a ional esou ces, leading in he las decades o some e olu ionizing p ac ical applica ions. Today machine lea ning is a subjec o in en- si e esea ch. The echnology gian s and uni e si ies a e ac i ely pu suing machine lea ning esea ch and de elopmen . Nume ous s a -ups ha e eme ged de eloping machine lea ning based p oduc s. A machine lea ning algo i hm is de eloped o a p oblem ela ed o some da a gene a ing p ocess. Cons uc ing a machine lea ning algo i hm in ol es speci ying i s a chi ec u e and aining i on a da a ob ained om he da a gene a ing p ocess. A ained algo i hm is hen able o make p edic ions o o he ypes o in elligen decisions on a new da a om he same da a gene a ing p ocess. The abili y o pe o m well on a new da a is called gene alizabili y, and is a c ucial p ope y o a machine lea ning algo i hm. On p ac ical le el, alida ing he pe o mance o he algo i hm is accomplished by es ing i on mo e da a. Resea ch and de elopmen c ucially equi es exis ence o adequa e in as uc u e su ounding i . Some o he impo an componen s o hese in as uc u e, ela ed o he p ojec s in his epo , a e: 3 CHAPTER 2. BACKGROUND • Da a acquisi ion/ da a collec ion amewo k. • So wa e o da a p ocessing. • Benchma king pla o m o algo i hms. 2.1.1 Da a in machine lea ning Pe haps, he single mos impo an i em in he abo e lis is he a ailabili y o da a. Da a is used o aining machine lea ning algo i hms as well as o es ing and quan- i a i ely e alua ing he pe o mance o algo i hms o e ime. Indeed, he impo ance o da a o machine lea ning can no be o e es ima ed. I is a gued ha da ase s a e a leas as impo an o a i icial in elligence de elopmen as ad ances in algo i hms hemsel es [20]. A ailabili y o adequa e da a is o u mos impo ance o companies which de elop machine lea ning based p oduc s. This is ue because o wo easons: Fi s ly, machine lea ning echnology equi es aining da a. Secondly, da a is equi ed o be able o es he pe o mance o he echnology a any gi en ime, equen ly o a speci ic use case. Tes ing and benchma king is impo an o wo pu poses: Fo he in e nal use, i p o ides o de elope s and manage s in o ma ion abou he s a us o he p oduc , and he abili y o see how algo i hms a e imp o ing wi h ime. Fo ex e nal use, company is able o p o ide conc e e and e i iable in o ma ion o i s clien s, in es o s and o he s akeholde s abou i s p oduc o speci ic applica ions. Machine lea ning p oblems in ol e making some kind o judgemen om he da a. A judgmen can be classi ying a da apoin in o a class o making some p edic ion om i . This judgmen is he a ge o he lea ning. Lea ning can be di ided in o wo ypes, supe ised lea ning and unsupe ised lea ning. In unsupe ised lea ning, one is lea ning om da a ha does no include explici in o ma ion abou he a ge . While in supe ised lea ning he unc ion ha maps he inpu da a poin s o he ou pu a ge s is lea ned om a da ase o sample inpu -ou pu pai s. Fo unsupe ised lea ning we only need samples o inpu da a. While, in he supe ised lea ning, we need a aining da ase which consis s o inpu da a samples oge he wi h labels which a e alues o he a ge aken as he g ound u h. The machine lea ning algo i hm lea ns how o make co ec judgmen s om such aining da a. As an example, o supe ised image classi ica ion ask, he aining da ase would consis o images each o which is assigned a class label. Building da ase s o supe ised lea ning comp ises a majo challenge. Labeling la ge da ase s is a cos ly p ocess, diligence is needed o au oma ize i , and some imes some deg ee o human in ol emen is necessa y. Me hodologies used o c ea e a da ase a y. Da a may be collec ed in a lab in con olled expe imen s. Da a can be sou ced om he in e ne , o hough c owdsou cing. Da a can be also ob ained by syn hesizing i . 4 2.2. COMPUTER VISION DATASETS Figu e 2.1: Example o images labeled wi h classes 2.1.2 So wa e o da a p ocessing C ea ing machine lea ning da ase s besides human inpu equen ly equi es da a p ocessing so wa e o a ious asks. The P ojec II in his epo is an example o de- eloping such so wa e o he compu e ision ask o came a objec pose es ima ion. 2.1.3 Benchma king pla o ms Fo machine lea ning companies i is a good p ac ice o ha e a benchma king ame- wo k. De elopmen o machine lea ning solu ions o complex p oblems is a con- s an ly e ol ing p ocess. In his si ua ion, exis ence o a p ope benchmak ing pla - o m is impo an o quan i ying e o s, o knowing pe o mance me ics o he cu en algo i hms, and o seeing how he models imp o e o e ime. In addi ion in he indus y applica ion, he e migh be a ious use cases o which he algo i hms apply. Then one needs o know wha a e he pe o mance me ics o each o he use cases and how hey imp o e o e ime. To achie e his one needs p ope e sioning sys em o models. One need o de ine use cases, and which da a is ele an o each o hem. One needs a da abase o s o e model esul s. And, one needs a dashboa d o isualiza ion o me ics. 2.2 Compu e ision da ase s One o he a eas whe e machine lea ning has had a emendous in luence in ecen yea s is compu e ision. In pa icula , he eeme gence o deep lea ning has e o- lu ionized compu e ision in he las en yea s o so. Mode n con olu ional neu al ne wo ks achie ed eno mous leaps in pa e n ecogni ion, pa e n de ec ion and o he compu e ision asks compa ed o wha was p e iously possible wi h adi ional im- age p ocessing me hods. 5 CHAPTER 2. BACKGROUND In he case o compu e ision, he da a on which algo i hms ope a e a e images. Supe ised aining and alida ion equi e image da a o be anno a ed, as o example, o image classi ica ion each image is anno a ed by a label speci ying i s class. The up- hea al o machine lea ning in compu e ision, besides de elopmen o new machine lea ning a chi ec u es and imp o emen in compu ing capaci y, has been accompa- nied wi h c ea ion o aining da ase s. A huge e o has been dedica ed o he ask o cons uc ing machine lea ning da ase s o a ious compu e ision applica ions. Some o he amous compu e ision da ase s include MNIST, ImageNe [5], CIFAR-10 [13], Mic oso COCO [14] and many o he s (see [19] o a long lis ). Fo compu e ision applica ions, da a collec ion in ol es aking images o eco d- ing ideos, which can be seen as sequences o images. In 3D compu e ision (see below) in addi ion o RGB images we also ha e dep h images. Fo supe ised lea n- ing, we need labels o images o ideo ames in ou da ase . F equen ly, he aw da a is no anno a ed wi h aining labels di ec ly. Ra he we ha e anno a ions abou he eco ding en i onmen , he wo ld. These anno a ions may include coo dina es o wo ld objec s, coo dina e ans o ma ion pa ame e s e c. T aining labels o some pa icula machine lea ning p oblem a e hen in e ed om hese pa ame e s. In ad- di ion, we may need o eco d in o ma ion such as e.g. ligh ing condi ions and ime o eco dings, as well as cha ac e is ics o wo ld objec s and/o pe sons, such as e.g. mo ion speed o an objec and age o a pe son. The decision on wha wo ld in o ma ion do we keep ack o du ing he eco dings is pa o he da a c ea ion p ocess. One impo an cha ac e is ics o aining da ase is a iabili y. In p ac ice, aining da ase is only a iny po ion o he eal da a o which he algo i hm is ying o lea n he p oblem. To ain a good model he aining da a has o ep esen he whole dis ibu ion o he eal da a which he model is ying o lea n. In compu e ision applica ions, he image o ideo iles ha we collec should show ex ensi e deg ee o a ia ion. Fi s we should ha e a iabili y in e ms o en i onmen , such as a iabili y in ligh ing condi ions. Secondly, we should ha e he p oblem speci ic a iabili y: I o example, ou model is ying o lea n dogs om images, in ou da ase we should ha e images o dogs o a ied size, colou , appea ing in a ious body posi ion, showing om di e en angles e c. In p ac ice one also has o conside he a iabili y o use cases o he machine lea ning applica ion. Fo example, i we a e s udying human a en ion sensing, we will wan o conside use cases o d i ing, e ail shopping, gaming e c. Aside om aining gene al models, i will be use ul o ain models o each pa icula use case. Fo his again, adequa e da a is needed. The e o e, we should know which pa o ou da a is ele an o which use case. These a e some o he conside a ions ha should be aking in o accoun when designing and building a compu e ision da ase s. 6 2.3. HUMAN DETECTION AND TRACKING 2.3 Human de ec ion and acking The objec i e o acking is o es ima e loca ion and pose o an objec om a ideo. The goal o human acking is o iden i y he pose o a ious body pa s (e.g. hands, inge s, head e c.) a each ame, and ack i o e ime. In applica ions, e en ually he objec o in e es may be beha iou ecogni ion, a en ion sensing, in en ion es ima ion, emo ion ecogni ion e c. Compu a ionally howe e , es ima ing pose o body pa s in ol es inding he loca ions o ce ain body landma ks, and es ima ing pa ame e s de ining he pose o he body pa s. T acking hen in ol es es ima ing hese da a a each ame o e ime. Figu e 2.2: Human pose es ima ion and acking om he wo k [1] Human de ec ion and acking da ase consis s o images o ideos con aining people. The aining labels a e pa ame e s such as loca ions o human bodies and human body landma ks and pa ame e s o a ious body poses such as head pose. A numbe o machine lea ning da ase s exis o human de ec ion and acking, many o hem specialized o mo e speci ic p oblems such as human pose es ima ion [2], ace ecogni ion (see [6]), head pose es ima ion [8], [9] eye acking [7], [21] and so on. They ha e been collec ed using di e en me hods and means. Some o hese we e c e- a ed h ough con olled da a collec ions, o he s we e collec ed hough c owdsou cing me hod. While, in o he app oaches image syn hesizing was used. 2.4 3D scanning and 3D compu e ision 3D scanning is a echnology which cap u es geome ic shape o objec s in he eal wo ld and, possibly, analyses cha ac e is ics such as colo . The e a e numbe o 3D 7 CHAPTER 2. BACKGROUND scanning me hods including lase scanning, ime-o - ligh and s uc u es-ligh echnolo- gies ( o mo e de ails see [3]). Popula mode n commodi y 3D scanning came as a e Realsence om In el and Kinec om Mic oso . The o me is suppo ed by an open sou ce c oss pla o m SDK [10]. These came as ha e bo h 3D scanning dep h s eam and an RGB s eam. F om he dep h measu emen s and he came a pin-hole model i is possible o compu e poin clouds, which a e se s o poin s in a 3D space. Poin clouds a e used o c ea e polygon meshes. Besides, 3D scanning de ices calib a e be ween dep h and RGB s eams, and gene a e ex u e mapping which de ines colo in o ma ion o he su ace scanned by he dep h senso . 3D compu e g aphics s o e geome ic objec s in 3D space, such as meshes, o isualizing hem in 2D o pe o ming compu a ional ope a ions on hem. 3D p o ides mo e in o ma ion han 2D images which can be exploi ed in compu - ing. Machine lea ning, and in pa icula deep lea ning, has been applied o dep h and colo da a o ecogni ion, de ec ion, acking e c. 8 Chap e 3 P ojec I: Machine Lea ning Da ase o Human De ec ion and T acking The Eu opean S a up uses da a o aining and benchma king i s machine lea ning algo i hms. The objec i e o he p ojec was o c ea e a specialized machine lea ning da ase o human de ec ion and acking. The da ase was o be comp ised o dep h and RGB ideo eco dings, oge he wi h anno a ions abou he eco ding en i onmen . The goal was o eco d mo e han 50 people o di e en appea ance wi hin a ying ambien condi ions and con ex s. 1 3.1 Eu opean S a up Da ase As a esul o he p ojec a Eu oean S a up Da ase was c ea ed. The da ase consis s o a ew hund ed eco dings by dep h sensing came as. Each eco ding is a ound 2 minu es long. Each o hem has an RGB s eam and a dep h s eam. The eco dings show a pe son doing simple body mo emen s ollowing he con en displayed on a sc een placed in on o hem. The eco dings show g ea deg ee o di e si y. Pa - icipan s o a ied age, gende , ace and appea ance whe e eco ded. The eco ding ambien ligh ing condi ions a iabili y was maximized. Fu he mo e, eco dings we e di e en ia ed by he sc een ype and he eco ding scena io. Th ee ypes o sc eens we e used o he eco dings. The eco dings we e done in a numbe o scena ios, o each o which speci ied ins uc ions we e gi en o he pa icipan s as o how o mo e hei head o body du ing he expe imen . Besides he ideos, he ollowing da a is included in he da ase . Fo each eco ding, sc een came a pose pa ame e s a e sa ed. Fo each eco ding, pixel coo dina es o 1 Fo con iden iali y easons we can no e eal he exac na u e o he da ase , he e o e he desc ip ion o he da ase and he da a collec ion p ocess a e gi en in gene al e ms. 9 CHAPTER 3. PROJECT I: MACHINE LEARNING DATASET FOR HUMAN DETECTION AND TRACKING elemen s o he sc een con en a e sa ed o abou 50 di e en ames a which i is assumed ha he pa icipan ’s body posi ion was engaged wi h he sc een con en as speci ied by he expe imen ules. Fo each eco ding, he e is a me ada a con aining in o ma ion abou he sc een ype, he eco ding scena io, as well as eco ding ambien condi ions and he pa icipan ’s cha ac e is ics. The ollowing ables summa ize he Eu opean S a up Da ase : Eu opean S a up Da ase Mo e han 200 Reco dings Reco ding RGB ideo Dep h ideo F ame-wise pa ame e s o he sc een con en Pa icipan Scena io Sc een ype 3.2 Da a collec ion The da a was collec ed om mo e han 50 people o e h ee weeks pe iod. The da ase was collec ed om olun ee pa icipan s. Wi h each pa icipan a ew ideo eco ding expe imen s we e made. Du ing a eco ding expe imen he pa icipan wa ches con en displayed on a sc een which changes h ough ime. The pa icipan mo es hei body ollowing he shown con en , o example poin ing some body pa s a a ious elemen s o he sc een con en . A dep h sensing came a is placed in on o he sc een acing he pa icipan , and eco ds he pa icipan (see Figu e 3.1). F om ime o ime he pa icipan “clicks” on an inpu de ice (such as a mouse o a space ba ) con i ming ha a ha momen hei body was posi ioned ela i e o he sc een con en as speci ied by he ules. This is when he pixel coo dina es o he sc een elemen s a e sa ed. In his way we ensu e ha a he momen s o "clicks" he pa icipan was p ope ly engaged wi h he sc een con en . Then, om he posi ion o he elemen s o he sc een con en in he wo ld i is possible o in e he g ound u h body pose pa ame e s which a e he aining labels o ou algo i hms. 10 3.2. DATA COLLECTION Figu e 3.1: Reco ding scene. The pa icipan is asked o change he body posi ion ollowing he con en shown on he sc een. As a esul , he aw da a p oduced by he eco ding expe imen s consis ed o ame- wise anno a ed ideo eco dings, which make up he co pus o he Eu opean S a up Da ase . The da a collec ion p ocess in ol ed a numbe o design and o ganiza ional chal- lenges and decisions. One had o make su e ha he p ocess was ollowed co ec ly by he pa icipan s and he o ganize s. Fo his pu pose a p ecise da a collec ion p o ocol was designed. In pa icula he p o ocol de ined he ules o engagemen by pa ici- pan s wi h he sc een con en . Also on he o ganiza ional side, since ou expe imen s in ol ed people, da a p i acy issues had o be aken in o accoun . In pa icula we ensu ed ha ou ac i i y was GDPR compa ible [18]. Logis ically, one had o secu e loca ions, olun ee pa icipan s and o he ma e ial o he eco dings. These esou ces had o exhibi su icien di e si y. The eco ding en i onmen s had o be di e se in illumina ion condi ion. As o pa icipan s, one had o secu e people o di e se age, gende , e hnici y and o he cha ac e is ics. In addi ion, we wan o eco d he pe son a di e en imes (o e ew days). The sc een on which he engagemen con en was displayed had o be o di e en size and mobili y. We used a LCD sc een, a p ojec o and a able which could be hand hold by he pa icipan . Resou ce Di e si y Dimension Loca ion Illumina ion condi ion Pa icipan Age, Gende , E hnici y, Clo hing ype, Time o eco ding Sc een Size, Mobili y 11 CHAPTER 4. PROJECT II: SCREEN CAMERA CALIBRATION TOOL The poin s whose coo dina es he calib a ion ool will collec a e he co ne s o a checke boa d pa e n displayed on a sc een. The pixel coo dina es o he checke boa d pa e n a e known. The p oblem is es ima ing hei coo dina es in WCS. The WCS is ixed wi h espec o he eco ding came a. The e o e, we can es ima e he WCS coo dina es o a poin which is isible by he eco ding came a. Howe e , he eco ding came a is placed so ha i can no see he sc een. This means ha we can no di ec ly es ima e coo dina es o a poin on a sc een by he eco ding came a. The solu ion is o in oduce ano he came a in he calib a ion p ocess. The new came a, which we will call an ex e nal came a is placed so ha i sees he sc een. A he same ime we ge a e e ence objec , and place i so ha bo h he eco ding came a and ex e - nal came a see i (see Figu e 4.3). Then, he ex e nal came a can ind coo dina es o a poin on a sc een wi h espec o i s own coo dina e sys em. While since he e e ence objec is isible by he bo h came as, bo h o hem can es ima e coo dina es o some e e ence poin s in hei espec i e coo dina e sys ems. These e e ence coo dina es hen a e used o ind he ans o ma ion om he ex e nal came a coo dina e sys em o he eco ding came a coo dina e sys em. Figu e 4.3: Calib a ion p ocess The calib a ion ool was implemen ed wi h an online mode and a eco ding-based mode. In he online mode he calib a ion ool uns on li e ideo da a om he wo came as, and collec s poin co espondences on he ly. In he eco ded mode, he calib a ion ool uns on sa ed eco dings. In he online mode he calib a ion p ocess wo ks as ollows: 1. The Sc een and he Reco ding Came a a e placed ixed in he posi ions in which hey will be du ing he da a collec ion expe imen . 2. The Ex e nal Came a is posi ioned in a way ha i usually sees he Sc een. 3. The Re e ence Objec is placed in a way ha i is usually isible by he bo h came as. 18 4.3. CALIBRATION TOOL 4. When he calib a ion ool s a s, a each li e ideo ame he ex e nal came a es ima es coo dina es o he co ne s o he checke boa d pa e n, while bo h he ex e nal came a and he eco ding came a es ima e coo dina es o some e e - ence poin s on he e e ence objec (see Sec ion 4.5). Using he coo dina es o he e e ence poin s es ima ed by bo h came as a ans o ma ion is ound be- ween he coo dina e sys ems o he wo came as. Using his ans o ma ion he checke boa d pa e n coo dina es a e ans o med o he WCS, and a e sa ed. A he same ime he pixel coo dina es o he checke boa d pa e n a e sa ed. A e ew ames, he poin s collec ion s ops. The collec ed poin co espondences a e ed o he linea eg ession which inds he calib a ion pa ame e s. In he eco ded mode he ool uns on eco ded ideos. Ano he di e ence o he eco ded mode om he li e mode is ha , he e e ence poin coo dina es a e no es ima ed by he eco ding came a ame-wise, bu o e he whole ideo. This means ha , unlike in he li e mode, he e e ence objec has o s ay ixed a he posi ion. Calib a ion in he eco ded mode is done as ollows: 1. The Sc een and he Reco ding Came a a e placed ixed in he posi ions in which hey will be du ing he da a collec ion expe imen . 2. The Re e ence Objec is placed in a way ha i is isible o he Reco ding Came a, and i s ays ixed. 3. The Reco ding Came a makes a eco ding o he Re e ence Objec and a ideo is sa ed, called In e nal Video. 4. The Ex e nal Came a is posi ioned in a way ha i usually sees bo h he Re e ence Objec and he Sc een, i makes a eco ding and sa es a ideo, called Ex e nal Video. 5. The calib a ion ool is un on he eco ded da a. Fi s i p ocesses he In e nal Video om whe e i ge he coo dina es o he Re e ence Poin s on he Re e ence Objec . Nex i p ocesses he Ex e nal Video. A each ame he coo dina es o he co ne s o he Checke boa d pa e ns a e es ima ed. A he same ime he coo dina es o he e e ence poin s a e es ima ed and using hem a ans o ma- ion be ween he wo came as is ound. Then, using his ans o ma ion he WCS coo dina es o he checke boa d co ne s a ha ame a e es ima ed and sa ed oge he wi h he pixel coo dina es. A e he ideo da a is p ocessed and poin co espondences a e collec ed he calib a ion pa ame e s a e ound by sol ing he linea eg ession p oblem. 19 CHAPTER 4. PROJECT II: SCREEN CAMERA CALIBRATION TOOL 4.4 Es ima ing 3D coo dina es o poin s As desc ibed abo e, he calib a ion p ocess in ol es es ima ing 3D coo dina es o poin s, such as co ne s o a checke boa d pa e n, o e e ence poin s on a e e ence objec . A each ideo ame a checke boa d pa e n is de ec ed using OpenCV unc ions [15]. This e u ns 2D coo dina es o he chessboa d co ne s on he image. Then, he 3D coo dina es a e es ima ed ei he by Pe spec i e-n-Poin me hod o by using dep h ideo da a. Pe spec i e-n-Poin me hod can es ima e 3D coo dina es o poin s om hei 2D p ojec ions on he image. OpenCV unc ion Sol ePnP is he OpenCV implemen a ion o his me hod which we used [15]. The al e na i e is o use 3D dep h da a, which b ings ex a p ecision o he es ima- ion. In his app oach, i s using he dep h da a we es ima e he plane on which he poin s lie (ou poin s always lie on a plane, ei he he sc een o he e e ence objec which is plana ), by i ing a plane o he poin cloud o he dep h ame. Nex we ind ays om he ocal poin o he came a going h ough he 2D p ojec ion o he poin s. Finally we ind he 3D coo dina es as in e sec ion poin s o he es ima ed plane and he ays. 4.5 T ans o ma ion be ween came a coo dina es The calib a ion ool in ol es inding a ans o ma ion o m coo dina e sys em o one came a o ano he one. This is done wi h help o a e e ence objec . Re e ence objec is an objec which is seen by bo h came as. The wo came as es ima e ce ain poin s on he e e ence objec , using which hen he ans o ma ion is ound. Figu e 4.4: Examples o A uco ma ke s. 20 4.5. TRANSFORMATION BETWEEN CAMERA COORDINATES The e e ence objec should ha e poin s which a e easily de ec able by a came a. We used a ca dboa d o which we a ached A uco ma ke s [16], [17]. A uco ma ke is a syn he ic squa e ma ke composed by a wide black bo de and a inne bina y ma ix which de e mines i s iden i ie . A uco lib a y [17] con ains unc ions o de ec ing, pose es ima ion and isualiza ion o A uco ma ke s, making A uco good ool o pose es ima ion applica ions. We placed se e al A uco ma ke s on ou e e ence objec , each A uco ma ke gi es ou e e ence poin s, each o which is uniquely iden i iable by he A uco ma ke iden i ie and he de e mined index among each ou on he same ma ke . In he calib a ion ool, bo h came as de ec hese uniquely iden i ied e e ence poin s. As a esul we ge 3D coo dina es o he e e ence poin s in he wo came a coo dina e sys ems. The ans o ma ion be ween hese coo dina e sys em hen is ound wi h Kabsch algo i hm [11] desc ibed below. Figu e 4.5: T ans o ma ion be ween he came a coo dina e sys ems. Gi en wo se s o co esponding poin s A and B (as o example, e e ence poin s es ima ed by he wo came as in ou se ing abo e) he algo i hm o ind op imal o a ion and ansla ion which aligns A o Bp oceeds as ollows: 1. Find cen oids o bo h se s: cen oidA= mean(A), cen oidB= mean(B).(4.9) 2. B ing bo h se s o he o igin: Ao=A−cen oidA, Bo=B−cen oidB.(4.10) 21 CHAPTER 4. PROJECT II: SCREEN CAMERA CALIBRATION TOOL 3. Find c oss-co a iance ma ix H=AT oBo. 4. Calcula e he SVD o he co a iance ma ix H=USV T. 5. Find he o a ion ma ix by R=V UT. 6. I de (R)<0, co ec Rby mul iplying i ’s hi d column by −1. 7. Find he ansla ion by = cen oidB−R(cen oidA). 4.6 Valida ion o calib a ion pa ame e s To alida e he calib a ion pa ame e s we implemen ed a isualiza ion ool. I uns on a ideo made by he ex e nal came a ilming he eco ding came a and he sc een. A each ame he sc een posi ion is es ima ed, hen using he ound sc een came a calib a ion pa ame e s he posi ion o he came a is ound and a came a schema is d awn on he ame. This allows us o make a isual compa ison o he es ima ed came a posi ion and he eal came a posi ion. 22 4.6. VALIDATION OF CALIBRATION PARAMETERS Figu e 4.6: Example o de ec ing a chessboa d pa e ns and A uco ma ke s. OpenCV lib a y is used. 23 Chap e 5 P ojec III: Benchma king Pla o m The Eu opean S a up de elops a cu ing-edge echnology which i s i es o cons an ly imp o e. The Eu opean S a up needs a mechanism which will allow seeing how i s echnology imp o es o e ime, bo h o in e nal and ex e nal use. The aim o his p ojec was o kick s a a pla o m o his pu pose. 5.1 Benchma king pla o m The benchma king pla o m consis s o a da abase o s o ing he ou pu o machine lea ning pipelines and a dashboa d o isualiza ion o me ics, as well as da a mo e- men modules o impo and expo o da a. 5.2 Me ics and use cases The p ocessing pipeline equi es mul iple s eps, such as ace de ec ion, head pose es i- ma ion, body pose es ima ion, e c. The me ics o in e es s a e quan i a i e measu es o he pe o mance o he algo i hms, such as how many alse de ec ions a e made, how many misses he sys em exhibi s, o how la ge a e he e o s. In he benchma king pla o m we keep ack o hese me ics. People de ec ion and acking pipelines change as algo i hms imp o e o e ime. In he benchma king pla o m each pipeline should be uniquely iden i iable. This is done using a gi ag. In addi ion we designed se e al use cases, each o hem co esponding o applica- ion use case such as beha iou acking while d i ing, o beha iou acking while e ail shopping. A use case is iden i ied by i ele an da a. Fo example, o a e ail 25 CHAPTER 5. PROJECT III: BENCHMARKING PLATFORM Figu e 5.1: Benchma king pla o m shopping use case we migh use ideos which show people passing by he came a as hey would in a e ail s o e aisles. 5.3 Da abase We buil a da abase o ou benchma king pla o m using MySQL. Ou da abase should s o es esul s o people de ec ion and acking expe imen s as well as ou pipelines, use cases, pe sons (expe imen pa icipan s) and eco dings. The da abase should also e lec he ela ionship among hese da a. Du ing he design p ocess, da a o be s o ed was iden i ied and da a in e ela ionships de ined. We build a no malized da abase wi h a ew ables. Some o he ables a e o da a om machine lea ning pipelines, o example a able o body landma k coo dina es and a able o head pose pa ame e s. O he ables a e o me ada a, o example a able o pipelines and a able o eco dings. We implemen ed da a expo module which expo s he esul s o unning a pipeline o he da abase. We also implemen ed modules ha impo me ada a abou pe sons and eco dings om he Eu opean S a up Da ase . Use case de inin ion mechanism was also implemen ed. 26 5.4. DASHBOARD 5.4 Dashboa d We designed a dashboa d wi h py hon Plo ly Dash lib a y [4]. The dashboa d isual- izes he people de ec ion and acking me ics. The use o he dashboa d can speci y pipeline and/o use case e c. Dash will load da a o ha speci ic eques , compu e me ics and isualize hem. Plo ly Dash de ines dashboa d page layou simila ly o HTML, and i comes wi h a ious widge s o da a isualiza ion. Dash suppo s eac i e p og amming s yle which we ollowed in ou implemen a ion. The inpu o he use is au oma ically p opaga ed h oughou he dashboa d and me ics and isualiza ions a e upda ed. 27