Automatic 3D Object Recognition and Localization for Robotic Grasping

FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO Au oma ic 3D Objec Recogni ion and Localiza ion o Robo ic G asping B uno Miguel Sil a Espí i o San o WORKING VERSION Mes ado In eg ado em Engenha ia Ele o écnica e de Compu ado es Supe iso : Gil Manuel Magalhães de And ade Gonçal es Second Supe iso : Liliana Pa ícia Saldanha An ão July 6, 2020 Abs ac Wi h he ad en o Indus y 4.0 and i s highly econ igu able manu ac u ing con ex , he ypical ixed-posi ion g asping sys ems a e no longe usable. This eali y unde lined he necessi y o ully au oma ic and adap able obo ic g asping sys ems. The de elopmen o a Compu e Vision sys em capable o de ec ing, iden i ying and es ima ing he 6D pose o an objec would b ing us close o ha eali y. Wi h ha in mind, he main pu pose o his hesis is o join Machine Lea ning models o de ec ion and pose es ima ion in o an au oma ic sys em o be used in a g asping en i onmen . To achie e his, ex ensi e esea ch was ca ied ou on he cu en s a e-o - he-a app oaches. The de eloped sys em uses Mask-RCNN and Dense usion models o he ecogni ion and pose es ima ion o objec s, espec i ely. The g asping is execu ed aking in o conside a ion bo h he pose and he objec ’s ID, as well as allowing o use and applica ion adap abili y h ough an ini ial con igu a ion. The sys em was es ed bo h on a alida ion da ase and in a eal wo ld en i onmen . The main esul s show ha he sys em has mo e di icul y wi h complex objec s, howe e , i shows p omising esul s o simple objec s, e en wi h aining on a educed da ase . I is also able o gene alize o objec s sligh ly di e en han he ones seen in aining. In g asping expe imen s, he e is a 60% success a e in he bes cases, o simple g asping a emp s. i ii Acknowledgemen s My deepes hanks o bo h my supe iso s. I was an absolu e pleasu e o wo k on his hesis wi h such g ea people and in a e y enjoyable en i onmen , despi e he di icul wo king si ua ions his yea . A lo was lea ned in de eloping his hesis and I could no ha e done i wi hou hei i eless help. B uno Miguel Sil a Espí i o San o iii i Con en s Lis o Figu es ii Lis o Tables ix Abb e ia ions xi 1 In oduc ion 1 1.1 Con ex ....................................... 1 1.2 Mo i a ion...................................... 2 1.3 P oblemDe ini ion ................................. 3 1.4 Objec i es...................................... 4 1.5 ThesisS uc u e................................... 4 2 Li e a u e Re iew 7 2.1 MachineLea ning.................................. 7 2.1.1 A i icial Neu al Ne wo ks . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2 PoseandRo a ions ................................. 13 2.3 Rela edWo k .................................... 13 2.3.1 Da ase s................................... 14 2.3.2 Machine Lea ning in Objec De ec ion and Recogni ion . . . . . . . . . 15 2.3.3 Machine Lea ning in Pose Es ima ion . . . . . . . . . . . . . . . . . . . 21 3 3D Objec Recogni ion and Localiza ion Sys em 31 3.1 Sys emO e iew.................................. 31 3.1.1 Ope a ionModes.............................. 32 3.2 Sys em’sModules.................................. 35 3.2.1 Da a Acquisi ion Module . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.2.2 Compu e Vision Module . . . . . . . . . . . . . . . . . . . . . . . . . . 36 4 Expe imen s and Resul s 41 4.1 Da ase ....................................... 41 4.2 O lineTes ingPhase ................................ 43 4.2.1 T aining he Objec De ec ion Model . . . . . . . . . . . . . . . . . . . 43 4.2.2 Pose Es ima ion Model T aining . . . . . . . . . . . . . . . . . . . . . . 45 4.2.3 Compu e Vision Module Tes ing . . . . . . . . . . . . . . . . . . . . . 46 4.3 OnlineTes ingPhase ................................ 47 4.3.1 Da a Acquisi ion Module Tes ing and Analysis . . . . . . . . . . . . . . 48 4.3.2 Objec De ec ion and Segmen a ion Tes ing . . . . . . . . . . . . . . . . 48 4.3.3 G aspingExpe imen s ........................... 54 i CONTENTS 4.4 Limi a ions ..................................... 56 5 Conclusions and Fu u e Wo k 59 5.1 Conclusions..................................... 59 5.2 Fu u eWo k..................................... 60 Re e ences 61 Lis o Figu es 2.1 Rep esen a ion o a pe cep on . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2 Rep esen a iono aae................................ 10 2.3 NNVSCNN .................................... 11 2.4 Illus a ion o he a chi ec u e o Fas e R-CNN . . . . . . . . . . . . . . . . . . 15 2.5 Illus a ion o he a chi ec u e o R-FCN . . . . . . . . . . . . . . . . . . . . . . 17 2.6 Anexampleo asco emap............................. 17 2.7 O e iew o he SSD a chi ec u e . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.8 O e iew o he o iginal YOLO ne wo k . . . . . . . . . . . . . . . . . . . . . . 19 2.9 O e iew o he SSD-6D model . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.10 Illus a ion o e e y iewpoin conside ed . . . . . . . . . . . . . . . . . . . . . 22 2.11 O e iew o he PoseCNN sys em . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.12 O e iew o he DPOD sys em . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.13 Illus a ion o he pose e inemen block . . . . . . . . . . . . . . . . . . . . . . 25 2.14 O e iew o he DenseFusion a chi ec u e . . . . . . . . . . . . . . . . . . . . . 26 2.15 Illus a ion o he pose e inemen ne wo k . . . . . . . . . . . . . . . . . . . . . 27 2.16 Illus a ion o he Hyb idPose model . . . . . . . . . . . . . . . . . . . . . . . . 28 2.17 O e iew o he DeepIM model . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.1 Illus a ion o he da a low in he sys em . . . . . . . . . . . . . . . . . . . . . . 31 3.2 O e iew o all ope a ion modes . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.3 S a e machine o he pose es ima ion module . . . . . . . . . . . . . . . . . . . . 34 3.4 ZED S e eo Came a and dimensions illus a ion . . . . . . . . . . . . . . . . . . 35 3.5 O e iew o he Mask-RCNN model . . . . . . . . . . . . . . . . . . . . . . . . 37 4.1 Dis ibu ion o numbe o ins ances pe objec . . . . . . . . . . . . . . . . . . . 41 4.2 Selec ion o illus a i e images om he da ase . . . . . . . . . . . . . . . . . . 42 4.3 O e iew o all he scena ios used in de ec ion es ing . . . . . . . . . . . . . . . 49 4.4 Illus a ion o one g asping posi ion pe objec . . . . . . . . . . . . . . . . . . . 54 4.5 RMS e o in dep h da a o 720p esolu ion . . . . . . . . . . . . . . . . . . . . 56 ii 2In oduc ion ML is a ield o esea ch dedica ed o compu e s’ abili y o in e ac wi h hei en i onmen and lea n om he da a collec ed. I is based on ma hema ical models, employed o enable he lea ning o pa e ns in da a, and is o en used o inc ease he sys em’s in elligence/ lexibili y o when p oblems canno be sol ed using adi ional explici p og amming. On he o he hand, Deep Lea ning is a b anch o ML ha desc ibes a se o modi ied ML echniques inspi ed by he biological ne ous sys em. Composed o ne wo ks o pa allel and concu en con olu ions, as well as o he ma hema ical ope a ions pe o med di ec ly on he inpu da a, i acqui es a g oup o ep esen a i e heu is ics be ween inpu and ou pu da a. Due o he con incing esul s ha ha e been achie ed in he scope o compu e ision, he e is an inc easing end owa ds he implemen a ion o DL algo i hms in obo ic g asping applica ions. Wi h ML and DL, obo s a e able o in e ac wi h hei en i onmen and espond o a ious s imuli in a comple ely au oma ed manne , enabling obo s o use came as o pe cei e hei en i onmen and e en "unde s and" i when using hese echniques. The main challenges in he ield o CV co e ed in his hesis a e objec de ec ion/ ecogni ion and pose es ima ion, which will be ackled wi h he use o ML models. In objec de ec ion and ecogni ion, he ask is o iden i y objec s in an image and a ibu e a seman ic classi ica ion (iden- i y he class i belongs o). As o pose es ima ion, he e is a need o es ima e he objec ’s posi ion and o ien a ion ela i e o he came a. 1.2 Mo i a ion As p e iously men ioned, objec de ec ion/ ecogni ion and pose es ima ion is o g ea impo ance o obo ic sys ems in se e al asks. One o hese asks is obo ic g asping: he ac o a obo ic manipula o handling objec s in an au oma ed manne . Fo ypical obo ic g asping o be accomplished, he obo needs o be able o know whe e and how an objec is posi ioned in i s coo dina e sys em. 6-dimensional (6D) poses, composed o 3 DOF o he posi ion and 3 o o ien a ion, a e o en used o his pu pose. Con a y o ypical 3D pose, 6D poses p o ide he o a ion, gi ing inpu no only on he "whe e" bu also on he "how." Ne e heless, when compa ed o humans, obo s p esen signi ican ly lowe success a es in g asping objec s in dynamic en i onmen s, as one would expec . Humans a e inhe en ly good a pe cei ing hei en i onmen and he objec s ha su ound hem, iden i ying hei cha ac e is ics, posi ion, and deciding na u ally whe e o g asp an objec and how much o ce o exe on i . Wi h he eali y o inc easing Human-Robo Collabo a ion solu ions in he indus y, humans, and obo s no only sha e physical space bu wo k oge he as a eam. I is o g ea impo ance o a eam o ha e he same pe cep ion and conside a ions o he asks hey will ul ill. Gi en his, ha ing g asping solu ions ha do no ake in o conside a ion he same aspec s as humans do, would lead o wo se eam pe o mance due o misma ches in g asping posi ions o hando e s, o e en due o g ippe damages in g asped ma e ial. Inspi ed by his, mo i a ion is p o ided o de elop solu ions ha imp o e manipula ion asks in lexible and collabo a i e indus ial en i onmen s, cha ac e is ic o he new indus ial eali y 1.3 P oblem De ini ion 3 imposed by Indus y 4.0. By p o iding obo s, in an indus ial en i onmen , wi h he abili y o lea n hei g asping asks au onomously in a simila manne o humans, collabo a i e obo ics is also empowe ed. This way, obo s a e gi en mo e pe cep ion o he objec s in a manipula ion ask. O e all, obo s a e able o wo k mo e in ui i ely wi h human ope a o s and e en assis hem in a ious asks in a sa e and sha ed en i onmen . 1.3 P oblem De ini ion The p oblem o objec de ec ion and pose es ima ion is one o he ac i e esea ch ha has su e ed inc emen al imp o emen s, as can be seen in chap e 2. Achie ing accu a e pose es ima ion o an objec would enable obo s o pe o m g asping in a ully au oma ic and e icien manne . Howe e , in o de o imp o e obo ic g asping, besides localiza ion, p o iding objec iden i i- ca ion can also be use ul. Fo his, ML is o en used, classi ying he seman ic class o he objec ha can hen be u ilized o adap ha g asping o he objec ’s cha ac e is ics. Fo ins ance, he g ipping s eng h ha he manipula o can apply wi hou causing damage o he objec can be in- e ed om i s classi ica ion, o e en choosing which objec o pick ou in a g oup o objec s, by i s cha ac e is ics. The e a e al eady many challenges wi h de ec ing and es ima ing he pose o objec s, a ising wi h he a iabili y o he su ounding en i onmen , such as a ying ligh condi ions, occlusion o objec s, and di e si y o objec dimensions. O he issues can also occu in e ms o he accu acy o he senso s used and he ypes o noise in oduced. Gi en his, he sys em o be de eloped needs o be accu a e and as enough o wo k in a eal indus ial en i onmen and be able o handle mul iple objec s o di e en dimensions, in si ua ion o occlusions o pa o he objec s. In sum, challenges ha a ise in objec de ec ion and pose es ima ion sys ems, by hemsel es, a e: •The ligh ing condi ions o he en i onmen ; •Occlusion and unca ion o ce ain objec s; •The need o models o be able o pe o m in eal- ime wi h low p ocessing speeds; •The need o high accu acy in indus ial en i onmen s whe e sys ems need o be sa e o ope a e. The e a e al eady many objec de ec ion/classi ica ion sys ems, as well as pose es ima ion models o g asping. Howe e , e y ew showing he bene i s o joining bo h app oaches o a mo e comple e and au onomous g asping, much less ully au oma ed o based on only came a ame in o ma ion (bo h RGB and/o dep h). The e o e, he e is a challenge o joining localiza ion and ca ego iza ion o a bi a y objec s in 3D o g asping, in eal- ime, using li e- eed came a moni o ing. This would ul ima ely b ing us a s ep close o seamless Human-Robo Collabo a ion. 4In oduc ion 1.4 Objec i es Gi en he p oblem de ined abo e, he pu pose o his hesis is o explo e and de elop an in elligen and ully au oma ed sys em o eal- ime objec ecogni ion and localiza ion de i ed om he analysis o da a om a s e eo ision came a. The de ec ion/ ecogni ion o a ious objec s and hei localiza ion wi h 6-DOF (posi ion and o ien a ion in 3D space) will be e ie ed o hen pe o m g asping asks acco ding o ha da a. This sys em should be able no only o ca ego ize and accu a ely loca e mul iple objec s bu also o be obus o occlusions. Finally, wi h his solu ion, a highe -le el obo g asping is hoped o be achie ed, imp o ing lexible and au onomous obo ic solu ions. In sum, he ollowing objec i es a e expec ed o be achie ed by he end o he hesis: 1. Ca y ou ex ensi e esea ch o he cu en s a e-o - he-a app oaches o objec de ec ion and ecogni ion, as well as pose es ima ion; 2. Gain an unde s anding o he e olu ion o models in his ield; 3. C ea e a sys em, based on ML, capable o de ec ing and classi ying an objec and es ima ing i s pose wi h 6-DOF; 4. The sys em needs o be ully au oma ed; 5. The sys em needs o be able o p ocess da a in eal- ime; 6. The ML model should be able o handle si ua ions o occlusion; 7. A obo ic manipula o should success ully and app op ia ely pe o m manipula ion asks o a ious objec s when ecei ing inpu s om he sys em; 1.5 Thesis S uc u e The ollowing ex is composed o ou chap e s: Li e a u e Re iew (chap e 2), 3D Objec Recog- ni ion and Localiza ion Sys em (chap e 3), Expe imen s and Resul s (chap e 4) and, inally, Con- clusions and Fu u e Wo k (chap e 5). Each chap e , as well as i s sec ions a e explained in his sec ion. In chap e 2, he main goal is o ca y ou ex ensi e esea ch o he ield in ques ion. In his chap e , we will i s b ie ly desc ibe Machine Lea ning and i s ca ego ies, ollowed by a mo e de ailed look in o A i icial Neu al Ne wo ks and some o i s a chi ec u es used in Compu e Vision. A sho o e iew o he de ini ion and cha ac e is ics o he objec ’s pose and o a ion will also be gi en. In chap e 3, i s , an o e iew o he comple e sys em o Au oma ic Objec Recogni ion and Localiza ion o obo ic g asping is gi en, as well as a look a i s da a low. A mo e in-dep h 1.5 Thesis S uc u e 5 desc ip ion is p o ided o he implemen a ion o his sys em, gi ing de ail no only on each ma- chine lea ning a chi ec u e bu also on each module and he sys em’s ope a ing modes as a S a e Machine. In chap e 4, he a ious expe imen s and esul s a e p esen ed, as well as a discussion o hose esul s and wha hey mean o he limi a ions o he sys em de eloped. In chap e 5, he conclusions a e made and a ew ideas o u u e wo k a e also p esen ed. 6In oduc ion Chap e 2 Li e a u e Re iew In his chap e , we will i s b ie ly desc ibe Machine Lea ning and i s ca ego ies, ollowed by a mo e de ailed look in o A i icial Neu al Ne wo ks and some o i s a chi ec u es used in Compu e Vision. A sho o e iew o he de ini ion and cha ac e is ics o he objec ’s pose and o a ion will also be gi en. Finally, he ela ed wo k ega ding objec de ec ion and pose es ima ion is p esen ed, de ailing commonly used da ase s, as well as Machine Lea ning me hods o bo h objec de ec ion and iden i ica ion and pose es ima ion. 2.1 Machine Lea ning Machine Lea ning is a subse o A i icial In elligence, and i deals wi h he abili y o compu e sys ems o ecognize pa e ns and in e om hem. This means ha , ins ead o asks ha ing o be di ec ly p og ammed, a ML model can be ained on sample da a and make decisions based on he gene aliza ions i makes om ha da a. ML is o g ea impo ance in au oma ion, enabling sys ems o pe o m bo h ou ine and com- plex asks (such as analyzing la ge se s o da a), wi h he added ad an age o making he sys em lexible o changes. I has p o en o be capable o sol ing a a ie y o asks, om image de ec ion o speech ecogni ion, ha can some imes be conside ed ha d o humans. This b anch o AI can be di ided in o h ee majo ca ego ies: 1) Unsupe ised lea ning, 2) Supe ised lea ning, and 3) Rein o cemen lea ning. O hose, only he i s wo will be e iewed, as hey a e he mos ele an o he opic in ques ion[2]. In Unsupe ised lea ning, he da a ed in o he model is no labeled. This means ha he ML model ies o ind he pa e ns o cha ac e is ic ea u es in he da a i ecei es, c ea ing a so o summa y wi hou being o e ed he eedback ha labeled aining da a would o e [2]. Me hods om his ca ego y gene ally can no be di ec ly applied o classi ica ion o eg ession p oblems, gi en ha he e is no p ecise knowledge o wha he ou pu da a migh be. Basically, his ca ego y exploi s he unlabeled da a a iance and sepa abili y o e alua e ea u e ele ance. Some o he mos known applica ions o unsupe ised lea ning include: 7 8Li e a u e Re iew •Clus e ing, which allows spli ing he da ase au oma ically in o g oups ("clus e s") acco d- ing o simila i y, o en ailing o ea da a poin s as indi iduals •Anomaly de ec ion, whe e he goal is o de ec a ypical da a poin s in he da ase au oma i- cally; •Associa ion mining, whe e se s o i ems ha occu oge he o en, a e iden i ied; •La en a iable models ha a e usually u ilized as me hods o p e-p ocessing da a (such as dimensionali y educ ion) o acili a ing da a isualiza ion ools. The main algo i hms in Unsupe ised Lea ning include K-means and P incipal Componen Analysis (PCA). On he o he hand, wi h Supe ised lea ning, he models ecei e labeled da a, e e ed o as " aining da a". This means ha he inpu consis s o he "objec " o be analyzed and he desi ed ou pu alue. This ype o lea ning is usually done in he con ex o classi ica ion (map inpu da a o ou pu class) o eg ession (map inpu da a o con inuous ou pu ). Usual algo i hms in supe ised lea ning include Logis ic Reg ession, Suppo Vec o Ma- chines, o A i icial Neu al Ne wo ks. The objec i e is he same in classi ica ion and eg ession: o ind pa icula ela ionships in he inpu da a ha enable he model o compu e he ou pu co - ec ly. In o he wo ds, he labels a e used o calcula e he co ec mapping unc ions o he inpu s ecei ed, which will enable he model o classi y new unlabeled inpu da a [3]. When pe o ming supe ised lea ning, model complexi y should be conside ed. The complex- i y o he model usually is dependen on he na u e o he aining da a. I he da ase is small, o i he da a be ween classes is clea ly dis inc , one should choose a low-complexi y model (wi h a high-complexi y model, i would likely o e - i , i.e., i would no be able o gene alize o o he da a poin s). The p oblem is ha labeled da a is ha d o ob ain since human anno a ion is bo ing. Labeling may equi e expe s o e en special de ices and can be e y ime-consuming. Gi en his, a possible solu ion is using semi-supe ised lea ning. This ype o lea ning is di e en om he me hods men ioned abo e since he inpu da a is a mix o labeled and unlabeled da a. Basically, i is an in e media e be ween supe ised and unsupe ised lea ning. The s anda d p ocess in ol es i s clus e ing simila da a using an unsupe ised lea ning al- go i hm, and hen, using he exis ing labeled da a, he unlabeled da a is labeled. The use o his me hod depends mainly on he applica ion and he se ing [3]. Wi h his b ie explana ion o he majo ca ego ies o ML and hei cha ac e is ics, a mo e in-dep h look a A i icial Neu al Ne wo ks, a e sa ile and widely used ML model, e y common in he opic o CV and objec de ec ion, will be gi en. 2.1 Machine Lea ning 9 2.1.1 A i icial Neu al Ne wo ks A sho in oduc ion o a i icial neu al ne wo ks and i s cha ac e is ics a e p o ided he e in b ie . This ype o model is one o he mos p e alen in ML, due o he ac ha ANNs a e some o he mos e sa ile and e ec i e algo i hms, and, as a esul , hey a e one o he mos signi ican subjec s in he ield o AI. Mos ANNs ha e, a hei basic le el, a i icial neu ons called "pe cep ons" (Fig. 2.1). Figu e 2.1: Rep esen a ion o a pe cep on. Image adap ed om [4] A pe cep on (o neu on) ecei es a ious inpu s (xi) mul iplied by i s co esponding weigh s (wi, numbe s exp essing he impo ance o he espec i e inpu s o he ou pu ) and akes he sum o hese esul s o p oduce a single ac i a ion (Eq. 2.1). The e can also be an addi ional e m, e e ed o as bias (b), ha ansla es how easy i is o ge he pe cep on o i e. a= n ∑ i=1 xi×wi+b(2.1) The pa ame e ain Eq. 2.1 is known as ac i a ion. The neu on’s ou pu (y) esul s in he ac i a ion o he pe cep on and is dependen on he alue aand he ac i a ion unc ion used. The ac i a ion unc ion is used o in oduce non-linea i y in o he ou pu o he pe cep on. This unc ion is chosen acco dingly o he na u e o he da a and he dis ibu ion o a ge a iables. Each ype o ac i a ion unc ion en ails a di e en ixed ma hema ical calcula ion. A adi ional A i icial Neu al Ne wo k is made up o in e connec ed laye s o pe cep ons (neu ons). The i s laye is known as he inpu laye , he las as he ou pu laye and he middle laye s a e e e ed o as hidden laye s. The inpu laye usually is equal o he numbe o di e en labels in a da ase , while he numbe o hidden laye s can ange om one o hund eds and a ies acco ding o he numbe o labels and compu a ion capabili y. As aining da a is ed h ough he ANN, he e o be ween he ac ual and he desi ed ou pu s o he ne wo k a e calcula ed using a cos /loss unc ion; his e o is hen used o adjus he weigh s and biases o he ANN. How hese adjus men s a e made depends on he op imiza ion algo i hm used, and on he pa ame e s o he ne wo k ha de ine he a ious ypes o a chi ec u es. 10 Li e a u e Re iew Mo eo e , he connec ions be ween neu ons can also be bi-di ec ional, leading o o he di - e en a chi ec u es. All o his con ibu es o he as amoun o ANNs in exis ence and he complexi y o his subjec [5]. In he nex subsec ions, some o he mos used ANN a chi ec u es in CV will be p esen ed b ie ly. 2.1.1.1 Au oencode s An Au oencode is an unsupe ised ANN, ha i s lea ns o comp ess and encode da a, and hen lea ns o econs uc back he da a om he encoded e sion. In a nu shell, an au oencode educes da a dimensions by lea ning how o igno e noise in he da a. Fo his o happen au oencode s ha e ou main pa s: encode ;bo lleneck,decode and econs uc ion loss. An example o an au oencode a chi ec u e is p esen ed in Fig. 2.2, whe e ˆxis he econs uc- ion o he o iginal inpu x. The encode akes i s inpu and ou pu s a ec o wi h i s encoded in o ma ion, comp essing and educing he inpu da a dimensions. The bo leneck is a laye in which he inpu ’s comp ession ep esen a ion is con ained ( he lowes possible dimensions o he inpu ). The decode ’s job is o hen ake his comp essed ec o and ou pu he closes ma ch o he o iginal inpu . The econs uc ion loss unc ion is hen minimized, which helps bo h he encode and decode lea n wi hou he need o labeled da a. This loss measu es how well he decode pe o ms by calcula ing how close he ou pu is o he o iginal da a. Howe e , i an encode lea ns o ma ch he o iginal inpu exac ly, he e is no use ulness o he ne wo k. Fo ha eason, es ic ions a e applied o limi how much o he inpu can be copied. This esul s in a unc ion ha p io i izes and only ou pu s he mos impo an cha ac e is ics o i s inpu [6]. Figu e 2.2: Gene al a chi ec u e o an Au oencode . 2.1 Machine Lea ning 11 2.1.1.2 Con olu ional Neu al Ne wo ks Ano he ype o ANN is he Con olu ional Neu al Ne wo k (CNN). Since, in a adi ional neu al ne wo k, all laye s a e ully connec ed, hey a e no able o ake in o accoun he image’s spa ial na u e (s uc u e and pa e n). In o he wo ds, all inpu pixels in he image would be ea ed he same way, independen ly o being a apa o close oge he . This is whe e CNNs a e mos use ul. They a e a class o deep neu al ne wo ks (an ANN wi h mul iple hidden laye s) op imized o da a in g id o m, which makes i especially use ul o image p ocessing. Fo his eason, CNNs ha e e ol ed o be he mos used me hod o sol ing a ious compu e ision p oblems [7] and will appea in he majo i y o Sec ion 2.3. CNNs a e iden ical o egula NN in e ms o being composed by neu ons, whe e he goal is o es ima e weigh s and bias. Howe e , CNNs ha e a much mo e e icien s uc u e o image p ocessing han NNs, gi en ha each neu on is only connec ed o a speci ic egion o he p e ious laye . The gene al a chi ec u e o a NN and CNN a e p esen ed in Fig. 2.3 o compa ison. Each laye o a CNN is ep esen ed as 3D olume [8]. Figu e 2.3: NN a chi ec u e VS CNN a chi ec u e. A s anda d NN is ep esen ed on he le , and a CNN on he igh . Each ci cle ep esen s a neu on [8]. This class o ne wo k uses a ian s o he ma hema ical ope a ion con olu ion. Con olu ion (deno ed as ∗) can be seen as he ma ix mul iplica ion o an inpu Iby a ke nel K(as seen in Eq. 2.2). Using a ke nel wi h a size signi ican ly in e io o he inpu size, a CNN can ex ac small ea u es o an image. This also esul s in ewe calcula ions and, he e o e, an inc ease in speed. C(i,j) = (I∗K)(i,j) = m ∑ i=1 n ∑ i=1 I(i−m)(j−n)K(m,n)(2.2) Typically, he laye s o hese ne wo ks a e made up o h ee s ages: con olu ion s age, de ec o s age, and pooling s age. In he con olu ion s age, mul iple pa allel con olu ions a e pe o med and esul in a collec ion o linea ac i a ions. A e ha , in he de ec o s age, a non-linea ac i a- ion unc ion is used on he linea ac i a ions. The p oblem a ises due o he ac ha hese laye s encode he in o ma ion in he p ecise posi- ion i is in, meaning ha i hese laye s p ocess he same inpu wi h a sligh change, i will esul in a di e en ou pu . This is whe e pooling unc ions a e use ul, making he ne wo k esis an , o in a ian , o small changes in he inpu . 18 Li e a u e Re iew •Using bo h VOC and COCO aining da a leads o inc eased accu acy: 83.6% and 82% s. 80.5% and 77.6%; •Using he i s 101 laye s o ResNe esul s in he bes accu acy: sa u a ion occu s a 80.5%; •The use o a RPN achie es supe io esul s han o he al e na i es: 79.5% s. 77.8% ( o he closes al e na i e). The a chi ec u e o he R-FCN emo es he expensi e compu a ion ha app oaches such as Fas e R-CNN use on each ROI (by passing each ROI h ough CNNs), in a o o a much mo e s aigh o wa d calcula ion o o e lap, enabling he model o be as e bu s ill main ain he accu- acy o e ed by a egion-based design. 2.3.2.3 SSD: Single Sho Mul ibox De ec o SSD (Fig. 2.7) [23] is one o he single-sho app oaches o objec de ec ion wi h he in en o being as and accu a e enough o be used in eal- ime applica ions. This de ec o emo es he need o egion p oposals, while s ill main aining accu acy. Figu e 2.7: O e iew o he SSD a chi ec u e. [23]. The i s s age in SSD is he gene a ion o ea u e maps a mul iple scales, which allows o he de ec ion o objec s o di e en sizes - ea u e maps wi h highe esolu ion a e able o de ec smalle objec s, while ones wi h lowe esolu ion can de ec bigge objec s. The backbone a chi- ec u e used is based on VGG, and o he con olu ional laye s a e appended a he end o allow o di e en scales. In he second s age, de ec ion is based on small con olu ional ke nels applied o each cell in he ea u e maps. Fo e e y cell, ou de aul bounding-boxes o di e en aspec a ios a e applied. These aspec a ios a e chosen o accommoda e objec s o di e en shapes and sizes (Eq. 4 in [23]). Fo each box, a class p obabili y sco e is compu ed, as well as di e en o se s o he size o he box (in o de o be e adjus he shape o he box o he shape o he objec ). This app oach is simila o ha o Fas e R-CNN bu applied o di e en esolu ions o ea u e maps. A pa icula ly in e es ing aspec in [22] is he use o "da a augmen a ion" o aining. This means ha , o each aining image, ei he he en i e image is used o a andom pa ch is ob ained 2.3 Rela ed Wo k 19 om i . In he case o a andom pa ch, i can also su e image dis o ions. The objec i e o da a augmen a ion is o enable he model o handle a ious shapes and sizes o objec s mo e obus ly. The model was e alua ed on he Pascal VOC and MS COCO da ase s, whe e he main conclu- sions we e he ollowing: •As wi h p e ious models, aining on bo h he VOC and COCO da ase s inc eases accu acy (Tables 1, 4 and 5 in [22]); •Using mul iple ou pu s laye s o mul iple ea u e map esolu ions esul s in highe accu acy (Table 3 in [22]); •Da a augmen a ion achie es be e esul s, as expec ed Table 6 in [22]); •SSD can achie e highe mAP han i s compe i o s using a smalle esolu ion inpu image Table 7 in [22]). 2.3.2.4 YOLO: You Only Look Once YOLO was i s in oduced in [24], su e ing subsequen al e a ions in [25] and some mino changes in [26]. I was in oduced as a single-sho app oach o objec de ec ion, by using a single ne wo k o sol e a eg ession p oblem o p edic bounding boxes and class p obabili ies o objec s in an image. The i s e sion o YOLO di ides i s inpu image in o a 7x7 g id, and each cell in he g id is esponsible o iden i ying an objec i i s cen e is loca ed wi hin he cell. Each cell p edic s a se o 2 bounding-boxes wi h associa ed con idence sco es, as well as class p obabili ies. The con idence sco es ake in o accoun he p obabili y o he boxes ha ing an objec wi hin hem and also he accu acy o he box in e ms o he shape o he objec . One p oblem ha can a ise om his app oach is he same objec being de ec ed by mo e han one cell. In his case, he cell wi h he highes con idence o a speci ic objec is chosen. The p edic ions o each cell a e made by classi ie s applied a e ea u e ex ac ion (which is done on he en i e image by he ne wo k in Fig. 2.8). Figu e 2.8: O e iew o he o iginal YOLO ne wo k. [24]. 20 Li e a u e Re iew In he second e sion o YOLO [25], bounding-boxes we e eplaced by he same ancho boxes used in Fas e R-CNN, since i made lea ning easie o he ne wo k and i also enabled each cell o p edic mo e han one objec . This had a small dec ease in accu acy, bu o e all ecall imp o ed. The numbe o ancho boxes chosen o his app oach was 5. Ano he change was eplacing he o iginal backbone ne wo k (Fig. 2.8) wi h a cus om ne wo k called Da kne -19, whose pu pose was o educe complexi y and imp o e accu acy. A signi ican imp o emen in his e sion was ha his model was able o de ec mo e han 9000 objec ca e- go ies, compa ed o he o iginal 20. The hi d e sion [26] came wi h mino changes, he mos impo an o which was he back- bone CNN being expanded o Da kne -53, a CNN wi h 53 con olu ional laye s ha is mo e pow- e ul han i s p edecesso , bu mo e e ec i e han o he used a ian s o ResNe . The mos ecen es esul s o YOLO we e p esen ed in [26]. They a e he esul s o es ing on he COCO da ase , and he ollowing conclusions we e made: •When compa ing mAP o he p ocessing speed (Fig. 3 in [26]), YOLO achie es he bes esul s; •In e ms o accu acy, YOLO is mo e accu a e han SSD, bu s ill in e io o wo-s age ap- p oaches, such as Fas e R-CNN. 2.3.2.5 Compa ison o Resul s The ollowing ables ea u e he esul s o mean a e age p ecision (in pe cen age) o each o he me hods in each da ase . The alues a e aken om he pape s ha in oduce each me hod. When a pape does no p esen esul s o a pa icula da ase , he alue is omi ed. Tables 2.1 and 2.2 p esen he esul s o he VOC07 and VOC12 da ase s, espec i ely. Table 2.1: Compa ison be ween me hods on he Pascal VOC07 da ase Au ho /Re e ence Yea Me hod mAP (%) Ren e al. [20] 2017 Fas e R-CNN 78.8 Dai e al. [22] 2016 R-FCN 83.6 Liu e al. [23] 2016 SSD 81.6 Redmon and Fa hadi [26] 2018 YOLO − Table 2.2: Compa ison be ween me hods on he Pascal VOC12 da ase Au ho /Re e ence Yea Me hod mAP (%) Ren e al. [20] 2017 Fas e R-CNN 75.9 Dai e al. [22] 2016 R-FCN 82.0 Liu e al. [23] 2016 SSD 80.0 Redmon and Fa hadi [26] 2018 YOLO − 2.3 Rela ed Wo k 21 Table 2.3 p esen s he esul s o he MS COCO da ase . On his da ase , mAP can be di ided in o wo me ics. The i s is he s anda d mAP when IoU be ween he objec bounding-box and he g ound- u h is 0.5. The second, is a mo e s ic me ic, o he mean a e age p ecision o e a ious h esholds, o an IoU be ween 0.5 and 0.95. Table 2.3: Compa ison be ween me hods on he MS COCO da ase Au ho /Re e ence Yea Me hod [email p o ec ed] (%) mAP@[.5,.95] (%) Ren e al. [20] 2017 Fas e R-CNN 42.7 21.9 Dai e al. [22] 2016 R-FCN 53.2 31.5 Liu e al. [23] 2016 SSD 46.5 26.8 Redmon and Fa hadi [26] 2018 YOLO 57.9 33.0 As can be seen, R-FCN and SSD a e he mos accu a e me hods on he VOC da ase s. SSD is he as es o he wo, being a one-sho app oach. On he MS COCO da ase , YOLO achie es he bes esul s. When IoU is be ween 0.5 and 0.95, he e is a signi ican d op in accu acy, since i is di icul o ob ain a high IoU. 2.3.3 Machine Lea ning in Pose Es ima ion 2.3.3.1 SSD-6D The objec i e o his ne wo k (Fig. 2.9), in oduced in [27], is o build upon he s uc u e o SSD o ob ain a collec ion o ou pu s: class p obabili ies, coo dina es o a 2D bounding-box, sco es o possible iewpoin s and in-plane o a ions. Figu e 2.9: O e iew o he SSD-6D model. [27]. The inpu image is passed h ough a backbone CNN based on Incep ionV4 o ou pu a se o ea u e maps a di e en scales. The same s uc u e as SSD is used o p edic ion o class and 22 Li e a u e Re iew eg ession o he bounding-box. The main changes occu in he sco ing o possible iewpoin s and in-plane o a ions. As is shown in Fig. 2.9, each map is con ol ed wi h a ke nel o shape (4 + C + V + R), whe e C ep esen s he numbe o objec classes, V he numbe o iewpoin s and R he numbe o in-plane o a ions. The iewpoin is he posi ion in 3D space om which he objec is iewed, which in luences he aspec o he objec as seen om he came a, while in-plane o a ion is seen as a ans o ma ion o he same iewpoin . As can be seen in Fig. 2.10, he iewpoin s a e sampled om a hal -sphe e a ound he objec . Fo symme ical objec s, only he a c in g een is sampled, and, o semi-symme ical objec s, only he poin s in ed a e used. In [27], i is a gued ha con olu ional laye s a e mo e e ec i e a sco ing a iewpoin and in-plane o a ion han using eg ession o p edic a se o ansla ions and o a ions. Fu he mo e, by sco ing iewpoin s wi h a con idence alue, all ha is abo e a ce ain h eshold can be accep ed, he eby dealing well wi h symme ical objec s. Figu e 2.10: Illus a ion o e e y iewpoin conside ed. [27]. Knowing he pa ame e s o he came a, he mos con iden sco es can be pooled o calcula e he ansla ion and he o a ion o he objec , esul ing in a se o 6-DOF pose hypo heses. Each hypo hesis goes h ough a pose e inemen s ep, which uses he I e a i e Closes Poin (ICP) al- go i hm (2.3.3.7). The bes pose is chosen by compa ing he e ined poses o he dep h da a om RGB-D. SSD-6D was e alua ed on bo h he Linemod and he Tejani da ase s. Howe e , since none o he o he me hods a e e alua ed on he las da ase , he e a e no compa isons o be made. Tejani da ase esul s will, he e o e, be omi ed. 2.3.3.2 PoseCNN PoseCNN [19] e olu ionized pose es ima ion algo i hms by decoupling pose es ima ion in o h ee sepa a e asks: seman ic labeling, 3D ansla ion es ima ion and 3D o a ion eg ession (as can be seen in Fig. 2.11). This app oach acili a es he job o he ne wo k by enabling i o model how each ask ela es o o he s. The backbone CNN akes an inpu image and ou pu s ea u e maps o di e en scales. The b anches co esponding o each ask hen use hese ea u e maps o calcula ions. 2.3 Rela ed Wo k 23 Figu e 2.11: O e iew o he PoseCNN sys em. [19]. In he seman ic labeling ask, wo 512 channel maps a e p ocessed in o de o ob ain one o he same scale as he o iginal image. He e, objec de ec ion is done by applying con olu ional ke nels o each pixel in he image and a ibu ing a seman ic label: each pixel is classi ied in o an objec class. This gi es be e esul s han he bounding-box app oaches used in SSD-6D, o example, since i handles occlusions be e . Fo 3D ansla ion es ima ion, he ne wo k needs o ou pu a ec o o coo dina es (Tx,Ty,Tz) o he objec cen e (in he came a coo dina e sys em). To ha ex en , each pixel ha belongs o an objec needs o o e o he cen e o he objec in he image coo dina e sys em, (cx,cy). A e ha , Txand Tycan be de i ed using Eq. 1 in [19]. Vo ing is implemen ed by a Hough o ing laye in he ne wo k and is done as ollows: 1. A eg ession ne wo k p edic s an a ay (nx,ny,Tz) o each pixel; 2. Each pixel o es o o he pixels (on whe he hey a e an objec cen e ) along he di ec ion o he ec o de ined by (nx,ny); 3. The pixel wi h he mos o es is chosen as he objec cen e . The mean o all alues o Tz o each pixel in an objec is chosen as he ue alue o Tz. Addi ionally, all pixels inside an objec a e known as inlie s. The bounding-box ha con ains all inlie s is also gene a ed in his s ep and is used o he 3D o a ion ask. In his ask, he objec i e is o ou pu a qua e nion, which ep esen s he es ima ed o a ion. The i s s ep is o apply 2 ROI pooling laye s using he bounding-boxes gene a ed in he p e ious ask. In ha way, he ea u e maps o each ROI can be ob ained. These ea u e maps a e hen added and ed in o 3 FC laye s, he las o which ou pu s a qua e nion. PoseCNN is e alua ed on he YCB-Video and he Occlusion LineMod da ase s. The ollowing conclusions can be made om ables 2 and 3 in [19]: •This model ou pe o ms coo dina e eg ession algo i hms; •Resul s a e a supe io when using pose e inemen ; •On he Occlusion LineMod da ase , PoseCNN wi h a pose e inemen algo i hm (ICP) achie es highe accu acy han o he me hods; 24 Li e a u e Re iew 2.3.3.3 DPOD: Dense Pose Objec De ec o DPOD [28] akes a di e en app oach o o he s men ioned p e iously. I s model (Fig. 2.12) is comp ised o h ee blocks: co espondence block, pose block, and a inal pose e inemen block. The co espondence block akes as inpu he desi ed image and uses an encode (based on he i s 12 laye s o ResNe ) and h ee decode s ha ou pu a co espondence map ( i s wo decode s) and he ID masks o each objec ( hi d decode ). Figu e 2.12: O e iew o he DPOD sys em. [28]. A co espondence map is a 2-channel image wi h alues om 0-255 ha maps a pixel in he image o a e ex in he objec 3D model. This esul s in mo e s aigh o wa d aining o he ne wo k and an o e all inc ease in quali y since i jus needs o sol e a colo classi ica ion p oblem o ma ch he image co espondence map o he 3D model co espondence map, ins ead o ha ing o eg ess he coo dina es o he objec . The ID masks a e a esul o he p obabili y ha an objec pixel belongs o a speci ic class (such as p e ious app oaches ha e used). Using he ID masks, he 3D model o each class can be ob ained. Bo h he co espondence map and he 3D models a e ed in o he pose block, which uses PnP+RANSAC o ou pu a o a ion ma ix Rand a ansla ion ec o T. As opposed o o he o ms o pose e inemen (2.3.3.7), he DPOD model u ilizes a pose e inemen block (Fig. 2.13) based on he ResNe a chi ec u e. This block akes in a pa ch o he o iginal image con aining he objec and a 3D ende ing o he objec in he es ima ed pose. These inpu s a e ed sepa a ely h ough wo b anches composed o he i s i e laye s o ResNe (E11 and E12), and he ou pu s a e sub ac ed and ed in o ano he ResNe -like ne wo k (E2). The di e ence ea u e ec o ha is ou pu ed is used o compu e he e o be ween he p e- dic ed pose and he ac ual pose. This is done by eeding i in o h ee sepa a e eg ession laye s (XY head, Z head and R head) ha co ec he coo dina es o ansla ion and he o a ion ma ix (as is seen in Fig. 2.13). 2.3 Rela ed Wo k 25 Figu e 2.13: Illus a ion o he pose e inemen block. [28]. The model was e alua ed on he LineMod and he Occlusion LineMod da ase s, ha ing ob- ained he ollowing conclusions: •Compa a i ely, DPOD achie es s a e-o - he-a esul s, wi h only PVNe (explained in Sec- ion 2.3.3.4) ha ing be e accu acy in pose es ima ion (Table 1 in [28]); •Pose e inemen imp o es accu acy by almos 10% (Table 1 in [28]); •DPOD’s no el pose e inemen echnique achie es be e esul s han DeepIM (Tables 1 and 5 in [28]). 2.3.3.4 PVNe : Pixel-wise Vo ing Ne wo k PVNe looks o imp o e upon he wo-s age app oach o keypoin de ec ion and pose es ima ion by using a model mo e obus o occlusion and unca ion. The main idea is o p edic , o each pixel in an objec , he uni ec o s om ha pixel o all he keypoin s in he objec - simila o wha was done in PoseCNN o he localiza ion o an objec cen e . This app oach is mo e obus o si ua ions whe e he keypoin s a e no isible in he image since i s posi ion can be in e ed om he o he isible pixels. In he i s s age o PVNe , a backbone CNN based on ResNe is used o ob ain class p ob- abili ies o each pixel (seman ic segmen a ion) and he uni ec o s ha ep esen he di ec ion om ha pixel o e e y keypoin . The o ing o each pixel is cha ac e ized in Eq. 1 and 2 o [29]. To educe a iance in localiza ion, he model needs o choose he keypoin s loca ed on he su ace o he objec . The ollowing algo i hm (Fa hes Poin Sampling) is used o choose a se o 8 keypoin s: 1. Add he cen e o he objec o he se o keypoin s; 2. Choose he a hes keypoin om he se and add i o he se ; 3. Repea he second s ep un il eigh keypoin s a e chosen. Finally, he model needs o compu e he 6-DOF pose using he se o keypoin s. This is done using an al e ed e sion o he PnP algo i hm ha akes in o accoun he mean and co a iance (unce ain y) associa ed wi h each keypoin (which a e calcula ed wi h Eq. 3 and 4 o [29]). 26 Li e a u e Re iew This me hod is e alua ed on he LineMod, Occlusion LineMod, and YCB-Video da ase s. I is concluded ha PVNe achie es he bes esul s in e ms o he ADD(-S) me ic when compa ed o o he app oaches (Table 3, 5, and 7 in [29]). 2.3.3.5 DenseFusion Mos app oaches so a ha e used 2D ea u es om RGB images and used dep h da a only o pose e inemen . DenseFusion’s [30] model (Fig. 2.14) akes ad an age o dep h in o ma ion om he i s s age by using RGB da a and poin cloud alues on a pe -pixel basis. Since bo h RGB and poin cloud da a a e e y di e en da a ypes, his model ea s each one sepa a ely and uses a pixel-wise dense usion algo i hm o combine hem. Figu e 2.14: O e iew o he DenseFusion a chi ec u e. [30]. In he i s s age, he same seman ic segmen a ion algo i hm applied in PoseCNN is used o ob ain segmen a ion masks o each objec . A bounding-box ha includes each mask is calcula ed and is used o ob ain an image c op in i s loca ion. The dep h da a o pixels inside he bounding box is also used in o de o con e i in o a 3D poin cloud ep esen a ion. The c opped image is ed in o an au oencode ne wo k (based on ResNe ) ha ou pu s colo in o ma ion. On he o he hand, poin cloud da a is p ocessed by a Poin Ne -like s uc u e o ob ain he geome ic in o ma ion o he objec . The usion algo i hm used combines colo (colo embeddings) and geome ic in o ma ion (ge- ome y embeddings) in o ea u e ec o s o each pixel in he objec . E e y ea u e ec o is hen p ocessed by a MLP wi h a e age pooling o ob ain a global ea u e ec o , which is hen appended o each pe -pixel ec o , c ea ing a pixel-wise ea u e (as shown in Fig. 2.14). The pixel-wise ea- u e is ed in o he pose p edic o ne wo k which ou pu s an es ima ion o he pose o e e y pixel wi h an associa ed con idence alue: Ri, i, and ci. 2.3 Rela ed Wo k 27 Figu e 2.15: Illus a ion o he pose e inemen ne wo k. [30]. DenseFusion also p esen s an i e a i e app oach o pose e inemen (Fig. 2.15) u ilizing a co ec ion ne wo k ha is composed o 4 FC laye s. Howe e , his ne wo k needs o lea n o co ec he pose ins ead o p edic ing a new one. The e o e, i s inpu s a e he colo embeddings o he image and he geome y embeddings o a ans o med poin cloud. This ans o med poin cloud is he esul o al e ing he inpu poin cloud by he ac o s ∆R and ∆ p edic ed by he pose esidual es ima o . This can be done i e a i ely, e ining he es ima ion u he and u he in each s ep. The inal pose p edic ed is ob ained by conca ena ing e e y pose ob ained in each i e a ion. E alua ion o he model is done on he LineMod and YCB-Video da ase s. The main conclu- sions ob ained in [30] a e he ollowing: 1. Wi h e inemen , DenseFusion achie es he highes accu acy when compa ed o he second bes me hod, PoseCNN+DeepIM ( ables 1 and 2 in [30]); 2. Pose e inemen inc eases pose accu acy by 4-8% ( ables 1 and 2 in [30]); 3. In Table 3 o [30], i can be seen ha DenseFusion is close o 200x as e han PoseCNN plus ICP; A pa icula ly ele an aspec ( o his hesis) in [30] is he use o DenseFusion in a obo ic g asping si ua ion. Ou o 60 a emp s, he obo has a 73% success a e while using DenseFusion as a pose p edic o . 2.3.3.6 Hyb idPose Mos app oaches ha e only used only one ype o in e media e ep esen a ion o an objec , he mos popula o which is a keypoin ep esen a ion. In [31], he au ho s a gue ha , in a eal- wo ld se ing, i is di icul o p edic keypoin s accu a ely based on a RGB image. Hyb idPose (Fig. 2.16) looks o sol e his p oblem by adding mo e in e media e ep esen a ions, enabling i s model o wo k be e in occlusion si ua ions and ha ing mo e in o ma ion abou he geome ical cha ac e is ics o he objec . 34 3D Objec Recogni ion and Localiza ion Sys em The Da a Acquisi ion Module has wo main unc ions. The i s unc ion deals wi h da a e ie al om he ZED came a; i uns on an in ini e loop, con inuously e ie ing he RGB image ame and he dep h map. The o he unc ion calcula es di e ences be ween consecu i e ames, using he S uc u al Simila i y (SSIM) algo i hm. This algo i hm ou pu s a alue, om 0 o 1, ha indica es how simila wo images a e, and he unc ion compa es his alue o a h eshold alue (calcula ions o his alue will be shown in Sec ion 4.3.1), in o de o indica e i he e we e changes be ween ames (such as an objec mo ing). In case o changes, he unc ion aises a global lag (indica ing ha a new ame is a ailable) and passes he new ame and i s co esponding dep h map o he Compu e Vision Module. Figu e 3.3: S a e machine o he pose es ima ion module In he Compu e Vision Module, a s a e machine (Fig. 3.3) is implemen ed. The e is one Ini ial s a e and ou main s a es: •De ec ion: implemen s he Objec De ec ion and Segmen a ion sys em explained in 3.2.2.1; •Localiza ion: implemen s he 6D Objec Pose Es ima ion sys em explained in 3.2.2.2; •Ready o G asp: chooses which objec o g asp and sends coo dina es o obo ic manipula- o ; 3.2 Sys em’s Modules 35 •G asping: wai s o g asping o inish and ies o de ec changes in ame. The ansi ion be ween s a es depends on he changes be ween ames ha a e de ec ed in module 1. I any objec in he ame mo es o disappea s, he whole s a e machine has o ese o he De ec ion s a e. 3.2 Sys em’s Modules 3.2.1 Da a Acquisi ion Module In he Da a Acquisi ion Module, he goal is o acqui e usable da a om he g asping wo kspace o he Compu e Vision module o p ocess and ex ac in o ma ion needed o g asping asks. In his case, he in o ma ion wan ed is 6D pose es ima ion and objec classi ica ion. Gi en his, and ega ding he equi emen o a 3D based solu ion in ou app oach, a dep h/3D came a was needed. In ou solu ion, gi en se e al epo ed uses and a ailabili y, he ZED s e eo ision came a and i s SKD we e used. The ZED came a was de eloped by S e eolabs and is composed o wo side-by-side RGB came as, which ou pu synch onized RGB images. The wo wide-angle lenses ha e 110◦ ield-o - iew, spaced a a baseline o 120 mm, allowing accu a e dep h es ima ion in he ange o 0.7 o 20 me e s. Came a pic u e, along wi h p oduc dimensions, a e shown in Fig. 3.4. The mos ele an cha ac e is ics o he ZED came a a e summa ized in Table 3.1. Figu e 3.4: ZED S e eo Came a and dimensions illus a ion Fo dep h acquisi ion, he ZED came a imi a es human ision, whe e each eye has a sligh ly di e en iew o he wo ld a ound, and by compa ing he wo iews, dep h can be in e ed. Like- wise, his came a, wi h i s sepa a e lens, can es ima e dep h by compa ing he pixels’ displacemen be ween he le and igh RGB images. The came a hen p o ides a dep h map (image con aining in o ma ion ega ding he dis ance o he su aces o scene objec s (z), o e e y pixel i,j) in he came a coo dina e sys em. This dep h is exp essed in me ic uni s and calcula ed om he le came a’s eye back o he g asping objec . 36 3D Objec Recogni ion and Localiza ion Sys em Table 3.1: ZED Came a ele an ea u es [1] Size and Weigh Dimensions: 175x30x33 mm Weigh : 159 g Indi idual image and dep h esolu ion HD2K: 2208 x 1242 (15 FPS) HD1080: 1920 x 1080 (30, 15 FPS) HD720: 1280 x 720 (60, 30, 15 FPS) WVGA: 672 x 376 (100, 60, 30, 15 FPS) Dep h Range: 1-20 m Fo ma : 32 bi s Baseline: 120 mm Lens Field o View: 110◦ /2.0 ape u e Connec i i y USB 3.0 (5 V / 380 mA) 0◦C o +45◦C SDK Sys em Requi emen s Windows o Linux Dual-co e 2.3 GHz 4 GB RAM N idia GPU The came a also p o ides a SDK o acili a ing came a con ol, bo h in C++ and Py hon. Fo ou implemen a ion, only he Py hon e sion was used. The ZED SDK is designed a ound OpenCV and CUDA lib a ies, wi h i s calib a ion and dep h es ima ion ou ines exploi ing CUDA’s pa allel GPU compu ing capabili ies. The SDK also o e s op ional dep h map p ocessing, includ- ing occlusion illing and edge sha pening. Un o una ely, hese buil -in pos -p ocessing echniques come wi h high compu a ional cos s, so hey could no be used in ou solu ion since eal- ime is a c ucial equi emen . The came a is con igu ed o un a 15 ps, wi h 720p esolu ion. The minimum dis ance o dep h es ima ion is 0.3m, and he dep h es ima ion was se oquali y mode, meaning i ou pu s mo e accu a e dep h alues a a small compu a ional cos . 3.2.2 Compu e Vision Module As shown in Fig. 3.1, he Compu e Vision Module is he mos impo an in he p oposed app oach, since i is h ough i s p ocessing capabili ies ha he objec ’s poses and ca ego ies a e iden i ied o pe o ming g asping ope a ions. This module is composed o wo di e en bu in e connec ed models: he Objec De ec ion and Segmen a ion model, and he Objec Pose es ima ion model. In he ollowing Subsec ions, de ails on hei implemen a ion will be p o ided. 3.2.2.1 De ec ion and Segmen a ion Model The objec de ec ion and segmen a ion model is composed o a p e-p ocessing s age and an in e - ence s age. In p e-p ocessing, he da a ecei ed om he Da a Acquisi ion module is p ocessed, so i can be ed in o he in e ence s age, which is made up o a Mask-RCNN model [34]. This s age 3.2 Sys em’s Modules 37 hen ou pu s a se o objec iden i ica ions, bounding-boxes, segmen a ion masks, and con idence alues o each objec in he came a ame. Figu e 3.5: O e iew o he Mask-RCNN model [34] In he p e-p ocessing s age, h ee RGB image ames and co esponding dep h maps a e e- cei ed by he pipeline, and an a e age o he pixel alues o bo h he RGB ames and he dep h maps a e calcula ed. This is o help educe he impac o small illumina ion di e ences, pixel shi ing, and andom noise when a ame is acqui ed, which in u n educes he licke ing e ec in he objec masks. This is discussed a g ea e leng h in Sec ion 4.3.2. A e wa d, he image ame is esized o i he inpu size o he in e ence s age. The Mask-RCNN model (o e iewed in Fig. ) is an ex ension o he Fas e R-CNN model 2.3.2.1, by applying a small FCN each ROI and ou pu ing a segmen a ion mask. As can be seen in he s a e-o - he-a analysis in sec ion 2.3.2.5, Fas e R-CNN is no he op-pe o ming model when i comes o pu e objec de ec ion, so why is i he one used? The answe is ha pu e objec de ec ion is no he only ac o in his p ojec . The inal ou pu should be he 6D pose o he objec , which means ha an addi ional model (in his case, he 6D pose es ima ion model discussed u he on, in Sec ion 3.2.2.2) is needed. One o he model equi emen s is he inpu o a segmen a ion mask o each objec and YOLO - he op-pe o ming model (in objec de ec ion), bo h in e ms o accu acy and speed - does no ou pu segmen a ion masks. Two di e en al e na i es we e analyzed: using YOLO wi h a segmen a ion mask ne wo k o e head (much like how Mask-RCNN imp o es upon Fas e R-CNN) o using Mask-RCNN ins ead. The conclusions we e ha Mask R-CNN is ac ually mo e accu a e and ma ginally as e han he o he al e na i e; he e o e, i is ac ually a be e choice o his implemen a ion. 3.2.2.2 Pose Es ima ion Model This sys em, as opposed o objec de ec ion and segmen a ion, wo ks on objec s one by one. I begins wi h a p e-p ocessing s age, which p ocesses he da a ecei ed om he objec de ec ion model o op imize he inpu s o he pose es ima ion s age. In pose es ima ion, a Dense usion 2.3.3.5 a chi ec u e is used o in e ence, and i ou pu s a se o o a ion qua e nions and ansla ion ec o s o each objec . A pose e inemen s age is hen applied o gi e a be e pose es ima ion. 38 3D Objec Recogni ion and Localiza ion Sys em Finally, a pos -p ocessing s age is execu ed o calcula e he inal op imal g asping poin coo dina es and g ippe pose o e e y objec in he ame. Due o he ZED came a limi a ions, he dep h map migh e u n alues such as in ini e and NaN. This occu s when poin s a e oo close/ a om he came a (in he case o in ini e alues), o when dep h canno be es ima ed due o occlusions (in he case o NaN). To make su e hese alues do no con ibu e o poo calcula ions, a mask o he dep h map is c ea ed whe e only eal alues a e masked. A e he dep h map mask is ob ained, i is mul iplied by he segmen a ion mask o ob ain all he eal dep h alues o only he poin s ha belong o he objec . Since he came a in he eal-wo ld en i onmen is posi ioned close o he objec s han he came a in he da ase , his means mo e poin s in he image will ep esen an objec , so choosing mo e poin s gi es us mo e geome ical in o ma ion abou he objec . Using mo e poin s is also a good way o making ou lie s (poin s in he mask wi h poo dep h alues) less in luen ial in he calcula ions. A se o 2000 poin s is hen andomly chosen om he objec dep h alues. The alue o 2000 poin s, a e some es ing, was chosen because i is a good ade-o be ween accu acy and speed. The ac ha he poin s a e chosen a andom also means ha he se o poin s will ha e a p ope dis ibu ion h oughou he whole objec . Wi h he poin s chosen, he poin cloud alue o each poin is calcula ed, ollowing Equa ions 3.1,3.2 and 3.3. z=dep h scale (3.1) x=(a−cx)×z x (3.2) y=(b−cy)×z y (3.3) In hese equa ions, dep h ep esen s he dep h alue o a poin , scale depends on he measu e- men uni s used by he came a, aand ba e he pixel coo dina es o a poin and cx,cy, x and y a e in insic came a pa ame e s. This is he same way he ZED came a calcula es i s poin cloud, howe e , he SDK calcula es a poin cloud o he whole image and each poin can be accessed indi idually, which inc eases compu a ion imes signi ican ly mo e. Following all his p e-p ocessing, he Dense usion model akes as inpu a c op o he came a image ame (in he dimensions o he objec ’s bounding-box), he calcula ed poin cloud, he 2000 poin s om he mask, and he objec ’s ID. The pose es ima ion is ou pu ed in he o m o o a ion qua e nions and a ansla ion ec o , which hen goes h ough wo i e a ions o pose e inemen , o p oduce be e esul s. Doing wo i e a ions is as e and has compa able accu acy o doing mo e i e a ions. The ou pu o Dense usion is no he inal pose es ima ion, howe e . These alues c ea e a ans o ma ion ma ix ha ans o ms he poin s in he objec 3D model om he canonical ame 3.2 Sys em’s Modules 39 de ined in he da ase o he came a coo dina e ame. To ob ain a inal pose es ima ion, pos - p ocessing is c ucial. In pos -p ocessing, he se o op imal g asping poin s (which we e loaded om one o he con igu a ion iles), a e ans o med acco ding o he ans o ma ion gi en by he Dense usion model. The cen e o his g oup o poin s is hen calcula ed o gi e he op imal poin o con ac o g asping. 40 3D Objec Recogni ion and Localiza ion Sys em Chap e 4 Expe imen s and Resul s 4.1 Da ase In o de o es and alida e ou p oposed sys em, and gi en ha a machine lea ning model is only as good as he da a i is ed, a da ase was chosen o ain and es he compu e ision module. The da ase used o ain bo h he objec de ec ion and he pose es ima ion models was he YCB- Video da ase , which uses 20 objec s om he YCB Objec Model Se [35]. Using a RGB-D came a, di e en g oups o he 20 objec s we e ilmed in di e en en i onmen s, wi h a ying backg ounds, di e en ligh ing condi ions, and si ua ions o occlusion. The came a mo ed in ela ion o he objec s o c ea e di e en iewpoin s o e e y objec , which esul ed in a o al o 133827 ames o ideo. Fu he mo e, he 3D poin cloud models o e e y objec we e ob ained wi h a scanning ig. F om a quan i a i e s andpoin , his da ase has 133827 RGB images (wi h 480x640 pixels) and hei co esponding dep h maps, as well as a collec ion o he poin cloud 3D models o all he objec s. Fo e e y ame and each o he objec s in he ame, he e a e anno a ions o class label, g ound- u h segmen a ion mask, and pose alues o he objec (gi en as a ans o ma ion, om he canonical ame o he objec o he came a coo dina e ame). The came a’s in insic pa ame e s and i s posi ions in he wo ld a e also gi en o each ame. Figu e 4.1: Dis ibu ion o numbe o ins ances pe objec 41 42 Expe imen s and Resul s In o de o e alua e i he da ase was imbalanced, he numbe o ins ances pe ype o objec in he da ase was plo ed. As can be seen in Fig. 4.1, he dis ibu ion o ins ances o each objec is ai ly balanced, and each objec has, a leas , 15000 ins ances. This con ibu es o a mo e balanced aining o he machine lea ning models and helps o educe bias. This cha ac e is ic explains why his da ase is one o he mos impo an and one he mos used o he aining and alida ion o he pose es ima ion models e iewed in Sec ion 2.3.3. F om a quali a i e s andpoin , he YCB-Video da ase p esen s a lo o ad an ages in e ms o a iabili y. One o hem is he di e ence in backg ound, which can be bo h unclu e ed (Fig. 4.2a) and clu e ed (Fig. 4.2e), as well as ligh o da k, and simple o complex (as is shown in all he images o Fig. 4.2). (a) (b) (c) (d) (e) ( ) Figu e 4.2: Selec ion o illus a i e images om he da ase When compa ing Figs. 4.2a and 4.2b, i can be seen ha he same objec s, in he same scene, a e shown in a a ie y o iewpoin s. This is mainly due o he mo emen o he came a, which helps o c ea e a subs an ial ange o poses o he aining o he pose es ima ion model. Ano he impo an aspec o he da ase is he change in ligh ing ha occu s be ween scenes. Compa ing Figs. 4.2b and 4.2d, he e a e examples o an o e exposed and an unde exposed image, espec- i ely. This a iabili y can help o make any model mo e obus o ligh ing changes. Finally, i is also impo an o men ion he way occlusions a e c ea ed o an objec . The a ying iewpoin s help c ea e na u al occlusions - when objec s o e lap - and unca ion o an objec - when i lea es he ield o iew o he came a (such as in Fig. 4.2c). Mo eo e , he a angemen o he objec s in he scene also helps o c ea e complex iewpoin s o an objec , whe e o e laps may occu ( o example, in igs. 4.2a,4.2d and 4.2e). All hese aspec s we e ele an o he choice o da ase , gi en ha wi h mo e a iabili y in di e en aspec s o each objec , he obus ness o he ained model will imp o e. 4.2 O line Tes ing Phase 43 4.2 O line Tes ing Phase In his sec ion, he aining and alida ion o each ML model in he Compu e Vision Module, will be discussed. The models we e ained and alida ed on he da ase analyzed in he sec ion abo e (Sec ion 4.1). A e his, he pose es ima ion model was es ed using as inpu s he ou pu s gi en by he objec de ec ion model. 4.2.1 T aining he Objec De ec ion Model The aining o his model was hea ily based on he idea o ans e lea ning. This me hod consis s o using p e- ained weigh s om a model - ob ained on one da ase - as a s a ing poin o aining on a di e en da ase . This app oach speeds up aining ime and inc eases pe o mance, bu only when he p e- ained model lea ned ele an gene al ea u es on a balanced da ase . In ou i s app oach, he p e- ained weigh s we e used o e e y b anch, excep he ne wo k heads; on he second app oach, he p e- ained weigh s we e u ilized only o he backbone ea u e ex ac ion ne wo k, excep o he inal laye s. The i s app oach esul ed in a 14% d op in mask gene a ion accu acy and a 5% d op in class label accu acy, in ela ion o he second app oach. Al hough he i s app oach only akes 33 hou s o ain compa ed o he 46 hou s o aining in he second app oach, he accu acy o he segmen a ion masks is pa amoun , so his is an accep able ade-o be ween speed and accu acy. Table 4.1: Compa ison o o iginal and ine- uned hype -pa ame e alues Hype -pa ame e O iginal alue Fine- uned alue Numbe _classes NA 21 Max_g _ins ances 100 10 De ec ion_min_con idence 0.8 0.7 ROI_miniba ch_size 512 128 RPN_nms_ h eshold 0.7 0.5 Lea ning_ a e 0.02 0.002 RPN_class_loss 1.0 1.0 RPN_bbox_loss 1.0 0.01 M cnn_class_loss 1.0 1.0 M cnn_bbox_loss 1.0 0.01 M cnn_mask_loss 1.0 10.0 Gi en ha he o iginal hype -pa ame e s o he model we e op imized o he COCO da ase , he hype -pa ame e s o he objec de ec ion model we e uned. Fo ha , Tenso boa d (a isu- aliza ion and op imiza ion ool o uning pa ame e s in Tenso low) was used. In Table 4.1, a compa ison be ween he o iginal and he ine- uned model pa ame e s is shown. The numbe _classes pa ame e akes he alue o 21, since he e a e 20 objec s in he da ase and one backg ound class. Max_g _classes was changed o 10 om 100 due o he ac ha each image in he da ase has a maximum o 10 objec ’s ins ances. To allow o mo e p edic ions 50 Expe imen s and Resul s da ase used in aining he model, i did no lea n how o di e en ia e hese ea u es in a complex objec co ec ly. In scene 2 (Fig. 4.3b), he h ee main changes we e he a ia ion o iewpoin o he bowl, he o e lap be ween one o he gela in boxes and he mug, and he pa ial occlusion o he ma ke by he smalle clamp. Like is shown in 4.7, his had an immedia e impac on he de ec ion o he gela in box. The e is only 50% accu a e de ec ions which co espond o he non-occluded box. The inaccu a e de ec ions o he bowl co espond o i being iden i ied as a mug, due o he shape in his iewpoin being simila o ha o a mug. The pa ial occlusion o he ma ke causes he ma ke o be unde ec ed. Name ID Coun Numbe o ames Accu a e De ec ion (%) Inaccu a e De ec ion (%) No de ec ion (%) A e age Con idence (0-1) Gela in box 8 2 153 50 0 50 0.85 Bowl 13 1 153 81.7 18.3 0 0.72 Mug 14 1 153 100 0 0 0.98 La ge ma ke 18 1 153 0 0 100 NA Clamp 19 2 153 0 74.5 25.5 NA Table 4.7: De ec ion esul s on scene 2 In scene 3 (Fig. 4.3c), he smalle gela in box is o e lapped on op o he bowl, c ea ing a pa ial occlusion o he bowl. This occlusion c ea es a iewpoin e y simila o he mug iewpoin , which explains he inaccu a e de ec ion a e (which can be seen in Table 4.8). The smalle gela in box is s ill well de ec ed e en on op o he bowl; howe e , he occluded gela in box emains unde ec ed, while i is occluded by he clamp. The ma ke , in an up igh posi ion, p esen s good de ec ion esul s. Name ID Coun Numbe o ames Accu a e De ec ion (%) Inaccu a e De ec ion (%) No de ec ion (%) A e age Con idence (0-1) Gela in box 8 2 116 50.0 0 50.0 0.89 Bowl 13 1 116 88.8 11.2 0 0.90 Mug 14 1 116 100 0 0 0.97 La ge ma ke 18 1 116 89.7 0 10.3 0.76 Clamp 19 3 116 0 66.7 33.3 NA Table 4.8: De ec ion esul s on scene 3 Scene 4 (Fig. 4.3d) p esen s all he objec s close oge he , wi h pa ial occlusions o he smalle gela in box. This p oximi y be ween objec s makes i ha d o he de ec ion model o 4.3 Online Tes ing Phase 51 co ec ly iden i y he objec s. In ac , he wo bes pe o ming objec s (acco ding o he esul s in Table 4.9) - he bowl and he mug - a e he mo e isola ed in he g oup. Name ID Coun Numbe o ames Accu a e De ec ion (%) Inaccu a e De ec ion (%) No de ec ion (%) A e age Con idence (0-1) Gela in box 8 2 115 4.3 0 95.7 0.72 Bowl 13 1 115 100 0 0 0.95 Mug 14 1 115 100 0 0 0.97 La ge ma ke 18 1 115 0 0 100 NA Clamp 19 2 115 0 44.8 55.2 NA Table 4.9: De ec ion esul s on scene 4 Scene 5 (Fig. 4.3e) p esen s a on al iew o he op o he bowl. Since his iewpoin is e y unique o his ype o objec , he de ec ion is 100% accu a e (Table 4.10). In a simila si ua ion o scene 4, he ma ke is unde ec ed due o i s p oximi y o he bowl and he wood block behind. Name ID Coun Numbe o ames Accu a e De ec ion (%) Inaccu a e De ec ion (%) No de ec ion (%) A e age Con idence (0-1) Gela in box 8 2 113 100 0 0 0.87 Bowl 13 1 113 100 0 0 0.87 Mug 14 1 113 72.6 0 27.4 0.74 La ge ma ke 18 1 113 0 0 100 NA Clamp 19 3 113 0 0 100 NA Table 4.10: De ec ion esul s on scene 5 As can be seen in Table 4.11, e en wi h he hea y occlusion o he bowl in scene 6 (Fig. 4.3 ), is s ill p esen s a easonable de ec ion a e. In his scene, he powe d ill was in oduced, howe e , due o i s occlusion, he iewpoin gene a ed is no ep esen a i e enough o he shape o he objec , in o de o he model o de ec i . 52 Expe imen s and Resul s Name ID Coun Numbe o ames Accu a e De ec ion (%) Inaccu a e De ec ion (%) No de ec ion (%) A e age Con idence (0-1) Gela in box 8 2 140 91.4 0 8.6 0.81 Bowl 13 1 140 88.6 0 11.4 0.83 Mug 14 1 140 100 0 0 0.97 Powe d ill 15 1 140 0 0 100 NA La ge ma ke 18 1 140 100 0 0 0.92 Clamp 19 2 140 0 0 100 NA Table 4.11: De ec ion esul s on scene 6 Fo scene 7, he esul s o Table 4.12 show ha e en when occlusion does no occu , ce ain posi ions o he gela in boxes cause he model no o be able o de ec i . This is due o he small amoun o iewpoin s in he educed da ase used o online es ing. All o he objec s, excep he clamp, ha e a 100% de ec ion a e. Name ID Coun Numbe o ames Accu a e De ec ion (%) Inaccu a e De ec ion (%) No de ec ion (%) A e age Con idence (0-1) Gela in box 8 2 209 50 0 50 0.83 Bowl 13 1 209 100 0 0 0.83 Mug 14 1 209 100 0 0 0.96 Powe d ill 15 1 209 100 0 0 0.83 La ge ma ke 18 1 209 100 0 0 0.92 Clamp 19 1 209 0 100 0 NA Table 4.12: De ec ion esul s on scene 7 Scene 8 (shown in Fig. 4.3h) p esen s a ai ly s anda d iew o he objec s, his ime wi h he emo al o he clamp om he wo kspace. Table 4.13 shows ha mos o he objec s a e well de ec ed, wi h he excep ion o he ma ke and he powe d ill. 4.3 Online Tes ing Phase 53 Name ID Coun Numbe o ames Accu a e De ec ion (%) Inaccu a e De ec ion (%) No de ec ion (%) A e age Con idence (0-1) Gela in box 8 2 143 97.2 0 2.8 0.83 Bowl 13 1 143 100 0 0 0.88 Mug 14 1 143 100 0 0 0.99 Powe d ill 15 1 143 24.5 0 75.5 0.73 La ge ma ke 18 1 143 0 0 100 NA Table 4.13: De ec ion esul s on scene 8 Finally, in scene 9 (Fig. 4.3i), e en wi h e y clea iewpoin s o he objec s, de ec ion esul s a e poo o he gela in box, he powe d ill and he clamp. Name ID Coun Numbe o ames Accu a e De ec ion (%) Inaccu a e De ec ion (%) No de ec ion (%) A e age Con idence (0-1) Gela in box 8 2 161 50 0 50 0.8 Bowl 13 1 161 100 0 0 0.91 Mug 14 1 161 100 0 0 0.99 Powe d ill 15 1 161 0 0 100 NA La ge ma ke 18 1 161 91.9 0 8.1 0.76 Clamp 19 2 161 0 50 50 NA Table 4.14: De ec ion esul s on scene 9 As can be seen, he educed da ase has a big impac on de ec ion pe o mance. The main issues wi h he use o so li le ins ances o each objec esul s in he ollowing p oblems: •Fo complex objec s (such as he clamp), he model needs mo e da a o be able o dis inguish ea u es be ween objec s, esul ing in si ua ions o inco ec classi ica ion; •E en o objec s wi h simple ea u es (like he gela in box, he bowl, and he ma ke ), he model o e - i ed o ce ain iewpoin s (which we e mo e p e alen in he educed da ase ), which esul ed in no de ec ion in ce ain objec posi ions; •When objec s we e oo close oge he , he model had ouble in co ec ly de ec ing hem, since his abili y is a mo e complex ea u e which needed mo e aining da a; Howe e , as in any o he p oblem, besides he issues, he e a e some posi i e aspec s wo h men ioning ega ding he model’s pe o mance: •The model is able o gene alize, and co ec ly iden i y objec s which a e di e en om he da ase , e en wi h limi ed da a. This is he case o he gela in boxes - which ha e di e en 54 Expe imen s and Resul s heigh and wid h in ela ion o he o iginal objec -, he bowl - which is alle and less wide han he o iginal model - and he mug - which has a di e en colo and is mo e na ow han he da ase model; •Wi h he limi ed amoun o aining da a, he model is s ill able o de ec accu a ely and wi h g ea con idence, he less complex objec s, like he gela in boxes, he bowl, and he mug; •E en wi h hea y occlusion (as seen in Fig. 4.3 ), he model can co ec ly de ec he bowl; One p oblem ha is common o bo h he educed da ase and he o iginal da ase is he lack o a ie y in he numbe o objec s pe image. Since his da ase was o iginally made o pose es ima ion, he main objec i e was o c ea e a complex en i onmen wi h p oximi y and occlusion be ween objec s. This is he igh app oach o pose es ima ion; howe e , in he objec de ec ion model’s aining, i can lead o o e - i ing. In es ing, when an objec was posi ioned alone in he wo kspace, he de ec ion a e d opped signi ican ly and was e y inconsis en . Howe e , when o he objec s we e posi ioned close o i , he de ec ion a e imp o ed and was mo e s able. This issue is due o o e - i ing o si ua ions wi h a g oup o objec s. 4.3.3 G asping Expe imen s In he hi d and inal es , he compu e ision module was alida ed. The es ing me hodology used was a g asping expe imen on he 5 o he 6 objec s ha we e chosen o es ing. This was due o he a ailable g ippe being oo small o be able o g asp he powe d ill. Figu e 4.4: Illus a ion o one g asping posi ion pe objec Each objec was placed in a di e en wo kspace posi ion, su ounded by o he objec s. A g asping a emp was made o each posi ion, o a o al o i e a emp s. Illus a ions o one o he 4.3 Online Tes ing Phase 55 posi ions o each objec a e p esen ed in Fig. 4.4. The esul s ob ained we e eco ded in Table 4.15. Scene Objec ID Success ul a emp s (%) A e age e o (cm) 1 8 60 2 2 8 60 5 3 13 20 2.7 4 14 80 0.5 5 18 60 3.5 6 19 40 0.75 Table 4.15: G asping esul s Fo each objec , he Compu e Vision Module an de ec ion, gene a ion o he segmen a ion mask, and pose es ima ion. The coo dina es o he cen e poin o he objec we e ob ained, and a ans o ma ion was applied om he cen e poin o he op imal g asping poin o enable co ec g asping. Due o he g ippe model no being eadily a ailable and also ime- es ic ions o es ing, he ans o ma ion o he se o op imal g asping poin s (discussed in Sec ion 3.2.2.2) was no implemen ed. Howe e , his es se es as a p oo o concep , since he ans o ma ion o all objec poin s is equi alen o he ans o ma ion o only a subse o op imal g asping poin s. Fu he mo e, since he p ima y pu pose o his p ojec is o de elop a sys em o objec ecog- ni ion and localiza ion, he main ocus was on he c ea ion o a pipeline wi h hose capabili ies whe e he pos -p ocessing s ep in ol ed in g asping was no he main ocus. The wo s pe o ming objec s a e he bowl and he clamp. These a e expec ed esul s when looking a bo h he esul s in Table 4.3 and in Sec ion 4.3.2. The clamp, being a complex objec wi h he wo s esul in Table 4.3 and wi h poo de ec ion esul s, was unlikely o p esen success ul g asping a emp s. As o he bowl, i was he second-wo s pe o ming objec in Table 4.15, due o being a symme ical objec . Bo h he gela in boxes ha e compa able success a es; howe e , he a e age e o is mo e signi ican o he la ge box, since small e o s in he pose’s es ima ion ha e a mo e subs an ial in luence in la ge objec s. One o he main sou ces o pose es ima ion e o du ing his phase o es ing was he incon- sis ency and inaccu acy o he segmen a ion masks ou pu ed by he objec de ec ion model. This was al eady an issue in Sec ion 4.2.2, howe e , he small acquisi ion e o s om he came a also con ibu e o he o e all de ec ion e o . Addi ionally, as is concluded in [1], he ZED came a has an associa ed e o o dep h da a (Fig. 4.5). Due o he wo kspace being a a maximum dis ance o 1.1m om he came a, his dep h es ima ion e o is e y small, howe e , i is s ill in luen ial. 56 Expe imen s and Resul s Figu e 4.5: RMS e o in dep h da a o 720p esolu ion [1] In his phase o es ing, gi en he eal- ime cons ains o g asping applica ions, he e was also a measu emen o he p ocessing ime o each componen : da a acquisi ion module, objec de ec ion, and segmen a ion model, and 6D objec pose es ima ion model. On a e age, ame acquisi ion and analysis ook 10ms. The objec de ec ion and objec localiza ion models ook on a e age 346ms and 29.6ms, espec i ely. Since he compu e ision module uns pa allel wi h ame acquisi ion, he whole p og am akes abou 375ms pe ame on a e age. This means i can un a abou 2.6 ps. A ame- a e o 2.6 ps is e y slow when applied o no mal ideo-playback o complex asks, such as eal- ime au onomous d i ing. Howe e , conside ing he no e y dynamic en i onmen on which his sys em aims o wo k in and conside ing he slow speed o a ull g asping a emp by he obo ic manipula o used o es ing (abou 5s), his ame- a e is adequa e. 4.4 Limi a ions Wi h he expe imen s and es s pe o med, some limi a ions in ou app oach we e ound. One o he limi a ions o he p oposed sys em is ha he ou pu o ideal g ippe o a ions is based on a p e iously c ea ed con igu a ion ile. While his is a alid solu ion ha esol es many e o s ha can occu when calcula ing hese o a ions au oma ically, while also allowing he use o choose he ideal g asping acco ding o his own p e e ence and applica ion, i also asks much o e head wo k om anyone who ies o implemen he sys em. Howe e , since he sys em’s main objec i e is o ou pu objec de ec ion and pose es ima ions o execu ing he g asping asks, i is no a e y impac ul limi a ion. O e all, he ade-o be ween allowing o use adap abili y and use o e head wo k may be bene icial. The de eloped sys em also a ec ed e o s in he acquisi ion o isual da a and ligh ing con- di ions due o he cha ac e is ics o he ML models used. Ligh ing is always a big pa o e o s 4.4 Limi a ions 57 in oduced in CV models, so his is an accep able limi a ion. The main way o sol ing his limi- a ion would be o ain he model on an e en bigge da ase wi h a ange o ligh ing condi ions, which would be oo cos ly. A se e e limi a ion is he ac ha he de ec ion a e d ops when an objec is by i sel in he wo kspace due o o e - i ing o si ua ions wi h a g oup o objec s. This is an issue o aining he de ec ion model on a pose es ima ion da ase . E en hough, when i comes o de ec ion, he sys em shows ha i gene alizes o objec s wi h di e en dimensions and cha ac e is ics om he da ase , in he case o pose es ima ion, objec s ha di e oo much om he da ase models will no be co ec ly es ima ed since he pose es ima ion pipeline akes in o accoun he canonical objec 3D model om he da ase . Ano he limi a ion is he ac ha he sys em canno choose he absolu e op imal objec o g asp in a clu e ed si ua ion. I only chooses he objec s ha i has he highes con idence o , wi hou aking in o accoun i s posi ion in ela ion o he o he objec s. 58 Expe imen s and Resul s Chap e 5 Conclusions and Fu u e Wo k 5.1 Conclusions In his hesis, he p oblem o de eloping an au oma ic objec ecogni ion and pose es ima ion sys- em was add essed. This sys em combines objec localiza ion and iden i ica ion o imp o e obo ic g asping. The main con ibu ion o his wo k was he de elopmen o a amewo k in eg a ing wo di e en ML models o c ea e a sys em o mo e comple e and au onomous g asping asks. Ex ensi e esea ch was ca ied ou on he cu en s a e o Machine Lea ning me hods, pa icu- la ly Con olu ional Neu al Ne wo ks. This esea ch esul ed in a deepe unde s anding o his ield and i s applica ions in Compu e Vision: om he mos basic o he mo e ad anced de elopmen s ha ha e occu ed h oughou he yea s. The main s a e-o - he-a app oaches o objec de ec- ion/ ecogni ion and pose es ima ion we e e iewed and compa ed, which led o mo e in o med de elopmen o he p oposed sys em. This hesis’ main ocus was on he in eg a ion o wo di e en ML models o de ec ion and pose es ima ion. Bo h Mask-RCNN and Dense usion we e co ec ly in eg a ed and show adequa e esul s in a eal es ing en i onmen . The de eloped sys em is capable o de ec ing mo emen s and di e ences be ween ideo ames, he e o e educing he need o cons an calcula ion and in e ence on each ideo ame. The p oposed solu ion uns en i ely au oma ically, a e an ini ial use pa ame e s’ selec ion and can gene a e g asping ou pu s om ideo da a a a a e o 2.6 ps. An in e es ing posi i e aspec o he p oposed solu ion is he abili y o a human o be in con ol o choosing he op imal g asping poin s o an objec and c ea ing a con igu a ion ile wi h hese poin s, hus allowing o use and applica ion adap abili y, while inc easing accu acy and speed o he sys em, since i does no es ima e hese poin s du ing un ime. The main limi a ions ound in he p oposed solu ion we e he o e head wo k needed o ou pu co ec g ippe o a ions o ce ain objec posi ions, he impac o ligh ing condi ions, and he dependency on p e iously scanned 3D models o he eal objec s. 59