Study of Machine Learning Algorithms To Detect Threats In Airport Passengers

S udy o Machine Lea ning Algo i hms To De ec Th ea s In Ai po Passenge s A Deg ee Thesis Submi ed o he Facul y o he Escola T`ecnica d’Enginye ia de Telecomunicaci´o de Ba celona Uni e si a Poli `ecnica de Ca alunya by Ad i´an de Jo ge S´anchez In pa ial ul ilmen o he equi emen s o he deg ee in TELECOMMUNICATION SYSTEMS ENGINEERING Ad iso s: Josep Vidal Manzano Olga Mu˜noz Medina Abs ac In he la es yea s, he ield o compu e ision has wi nessed con inual ad- ancemen s. One o he mos s a ed ad ancemen is Con olu ion Neu al Ne wo ks (CNNs). Deep Lea ning echniques ha e p o en o pe o m e y well on a la ge a ie y o p oblems and ields (i.e. Biology, Physics, Compu e Science, Ma hema - ics, e c.). I s g ea powe and lexibili y is achie ed by lea ning o ep esen he wo ld as a nes ed hie a chy o concep s, wi h each concep de ined in ela ion o simple concep s, and mo e abs ac ep esen a ions compu ed in e ms o less abs ac ones. Tha is, o de ec a complex shape, he image passes h ough he i s laye , in which i s neu ons a e exci ed wi h basic shapes, be hey con ou s, squa es, ci cles, e c... As he dep h o he ne wo k inc eases, hey also augmen he pa ame e s on which he laye in ques ion depends, and he e o e, i is able o d aw o ep esen much mo e de ailed cha ac e is ics o he incoming image. My esea ch goal in his hesis is o de elop a Deep Lea ning model ha , p o ided a millime e -wa e image, iden i ies he p esence o h ea s unde a a ie y o objec ypes, clo hing ypes, and body ypes. The ained model will be p esen ed o Kaggle, a web page ha hos s da a science compe i ions o use s a ound he wo ld. Apa om he applica ion o he designed so wa e in he p oblem p oposed in he compe i ion, an- o he sui able applica ion could be a gene ic en y-secu i y sys em, in which i is common p ac ise o employ a ga eway me al de ec o . In his case he gi en scene would be a single subjec s anding in on o he image and he sys em would be designed o de ec any o eign objec s being ca ied by he subjec . 1 Resumen En los ´ul imos a˜nos, el campo de la isi´on po compu ado ha sido es igo de a ances con inuos. Uno de los a ances m´as des acados es el desa ollo de las llamadas edes neu onales con olucionales. Las ´ecnicas de ap endizaje p o undo han demos ado unciona muy bien en una g an a iedad de p oblemas y campos (po ejemplo, Biolog´ıa, F´ısica, In o m´a ica, Ma em´a icas, e c.). Su g an pode y lexibilidad se log a al ap ende a ep esen a el mundo como una je a qu´ıa de concep os anidada, con cada concep o de inido en elaci´on con concep os m´as simples, y ep- esen aciones m´as abs ac as calculadas en ´e minos de menos abs ac as. Es deci , pa a de ec a una o ma compleja, la imagen pasa po la p ime a capa, en la cual sus neu onas se exci an con o mas b´asicas, ya sean con- o nos, cuad ados, c´ı culos, e c... A medida que la p o undidad de la ed augmen a, ambi´en augmen an los pa ´ame os de los que depende la capa en cues i´on, y po ende, es capaz de dibuja o ep esen a unas ca ac e ´ıs icas mucho m´as de alladas de la imagen en an e. Mi obje i o de in es igaci´on en es a esis es desa olla un modelo de ap en- dizaje p o undo que, con una imagen de ondas milim´e icas, iden i ique la p esencia de amenazas bajo una a iedad de ipos de obje os, ipos de opa y ipos de cue pos. El modelo en enado se p esen a ´a a Kaggle, una p´agina web que o ganiza concu sos de ciencia de da os pa a usua ios de odo el mundo. Adem´as de la aplicaci´on del so wa e dise˜nado en el p oblema p opues o en la com- pe e ici´on, o a aplicaci´on adecuada pod ´ıa se un sis ema de segu idad de en ada gen´e ico, en el que es una p ´ac ica com´un emplea un de ec o de me ales de pue a de enlace. En es e caso, la escena dada se ´ıa un suje o ´unico de pie en e al gene ado de im´agenes y el sis ema es a ´ıa dise˜nado pa a de ec a cualquie obje o ex a˜no anspo ado po el suje o. 2 Resum En els ´ul ims anys, el camp de la isi´o pe compu ado ha es a es imoni d’a en¸cos con inus. Un dels a en¸cos m´es des aca s s´on les Xa xes Neu onals Con olucionals (CNN en angl`es). Les `ecniques d’ap enen a ge p o und han demos a eni un g an endimen en una g an a ie a de p oblemes i camps (pe exemple, Biologia, F´ısica, In- o m`a ica, Ma em`a iques, e c.). El seu g an pode i lexibili a s’aconsegueix ap enen a ep esen a el m´on com una je a quia de concep es anidada, amb cada concep e de ini en elaci´o amb concep es m´es senzills i ep esen acions m´es abs ac es calculades en e mes menys abs ac es (els de les capes an- e io s). ´ Es a di , pe de ec a una o ma complexa, la ima ge passa pe la p ime a capa, en la qual les se es neu onas s’exci en amb o mes b`asiques, ja siguin con o ns, quad a s, ce cles, e c... A mida que la p o undi a de la xa xa augmen a, amb´e augmen en els pa `ame es dels que dep`en la capa en q¨ues i´o, y po an , ´es capa¸c de dibuixa o ep esen a unes ca ac e ´ıs iques mol m´es de allades de la ima ge en an . El meu objec iu d’in es igaci´o en aques a esi ´es desen olupa un model d’Ap enen a ge P o und que, donada una ima ge d’ona mil·lim`e ica, iden i- iqui la p es`encia d’amenaces so a una a ie a de ipus d’objec es, ipus de oba i ipus de cos. El model en ena es p esen a `a a Kaggle, una p`agina web que o gani za compe icions de ci`encia de dades pe a usua is de o el m´on. A pa de l’aplicaci´o de l’algo i me dissenya en el p oblema p oposa en la compe ici´o, una al a aplicaci´o adequada pod ia se un sis ema gen`e ic de d’en ada de segu e a , en el qual ´es p `ac ica habi ual u ili za un de ec o de me all d’en ada. En aques cas, l’escena donada se ia un subjec e ´unic en on del gene ado d’ima ges i el sis ema es a `a dissenya pe de ec a qualse ol objec e es ange anspo a pe l’assump e. 3 Acknowledgemen s Fi s o all, I wan o hank my u o s, Josep Vidal Manzano and Olga Mu˜noz Medina, o helping me h oughou he en i e de elopmen o his hesis. I app ecia e he insigh s hey ha e gi en o me, as well as he pa ience o eaching and ad ising me. I would also like o hank Josep Pujal, and in gene al, he TSC (Teo ia del Senyal i Comunicacions) depa men o hei backup in echnical ques ions and he memmo y esou ces p o ided o he de elopmen o he hesis. Las bu no leas , I wan o hank my amily o being always by my side and chee ing me along he du a ion o my s udies. 4 Re ision His o y - App o al Reco d Re ision His o y and App o al Reco d Re ision Da e Pu pose 0 20/12/2017 C ea ion 1 8/01/2018 Re ision 2 15/01/2018 Re ision 3 20/01/2018 Re ision 4 25/01/2018 Upload Documen Dis ibu ion Lis Name E-mail Ad i´an de Jo ge S´anchez adjsanc[email p o ec ed] Olga Mu˜noz Medina [email protected] Josep Vidal Manzano josep. [email protected] 5 Con en s 1 In oduc ion 11 1.1 O e iew.............................. 11 1.2 Requi emen s and Speci ica ions . . . . . . . . . . . . . . . . . 12 1.3 Wo kPlan............................. 13 1.4 Gan Diag am .......................... 14 2 S a e o he a 15 2.1 Neu alNe wo ks ......................... 15 2.1.1 Pe cep on......................... 16 2.1.2 Ac i a ion Func ions . . . . . . . . . . . . . . . . . . . 17 2.1.3 Backp opaga ion . . . . . . . . . . . . . . . . . . . . . 19 2.1.4 Con olu ional Neu al Ne wo ks . . . . . . . . . . . . . 21 2.2 DeepLea ning........................... 28 2.2.1 Objec Recogni ion . . . . . . . . . . . . . . . . . . . . 28 3 Me hodology 31 3.1 P oblem S a emen . . . . . . . . . . . . . . . . . . . . . . . . 31 3.1.1 Pe o mance C i e ia . . . . . . . . . . . . . . . . . . . 31 3.1.2 P o ided Da a . . . . . . . . . . . . . . . . . . . . . . . 32 3.2 Da aAnalysis........................... 33 3.2.1 Da a S uc u e . . . . . . . . . . . . . . . . . . . . . . 33 3.2.2 Da a Visualiza ion . . . . . . . . . . . . . . . . . . . . 35 3.3 Da aP epa a ion......................... 36 3.3.1 Da a P ep ocessing . . . . . . . . . . . . . . . . . . . . 37 3.3.2 Da a Segmen a ion . . . . . . . . . . . . . . . . . . . . 41 3.4 Building he Model . . . . . . . . . . . . . . . . . . . . . . . . 42 3.4.1 F amewo k ........................ 43 3.4.2 Model A chi ec u e . . . . . . . . . . . . . . . . . . . . 43 3.4.3 Model T aining . . . . . . . . . . . . . . . . . . . . . . 46 6 4 Resul s 51 4.1 E alua ion Me ics . . . . . . . . . . . . . . . . . . . . . . . . 51 4.2 P edic ions ............................ 52 5 Conclusions 57 777 Lis o Figu es 2.1 Single laye Neu al Ne wo k, also called pe cep on. I ecei es a se o inpu s (3 in his case), which a e mul iplied by he weigh s and hen adds a bias. The simples ac i a ion unc ion is he s ep unc ion (see Eq. 2.1). . . . . . . . . . . . . . . . . 16 2.2 Sigmoid Func ion. The slope in he sigmoid unc ion end o sa u a e ela i ely low inpu s, bu is a so e sion o he anh. 18 2.3 Tanh Func ion. No e he di e en slope in ela ion o Sigmoid (seeFig.2.2)............................. 19 2.4 Fo simplici y, we can image he e o unc ion as a hill in which we a e ying o igu e ou he coo dina es (i.e. weigh s) o i s lowes place. F om Adi Deshpande. . . . . . . . . . . . 21 2.5 3-laye ne wo k wi h 4 neu ons in each hidden laye and 1 neu on in he ou pu laye . No e how he inpu laye is no coun ed, and he neu ons belonging o he same laye a e no connec ed.............................. 22 2.6 One o he i s Con olu ional Neu al Ne wo ks, by Peemen e al., 2011. This a chi ec u e was speci ically used o digi ecogni ion asks.......................... 23 2.7 An example inpu olume in ed (e.g. a 32x32x3 CIFAR-10 image), and a example il e in blue (con olu ional o dep h = 5). F om: CS231-n .............................. 24 2.8 The combina ion o wo 3x3 il e s a anged consecu i ely h ough laye s gi es a ecep i e ield o a single 5x5 il e in a laye . F om:gi books .......................... 24 2.9 The pooling laye akes, in his case, a pa ch o size 2x2 om he inpu and downsamples i h ough he inpu wi h s ide 2, aking he max ou o he 4 elemen s. F om: CS231-n . . . 26 2.10 Regula neu al ne wo k wi h a single ully connec ed laye . . . 27 2.11 R-CNN a chi ec u e by Ross Gi shick e al. which in ol es localiza ion + classi ica ion. . . . . . . . . . . . . . . . . . . . 29 2.12 Main scheme o a Gene a i e Ad e sa ial Ne wo k. . . . . . . 30 8 Chap e 2 S a e o he a In o de o achie e he co ec unde s anding o he di e en ields ha had o be add essed in his p ojec , a deep esea ch has been done h oughou he du a ion o i . This chap e gi es a b ie in oduc ion o he backg ound needed o de elop his hesis. The sec ion i s ly add esses he de ini ion o Neu al Ne wo k, i s a chi ec u e as i s di e en pa s. Followed by some o he common ypes o Neu al Ne - wo ks, his sec ion la e ocuses on Con olu ional Neu al Ne wo ks (CNN’s) by de ining i s a ious ypes o laye s The second sec ion desc ibes wo common ne wo ks used in objec ecogni- ion asks, which will gi e a b oade pe spec i e o he eade in how his p oblems a e commonly ea ed. By he end o his chap e , he necessa y backg ound o his hesis will ha e been in oduced o he eade . Neu al Ne wo ks The main concep o Neu al Ne wo k can be da ed o 1959, when a biological model, p oposed by Nobel lau ea es Hubel and Wiesel[22], was based on hei disco e y o wo ypes o cells in he p ima y isual co ex o a ca : simple cells and complex cells. Di e en pa e ns o ligh s imuli we e ied and i was obse ed ha a kind o pa e n ha may be good o some o he neu ons may be no longe e ec i e o he o he s. Successi e expe imen s ga e an unde s anding o which pa s o ha co ex we e s imula ed h ough di e en pa e ns, hence like ” il e s” ha beha e di e en ly o he same da a p esen ed. 15 Pe cep on The i s unc ional ne wo ks wi h many laye s we e published by I akhnenko and Lapa in 1965[1], becoming he G oup Me hod o Da a Handling (GMDH): a amily o induc i e algo i hms o ma hema ical modeling o mul i-pa ame ic da ase s ha pe o ms a pa ame ic op imiza ion o models. Ne e heless, i was no un il 1975 ha Neu al Ne wo ks egained in e es , when Paul We bos in oduced his pape [23] and p esen ed wha we may know now as backp opaga ion, which will be desc ibed la e (see Sec ion 2.1.3). x2w2Σ Ac i a ion Func ion y Ou pu x1w1 x3w3 Weigh s Bias b Inpu s Figu e 2.1: Single laye Neu al Ne wo k, also called pe cep on. I ecei es a se o inpu s (3 in his case), which a e mul iplied by he weigh s and hen adds a bias. The simples ac i a ion unc ion is he s ep unc ion (see Eq. 2.1). The mos basic neu al ne wo k, which all a ia ions de i e om, is a single- laye pe cep on, which consis s o a single laye o ou pu nodes; he inpu s a e ed di ec ly o he ou pu s ia a se ies o weigh s. In his way i can be conside ed he simples kind o eed- o wa d ne wo k (see Fig. 2.1). In common Neu al Ne wo k implemen a ions, he signal a a connec ion be- ween a i icial neu ons is a eal numbe , and he ou pu o each a i icial neu on is compu ed by a non-linea unc ion (see subsec ion2.1.2) o he sum o i s inpu s. Conside ing he s ep unc ion as ac i a ion unc ion, he ou pu o a pe cep on o an inpu ec o xis: (x) = (1 i w·x+b > 0 0 o he wise (2.1) whe e wT·x=Pm i=1 wixi, m is he numbe o inpu s o he pe cep on, and b is he bias gi en o i . In his example, he ac i a ion unc ion is he s ep 161616 unc ion. A i icial neu ons and connec ions ypically ha e a weigh ha adjus s as lea ning p oceeds. The weigh inc eases o dec eases he s eng h o he sig- nal a a connec ion. These neu ons may ha e a h eshold such ha only i he signal i sel c osses ha h eshold, he signal is sen . Typically, hese neu ons a e o ganized in laye s. Di e en laye s may pe o m di e en kinds o ans o ma ions on hei inpu s. Signals a el acc oss all he laye s, om he i s (inpu ) o he las (ou pu ) laye . I he signal c osses each laye once (i.e. he e a e no cycles o loops), we speak o Feed-Fo wa d Neu al Ne wo ks. I on he o he hand, in o ma- ion low bo h ways we speak o Recu en Neu al Ne wo ks. This hesis will ocus on eed- o wa d ne wo ks. Ac i a ion Func ions When we do no ha e an ac i a ion unc ion he weigh s and bias would simply do a linea ans o ma ion. A linea equa ion is simple o sol e bu is limi ed in i s capaci y o sol e complex p oblems. A neu al ne wo k wi hou a non-linea ac i a ion unc ion is essen ially jus a linea eg ession model. I we s ack nlaye s and each o hem applies a linea ans o ma ion o i s inpu , a he n h laye we will s ill ha e a linea unc ion. Compac ing all he linea ans o ma ions applied by all di e en laye s (suming all he do - p odu s o he laye s and mul iplying by each o hei weigh s, ecu si ely) we can squash his ne wo k in o a single-laye one. The main eason behind using ac i a ion unc ions is o in oduce a non- linea i y in he model, ha is able o simula e i ing a es, o p obabili ies o being exci ed gi en ce ain inpu o no . These unc ions map he ou pu o a neu on o some hing ha is bounded (e.g. be ween 0 and 1). Some o he mos common ac i a ion unc ions a e desc ibed below. Sigmoid The sigmoid non-linea i y akes a eal- alued numbe and educes i in o he ange [0,1]. Essen ially, la ge nega i e numbe s become 0 and la ge posi i e numbe s become 1. I can be exp essed as: σ(x) = 1 1 + e−x(2.2) 171717 This ans o ma ion has been used his o ically since i has a nice in e p e a- ion as he i ing a e o a neu on: om no i ing a all, o ully-sa u a ed i ing a an assumed maximum equency, 0 and 1 espec i ely. Howe e , i has wo d awbacks. Figu e 2.2: Sigmoid Func ion. The slope in he sigmoid unc- ion end o sa u a e ela i ely low inpu s, bu is a so e sion o he anh. As de ailed be o e, sigmoids end o ou pu numbe s nea o 0 o 1, sa u a - ing and kill g adien s. A e y undesi able p ope y o he sigmoid neu on is ha when he neu on’s ac i a ion sa u a es a ei he ail o 0 o 1, he g a- dien a hese egions is almos ze o. Du ing backp opaga ion (see Sec ion 2.1.3), his (local) g adien will be mul iplied o he g adien o his ga e’s ou pu . The e o e, i he local g adien is e y small, i will e ec i ely “kill” he g adien and almos no signal will low h ough he neu on o i s weigh s and so o i s da a. On he o he hand, i he ini ial weigh s a e oo la ge hen mos neu ons would become sa u a ed and he ne wo k will ba ely lea n. Following he i s disad an age, sigmoid ou pu s a e no ze o-cen e ed. This has implica ions on he dynamics du ing g adien descen , because i he da a coming in o a neu on is always posi i e (see Fig. 2.2), hen he g adien on he weigh s will du ing backp opaga ion become ei he all be posi i e, o all nega i e (depending on he g adien o he whole exp ession). This could in oduce zig-zagging dynamics in he g adien upda es o he weigh s. How- e e , i has less consequences compa ed o he sa u a ed ac i a ion p oblem abo e. 181818 Hype bolic Tangen The anh squashes a eal- alued numbe o he ange [−1,1]. Like he sig- moid neu on, i s ac i a ions sa u a e, bu he sigmoid neu on’s ou pu is ze o-cen e ed. The e o e, in p ac ice he anh non-linea i y is always p e- e ed o he sigmoid (see 2.2) nonlinea i y. Figu e 2.3: Tanh Func ion. No e he di e en slope in ela ion o Sigmoid (see Fig.2.2). In his pape [2], hey mainly p esen di e en unc ions wi h sa u a ed be- ha iou s and ”penalize” he nega i e ou pu s wi h a cons an (i.e. α∈[0,1]), o ha e a educed impac in ha ou pu ). A e-scaled Sigmoid ac i a ion is p oposed in he pape o make deep Sigmoid ne wo k ainable. I also has o be no ed ha hese desc ibed unc ions ha e a simple de i a i e, hus gi ing good compu a ional e iciency when backp opaga ing h ough he g adien s o each laye . Backp opaga ion A se o ques ions a ise like how do how do he il e s in each laye know wha alues o ha e? O how does he ully connec ed laye know wha ac i a ion maps o look a ? The e is a e m o e e o he lea ning p ocess, i is called backp opaga ion. Backp opaga ion can be sepa a ed in o 3 dis inc sec ions: he loss unc ion, he backwa d pass, and he weigh upda e. •Loss Func ion: When he da a has gone h ough all he ne wo k and so inishes he o wa d pass, since all o he weigh s o il e alues 191919 a e andomly ini ialized, he ou pu does no gi e p e e ence o any class/ca ego y in pa icula . The ne wo k, wi h i s cu en weigh s, is no able o make any easonable conclusion abou wha he classi ica- ion migh be. To cope wi h his, a loss unc ion is compu ed wi h he ou pu s o he ne wo k and he labels o he da a. Loss unc ions a e used o ep esen he p ice paid o inaccu acy o p edic ions in classi ica ion p oblems. Typical loss unc ions ha a e used: Mean Squa ed E o (MSE) and C oss-En opy. 1 N N X i=1 kˆyi−yik2 L(w) = 1 N N X n=1 [ynlog ˆy+ (1 −yn) log(1 −ˆyn)] (2.3) whe e ynis he ue label and ˆynis he p edic ed p obabili y. Ac ually, we wan o ge o a poin whe e he p edic ed label is he same as he aining label ( his means ha ou ne wo k go i s p edic- ion igh ). In o de o ge he e, we wan o minimize he amoun o loss we ha e. Tha is done by inding ou which weigh s mos di ec ly con ibu e o he loss (o e o ) o he ne wo k. In his pape [15], hey analyze a wide ange o losses ( om he p e- iously desc ibed (2.1.3) o some ancy unc ion losses like Tanimo o, Chebyshe o Cauchy-Schwa z Di e gence) •Backwa d Pass: The nex s ep is o de e mine which weigh s con- ibu e mos o he loss and ind ways o adjus hem so ha he loss dec eases. This is he ma hema ical equi alen o a dL/dW whe e W a e he weigh s a a pa icula laye , and Lis he unc ion chosen o he p oblem. •Weigh Upda e: Once he de i a e is compu ed ( he g adien o he loss depending on he weigh s gi en), i is he u n o make he weigh s know how hey a e con ibu ing o he loss. Ma hema ically, his is done by upda e hem so ha hey change in he opposi e di ec ion o he g adien . The simples o m o upda e is o change he pa ame e s along he nega i e g adien di ec ion (since he g adien indica es he di ec ion o inc ease, bu we usually wish o minimize a loss unc ion). Wn+1 =Wn−µ∇W(L) (2.4) 202020 Figu e 2.4: Fo simplici y, we can image he e o unc ion as a hill in which we a e ying o igu e ou he coo dina es (i.e. weigh s) o i s lowes place. F om Adi Deshpande. whe e ∇(L) is he g adien o he loss, L, in espec wi h he weigh s W, and µis he lea ning a e. The lea ning a e is a pa ame e ha is chosen by he p og amme . A high lea ning a e means ha bigge s eps a e aken in he weigh upda es and hus, i may ake less ime o he model o con e ge o an op imal se o weigh s. Howe e , a lea ning a e ha is oo high esul in jumps ha a e oo la ge and no p ecise enough o each he op imal poin (i keeps oscilla ing a ound i ). The p og am will epea his p ocess o a ixed numbe o i e a ions o each se o aining examples, i.e. examples o which he class hey belong o is known, commonly called mini-ba ch g adien descend, o o each image, s ochas ic g adien descen . Con olu ional Neu al Ne wo ks Unlike a egula Neu al Ne wo k, he laye s o a Con olu ional Neu al Ne - wo k ha e neu ons a anged in 3 dimensions: wid h, heigh , dep h. CNNs assume ha he inpu ec o is an image, op imizing he a chi ec u e p o i - ing om he ansla ional in a iance ea u e o images. The wo d dep h he e e e s o he hi d dimension o an ac i a ion olume, no o he dep h o a ull Neu al Ne wo k, which e e s o he o al numbe o laye s in a ne wo k (see Fig. 2.5). 212121 Regula single laye , ully-connec ed Neu al Ne wo ks do no scale well o ull images. In CIFAR-10, which is one o he mos popula da ase s (see CIFAR-10 ), images a e only o size 32x32x3 (32 wide, 32 heigh , 3 colo channels), so a single ully-connec ed neu on in a i s hidden laye o a egula Neu al Ne wo k would ha e 32x32x3 = 3072 weigh s. This amoun s ill seems manageable, bu clea ly his ully-connec ed s uc u e does no scale o la ge images. Fo example, an image o a bigge size, e.g 200x200x3, would lead o neu ons (see Fig. 2.1) ha ha e 200x200x3 = 120,000 weigh s. Con olu ional Neu al Figu e 2.5: 3-laye ne wo k wi h 4 neu ons in each hidden laye and 1 neu on in he ou pu laye . No e how he inpu laye is no coun ed, and he neu ons belonging o he same laye a e no connec ed. Ne wo ks p o i om he p io knowledge o he inpu being an image. Due o he ansla ionally-in a ian s uc u e o he image, i he neu ons a e de ec ing a ho izon al edge o some kind o blob-like pa e n a some loca ion in he image, i should in ui i ely be use ul a some o he loca ion as well. This is he eason behind using small il e s1: i does no ma e whe e he a ge is bu he co ec unde s anding o i , which is done in he ollowing laye s by ex ac ing high-le el ea u es o he da a. 1Con olu ion is a neighbo hood ope a ion in which each ou pu pixel is he weigh ed sum o neighbo ing inpu pixels. The ma ix o weigh s is called he con olu ion ke nel, o also known as il e . 222222 Figu e 2.6: One o he i s Con olu ional Neu al Ne wo ks, by Peemen e al., 2011. This a chi ec u e was speci ically used o digi ecogni ion asks. Laye Types in CNNs Con olu ional Neu al Ne wo ks a e buil by s acking laye s on op o each o he . These laye s pe o m di e en ac ions based on hei inpu s and he mos used ones a e Con olu ional, Pooling and Fully Connec ed Laye s. Con olu ional Laye s I s pa ame e s consis o a se o lea nable il e s. E e y il e is small spa ially, along wid h and heigh ( ypical alues a e 3x3, 5x5 and a ely 7x7), bu ex ends h ough he ull dep h o he inpu olume. Du ing he o wa d pass, we slide o mo e p ecisely con ol e each il e ac oss he wid h and heigh o he inpu olume and compu e do p oduc s be- ween he en ies o he il e and he inpu a any posi ion. As we slide he il e o e he wid h and heigh o he inpu olume we will p oduce a 2- dimensional ac i a ion map ha gi es he esponses o ha il e a di e en egions o he image. I we ex end he numbe o il e s o a dep h d, we will ha e an en i e se o weigh s and each o hem will p oduce a sepa a e 2-dimensional ac i a- ion map. A he end o he con olu ional laye , dac i a ion maps will be s acked along he dep h dimension and p oduce he ou pu olume. In Fig. 2.6, numbe s C1and C2co espond o he numbe o ac i a ion maps in each con olu ional laye , espec i ely. 232323 Figu e 2.7: An example inpu olume in ed (e.g. a 32x32x3 CIFAR-10 image), and a example il e in blue (con olu ional o dep h = 5). F om: CS231-n As an example, no e ha in Fig. 2.7 he e a e mul iple neu ons (5 in his example) along he dep h, all looking a he same egion in he inpu . Each neu on in he con olu ional laye is connec ed only o ha egion o he inpu space bu o he ull dep h, in his case 3, all colo channels. The ne wo k will lea n he weigh s o he di e en il e s ha ope a e on he image h ough he con olu ion ope a ion. The adap a ion o ha weigh s o some ype o isual ea u e such as an edge o some o ien a ion o pa e n is achie ed by backp opaga ion (see Sec ion 2.1.3). Mos o he bes - a ed me hods and algo i hms in Con Ne s[3][20], play wi h he ecep i e ields o neu ons: i is p e e ed o s ack small il e s on op o each o he han ha ing a la ge ecep i e ield on a single il e . This pape [11], among he o he ones, discuss he ad an ages and disad an ages o he main il e sizes and i ’s di e en possible conca ena ions. Figu e 2.8: The combina ion o wo 3x3 il e s a anged con- secu i ely h ough laye s gi es a ecep i e ield o a single 5x5 il e in a laye . F om: gi books 242424 Chap e 3 Me hodology This chap e includes all ele an in o ma ion abou he da a p ep ocessing s age, so wa e used and machine lea ning echniques applied in o de o ain he inal ne wo k ha will de ec h ea s om millime e -wa e images. P oblem S a emen The main objec i e o his p ojec is o build a model using deep lea ning o be able o disce n be ween subjec s ca ying h ea s o no . The model has been p esen ed o a compe i ion hos ed by Kaggle, a web pla o m ha o ganizes da a science compe i ions o use s om all o e he wo ld. Pe o mance C i e ia The p oblem p oposed is o p edic he p obabili y ha a gi en body zone, ou o 17 o al body zones, has a h ea p esen . The main e alua ion me hod used o classi y he models o he compe i o s is he c oss en opy loss, in his case, a binomial c oss en opy loss, which can be w i en as: L(w) = 1 N N X n=1 [ynlog ˆy+ (1 −yn) log(1 −ˆyn)] (3.1) whe e ˆynis he p edic ed p obabili y o he scan ha ing a h ea in he gi en body zone; Nis 17(numbe o h ea zones) ×numbe o scans (subjec s wi h unique scan id) in he es se ; ynis 1 i a h ea is p esen , 0 o he wise; log is he na u al (base e) loga i hm. 31 The wo las pe o mance measu es p o ide us a deepe pe spec i e o how he machine is pe o ming, alongside o he co ec o inco ec p edic ions (see Eq. (4.1)). The idea behind op imizing ha ype o loss is because i measu es he dissimila i y be ween wo ec o s, in his case he p edic ed ou pu s and he labels. I we minimize his loss, we a e making hese wo p e iously men ioned ec o s mo e simila in he da a space, con ibu ing o he lea ning o i . The model has o exploi he di e ence be ween he e lec i i y o he body, mainly composed by wa e , and he e lec i i y o h ea s, ha come in a wide ange o shapes and ma e ials. The ne wo k will be ained using a i ual p i a e ne wo k connec ing o he compu a ion’s emo e se e CALCULA, om Teo ia del Senyal i Comuni- cacions (TSC), which in u n belongs o UPC. I s ou pu s will be e alua ed conside ing, apa om he loss which is he main me ic o ake in o accoun , some o he e alua ion me ics such as he accu acy, speci i y and sensi i i y. P o ided Da a The da ase was p o ided by he T anspo a ion Secu i y Adminis a ion (TSA) o Kaggle and la e on uploaded o he compe i o s. The da ase con ains a la ge numbe o body scans acqui ed by a new gene a ion o millime e -wa e scanne called he High De ini ion-Ad anced Imaging Tech- nology (HD-AIT) sys em. The images in he da ase we e designed o cap u e eal scanning condi ions. They we e comp ised o olun ee s wea ing di e en clo hing ypes ( om ligh summe clo hes o hea y win e clo hes), wi h di e en body mass in- dices, di e en gende s, di e en numbe s o h ea s, and di e en ypes o h ea s. Due o es ic ions on e ealing he ypes o h ea s o which he TSA sc eens, he h ea s in he images we e ”ine ” objec s wi h a ying ma e ial p ope ies. In he i s s age o he con es , he compe i o s we e asked o use he gi en da ase and y o de ec h ea s on hem. The p oposed model o algo i hm had o gene alize well o o he ypes o subjec s (mainly om s age 2). The second s age consis ed o 1388 subjec s who had much mo e di e ences in e ms o body shapes and heigh , oge he wi h ha ing unlabeled da a. In his hesis, his da a was used as es da a o he s age 1 models. The main wo k was done on he s age 1 da ase , hough. 323232 Da a Analysis Ha ing in oduced he p oblem and he da ase gi en o sol e i , he nex s ep gi es a b ew explana ion o he s uc u e o he da a oge he wi h a isualiza ion o he di e en angle iews. Da a S uc u e The da a o each scan pe o med by he HD-AIT sys em is e e ed o as an HD-AIT F ame. A ame consis s o he ollowing ou bina y iles: •.ahi = calib a ed objec aw da a ile (2.26GB pe ile) •.aps = p ojec ed image angle sequence ile (10.3MB pe ile) •.a3d = combined image 3D ile (330MB pe ile) •.a3daps = combined image angle sequence ile (41.2MB pe ile) The ou iles gene a ed by he HD-AIT p og am ha e a common ile s uc- u e. All ou iles a e bina y and include a 512 by e heade ollowed by he ile’s da a. The heade mos ly con ains echnical scan pa ame e s like he equency wi h which his image has been cap u ed, dimensions (x, y, z) inc emen be ween ames, ime, e c... and is la gely iden ical ac oss all im- ages. Wi h he excep ion o he ield da a scale ac o , ha gi es us he scale o he da a comp ession in o de o i a ce ain numbe o ma (in his case uin 16), he o he ields a e no used in his hesis. The subjec s we e exposed o a scan ha made a ull o a ion and ook snap- sho s e e y 22’5o, esul ing on 16 ames which depic ed he scene scanned. Fo a subjec , he e a e 16 iews o 620x512 pixels so basically he enso s o he da ase a e o shape: 16x620x512. The i s s age o he compe i ion consis ed in a se o labelled subjec s, con- c e ely, 1147, wi h wha i was made he aining, alida ion and es se . In Fig. 3.1 he zone segmen a ion p oposed in he compe i ion is shown. 333333 The compe i ion p o ided labels o he o al 1147 subjec s, which made he p oblem all in o he ca ego y o supe ised lea ning1. By using he labels o each o he subjec , I could check wha body zones we e mo e p one o ha e a h ea . In o he wo ds, he numbe o subjec s ca ying a h ea we e coun ed o e e y body zone and hen, di ided by he numbe o subjec s, gi ing he co esponding pa s pe uni o each o he zones. The esul s a e summa ized in he nex able 3.1, we e he co esponding p.p.u. (pa s pe uni ) is o de ed in descendan mode. Figu e 3.1: P oposed h ea zones by he TSA. I also has o be no ed ha , due o he a ying physical condi ions o he subjec s, h ea s in conc e e body pa s we e disguised be e on di e en subjec s han o he s, which in u n made he di icul y o h ea de ec ion ise in ha egion. The desc ip ion o each body zone can be seen in Table 3.1. 1Supe ised Lea ning is he ield o Machine Lea ning whe e he da ase gi en o sol e a p oblem is labeled ( he algo i hm can es ima e a ype o loss o e o unc ion based on he knowledge o he inpu ’s class/ca ego y) 343434 Body Zone Body Desc ip ion # Th ea s Pe cen age 1 Righ Bicep 133 0.115955 2 Righ Fo ea m 126 0.109852 8 Uppe Righ Hip/Thigh 124 0.108108 14 Le Cal 122 0.106364 15 Righ Ankle Bone 118 0.102877 11 Lowe Righ Thigh 116 0.101133 6 Righ Rib Cage/Abs 116 0.101133 13 Righ Cal 110 0.095902 16 Le Ankle Bone 109 0.095031 4 Le Fo ea m 108 0.094159 5 Uppe Ches 106 0.092415 3 Le Bicep 104 0.090671 12 Lowe Le Thigh 101 0.088056 10 Uppe Le Hip/Thigh 100 0.087184 17 Uppe Back 95 0.082825 7 Le Rib Cage/Abs 93 0.081081 9 G oin (Sensi i e A ea) 90 0.078466 Table 3.1: Summa y o he numbe o h ea s depending on body zones plus he desc ip ion o each body pa . Da a Visualiza ion Be o e add esing he p ep ocessing and ne wo k implemen a ion pa , a i- sual analysis on he da a has been made in o de o p oceed co ec ly owa ds a a ional solu ion. As i will be seen, he e a e some pa s o he body ha a e no be isible in ce ain scans, bu he i s p ocedu e in his hesis has been o ake all he da a in o accoun because, i i ’s no ele an , he algo i hm will disca d i a any ime (won’ ac i a e neu ons o ha image, because i will no see any h ea ). Lo s o di ec human isualiza ion o he di e en subjec s was done in o de o look o di e en app oaches. The ollowing pic u e (see Fig. 3.2) shows he summa y o 16 iews o a single subjec . No e ha his images we e used be o e p ep ocessing, wi h he pu pose o es ing ou isual pe cep ion o he h ea s. 353535 Figu e 3.2: 16 slices o a ull 360 deg ee image o a subjec . I looked closely, his example shows a h ea o e he le knee ( h ea zone 12). Da a P epa a ion Following he inspec ion o he da a by isualising and manipula ing i , i is needed o p epa e he da a o he inpu pipeline o he models. The main a chi ec u e o his algo i hm consis s in a p ep ocessing module, ha segmen a es he image in o 17 di e en body zones and no malizes he da a, and a ne wo k module wi h 17 independen models, i.e. each body zone is ea ed sepa a ely. A e his sec ion, he images will be eady o be ed o each o he di e en se en een models (i.e. one o each body zone). The gi en images we e o shape 16x620x512 (wid h and heig h espec i ely) and had o be educed o pa ches o size 16x224x224 in o de o apply o he s anda d Con Ne inpu sizes. 363636 Da a P ep ocessing When wo king wi h CNNs we no mally use aw images as inpu da a, bu some p ep ocessing mus be done be o e. CNNs lea n by con inually adding o he weigh s, g adien e o ec o s (mul iplied by a lea ning a e) ob ained om a backp opaga ion h ough many ma ices ba ch by ba ch. I is in ou in e es o ensu e ha e e y image has a simila ange o alues in o de o a oid ha he g adien s un ou o con ol, and he way o do i is by sub ac ing he mean o he whole image da ase o each sample. This way, he g adien s ac uni o mly o each channel. O he wise, he lea ning a e would cause co ec ions in each dimension ha would di e , and compensa - ing a co ec ion in one weigh dimension migh imply unde compensa ing in ano he , p o oking di icul ies o he loss o s abilize. Mos o he pixels om he aw image we e in he ange [0, 20] (da kes pixels). No e how much noise is p oduced by his ype o scans. Figu e 3.3: An example aw da a image co esponding o he i s iew/dep h slice, whe e he subjec is acing on wa ds. As i may be no iced looking a he his og am, he a iance o he pixels is no ha high, since almos 3/4 o he o al pixels a e in he ange [0, 20] (da kes pixels). This ac ually is undesi able because we wan he maximum di e en in ensi ies (no compac ed in o a egion) while p ese ing he isual in o ma ion in ac . 373737 Figu e 3.4: His og am o he aw image. The echnique applied o ampli y he con as in an image is called his- og am2equaliza ion. The equalized image is shown below, oge he wi h i s his og am. Figu e 3.5: Image a e equalizing. No e ha mos o he noise has been emo ed. 2An image his og am is a ype o his og am ha ac s as a g aphical ep esen a ion o he onal dis ibu ion in a digi al image. I gi es he numbe o pixels wi h a conc e e onal alue. 383838 Equaliza ion was pe o med using a CLAHE (Con as Limi ed Adap i e His og am Equaliza ion) echnique (wi h a c 2 unc ion, see 3.4.1 o ame- wo k in o ma ion). This echnique consis s in adap ing he his og am equal- iza ion o small egions o he image, o p ese e hem o o e -b igh en/o e - da ken because o conside ing global con as ins ead. Then each o hese blocks a e his og am equalized as usual. A e equaliza ion, o emo e a i- ac s in ile bo de s, bilinea in e pola ion is applied. A ile size o 8x8 was used because, expe imen ally, ga e he bes esul s. Figu e 3.6: Co esponding his og am a e CLAHE applied. No ice ha he ange o he alues has no inc eased, bu he image has signi an ly educed i s noise plus he h ea (body zone 14, le cal ) is be e de ined. Following he equaliza ion, a da a cen e ing, also called mean sub ac ion was done. When he da a consis s o images, he common p ocedu e in- ol es sub ac ing he mean o he image o all he pixels. Basically, we can s a de ining he mean o a ec o xo n alues (e.g. a pixel column/ ow o an image): E{x}=µ=1 n n X i=1 xi(3.2) F om 3.2, i we ex end i o a wo-dimensional case (e.g. he ull image): E{x}=µ=1 nm m X j=1 n X i=1 xij (3.3) 393939 Now i we compu e he mean o each iew, sepa a edly, as 3.3, we’ll end up wi h µ0, µ1, ..., µ15 means o each iew. Now he mean o each iew (o channel, as i was explained be o e) is subs ac ed om he co esponding iew i sel . I we de ine a single iew/ ame/slice as Xc, whe e c ∈1, . . . , C and Cis he o al numbe o channels in his case 16: ∀xij ∈Xc, µc=1 nm m X j=1 n X i=1 xij, Xc=Xc−µcI (3.4) The eason behind subs ac ing he mean is such as cen e ing he cloud o da a a ound he o igin along e e y dimension. In deep lea ning, i is used o a oid anishing/exploding g adien s due o a bias in he aw da ase . Mo eo e , a e ze o cen e ing, he esponse om a andomly ini ialized il e (con olu ional laye ) is s ill close o a ze o-cen e ed dis ibu ion. This would loca e he ne wo k’s ini ial esponses nea he 0 o he nonlinea i y ( elu, sigmoid, leaky- elu, e c.). A his poin small changes in he il e esponse ha e a la ge e ec on he nonlinea i y esponse (changing om 0 o posi i e in he elu, and quickly changing signs in he sigmoid o leaky- elu), hus yielding la ge g adien s e en wi h small changes in he il e esponse. Ano he p ep ocessing echnique applied o he da ase is no malizing he da a dimensions so ha hey a e o app oxima ely he same scale. The com- mon p ocedu e is o di ide each dimension by i s s anda d de ia ion, once i has been ze o-cen e ed. In case o images i consis s in educing he ange om [0,255] o [0,1], so we p ese e he ela ion be ween pixels bu we ha e his pixels cons ained o 0 and 1 (0 ≤I≤1). As in images pixels a e commonly wide-dis ibu ed h ough he di e en le els (i no , as i was ou case, we apply his og am equaliza ion) i is no ha impo an . Howe e , in classi ie s ha calcula e he dis ance be ween wo poin s by he Euclidean dis ance would ha e a g ea p oblem in sol ing an op imiza ion p oblem, since i he ea u es a e non-scaled, he mos a ian ea u es will ha e dominion o e he o he s when op imizing ( hey will gi e la ge alues in dis ance). Thus, no malizing is a way o equally dis ibu e he p opo ion o con ibu ion o each ea u e o he inal dis ance. 404040 Figu e 3.10: A no mal dis ibu ion wi h µ= 0 and σ= 0.1 •Xa ie [9] No mal Ini ialize : This ini ialize is designed o keep he scale o he g adien s oughly he same in all laye s. I simply is a no mal dis ibu ion wi h mean 0 and s anda d de ia ion σwhe e V a (w) = σ2=2 nin +nou (3.6) and nin, nou a e he numbe o inpu neu ons and ou pu neu ons o he laye , espec i ely. A simple explana ion is ha i ini ializes he weigh s in a way ha he signal does no ei he sh ink o inc ease when i passes h ough all he laye s, and he e o e a he end o he ne wo k we do no ha e useless neu ons by anished/exploded g adien . Howe e , in his pape [12], discuss ha o ReLU neu ons, Xa ie ini ializa ion migh no wo k ha well. This has o do wi h an app oxima ion aken in Xa ie , which is ha he inpu is ze o-cen e ed and conc e ely in ReLU neu ons he mean is g ea e han 0, µ > 0. •He[12] Ini ializa ion: This is a modi ica ion on o Xa ie ini ialize , which akes in o accoun bo h inpu and ou pu uni s by aking he ha monic mean o hem. In he pape om Kaiming He e al., hey demons a e ha o deep ne wo ks, in his pape 22 and 30 laye s, Xa ie ini ialize akes longe o educe he e o in he 22-laye ne - wo k, and in he 30-laye ne wo k i comple ely s alls, which is e i ied by hei g adien s, ha ge diminished. The cons ain added o he weigh s ini ializa ion is V a (w) = 2 nin (3.7) 474747 Op imize s When backp opaga ing h ough he ne wo k, he algo i hm upda es he weigh s based on a hype pa ame e called lea ning a e. This lea ning a e speci y how much o he g adien o he loss ∇L(wn) (which is unc ion o he weigh s assigned in he p e ious s ep) is subs ac ed o ha weigh . Then a new weigh wn+1 =wn−µ∇L(wn) is assigned. This is called s ochas ic g adien descend. Many imp o emen s on he basic s ochas ic g adien descen algo i hm ha e been p oposed and used. The need o se an op imized lea ning a e was c ucial, since se ing his pa ame e oo high can cause he algo i hm o di- e ge, when on he o he hand, se ing i oo low makes i slow o con e ge. The ollowing summa y p o ides in o ma ion abou he di e en op imize s used in he p oblem. •Momen um: This op imize eme ged om a physical poin o iew. I uses physical law o mo ion o go pass h ough local op ima (small hills). In ui i ely, adding momen um will also make he con e gence as e , as we’ e accumula ing speed, so he g adien s ep will be la ge . The objec i e unc ions o deep a chi ec u es ha e complex o ms (wi h local op ima) and hus s anda d SGD can lead o e y slow con e gence pa icula ly a e he ini ial s eep gains. n+1 =γ∗ n+µ∇L(wn), wn+1 =wn+ n+1 (3.8) The main di e ence om he g adien descend me hod shown abo e, whe e he g adien di ec ly is in eg a ed wi h he posi ion (weigh s), is ha he physics iew sugges s an upda e in which he g adien only di ec ly in luences he eloci y, which in u n has an e ec on he posi- ion. The upda e o he eloci y is gi en he old eloci y alue and new G adien Descen s ep µ∇L(wn−1). We also decay ou pas eloci y so ha we only conside he mos ecen eloci ies wi h γ= 0.9 which is he mos comomn alue used. •AdaG ad: Adap i e G adien [7] is ano he app oach o he G adien Descen p oblem, bu om ano he poin o iew. The p oblem wi h lea ning a e µin G adien Descen is ha i is cons an and a ec s all he pa ame e s. To sol e his, he sum o squa ed o all o ou pa- ame e s’ g adien , and use ha o no malize he lea ning a e µ. I we de ine gn,i as he g adien o he objec i e unc ion wi h ela ion o he pa ame e wia ime s ep nwe ha e: 484848 gn,i =∇L(wn,i), wn+1,i =wn,i −µ pGn,ii +gn,i (3.9) whe e Gn∈ <dxd is he diagonal ma ix whe e each diagonal elemen is he sum o he squa es o he g adien s w. . wn,i. Now he lea ning a e applied o each pa ame e will be smalle o la ge depending on how he pas g adien s beha ed: pa ame e s ha go a big upda e will be slowed down while pa ame e s ha ecei ed li le upda es will ha e bigge lea ning a e o accele a e he lea ning p ocess. In [13], om Penning on e al., Adag ad was used o ain GloVe wo d embeddings, as in equen wo ds equi e much la ge up- da es han equen ones. Howe e , Adag ad’s main weakness is i s accumula ion o he squa ed g adi- en s in he denomina o : Since e e y added e m is posi i e, he accumula ed sum keeps g owing du ing aining. This in u n causes he lea ning a e o sh ink and e en ually become in ini esimally small, a which poin he algo- i hm is no longe able o acqui e addi ional knowledge. Regula iza ion A cen al p oblem in machine lea ning is how o make an algo i hm ha will pe o m well no jus on he aining da a, bu also on new inpu s. Many s a egies used in machine lea ning a e explici ly designed o educe he es e o , possibly a he expense o inc eased aining e o . These s a egies a e known as egula iza ion. I is a way o ensu e ha he ne wo k does no o e i o he aining images, bu a he makes obus connec ions be ween neu ons, inducing he model o be mo e spa se in he laye ac i a ions. D opou is an ex emely e ec i e, simple and ecen ly in oduced egula - iza ion echnique by S i as a a e al. in [21] ha complemen s he o he me hods (L1, L2, maxno m). In ou case, only d opou wi h a a e o 0.5 was used. The e is mainly one eason, apa om he di e en esea ch in [8] whe e i is shown ha he op imum ange o he d opou a e o neu ons is be ween [0.4, 0.6], is ha wi h a neu on ac i a ion p obabili y o 0.5, we ge an equally p obable dis ibu ion o se s wi h di e en neu on ac i a ions (i.e. maximum a iabili y be ween se s o neu ons). While aining, d opou is implemen ed by only keeping a neu on ac i e wi h some p obabili y p(a hype pa ame e ), o se ing i o ze o o he wise. When es ing, we emo e 494949 Figu e 3.11: D opou educes he numbe o connec ions in he ne wo k, enhancing ea u e de ec ion by cu ing some o he in o ma ion gi en o he nex laye , he e o e inducing he model o de elop mo e obus connec ions be ween neu ons he d opou laye s in o de o es he po en ial o all he connec ions c ea ed while aining. The di e en ne wo ks we e ained wi h a ba ch size o 128 and o 50 epochs. In he pic u es i will be seen ha he loss d ops a abou he hal o epochs, hence, a checkpoin a gumen was passed o he unc ion used o i he model, so he model was sa ed in case ha alida ion loss did no educe o 5 epochs. 505050 Chap e 4 Resul s In his sec ion, he esul s om he VGG-16 ained ne wo k a e shown. Some o he di e en con igu a ions a e discussed along wi h he di e en esul in e p e a ions. Much o he wo k was done in he weigh ini ial- iza ion pa . Mos o he p oblems o aining his ne wo k was he high numbe o pa ame e s o une: ha ing a bad weigh ini ializa ion in he i s laye becomes a disas e wi h mo e and mo e laye s, because i keeps ge ing mul iplied by a alue ha is no ” ele an ” o he ne wo k, hence explod- ing/diminishing g adien s. A Ba ch No maliza ion laye is p oposed o cope wi h his p oblem. E alua ion Me ics Fi s o all, we de ine he e alua ion me ics used o measu e he pe o mance o ou deep lea ning model. The mos impo an one among Accu acy is C oss-En opy Loss, ha is he los used o desc ibe how good/bad is ou model app oxima ing he g ound u h o he da a. Also, he compu a ion o he weigh /pa ame e upda e ha depends di ec ly o he g adien o his loss, di e s acco ding he op imize used. O he me ics aken in o accoun a e he speci i y and sensi i i y, which a e desc ibed ma hema ically as: Sensi i i y =TP TP +FN , Speci i y =TN TN +FP (4.1) whe e TP, ue posi i e, could be de ined as he co ec hi /p edic ion ( h ea p esence) o a subjec whe eas TN, ue nega i e, could be de ined as he co ec ejec ion ( h ea absence) o a subjec . 51 A Con usion Ma ix will be used o ep esen his alues. I s name s ems om he ac ha i makes i easy o see i he sys em is con using wo classes (i.e. ou pu ing h ea p esence he e is no h ea and ice e sa). whe e we T ue Label Posi i e Nega i e Ne wo k Ou pu Posi i e TP FP TP +FP Nega i e FN TN FN +TN TP +FN FP +TN N Table 4.1: Con usion Ma ix o a p oblem wi h 2 classes. can easily calcula e speci i y and sensi i i y. Wi h he i s column, we can compu e he o me ; wi h he second column, we can compu e he la e . The goal o he model is o educe he c oss-en opy be ween ue labels and p edic ed ones. This loss inc eases as he p edic ed p obabili y di e ges om he ac ual label. So p edic ing a p obabili y o .012 when he ac ual obse - a ion label is 1 would be bad and esul in a high log loss. Ma hema ically, i can be w i en as L(w) = 1 N N X n=1 [ynlog ˆy+ (1 −yn) log(1 −ˆyn)] (4.2) P edic ions As i is explained in Sec ion 3.1.2, aining wo k was done in he s age 1 da a. This da ase consis ed o 1147 labeled images, which we e ed o he models a e he p ocedu es desc ibed in Da a P epa a ion (Sec ion 3.3). This sec ion pu s up all he esul s gi en by he di e en ne wo k con igu- a ions. D opou Laye s wi h p obabili y a e o 0.5 (see Fig. 3.11) we e placed in he 2 ully connec ed laye s p e ious o he so max classi ie laye (ou pu laye ). The ba ch size used in all he aining con igu a ions has been changing be ween 64/128 images (16 iews implici ly), bu wi hou no- able changes. I ha e added he me ics o he bes lea ning a e o each op imize , only. Two o he bes pe o ming models we e he ones assigned o h ea zone 9 (Sensi i e A ea) and h ea zone 16 (Le Ankle Bone). In he ollowing pic u e, we can see he loss achie ed a s ep 284 o he body zone 16. 525252 The i s esul s shown co espond o Momen um (see 3.8 op imized mod- els, whe e he accu acy was inc edibly apid o each he maximum in each o he body zones. They a e summa ized in he able below. The es ba ch Zone Val. Accu acy X-En opy Loss 1 0.8678 0.3357 2 0.8952 0.2641 3 0.8849 0.2838 4 0.9237 0.2037 5 0.8975 0.2595 6 0.8634 0.2845 7 0.9158 0.2163 8 0.9265 0.1965 9 0.9079 0.2238 10 0.8942 0.2397 11 0.9037 0.2302 12 0.9182 0.2154 13 0.9323 0.1784 14 0.8854 0.2508 15 0.8713 0.2632 16 0.9345 0.1696 17 0.9215 0.1858 Table 4.2: Accu acy and C oss-En opy loss o e e y h ea zone wi h he Momen um Op imize + lea ning a e = 10−4 was o shape 229x16x224x224 (20% o 1147). As i can be seen, he di e en accu acies achie ed a e a ela i e good esul . The Momen um upda e p o- ided he as es way o con e ge in o he local op ima, bu i seemed o s all when eaching i . Con usion Ma ix om h ea zone 16, which is he bes model in e ms o Speci i y and Sensi i i y, can be seen below. (see Table ??).. T ue Label Posi i e Nega i e Ne wo k Ou pu Posi i e 12 9 21 Nega i e 6 202 208 18 211 229 Table 4.3: Con usion Ma ix o h ea zone 16 wi h Momen um Op imize 535353 Speci i y and Sensi i i y a e easily compu ed om he Con usion Ma ix. Fo his h ea zone, Sensi i i y is compu ed as Se =12 18 = 0.66 and Speci i y = 202 211 = 0.9573. In his p oblem, whe e he e was no a balanced da ase , ( 10% h ea p es- ence) i was ha d o ge a high Sensi i i y (co ec h ea p esence ou pu ), as he ne wo ks we e ained wi h much mo e ”non-ca ying h ea ” subjec s da a han wi h images ha con ained a h ea in i . Howe e , I did no wan o balance he da ase as he e al eady we e ew images. An image o alida ion accu acy/loss on h ea zone 9 is gi en below. Figu e 4.1: Valida ion Accu acy and Loss in body zone 9 wi h Momen um Op imize . No e ha Accu acy eaches he maximum a ea lie epochs han he Loss. Ano he op imize used was AdaG ad, which is a di e en me hod o upda e he pa ame e s, adap ing he lea ning a e sepa a ely o e e y dimension (see ??) acco ding o he loss p oduced by hem. The nex able summa izes he esul s on Adap i e G adien op imized models. This op imize was used in o de o p e en weigh s o aking he same alue, and hus dieing while aining. I has o be no ed ha his also has conse- quences: he g adien is mo e likely o pe o m zigzags due o he inc eased a iabili y o he alues o i s weigh s. 545454 Zone Val. Accu acy X-En opy Loss 1 0.8431 0.3656 2 0.8682 0.2941 3 0.8537 0.3338 4 0.9024 0.2253 5 0.8634 0.2885 6 0.8429 0.3245 7 0.8758 0.2662 8 0.8872 0.2415 9 0.9170 0.2107 10 0.8812 0.2508 11 0.8736 0.2609 12 0.9042 0.2253 13 0.8785 0.2784 14 0.8854 0.2397 15 0.8713 0.2632 16 0.9126 0.2184 17 0.9028 0.2305 Table 4.4: Accu acy and C oss-En opy loss o e e y h ea zone wi h he AdaG ad Op imize + lea ning a e = 10−4 . Wi h his op imize , he models pe o med sligh ly wo se han he Mo- men um ones. Mo eo e , he model wi h highes accu acy and lowes loss was 9, di e ing om he esul wi h Momen um (highes accu acy achie ed wi h body zone 16). See Fig. 3.1) o a isual ep esen a ion o each body zone along wi h i s assigned numbe . Con usion Ma ix om h ea zone 9 can be seen below. T ue Label Posi i e Nega i e Ne wo k Ou pu Posi i e 11 11 22 Nega i e 8 199 207 19 210 229 Table 4.5: Con usion Ma ix o h ea zone 9 wi h AdaG ad op imize . This op imize wo ked a bi wo se han he Momen um. Fi s , he e we e p oblems ini ializing he weigh s: when applying Xa ie ini ializa ion (see 555555 3.4.3), he weigh s we e, a e ew epochs, diminishing un il becoming 0. I was when He ini ializa ion (see 3.4.3) was applied ha he weigh s began o become spa se and s a o lea n di e en ea u es h ough he laye s o he ne wo k. One eason is ha He ini ializa ion akes in o accoun he non- linea i y o he ac i a ion unc ion, while Xa ie s ays in he linea egion. In he nex igu e, alida ion accu acy and loss a e shown. Figu e 4.2: Valida ion Accu acy and Loss in body zone 9 wi h AdaG ad Op imize . No e ha he slope o he cu e on he loss is di e en han in Fig 4.1 None heless, in compa ison o Momen um, om accu acy and loss o sensi i - i y and speci i y, he esul s on his Op imize we e wo se han he ob ained wi h he Momen um Op imize , al hough a p io i, i seemed ha he Adap- i e G adien was a solu ion o he o me . In he ollowing able, we can see he alida ion accu acy and loss o AdaG ad ained models. 565656