S udy o Machine Lea ning
Algo i hms To De ec Th ea s In
Ai po Passenge s
A Deg ee Thesis
Submi ed o he Facul y o he
Escola T`ecnica d’Enginye ia de Telecomunicaci´o de
Ba celona
Uni e si a Poli `ecnica de Ca alunya
by
Ad i´an de Jo ge S´anchez
In pa ial ul ilmen
o he equi emen s o he deg ee in
TELECOMMUNICATION SYSTEMS
ENGINEERING
Ad iso s:
Josep Vidal Manzano
Olga Mu˜noz Medina
Abs ac
In he la es yea s, he ield o compu e ision has wi nessed con inual ad-
ancemen s. One o he mos s a ed ad ancemen is Con olu ion Neu al
Ne wo ks (CNNs).
Deep Lea ning echniques ha e p o en o pe o m e y well on a la ge a ie y
o p oblems and ields (i.e. Biology, Physics, Compu e Science, Ma hema -
ics, e c.). I s g ea powe and lexibili y is achie ed by lea ning o ep esen
he wo ld as a nes ed hie a chy o concep s, wi h each concep de ined in
ela ion o simple concep s, and mo e abs ac ep esen a ions compu ed in
e ms o less abs ac ones. Tha is, o de ec a complex shape, he image
passes h ough he i s laye , in which i s neu ons a e exci ed wi h basic
shapes, be hey con ou s, squa es, ci cles, e c... As he dep h o he ne wo k
inc eases, hey also augmen he pa ame e s on which he laye in ques ion
depends, and he e o e, i is able o d aw o ep esen much mo e de ailed
cha ac e is ics o he incoming image.
My esea ch goal in his hesis is o de elop a Deep Lea ning model ha ,
p o ided a millime e -wa e image, iden i ies he p esence o h ea s unde a
a ie y o objec ypes, clo hing ypes, and body ypes.
The ained model will be p esen ed o Kaggle, a web page ha hos s da a
science compe i ions o use s a ound he wo ld. Apa om he applica ion
o he designed so wa e in he p oblem p oposed in he compe i ion, an-
o he sui able applica ion could be a gene ic en y-secu i y sys em, in which
i is common p ac ise o employ a ga eway me al de ec o . In his case he
gi en scene would be a single subjec s anding in on o he image and he
sys em would be designed o de ec any o eign objec s being ca ied by he
subjec .
1
Resumen
En los ´ul imos a˜nos, el campo de la isi´on po compu ado ha sido es igo
de a ances con inuos. Uno de los a ances m´as des acados es el desa ollo de
las llamadas edes neu onales con olucionales.
Las ´ecnicas de ap endizaje p o undo han demos ado unciona muy bien
en una g an a iedad de p oblemas y campos (po ejemplo, Biolog´ıa, F´ısica,
In o m´a ica, Ma em´a icas, e c.). Su g an pode y lexibilidad se log a al
ap ende a ep esen a el mundo como una je a qu´ıa de concep os anidada,
con cada concep o de inido en elaci´on con concep os m´as simples, y ep-
esen aciones m´as abs ac as calculadas en ´e minos de menos abs ac as.
Es deci , pa a de ec a una o ma compleja, la imagen pasa po la p ime a
capa, en la cual sus neu onas se exci an con o mas b´asicas, ya sean con-
o nos, cuad ados, c´ı culos, e c... A medida que la p o undidad de la ed
augmen a, ambi´en augmen an los pa ´ame os de los que depende la capa en
cues i´on, y po ende, es capaz de dibuja o ep esen a unas ca ac e ´ıs icas
mucho m´as de alladas de la imagen en an e.
Mi obje i o de in es igaci´on en es a esis es desa olla un modelo de ap en-
dizaje p o undo que, con una imagen de ondas milim´e icas, iden i ique la
p esencia de amenazas bajo una a iedad de ipos de obje os, ipos de opa
y ipos de cue pos.
El modelo en enado se p esen a ´a a Kaggle, una p´agina web que o ganiza
concu sos de ciencia de da os pa a usua ios de odo el mundo. Adem´as
de la aplicaci´on del so wa e dise˜nado en el p oblema p opues o en la com-
pe e ici´on, o a aplicaci´on adecuada pod ´ıa se un sis ema de segu idad de
en ada gen´e ico, en el que es una p ´ac ica com´un emplea un de ec o de
me ales de pue a de enlace. En es e caso, la escena dada se ´ıa un suje o
´unico de pie en e al gene ado de im´agenes y el sis ema es a ´ıa dise˜nado
pa a de ec a cualquie obje o ex a˜no anspo ado po el suje o.
2
Resum
En els ´ul ims anys, el camp de la isi´o pe compu ado ha es a es imoni
d’a en¸cos con inus. Un dels a en¸cos m´es des aca s s´on les Xa xes Neu onals
Con olucionals (CNN en angl`es).
Les `ecniques d’ap enen a ge p o und han demos a eni un g an endimen
en una g an a ie a de p oblemes i camps (pe exemple, Biologia, F´ısica, In-
o m`a ica, Ma em`a iques, e c.). El seu g an pode i lexibili a s’aconsegueix
ap enen a ep esen a el m´on com una je a quia de concep es anidada, amb
cada concep e de ini en elaci´o amb concep es m´es senzills i ep esen acions
m´es abs ac es calculades en e mes menys abs ac es (els de les capes an-
e io s). ´
Es a di , pe de ec a una o ma complexa, la ima ge passa pe la
p ime a capa, en la qual les se es neu onas s’exci en amb o mes b`asiques,
ja siguin con o ns, quad a s, ce cles, e c... A mida que la p o undi a de la
xa xa augmen a, amb´e augmen en els pa `ame es dels que dep`en la capa en
q¨ues i´o, y po an , ´es capa¸c de dibuixa o ep esen a unes ca ac e ´ıs iques
mol m´es de allades de la ima ge en an .
El meu objec iu d’in es igaci´o en aques a esi ´es desen olupa un model
d’Ap enen a ge P o und que, donada una ima ge d’ona mil·lim`e ica, iden i-
iqui la p es`encia d’amenaces so a una a ie a de ipus d’objec es, ipus de
oba i ipus de cos.
El model en ena es p esen a `a a Kaggle, una p`agina web que o gani za
compe icions de ci`encia de dades pe a usua is de o el m´on. A pa de
l’aplicaci´o de l’algo i me dissenya en el p oblema p oposa en la compe ici´o,
una al a aplicaci´o adequada pod ia se un sis ema gen`e ic de d’en ada de
segu e a , en el qual ´es p `ac ica habi ual u ili za un de ec o de me all
d’en ada. En aques cas, l’escena donada se ia un subjec e ´unic en on
del gene ado d’ima ges i el sis ema es a `a dissenya pe de ec a qualse ol
objec e es ange anspo a pe l’assump e.
3
Acknowledgemen s
Fi s o all, I wan o hank my u o s, Josep Vidal Manzano and Olga Mu˜noz
Medina, o helping me h oughou he en i e de elopmen o his hesis. I
app ecia e he insigh s hey ha e gi en o me, as well as he pa ience o
eaching and ad ising me.
I would also like o hank Josep Pujal, and in gene al, he TSC (Teo ia del
Senyal i Comunicacions) depa men o hei backup in echnical ques ions
and he memmo y esou ces p o ided o he de elopmen o he hesis.
Las bu no leas , I wan o hank my amily o being always by my side
and chee ing me along he du a ion o my s udies.
4

Re ision His o y - App o al
Reco d
Re ision His o y and App o al Reco d
Re ision Da e Pu pose
0 20/12/2017 C ea ion
1 8/01/2018 Re ision
2 15/01/2018 Re ision
3 20/01/2018 Re ision
4 25/01/2018 Upload
Documen Dis ibu ion Lis
Name E-mail
Ad i´an de Jo ge S´anchez adjsanc[email p o ec ed]
Olga Mu˜noz Medina [email protected]
Josep Vidal Manzano josep. [email protected]
5
Con en s
1 In oduc ion 11
1.1 O e iew.............................. 11
1.2 Requi emen s and Speci ica ions . . . . . . . . . . . . . . . . . 12
1.3 Wo kPlan............................. 13
1.4 Gan Diag am .......................... 14
2 S a e o he a 15
2.1 Neu alNe wo ks ......................... 15
2.1.1 Pe cep on......................... 16
2.1.2 Ac i a ion Func ions . . . . . . . . . . . . . . . . . . . 17
2.1.3 Backp opaga ion . . . . . . . . . . . . . . . . . . . . . 19
2.1.4 Con olu ional Neu al Ne wo ks . . . . . . . . . . . . . 21
2.2 DeepLea ning........................... 28
2.2.1 Objec Recogni ion . . . . . . . . . . . . . . . . . . . . 28
3 Me hodology 31
3.1 P oblem S a emen . . . . . . . . . . . . . . . . . . . . . . . . 31
3.1.1 Pe o mance C i e ia . . . . . . . . . . . . . . . . . . . 31
3.1.2 P o ided Da a . . . . . . . . . . . . . . . . . . . . . . . 32
3.2 Da aAnalysis........................... 33
3.2.1 Da a S uc u e . . . . . . . . . . . . . . . . . . . . . . 33
3.2.2 Da a Visualiza ion . . . . . . . . . . . . . . . . . . . . 35
3.3 Da aP epa a ion......................... 36
3.3.1 Da a P ep ocessing . . . . . . . . . . . . . . . . . . . . 37
3.3.2 Da a Segmen a ion . . . . . . . . . . . . . . . . . . . . 41
3.4 Building he Model . . . . . . . . . . . . . . . . . . . . . . . . 42
3.4.1 F amewo k ........................ 43
3.4.2 Model A chi ec u e . . . . . . . . . . . . . . . . . . . . 43
3.4.3 Model T aining . . . . . . . . . . . . . . . . . . . . . . 46
6
4 Resul s 51
4.1 E alua ion Me ics . . . . . . . . . . . . . . . . . . . . . . . . 51
4.2 P edic ions ............................ 52
5 Conclusions 57
777
Lis o Figu es
2.1 Single laye Neu al Ne wo k, also called pe cep on. I ecei es
a se o inpu s (3 in his case), which a e mul iplied by he
weigh s and hen adds a bias. The simples ac i a ion unc ion
is he s ep unc ion (see Eq. 2.1). . . . . . . . . . . . . . . . . 16
2.2 Sigmoid Func ion. The slope in he sigmoid unc ion end o
sa u a e ela i ely low inpu s, bu is a so e sion o he anh. 18
2.3 Tanh Func ion. No e he di e en slope in ela ion o Sigmoid
(seeFig.2.2)............................. 19
2.4 Fo simplici y, we can image he e o unc ion as a hill in
which we a e ying o igu e ou he coo dina es (i.e. weigh s)
o i s lowes place. F om Adi Deshpande. . . . . . . . . . . . 21
2.5 3-laye ne wo k wi h 4 neu ons in each hidden laye and 1
neu on in he ou pu laye . No e how he inpu laye is no
coun ed, and he neu ons belonging o he same laye a e no
connec ed.............................. 22
2.6 One o he i s Con olu ional Neu al Ne wo ks, by Peemen
e al., 2011. This a chi ec u e was speci ically used o digi
ecogni ion asks.......................... 23
2.7 An example inpu olume in ed (e.g. a 32x32x3 CIFAR-10 image),
and a example il e in blue (con olu ional o dep h = 5). F om:
CS231-n .............................. 24
2.8 The combina ion o wo 3x3 il e s a anged consecu i ely h ough
laye s gi es a ecep i e ield o a single 5x5 il e in a laye .
F om:gi books .......................... 24
2.9 The pooling laye akes, in his case, a pa ch o size 2x2 om
he inpu and downsamples i h ough he inpu wi h s ide
2, aking he max ou o he 4 elemen s. F om: CS231-n . . . 26
2.10 Regula neu al ne wo k wi h a single ully connec ed laye . . . 27
2.11 R-CNN a chi ec u e by Ross Gi shick e al. which in ol es
localiza ion + classi ica ion. . . . . . . . . . . . . . . . . . . . 29
2.12 Main scheme o a Gene a i e Ad e sa ial Ne wo k. . . . . . . 30
8
Chap e 2
S a e o he a
In o de o achie e he co ec unde s anding o he di e en ields ha had
o be add essed in his p ojec , a deep esea ch has been done h oughou
he du a ion o i . This chap e gi es a b ie in oduc ion o he backg ound
needed o de elop his hesis.
The sec ion i s ly add esses he de ini ion o Neu al Ne wo k, i s a chi ec u e
as i s di e en pa s. Followed by some o he common ypes o Neu al Ne -
wo ks, his sec ion la e ocuses on Con olu ional Neu al Ne wo ks (CNN’s)
by de ining i s a ious ypes o laye s
The second sec ion desc ibes wo common ne wo ks used in objec ecogni-
ion asks, which will gi e a b oade pe spec i e o he eade in how his
p oblems a e commonly ea ed. By he end o his chap e , he necessa y
backg ound o his hesis will ha e been in oduced o he eade .
Neu al Ne wo ks
The main concep o Neu al Ne wo k can be da ed o 1959, when a biological
model, p oposed by Nobel lau ea es Hubel and Wiesel[22], was based on hei
disco e y o wo ypes o cells in he p ima y isual co ex o a ca : simple
cells and complex cells. Di e en pa e ns o ligh s imuli we e ied and
i was obse ed ha a kind o pa e n ha may be good o some o he
neu ons may be no longe e ec i e o he o he s. Successi e expe imen s
ga e an unde s anding o which pa s o ha co ex we e s imula ed h ough
di e en pa e ns, hence like ” il e s” ha beha e di e en ly o he same
da a p esen ed.
15

Pe cep on
The i s unc ional ne wo ks wi h many laye s we e published by I akhnenko
and Lapa in 1965[1], becoming he G oup Me hod o Da a Handling (GMDH):
a amily o induc i e algo i hms o ma hema ical modeling o mul i-pa ame ic
da ase s ha pe o ms a pa ame ic op imiza ion o models. Ne e heless,
i was no un il 1975 ha Neu al Ne wo ks egained in e es , when Paul
We bos in oduced his pape [23] and p esen ed wha we may know now as
backp opaga ion, which will be desc ibed la e (see Sec ion 2.1.3).
x2w2Σ
Ac i a ion
Func ion
y
Ou pu
x1w1
x3w3
Weigh s
Bias
b
Inpu s
Figu e 2.1: Single laye Neu al Ne wo k, also called pe cep on. I ecei es
a se o inpu s (3 in his case), which a e mul iplied by he weigh s and hen
adds a bias. The simples ac i a ion unc ion is he s ep unc ion (see Eq.
2.1).
The mos basic neu al ne wo k, which all a ia ions de i e om, is a single-
laye pe cep on, which consis s o a single laye o ou pu nodes; he inpu s
a e ed di ec ly o he ou pu s ia a se ies o weigh s. In his way i can be
conside ed he simples kind o eed- o wa d ne wo k (see Fig. 2.1).
In common Neu al Ne wo k implemen a ions, he signal a a connec ion be-
ween a i icial neu ons is a eal numbe , and he ou pu o each a i icial
neu on is compu ed by a non-linea unc ion (see subsec ion2.1.2) o he sum
o i s inpu s. Conside ing he s ep unc ion as ac i a ion unc ion, he ou pu
o a pe cep on o an inpu ec o xis:
(x) = (1 i w·x+b > 0
0 o he wise (2.1)
whe e wT·x=Pm
i=1 wixi, m is he numbe o inpu s o he pe cep on, and
b is he bias gi en o i . In his example, he ac i a ion unc ion is he s ep
161616
unc ion.
A i icial neu ons and connec ions ypically ha e a weigh ha adjus s as
lea ning p oceeds. The weigh inc eases o dec eases he s eng h o he sig-
nal a a connec ion. These neu ons may ha e a h eshold such ha only i
he signal i sel c osses ha h eshold, he signal is sen . Typically, hese
neu ons a e o ganized in laye s. Di e en laye s may pe o m di e en kinds
o ans o ma ions on hei inpu s.
Signals a el acc oss all he laye s, om he i s (inpu ) o he las (ou pu )
laye . I he signal c osses each laye once (i.e. he e a e no cycles o loops),
we speak o Feed-Fo wa d Neu al Ne wo ks. I on he o he hand, in o ma-
ion low bo h ways we speak o Recu en Neu al Ne wo ks. This hesis will
ocus on eed- o wa d ne wo ks.
Ac i a ion Func ions
When we do no ha e an ac i a ion unc ion he weigh s and bias would
simply do a linea ans o ma ion. A linea equa ion is simple o sol e bu is
limi ed in i s capaci y o sol e complex p oblems. A neu al ne wo k wi hou
a non-linea ac i a ion unc ion is essen ially jus a linea eg ession model.
I we s ack nlaye s and each o hem applies a linea ans o ma ion o i s
inpu , a he n h laye we will s ill ha e a linea unc ion. Compac ing all
he linea ans o ma ions applied by all di e en laye s (suming all he do -
p odu s o he laye s and mul iplying by each o hei weigh s, ecu si ely)
we can squash his ne wo k in o a single-laye one.
The main eason behind using ac i a ion unc ions is o in oduce a non-
linea i y in he model, ha is able o simula e i ing a es, o p obabili ies
o being exci ed gi en ce ain inpu o no . These unc ions map he ou pu
o a neu on o some hing ha is bounded (e.g. be ween 0 and 1). Some o
he mos common ac i a ion unc ions a e desc ibed below.
Sigmoid
The sigmoid non-linea i y akes a eal- alued numbe and educes i in o he
ange [0,1]. Essen ially, la ge nega i e numbe s become 0 and la ge posi i e
numbe s become 1. I can be exp essed as:
σ(x) = 1
1 + e−x(2.2)
171717
This ans o ma ion has been used his o ically since i has a nice in e p e a-
ion as he i ing a e o a neu on: om no i ing a all, o ully-sa u a ed
i ing a an assumed maximum equency, 0 and 1 espec i ely. Howe e , i
has wo d awbacks.
Figu e 2.2: Sigmoid Func ion. The slope in he sigmoid unc-
ion end o sa u a e ela i ely low inpu s, bu is a so e sion
o he anh.
As de ailed be o e, sigmoids end o ou pu numbe s nea o 0 o 1, sa u a -
ing and kill g adien s. A e y undesi able p ope y o he sigmoid neu on is
ha when he neu on’s ac i a ion sa u a es a ei he ail o 0 o 1, he g a-
dien a hese egions is almos ze o. Du ing backp opaga ion (see Sec ion
2.1.3), his (local) g adien will be mul iplied o he g adien o his ga e’s
ou pu . The e o e, i he local g adien is e y small, i will e ec i ely “kill”
he g adien and almos no signal will low h ough he neu on o i s weigh s
and so o i s da a. On he o he hand, i he ini ial weigh s a e oo la ge hen
mos neu ons would become sa u a ed and he ne wo k will ba ely lea n.
Following he i s disad an age, sigmoid ou pu s a e no ze o-cen e ed. This
has implica ions on he dynamics du ing g adien descen , because i he da a
coming in o a neu on is always posi i e (see Fig. 2.2), hen he g adien on
he weigh s will du ing backp opaga ion become ei he all be posi i e, o
all nega i e (depending on he g adien o he whole exp ession). This could
in oduce zig-zagging dynamics in he g adien upda es o he weigh s. How-
e e , i has less consequences compa ed o he sa u a ed ac i a ion p oblem
abo e.
181818
Hype bolic Tangen
The anh squashes a eal- alued numbe o he ange [−1,1]. Like he sig-
moid neu on, i s ac i a ions sa u a e, bu he sigmoid neu on’s ou pu is
ze o-cen e ed. The e o e, in p ac ice he anh non-linea i y is always p e-
e ed o he sigmoid (see 2.2) nonlinea i y.
Figu e 2.3: Tanh Func ion. No e he di e en slope in ela ion
o Sigmoid (see Fig.2.2).
In his pape [2], hey mainly p esen di e en unc ions wi h sa u a ed be-
ha iou s and ”penalize” he nega i e ou pu s wi h a cons an (i.e. α∈[0,1]),
o ha e a educed impac in ha ou pu ). A e-scaled Sigmoid ac i a ion is
p oposed in he pape o make deep Sigmoid ne wo k ainable.
I also has o be no ed ha hese desc ibed unc ions ha e a simple de i a i e,
hus gi ing good compu a ional e iciency when backp opaga ing h ough he
g adien s o each laye .
Backp opaga ion
A se o ques ions a ise like how do how do he il e s in each laye know wha
alues o ha e? O how does he ully connec ed laye know wha ac i a ion
maps o look a ?
The e is a e m o e e o he lea ning p ocess, i is called backp opaga ion.
Backp opaga ion can be sepa a ed in o 3 dis inc sec ions: he loss unc ion,
he backwa d pass, and he weigh upda e.
•Loss Func ion: When he da a has gone h ough all he ne wo k and
so inishes he o wa d pass, since all o he weigh s o il e alues
191919
a e andomly ini ialized, he ou pu does no gi e p e e ence o any
class/ca ego y in pa icula . The ne wo k, wi h i s cu en weigh s, is
no able o make any easonable conclusion abou wha he classi ica-
ion migh be.
To cope wi h his, a loss unc ion is compu ed wi h he ou pu s o he
ne wo k and he labels o he da a. Loss unc ions a e used o ep esen
he p ice paid o inaccu acy o p edic ions in classi ica ion p oblems.
Typical loss unc ions ha a e used: Mean Squa ed E o (MSE) and
C oss-En opy.
1
N
N
X
i=1 kˆyi−yik2
L(w) = 1
N
N
X
n=1
[ynlog ˆy+ (1 −yn) log(1 −ˆyn)]
(2.3)
whe e ynis he ue label and ˆynis he p edic ed p obabili y.
Ac ually, we wan o ge o a poin whe e he p edic ed label is he
same as he aining label ( his means ha ou ne wo k go i s p edic-
ion igh ). In o de o ge he e, we wan o minimize he amoun o
loss we ha e. Tha is done by inding ou which weigh s mos di ec ly
con ibu e o he loss (o e o ) o he ne wo k.
In his pape [15], hey analyze a wide ange o losses ( om he p e-
iously desc ibed (2.1.3) o some ancy unc ion losses like Tanimo o,
Chebyshe o Cauchy-Schwa z Di e gence)
•Backwa d Pass: The nex s ep is o de e mine which weigh s con-
ibu e mos o he loss and ind ways o adjus hem so ha he loss
dec eases. This is he ma hema ical equi alen o a dL/dW whe e W
a e he weigh s a a pa icula laye , and Lis he unc ion chosen o
he p oblem.
•Weigh Upda e: Once he de i a e is compu ed ( he g adien o he
loss depending on he weigh s gi en), i is he u n o make he weigh s
know how hey a e con ibu ing o he loss. Ma hema ically, his is
done by upda e hem so ha hey change in he opposi e di ec ion o
he g adien . The simples o m o upda e is o change he pa ame e s
along he nega i e g adien di ec ion (since he g adien indica es he
di ec ion o inc ease, bu we usually wish o minimize a loss unc ion).
Wn+1 =Wn−µ∇W(L) (2.4)
202020

Figu e 2.4: Fo simplici y, we can image he e o unc ion
as a hill in which we a e ying o igu e ou he coo dina es
(i.e. weigh s) o i s lowes place. F om Adi Deshpande.
whe e ∇(L) is he g adien o he loss, L, in espec wi h he weigh s
W, and µis he lea ning a e.
The lea ning a e is a pa ame e ha is chosen by he p og amme .
A high lea ning a e means ha bigge s eps a e aken in he weigh
upda es and hus, i may ake less ime o he model o con e ge o
an op imal se o weigh s. Howe e , a lea ning a e ha is oo high
esul in jumps ha a e oo la ge and no p ecise enough o each he
op imal poin (i keeps oscilla ing a ound i ).
The p og am will epea his p ocess o a ixed numbe o i e a ions o each
se o aining examples, i.e. examples o which he class hey belong o
is known, commonly called mini-ba ch g adien descend, o o each image,
s ochas ic g adien descen .
Con olu ional Neu al Ne wo ks
Unlike a egula Neu al Ne wo k, he laye s o a Con olu ional Neu al Ne -
wo k ha e neu ons a anged in 3 dimensions: wid h, heigh , dep h. CNNs
assume ha he inpu ec o is an image, op imizing he a chi ec u e p o i -
ing om he ansla ional in a iance ea u e o images. The wo d dep h he e
e e s o he hi d dimension o an ac i a ion olume, no o he dep h o a
ull Neu al Ne wo k, which e e s o he o al numbe o laye s in a ne wo k
(see Fig. 2.5).
212121
Regula single laye , ully-connec ed Neu al Ne wo ks do no scale well
o ull images. In CIFAR-10, which is one o he mos popula da ase s (see
CIFAR-10 ), images a e only o size 32x32x3 (32 wide, 32 heigh , 3 colo
channels), so a single ully-connec ed neu on in a i s hidden laye o a
egula Neu al Ne wo k would ha e 32x32x3 = 3072 weigh s. This amoun
s ill seems manageable, bu clea ly his ully-connec ed s uc u e does no
scale o la ge images.
Fo example, an image o a bigge size, e.g 200x200x3, would lead o neu ons
(see Fig. 2.1) ha ha e 200x200x3 = 120,000 weigh s. Con olu ional Neu al
Figu e 2.5: 3-laye ne wo k wi h 4 neu ons in each hidden
laye and 1 neu on in he ou pu laye . No e how he inpu
laye is no coun ed, and he neu ons belonging o he same
laye a e no connec ed.
Ne wo ks p o i om he p io knowledge o he inpu being an image. Due
o he ansla ionally-in a ian s uc u e o he image, i he neu ons a e
de ec ing a ho izon al edge o some kind o blob-like pa e n a some loca ion
in he image, i should in ui i ely be use ul a some o he loca ion as well.
This is he eason behind using small il e s1: i does no ma e whe e he
a ge is bu he co ec unde s anding o i , which is done in he ollowing
laye s by ex ac ing high-le el ea u es o he da a.
1Con olu ion is a neighbo hood ope a ion in which each ou pu pixel is he weigh ed
sum o neighbo ing inpu pixels. The ma ix o weigh s is called he con olu ion ke nel,
o also known as il e .
222222
Figu e 2.6: One o he i s Con olu ional Neu al Ne wo ks,
by Peemen e al., 2011. This a chi ec u e was speci ically
used o digi ecogni ion asks.
Laye Types in CNNs
Con olu ional Neu al Ne wo ks a e buil by s acking laye s on op o each
o he . These laye s pe o m di e en ac ions based on hei inpu s and he
mos used ones a e Con olu ional, Pooling and Fully Connec ed Laye s.
Con olu ional Laye s I s pa ame e s consis o a se o lea nable il e s.
E e y il e is small spa ially, along wid h and heigh ( ypical alues a e 3x3,
5x5 and a ely 7x7), bu ex ends h ough he ull dep h o he inpu olume.
Du ing he o wa d pass, we slide o mo e p ecisely con ol e each il e ac oss
he wid h and heigh o he inpu olume and compu e do p oduc s be-
ween he en ies o he il e and he inpu a any posi ion. As we slide he
il e o e he wid h and heigh o he inpu olume we will p oduce a 2-
dimensional ac i a ion map ha gi es he esponses o ha il e a di e en
egions o he image.
I we ex end he numbe o il e s o a dep h d, we will ha e an en i e se
o weigh s and each o hem will p oduce a sepa a e 2-dimensional ac i a-
ion map. A he end o he con olu ional laye , dac i a ion maps will be
s acked along he dep h dimension and p oduce he ou pu olume. In Fig.
2.6, numbe s C1and C2co espond o he numbe o ac i a ion maps in each
con olu ional laye , espec i ely.
232323
Figu e 2.7: An example inpu olume in ed (e.g. a 32x32x3
CIFAR-10 image), and a example il e in blue (con olu ional o
dep h = 5). F om: CS231-n
As an example, no e ha in Fig. 2.7 he e a e mul iple neu ons (5 in
his example) along he dep h, all looking a he same egion in he inpu .
Each neu on in he con olu ional laye is connec ed only o ha egion o
he inpu space bu o he ull dep h, in his case 3, all colo channels.
The ne wo k will lea n he weigh s o he di e en il e s ha ope a e on he
image h ough he con olu ion ope a ion. The adap a ion o ha weigh s o
some ype o isual ea u e such as an edge o some o ien a ion o pa e n is
achie ed by backp opaga ion (see Sec ion 2.1.3).
Mos o he bes - a ed me hods and algo i hms in Con Ne s[3][20], play wi h
he ecep i e ields o neu ons: i is p e e ed o s ack small il e s on op o
each o he han ha ing a la ge ecep i e ield on a single il e . This pape [11],
among he o he ones, discuss he ad an ages and disad an ages o he main
il e sizes and i ’s di e en possible conca ena ions.
Figu e 2.8: The combina ion o wo 3x3 il e s a anged con-
secu i ely h ough laye s gi es a ecep i e ield o a single 5x5
il e in a laye . F om: gi books
242424
Chap e 3
Me hodology
This chap e includes all ele an in o ma ion abou he da a p ep ocessing
s age, so wa e used and machine lea ning echniques applied in o de o
ain he inal ne wo k ha will de ec h ea s om millime e -wa e images.
P oblem S a emen
The main objec i e o his p ojec is o build a model using deep lea ning
o be able o disce n be ween subjec s ca ying h ea s o no . The model
has been p esen ed o a compe i ion hos ed by Kaggle, a web pla o m ha
o ganizes da a science compe i ions o use s om all o e he wo ld.
Pe o mance C i e ia
The p oblem p oposed is o p edic he p obabili y ha a gi en body zone,
ou o 17 o al body zones, has a h ea p esen . The main e alua ion me hod
used o classi y he models o he compe i o s is he c oss en opy loss, in
his case, a binomial c oss en opy loss, which can be w i en as:
L(w) = 1
N
N
X
n=1
[ynlog ˆy+ (1 −yn) log(1 −ˆyn)] (3.1)
whe e ˆynis he p edic ed p obabili y o he scan ha ing a h ea in he gi en
body zone; Nis 17(numbe o h ea zones) ×numbe o scans (subjec s
wi h unique scan id) in he es se ; ynis 1 i a h ea is p esen , 0 o he wise;
log is he na u al (base e) loga i hm.
31

The wo las pe o mance measu es p o ide us a deepe pe spec i e o how
he machine is pe o ming, alongside o he co ec o inco ec p edic ions
(see Eq. (4.1)). The idea behind op imizing ha ype o loss is because i
measu es he dissimila i y be ween wo ec o s, in his case he p edic ed
ou pu s and he labels. I we minimize his loss, we a e making hese wo
p e iously men ioned ec o s mo e simila in he da a space, con ibu ing o
he lea ning o i .
The model has o exploi he di e ence be ween he e lec i i y o he body,
mainly composed by wa e , and he e lec i i y o h ea s, ha come in a
wide ange o shapes and ma e ials.
The ne wo k will be ained using a i ual p i a e ne wo k connec ing o he
compu a ion’s emo e se e CALCULA, om Teo ia del Senyal i Comuni-
cacions (TSC), which in u n belongs o UPC. I s ou pu s will be e alua ed
conside ing, apa om he loss which is he main me ic o ake in o accoun ,
some o he e alua ion me ics such as he accu acy, speci i y and sensi i i y.
P o ided Da a
The da ase was p o ided by he T anspo a ion Secu i y Adminis a ion
(TSA) o Kaggle and la e on uploaded o he compe i o s. The da ase
con ains a la ge numbe o body scans acqui ed by a new gene a ion o
millime e -wa e scanne called he High De ini ion-Ad anced Imaging Tech-
nology (HD-AIT) sys em.
The images in he da ase we e designed o cap u e eal scanning condi ions.
They we e comp ised o olun ee s wea ing di e en clo hing ypes ( om
ligh summe clo hes o hea y win e clo hes), wi h di e en body mass in-
dices, di e en gende s, di e en numbe s o h ea s, and di e en ypes o
h ea s. Due o es ic ions on e ealing he ypes o h ea s o which he
TSA sc eens, he h ea s in he images we e ”ine ” objec s wi h a ying
ma e ial p ope ies.
In he i s s age o he con es , he compe i o s we e asked o use he gi en
da ase and y o de ec h ea s on hem. The p oposed model o algo i hm
had o gene alize well o o he ypes o subjec s (mainly om s age 2). The
second s age consis ed o 1388 subjec s who had much mo e di e ences in
e ms o body shapes and heigh , oge he wi h ha ing unlabeled da a. In
his hesis, his da a was used as es da a o he s age 1 models. The main
wo k was done on he s age 1 da ase , hough.
323232
Da a Analysis
Ha ing in oduced he p oblem and he da ase gi en o sol e i , he nex
s ep gi es a b ew explana ion o he s uc u e o he da a oge he wi h a
isualiza ion o he di e en angle iews.
Da a S uc u e
The da a o each scan pe o med by he HD-AIT sys em is e e ed o as an
HD-AIT F ame. A ame consis s o he ollowing ou bina y iles:
•.ahi = calib a ed objec aw da a ile (2.26GB pe ile)
•.aps = p ojec ed image angle sequence ile (10.3MB pe ile)
•.a3d = combined image 3D ile (330MB pe ile)
•.a3daps = combined image angle sequence ile (41.2MB pe ile)
The ou iles gene a ed by he HD-AIT p og am ha e a common ile s uc-
u e. All ou iles a e bina y and include a 512 by e heade ollowed by
he ile’s da a. The heade mos ly con ains echnical scan pa ame e s like
he equency wi h which his image has been cap u ed, dimensions (x, y, z)
inc emen be ween ames, ime, e c... and is la gely iden ical ac oss all im-
ages. Wi h he excep ion o he ield da a scale ac o , ha gi es us he scale
o he da a comp ession in o de o i a ce ain numbe o ma (in his case
uin 16), he o he ields a e no used in his hesis.
The subjec s we e exposed o a scan ha made a ull o a ion and ook snap-
sho s e e y 22’5o, esul ing on 16 ames which depic ed he scene scanned.
Fo a subjec , he e a e 16 iews o 620x512 pixels so basically he enso s
o he da ase a e o shape: 16x620x512.
The i s s age o he compe i ion consis ed in a se o labelled subjec s, con-
c e ely, 1147, wi h wha i was made he aining, alida ion and es se . In
Fig. 3.1 he zone segmen a ion p oposed in he compe i ion is shown.
333333
The compe i ion p o ided labels o he o al 1147 subjec s, which made
he p oblem all in o he ca ego y o supe ised lea ning1. By using he labels
o each o he subjec , I could check wha body zones we e mo e p one o
ha e a h ea . In o he wo ds, he numbe o subjec s ca ying a h ea we e
coun ed o e e y body zone and hen, di ided by he numbe o subjec s,
gi ing he co esponding pa s pe uni o each o he zones. The esul s a e
summa ized in he nex able 3.1, we e he co esponding p.p.u. (pa s pe
uni ) is o de ed in descendan mode.
Figu e 3.1: P oposed h ea zones by he TSA.
I also has o be no ed ha , due o he a ying physical condi ions o he
subjec s, h ea s in conc e e body pa s we e disguised be e on di e en
subjec s han o he s, which in u n made he di icul y o h ea de ec ion
ise in ha egion. The desc ip ion o each body zone can be seen in Table
3.1.
1Supe ised Lea ning is he ield o Machine Lea ning whe e he da ase gi en o sol e
a p oblem is labeled ( he algo i hm can es ima e a ype o loss o e o unc ion based on
he knowledge o he inpu ’s class/ca ego y)
343434
Body Zone Body Desc ip ion # Th ea s Pe cen age
1 Righ Bicep 133 0.115955
2 Righ Fo ea m 126 0.109852
8 Uppe Righ Hip/Thigh 124 0.108108
14 Le Cal 122 0.106364
15 Righ Ankle Bone 118 0.102877
11 Lowe Righ Thigh 116 0.101133
6 Righ Rib Cage/Abs 116 0.101133
13 Righ Cal 110 0.095902
16 Le Ankle Bone 109 0.095031
4 Le Fo ea m 108 0.094159
5 Uppe Ches 106 0.092415
3 Le Bicep 104 0.090671
12 Lowe Le Thigh 101 0.088056
10 Uppe Le Hip/Thigh 100 0.087184
17 Uppe Back 95 0.082825
7 Le Rib Cage/Abs 93 0.081081
9 G oin (Sensi i e A ea) 90 0.078466
Table 3.1: Summa y o he numbe o h ea s depending on
body zones plus he desc ip ion o each body pa .
Da a Visualiza ion
Be o e add esing he p ep ocessing and ne wo k implemen a ion pa , a i-
sual analysis on he da a has been made in o de o p oceed co ec ly owa ds
a a ional solu ion.
As i will be seen, he e a e some pa s o he body ha a e no be isible in
ce ain scans, bu he i s p ocedu e in his hesis has been o ake all he
da a in o accoun because, i i ’s no ele an , he algo i hm will disca d i
a any ime (won’ ac i a e neu ons o ha image, because i will no see
any h ea ).
Lo s o di ec human isualiza ion o he di e en subjec s was done in o de
o look o di e en app oaches. The ollowing pic u e (see Fig. 3.2) shows
he summa y o 16 iews o a single subjec . No e ha his images we e
used be o e p ep ocessing, wi h he pu pose o es ing ou isual pe cep ion
o he h ea s.
353535
Figu e 3.2: 16 slices o a ull 360 deg ee image o a subjec . I
looked closely, his example shows a h ea o e he le knee
( h ea zone 12).
Da a P epa a ion
Following he inspec ion o he da a by isualising and manipula ing i , i
is needed o p epa e he da a o he inpu pipeline o he models. The
main a chi ec u e o his algo i hm consis s in a p ep ocessing module, ha
segmen a es he image in o 17 di e en body zones and no malizes he da a,
and a ne wo k module wi h 17 independen models, i.e. each body zone is
ea ed sepa a ely.
A e his sec ion, he images will be eady o be ed o each o he di e en
se en een models (i.e. one o each body zone). The gi en images we e o
shape 16x620x512 (wid h and heig h espec i ely) and had o be educed o
pa ches o size 16x224x224 in o de o apply o he s anda d Con Ne inpu
sizes.
363636

Da a P ep ocessing
When wo king wi h CNNs we no mally use aw images as inpu da a, bu
some p ep ocessing mus be done be o e. CNNs lea n by con inually adding
o he weigh s, g adien e o ec o s (mul iplied by a lea ning a e) ob ained
om a backp opaga ion h ough many ma ices ba ch by ba ch.
I is in ou in e es o ensu e ha e e y image has a simila ange o alues in
o de o a oid ha he g adien s un ou o con ol, and he way o do i is by
sub ac ing he mean o he whole image da ase o each sample. This way,
he g adien s ac uni o mly o each channel. O he wise, he lea ning a e
would cause co ec ions in each dimension ha would di e , and compensa -
ing a co ec ion in one weigh dimension migh imply unde compensa ing in
ano he , p o oking di icul ies o he loss o s abilize.
Mos o he pixels om he aw image we e in he ange [0, 20] (da kes
pixels). No e how much noise is p oduced by his ype o scans.
Figu e 3.3: An example aw da a image co esponding o he
i s iew/dep h slice, whe e he subjec is acing on wa ds.
As i may be no iced looking a he his og am, he a iance o he pixels is
no ha high, since almos 3/4 o he o al pixels a e in he ange [0, 20]
(da kes pixels). This ac ually is undesi able because we wan he maximum
di e en in ensi ies (no compac ed in o a egion) while p ese ing he isual
in o ma ion in ac .
373737
Figu e 3.4: His og am o he aw image.
The echnique applied o ampli y he con as in an image is called his-
og am2equaliza ion. The equalized image is shown below, oge he wi h i s
his og am.
Figu e 3.5: Image a e equalizing. No e ha mos o he
noise has been emo ed.
2An image his og am is a ype o his og am ha ac s as a g aphical ep esen a ion o
he onal dis ibu ion in a digi al image. I gi es he numbe o pixels wi h a conc e e
onal alue.
383838
Equaliza ion was pe o med using a CLAHE (Con as Limi ed Adap i e
His og am Equaliza ion) echnique (wi h a c 2 unc ion, see 3.4.1 o ame-
wo k in o ma ion). This echnique consis s in adap ing he his og am equal-
iza ion o small egions o he image, o p ese e hem o o e -b igh en/o e -
da ken because o conside ing global con as ins ead. Then each o hese
blocks a e his og am equalized as usual. A e equaliza ion, o emo e a i-
ac s in ile bo de s, bilinea in e pola ion is applied. A ile size o 8x8 was
used because, expe imen ally, ga e he bes esul s.
Figu e 3.6: Co esponding his og am a e CLAHE applied.
No ice ha he ange o he alues has no inc eased, bu he image has
signi an ly educed i s noise plus he h ea (body zone 14, le cal ) is be e
de ined.
Following he equaliza ion, a da a cen e ing, also called mean sub ac ion
was done. When he da a consis s o images, he common p ocedu e in-
ol es sub ac ing he mean o he image o all he pixels.
Basically, we can s a de ining he mean o a ec o xo n alues (e.g. a
pixel column/ ow o an image):
E{x}=µ=1
n
n
X
i=1
xi(3.2)
F om 3.2, i we ex end i o a wo-dimensional case (e.g. he ull image):
E{x}=µ=1
nm
m
X
j=1
n
X
i=1
xij (3.3)
393939
Now i we compu e he mean o each iew, sepa a edly, as 3.3, we’ll end
up wi h µ0, µ1, ..., µ15 means o each iew. Now he mean o each iew (o
channel, as i was explained be o e) is subs ac ed om he co esponding
iew i sel . I we de ine a single iew/ ame/slice as Xc, whe e c ∈1, . . . , C
and Cis he o al numbe o channels in his case 16:
∀xij ∈Xc, µc=1
nm
m
X
j=1
n
X
i=1
xij,
Xc=Xc−µcI
(3.4)
The eason behind subs ac ing he mean is such as cen e ing he cloud o
da a a ound he o igin along e e y dimension. In deep lea ning, i is used o
a oid anishing/exploding g adien s due o a bias in he aw da ase .
Mo eo e , a e ze o cen e ing, he esponse om a andomly ini ialized il e
(con olu ional laye ) is s ill close o a ze o-cen e ed dis ibu ion. This would
loca e he ne wo k’s ini ial esponses nea he 0 o he nonlinea i y ( elu,
sigmoid, leaky- elu, e c.). A his poin small changes in he il e esponse
ha e a la ge e ec on he nonlinea i y esponse (changing om 0 o posi i e
in he elu, and quickly changing signs in he sigmoid o leaky- elu), hus
yielding la ge g adien s e en wi h small changes in he il e esponse.
Ano he p ep ocessing echnique applied o he da ase is no malizing he
da a dimensions so ha hey a e o app oxima ely he same scale. The com-
mon p ocedu e is o di ide each dimension by i s s anda d de ia ion, once i
has been ze o-cen e ed.
In case o images i consis s in educing he ange om [0,255] o [0,1], so
we p ese e he ela ion be ween pixels bu we ha e his pixels cons ained
o 0 and 1 (0 ≤I≤1). As in images pixels a e commonly wide-dis ibu ed
h ough he di e en le els (i no , as i was ou case, we apply his og am
equaliza ion) i is no ha impo an .
Howe e , in classi ie s ha calcula e he dis ance be ween wo poin s by he
Euclidean dis ance would ha e a g ea p oblem in sol ing an op imiza ion
p oblem, since i he ea u es a e non-scaled, he mos a ian ea u es will
ha e dominion o e he o he s when op imizing ( hey will gi e la ge alues
in dis ance). Thus, no malizing is a way o equally dis ibu e he p opo ion
o con ibu ion o each ea u e o he inal dis ance.
404040
Figu e 3.10: A no mal dis ibu ion wi h µ= 0 and σ= 0.1
•Xa ie [9] No mal Ini ialize : This ini ialize is designed o keep he
scale o he g adien s oughly he same in all laye s. I simply is a
no mal dis ibu ion wi h mean 0 and s anda d de ia ion σwhe e
V a (w) = σ2=2
nin +nou
(3.6)
and nin, nou a e he numbe o inpu neu ons and ou pu neu ons o
he laye , espec i ely.
A simple explana ion is ha i ini ializes he weigh s in a way ha he
signal does no ei he sh ink o inc ease when i passes h ough all he
laye s, and he e o e a he end o he ne wo k we do no ha e useless
neu ons by anished/exploded g adien . Howe e , in his pape [12],
discuss ha o ReLU neu ons, Xa ie ini ializa ion migh no wo k
ha well. This has o do wi h an app oxima ion aken in Xa ie , which
is ha he inpu is ze o-cen e ed and conc e ely in ReLU neu ons he
mean is g ea e han 0, µ > 0.
•He[12] Ini ializa ion: This is a modi ica ion on o Xa ie ini ialize ,
which akes in o accoun bo h inpu and ou pu uni s by aking he
ha monic mean o hem. In he pape om Kaiming He e al., hey
demons a e ha o deep ne wo ks, in his pape 22 and 30 laye s,
Xa ie ini ialize akes longe o educe he e o in he 22-laye ne -
wo k, and in he 30-laye ne wo k i comple ely s alls, which is e i ied
by hei g adien s, ha ge diminished. The cons ain added o he
weigh s ini ializa ion is
V a (w) = 2
nin
(3.7)
474747

Op imize s
When backp opaga ing h ough he ne wo k, he algo i hm upda es he
weigh s based on a hype pa ame e called lea ning a e. This lea ning a e
speci y how much o he g adien o he loss ∇L(wn) (which is unc ion o he
weigh s assigned in he p e ious s ep) is subs ac ed o ha weigh . Then
a new weigh wn+1 =wn−µ∇L(wn) is assigned. This is called s ochas ic
g adien descend.
Many imp o emen s on he basic s ochas ic g adien descen algo i hm ha e
been p oposed and used. The need o se an op imized lea ning a e was
c ucial, since se ing his pa ame e oo high can cause he algo i hm o di-
e ge, when on he o he hand, se ing i oo low makes i slow o con e ge.
The ollowing summa y p o ides in o ma ion abou he di e en op imize s
used in he p oblem.
•Momen um: This op imize eme ged om a physical poin o iew.
I uses physical law o mo ion o go pass h ough local op ima (small
hills). In ui i ely, adding momen um will also make he con e gence
as e , as we’ e accumula ing speed, so he g adien s ep will be la ge .
The objec i e unc ions o deep a chi ec u es ha e complex o ms (wi h
local op ima) and hus s anda d SGD can lead o e y slow con e gence
pa icula ly a e he ini ial s eep gains.
n+1 =γ∗ n+µ∇L(wn),
wn+1 =wn+ n+1
(3.8)
The main di e ence om he g adien descend me hod shown abo e,
whe e he g adien di ec ly is in eg a ed wi h he posi ion (weigh s),
is ha he physics iew sugges s an upda e in which he g adien only
di ec ly in luences he eloci y, which in u n has an e ec on he posi-
ion. The upda e o he eloci y is gi en he old eloci y alue and new
G adien Descen s ep µ∇L(wn−1). We also decay ou pas eloci y so
ha we only conside he mos ecen eloci ies wi h γ= 0.9 which is
he mos comomn alue used.
•AdaG ad: Adap i e G adien [7] is ano he app oach o he G adien
Descen p oblem, bu om ano he poin o iew. The p oblem wi h
lea ning a e µin G adien Descen is ha i is cons an and a ec s
all he pa ame e s. To sol e his, he sum o squa ed o all o ou pa-
ame e s’ g adien , and use ha o no malize he lea ning a e µ. I
we de ine gn,i as he g adien o he objec i e unc ion wi h ela ion o
he pa ame e wia ime s ep nwe ha e:
484848
gn,i =∇L(wn,i),
wn+1,i =wn,i −µ
pGn,ii +gn,i (3.9)
whe e Gn∈ <dxd is he diagonal ma ix whe e each diagonal elemen
is he sum o he squa es o he g adien s w. . wn,i.
Now he lea ning a e applied o each pa ame e will be smalle o
la ge depending on how he pas g adien s beha ed: pa ame e s ha
go a big upda e will be slowed down while pa ame e s ha ecei ed
li le upda es will ha e bigge lea ning a e o accele a e he lea ning
p ocess. In [13], om Penning on e al., Adag ad was used o ain
GloVe wo d embeddings, as in equen wo ds equi e much la ge up-
da es han equen ones.
Howe e , Adag ad’s main weakness is i s accumula ion o he squa ed g adi-
en s in he denomina o : Since e e y added e m is posi i e, he accumula ed
sum keeps g owing du ing aining. This in u n causes he lea ning a e o
sh ink and e en ually become in ini esimally small, a which poin he algo-
i hm is no longe able o acqui e addi ional knowledge.
Regula iza ion
A cen al p oblem in machine lea ning is how o make an algo i hm ha will
pe o m well no jus on he aining da a, bu also on new inpu s. Many
s a egies used in machine lea ning a e explici ly designed o educe he es
e o , possibly a he expense o inc eased aining e o . These s a egies
a e known as egula iza ion. I is a way o ensu e ha he ne wo k does no
o e i o he aining images, bu a he makes obus connec ions be ween
neu ons, inducing he model o be mo e spa se in he laye ac i a ions.
D opou is an ex emely e ec i e, simple and ecen ly in oduced egula -
iza ion echnique by S i as a a e al. in [21] ha complemen s he o he
me hods (L1, L2, maxno m). In ou case, only d opou wi h a a e o 0.5
was used. The e is mainly one eason, apa om he di e en esea ch in
[8] whe e i is shown ha he op imum ange o he d opou a e o neu ons
is be ween [0.4, 0.6], is ha wi h a neu on ac i a ion p obabili y o 0.5, we
ge an equally p obable dis ibu ion o se s wi h di e en neu on ac i a ions
(i.e. maximum a iabili y be ween se s o neu ons). While aining, d opou
is implemen ed by only keeping a neu on ac i e wi h some p obabili y p(a
hype pa ame e ), o se ing i o ze o o he wise. When es ing, we emo e
494949
Figu e 3.11: D opou educes he numbe o connec ions in
he ne wo k, enhancing ea u e de ec ion by cu ing some o
he in o ma ion gi en o he nex laye , he e o e inducing he
model o de elop mo e obus connec ions be ween neu ons
he d opou laye s in o de o es he po en ial o all he connec ions c ea ed
while aining.
The di e en ne wo ks we e ained wi h a ba ch size o 128 and o 50
epochs. In he pic u es i will be seen ha he loss d ops a abou he hal o
epochs, hence, a checkpoin a gumen was passed o he unc ion used o i
he model, so he model was sa ed in case ha alida ion loss did no educe
o 5 epochs.
505050
Chap e 4
Resul s
In his sec ion, he esul s om he VGG-16 ained ne wo k a e shown.
Some o he di e en con igu a ions a e discussed along wi h he di e en
esul in e p e a ions. Much o he wo k was done in he weigh ini ial-
iza ion pa . Mos o he p oblems o aining his ne wo k was he high
numbe o pa ame e s o une: ha ing a bad weigh ini ializa ion in he i s
laye becomes a disas e wi h mo e and mo e laye s, because i keeps ge ing
mul iplied by a alue ha is no ” ele an ” o he ne wo k, hence explod-
ing/diminishing g adien s. A Ba ch No maliza ion laye is p oposed o cope
wi h his p oblem.
E alua ion Me ics
Fi s o all, we de ine he e alua ion me ics used o measu e he pe o mance
o ou deep lea ning model. The mos impo an one among Accu acy is
C oss-En opy Loss, ha is he los used o desc ibe how good/bad is ou
model app oxima ing he g ound u h o he da a. Also, he compu a ion
o he weigh /pa ame e upda e ha depends di ec ly o he g adien o his
loss, di e s acco ding he op imize used. O he me ics aken in o accoun
a e he speci i y and sensi i i y, which a e desc ibed ma hema ically as:
Sensi i i y =TP
TP +FN ,
Speci i y =TN
TN +FP
(4.1)
whe e TP, ue posi i e, could be de ined as he co ec hi /p edic ion ( h ea
p esence) o a subjec whe eas TN, ue nega i e, could be de ined as he
co ec ejec ion ( h ea absence) o a subjec .
51
A Con usion Ma ix will be used o ep esen his alues. I s name s ems
om he ac ha i makes i easy o see i he sys em is con using wo classes
(i.e. ou pu ing h ea p esence he e is no h ea and ice e sa). whe e we
T ue Label
Posi i e Nega i e
Ne wo k Ou pu Posi i e TP FP TP +FP
Nega i e FN TN FN +TN
TP +FN FP +TN N
Table 4.1: Con usion Ma ix o a p oblem wi h 2 classes.
can easily calcula e speci i y and sensi i i y. Wi h he i s column, we can
compu e he o me ; wi h he second column, we can compu e he la e .
The goal o he model is o educe he c oss-en opy be ween ue labels and
p edic ed ones. This loss inc eases as he p edic ed p obabili y di e ges om
he ac ual label. So p edic ing a p obabili y o .012 when he ac ual obse -
a ion label is 1 would be bad and esul in a high log loss. Ma hema ically,
i can be w i en as
L(w) = 1
N
N
X
n=1
[ynlog ˆy+ (1 −yn) log(1 −ˆyn)] (4.2)
P edic ions
As i is explained in Sec ion 3.1.2, aining wo k was done in he s age 1
da a. This da ase consis ed o 1147 labeled images, which we e ed o he
models a e he p ocedu es desc ibed in Da a P epa a ion (Sec ion 3.3).
This sec ion pu s up all he esul s gi en by he di e en ne wo k con igu-
a ions. D opou Laye s wi h p obabili y a e o 0.5 (see Fig. 3.11) we e
placed in he 2 ully connec ed laye s p e ious o he so max classi ie laye
(ou pu laye ). The ba ch size used in all he aining con igu a ions has
been changing be ween 64/128 images (16 iews implici ly), bu wi hou no-
able changes. I ha e added he me ics o he bes lea ning a e o each
op imize , only. Two o he bes pe o ming models we e he ones assigned
o h ea zone 9 (Sensi i e A ea) and h ea zone 16 (Le Ankle Bone). In
he ollowing pic u e, we can see he loss achie ed a s ep 284 o he body
zone 16.
525252

The i s esul s shown co espond o Momen um (see 3.8 op imized mod-
els, whe e he accu acy was inc edibly apid o each he maximum in each
o he body zones. They a e summa ized in he able below. The es ba ch
Zone Val. Accu acy X-En opy Loss
1 0.8678 0.3357
2 0.8952 0.2641
3 0.8849 0.2838
4 0.9237 0.2037
5 0.8975 0.2595
6 0.8634 0.2845
7 0.9158 0.2163
8 0.9265 0.1965
9 0.9079 0.2238
10 0.8942 0.2397
11 0.9037 0.2302
12 0.9182 0.2154
13 0.9323 0.1784
14 0.8854 0.2508
15 0.8713 0.2632
16 0.9345 0.1696
17 0.9215 0.1858
Table 4.2: Accu acy and C oss-En opy loss o e e y h ea
zone wi h he Momen um Op imize + lea ning a e = 10−4
was o shape 229x16x224x224 (20% o 1147). As i can be seen, he di e en
accu acies achie ed a e a ela i e good esul . The Momen um upda e p o-
ided he as es way o con e ge in o he local op ima, bu i seemed o s all
when eaching i . Con usion Ma ix om h ea zone 16, which is he bes
model in e ms o Speci i y and Sensi i i y, can be seen below. (see Table
??)..
T ue Label
Posi i e Nega i e
Ne wo k Ou pu Posi i e 12 9 21
Nega i e 6 202 208
18 211 229
Table 4.3: Con usion Ma ix o h ea zone 16 wi h Momen um Op imize
535353
Speci i y and Sensi i i y a e easily compu ed om he Con usion Ma ix. Fo
his h ea zone, Sensi i i y is compu ed as Se =12
18 = 0.66 and Speci i y =
202
211 = 0.9573.
In his p oblem, whe e he e was no a balanced da ase , ( 10% h ea p es-
ence) i was ha d o ge a high Sensi i i y (co ec h ea p esence ou pu ),
as he ne wo ks we e ained wi h much mo e ”non-ca ying h ea ” subjec s
da a han wi h images ha con ained a h ea in i . Howe e , I did no
wan o balance he da ase as he e al eady we e ew images. An image o
alida ion accu acy/loss on h ea zone 9 is gi en below.
Figu e 4.1: Valida ion Accu acy and Loss in body zone 9
wi h Momen um Op imize . No e ha Accu acy eaches he
maximum a ea lie epochs han he Loss.
Ano he op imize used was AdaG ad, which is a di e en me hod o upda e
he pa ame e s, adap ing he lea ning a e sepa a ely o e e y dimension
(see ??) acco ding o he loss p oduced by hem. The nex able summa izes
he esul s on Adap i e G adien op imized models.
This op imize was used in o de o p e en weigh s o aking he same alue,
and hus dieing while aining. I has o be no ed ha his also has conse-
quences: he g adien is mo e likely o pe o m zigzags due o he inc eased
a iabili y o he alues o i s weigh s.
545454
Zone Val. Accu acy X-En opy Loss
1 0.8431 0.3656
2 0.8682 0.2941
3 0.8537 0.3338
4 0.9024 0.2253
5 0.8634 0.2885
6 0.8429 0.3245
7 0.8758 0.2662
8 0.8872 0.2415
9 0.9170 0.2107
10 0.8812 0.2508
11 0.8736 0.2609
12 0.9042 0.2253
13 0.8785 0.2784
14 0.8854 0.2397
15 0.8713 0.2632
16 0.9126 0.2184
17 0.9028 0.2305
Table 4.4: Accu acy and C oss-En opy loss o e e y h ea
zone wi h he AdaG ad Op imize + lea ning a e = 10−4
.
Wi h his op imize , he models pe o med sligh ly wo se han he Mo-
men um ones. Mo eo e , he model wi h highes accu acy and lowes loss
was 9, di e ing om he esul wi h Momen um (highes accu acy achie ed
wi h body zone 16). See Fig. 3.1) o a isual ep esen a ion o each body
zone along wi h i s assigned numbe . Con usion Ma ix om h ea zone 9
can be seen below.
T ue Label
Posi i e Nega i e
Ne wo k Ou pu Posi i e 11 11 22
Nega i e 8 199 207
19 210 229
Table 4.5: Con usion Ma ix o h ea zone 9 wi h AdaG ad op imize .
This op imize wo ked a bi wo se han he Momen um. Fi s , he e we e
p oblems ini ializing he weigh s: when applying Xa ie ini ializa ion (see
555555
3.4.3), he weigh s we e, a e ew epochs, diminishing un il becoming 0. I
was when He ini ializa ion (see 3.4.3) was applied ha he weigh s began
o become spa se and s a o lea n di e en ea u es h ough he laye s o
he ne wo k. One eason is ha He ini ializa ion akes in o accoun he non-
linea i y o he ac i a ion unc ion, while Xa ie s ays in he linea egion.
In he nex igu e, alida ion accu acy and loss a e shown.
Figu e 4.2: Valida ion Accu acy and Loss in body zone 9 wi h
AdaG ad Op imize . No e ha he slope o he cu e on he
loss is di e en han in Fig 4.1
None heless, in compa ison o Momen um, om accu acy and loss o sensi i -
i y and speci i y, he esul s on his Op imize we e wo se han he ob ained
wi h he Momen um Op imize , al hough a p io i, i seemed ha he Adap-
i e G adien was a solu ion o he o me . In he ollowing able, we can see
he alida ion accu acy and loss o AdaG ad ained models.
565656