ŝ
,LJƉĞƌƉĂƌĂŵĞƚĞƌƐKƉƚŝŵŝnjĂƚŝŽŶŽŶEĞƵƌĂů
EĞƚǁŽƌŬƐĨŽƌŽŶĚdƌĂĚŝŶŐ
&ƌĂŶĐŝƐĐŽhƌďĂŶŽ&ŽŶƐĞĐĂ
dƌĂďĂůŚŽĚĞWƌŽũĞƚŽĂƉƌĞƐĞŶƚĂĚŽ
ĐŽŵŽƌĞƋƵŝƐŝƚŽƉĂƌĐŝĂůƉĂƌĂ
ŽďƚĞŶĕĆŽĚŽŐƌĂƵĚĞDĞƐƚƌĞĞŵƐƚĂƚşƐƚŝĐĂĞ'ĞƐƚĆŽĚĞ
/ŶĨŽƌŵĂĕĆŽ
ŝ
>KDD'/
Tí ulo: Hype pa ame e s Op imiza ion on Neu al Ne wo ks o
Bond T ading
F ancisco U bano Fonseca
MEGI
2018
ŝ
Uni e sidade No a de Lisboa
In o ma ion Managemen School
Hype pa ame e s Op imiza ion on Neu al Ne wo ks o
Bond T ading
F ancisco Fonseca
Submi ed in pa ul ilmen o he equi emen s o he deg ee o
Mas e o Science in S a is ics and In o ma ion Managemen
o he Uni e sidade No a de Lisboa. 2018
Abs ac
A i icial Neu al Ne wo ks ha e been ecen ly spo ligh ed as de ac o ools used o classi ica ion. Thei
abili y o deal wi h complex decision bounda ies makes hem po en ially sui able o wo k on ading
wi hin inancial ma ke s, namely on Bonds. Such classi ie aces high lexibili y on i s pa ame e s
in pa allel wi h g ea modula i y o i s echniques, a ising hus he need o e icien ly op imize i s
hype pa ame e s. To de e mine he mos e icien sea ch me hod o op imize almos he majo i y o he
Neu al Ne wo ks hype pa ame e s, we ha e compa ed he esul s ob ained by he manual, e olu iona y
(gene ic algo i hm) and andom sea ch me hods. The sea ch me hods compe e on se e al me ics om
which we aim o es ima e he gene aliza ion capabili y, i.e. he capaci y o co ec ly p edic on unseen
da a. We ha e ound he manual me hod o p esen be e gene aliza ion esul s han he emaining
au oma ic me hods. Also, no bene i was ound on he di ec ion p o ided by he gene ic sea ch me hod
when compa ed o he pu ely andom. Such esul s demons a e he impo ance o human o e sigh
du ing he hype pa ame e s op imiza ion and weigh aining phases, capable o analyzing in pa allel
mul iple me ics and da a isualiza ion echniques, a p ocess c i ical o a oid subop imal solu ions
when na iga ing complex hype spaces.
i
ii
Acknowledgemen s
I am p o oundly g a e ul o e e yone who ha e inspi ed o helped me h oughou he de elopmen o
his wo k, ei he di ec ly o indi ec ly. This achie emen would no ha e been possible wi hou hem.
To my supe iso I o Gon¸cal es and co-supe iso Mau o Cas elli o all he guidance.
To my pa en s and amily o he cons an encou agemen and op imism.
To Inˆes o all he joy and ne e -ending suppo .
To Guilhe me and Paulo o he iendship.
To Diogo, Paulo, Be o, Di and I o o he companionship and cama ade ie.
To Jo˜ao, Ana, Ra a, Ve a and Vanda o he mo i a ion and coaching.
To S´e gio o in oducing me o he wo ld o Machine Lea ning and o all he guidance and discussions
which in luenced deeply he cou se o his wo k.
To Ca ina, Tiago and Jo˜ao o he chee ulness and good spi i .
To Rica do o belie ing in me and always eaching me how o be a be e pe son.
To he Py hon communi y o all he open-minded dis ibu ion o con en s.
iii
i
Lis o Figu es
2.1 A simple ully connec ed neu al ne wo k wi h one hidden laye . . . . . . . . . . . . . . 6
2.2 Neu al Ne wo k Backp opaga ion T aining Li e Cycle. . . . . . . . . . . . . . . . . . . 11
4.1 PV o Bonds A and B h oughou hei li e cycle. . . . . . . . . . . . . . . . . . . . . . 22
4.2 P esen Value wi h posi i ely-sloped in e es a e cu e. . . . . . . . . . . . . . . . . . 24
5.1 O e all da ase in ma ix o m. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.2 G aphic in e p e a ion o sampling he da ase . . . . . . . . . . . . . . . . . . . . . . . 35
5.3 G aphic in e p e a ion o passing a da ase sample h oughou he neu al ne wo k. . . 36
6.1 Ex a T ees, Linea SVM and Pe cep on ea u es weigh s/ impo ances. . . . . . . . . 47
6.2 Pea son’s co ela ion o he a iables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
6.3 A Recu si e Fea u e Elimina ion s udy using Logis ic Reg ession. . . . . . . . . . . . . 49
6.4 2D and 3D PCA pe class label. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
7.1 Rep esen a ion o an au oencode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
7.2 Lea ning Cu e s o he Au oencode aining. . . . . . . . . . . . . . . . . . . . . . . 56
8.1 Accu acy, LogLoss and Sensi i i y box plo s. . . . . . . . . . . . . . . . . . . . . . . . 64
8.2 Speci ici y, False Posi i e Ra e and P ecision box plo s. . . . . . . . . . . . . . . . . . 65
xi
8.3 AUC, Youden’s Index and Disc iminan Powe box plo s. . . . . . . . . . . . . . . . . 66
8.4 F-Sco e and P o i box plo s. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
8.5 Tes ing Pe iod Financial Gain by Sea ch Me hod. . . . . . . . . . . . . . . . . . . . . . 67
8.6 Posi i e and Nega i e Likelihoods box plo s. . . . . . . . . . . . . . . . . . . . . . . . . 68
8.7 T ue Posi i e s False Posi i e Resul s pe Sea ch Me hod. . . . . . . . . . . . . . . . 69
A.1 Con ou Plo s o he Disc iminan Powe me ic. . . . . . . . . . . . . . . . . . . . . . 77
A.2 Con ou Plo s o he F-Measu e me ic. . . . . . . . . . . . . . . . . . . . . . . . . . . 77
A.3 Con ou Plo s o he Youden’s Index me ic. . . . . . . . . . . . . . . . . . . . . . . . . 77
xii
Chap e 1
In oduc ion
1.1 Mo i a ion and Objec i es
The ul ima e goal o his wo k is o c ea e an e ec i e and p ac ical ading ool o an Asse Man-
agemen company. The sea ch o al e na i es o he adi ional in es men decision me hods, e.g.
ading based on newspape a icles o on classical economic heo y models, gi es machine lea ning
echniques an impo an spo ligh o hei consolida ion as de ac o ools in he indus y. Acco dingly,
by inco po a ing s a e-o - he-a neu al ne wo ks’ da a p ocessing capabili ies, we aim o implemen
he algo i hm as he ul ima e in es men decision make .
Conside ing he gene alized ecogni ion o neu al ne wo ks as classi ie ools, we will ake ad an age o
hei modula design o mul iple uning (hype )pa ame e s and s udy hei combina o ial op imiza ion.
1.2 Con ibu ions and Impo ance o he Topic
The o e all con ibu ion o his wo k is dual wi hin he ields o Economics (in pa icula , Financial
Ma ke s) and Da a Science ( ocusing on Machine Lea ning).
Wi hin he Da a Science ield, he wo k aims o compa e he E olu iona y, Random and Manual
hype pa ame e s op imiza ion sea ch p ocedu es h ough hei ease-o -use, e iciency and bes solu-
ion encoun e ed. The applicabili y on Neu al Ne wo ks is highly ele an due o hei as i ude o
op imiza ion pa ame e s, along wi h he echniques ha keep appea ing wi hin he ield which a e
modula -like and easy o implemen . The wo k he eby p oduced aims o p o ide a ele an baseline
1
2Chap e 1. In oduc ion
o bo h academia and indus y on implemen ing neu al ne wo ks and hei op imiza ion. Thei con-
s uc ion as modula building blocks is, a he same ime, a ad an age and a cu se — such will be he
main opic o ou wo k. The lexibili y will be p o en o be a posi i e cha ac e is ic up o a ce ain
poin whe e i s a s o become a bu den. Ou wo k aims o p o ide a comple e desc ip ion on how o
implemen a machine lea ning app oach o p oblem-sol ing, as well as o desc ibe and es he main
amewo k on implemen ing such echniques.
Wi h wha conce ns he Financial Ma ke s, we aim o s udy he possibili y and he consequen ca-
paci y o p edic abili y o a Bond T ading Algo i hm. Bonds a e inancial secu i ies ha usually ha e
low ele ance on cen alized ading ma ke s, being aded mo e on O e -The-Coun e , and usually
highe Bid-Ask sp eads 1when compa ed o S ocks. This c ea es a deg ee o di icul y which we in end
o o e come and, possibly, coun e ac . We will s udy he abili y o p edic he u u e mo emen on
Bonds’ p icing which, i ound o be success ul, may p o ide ma ke deepening and inc ease liquidi y,
by inc easing po olio o a ion (when compa ing o a passi e s a egy o buy-and-hold o ecei e he
coupons pe iodically).
1.3 Me hodology
The wo k he eby p oduced aims o achie e he ollowing p ope ies: simplici y, ep oducibili y, ease-
o -use and pipeline-abili y.
Simplici y aims o gain in ui i e con ol o e he p ocesses o he lea ning scheme. The logic behind
i is ha i we keep he algo i hm simple and clea enough o he human mind, we can coun e ac
ad e si ies easily and e icien ly, because we will be able o iden i y he sou ce o he p oblem.
The p ac ical app oach equi es ease-o -use o he o e all algo i hm, such ha all he necessa y s eps
o he deploymen (c ea ion, debugging and implemen a ion) can be sha ed wi hin he company. This
o en elies on ha ing he sc ip well documen ed and simple.
In ha mony wi h he sc ip being easy o use, i mus also be ep oducible. All he necessa y s eps o
i s success ul implemen a ion ( om collec ing he da a o making he in es men decisions) mus be
s ess- es ed and obus enough o main aining s abili y when acing changes.
The Pipeline-abili y is a cha ac e is ic o he coding i sel , which o ces us o, when con on ed wi h
mul iple solu ions o he same p oblem, a o simple solu ions which can be pipelined, as a combi-
na ion o mul iple blocks, conjuga ing mul iple modules.
1The Bid-Ask sp ead can be measu e by he di e ence be ween he Ask and Bid p ices o he secu i y.
1.4. S uc u e 3
1.4 S uc u e
We i s p esen indi idually he main opics o ou wo k: Neu al Ne wo ks (In oduc ion and
S a e o he A ) and Bonds, on chap e s 2, 3 and 4, espec i ely, o p o ide he non-specialis eade
su icien knowledge o comp ehend he emainde o he ex .
The p ac ical de elopmen is hen di ided in o h ee main opics, by hei o de o appea ance on he
wo k: The Da abase (5), whe e we explain he p ocess and speci ici ies o c ea ing he da abase;
The Model (6), which con ains he co e opics on de eloping he algo i hm, and Deploymen (7)
whe e we p esen he implemen a ion o he wo k.
Finally, he discussion he Resul s (8) and he Conclusions (9) p esen he main indings and u u e
wo k.
1.5 Resea ch Ques ions
The wo k is cons uc ed o p o ide answe s o he ollowing ques ions:
H1. A e he E olu iona y and Random sea ch p ocedu es e icien , in hei ime-cos o applica ion?
H2. Does he di ec ion p o ided by he E olu iona y Sea ch p ocedu e p o ides an ad an age o e
a pu e Random Sea ch p ocedu e?
We belie e such enqui ies may p o ide ele an di ec ions o u u e p ac i ione s on how o add ess
he p oblem o op imizing a la ge numbe o hype pa ame e s o a lea ning algo i hm.
Chap e 2
Neu al Ne wo ks: An In oduc ion
This sec ion in ends o p o ide a p esen a ion and b ie e iew o he gene al concep s ha su ound
he ield o A i icial Neu al Ne wo ks (ANNs). This is pa icula ly aimed o allow non-expe s on he
ield o comp ehend he wo k done below.
Neu onal Ne wo ks a e compu a ional means which mimic he mechanisms and beha iou o he human
b ain (P. Fonseca, 1995), in pa icula , he abili y o deal wi h complex, non-linea pa e n ecogni ion
asks (Fi ko -No is e al., 2012). Simila o a human b ain, he ANN a e pa allel p ocessing s uc u es
(Moja ad e al., 2011) o densely connec ed mul iple neu ons ecei ing, p ocessing and ou pu ing
in o ma ion. In eed o wa d ANNs, which will be ou ocus, all connec ions a e di ec ed om inpu s
owa ds he ou pu s (Moja ad e al., 2011), con as ing wi h ecu en ne wo ks in which connec ions
amongs nodes a e allowed o e ain in o ma ion abou pas inpu s (Pascanu e al., 2013).
Neu al ne wo ks a e well es ablished ools o classi ica ion (Fi ko -No is e al., 2012; Janocha and
Cza necki, 2017) wi h good pe o mance on di e se ields, om which we include medicine (Kagua a e
al., 2014; Lu e al., 2001; Moja ad e al., 2011), compu e -aided de ec ion and design (Zu e al., 2009),
hid ology (Pio owski and Napio kowski, 2013). Along wi h he capaci y o cap u e complex and non-
linea in e ac ions, hey can also ake in o accoun he in e - ela ions be ween a iables (Moja ad
e al., 2011). Thei success is also a esul o he de elopmen o simple, b oadly applicable echniques
(Neelakan an e al., 2015), such as d opou (see sec ion 3.1), inno a i e ac i a ions unc ions (see
2.1.2) o weigh ini ializa ion (see 2.2.1).
4
2.1. A chi ec u e 5
2.1 A chi ec u e
The a chi ec u e o an ANN, i.e. he pa e n be ween he neu ons (Ng, 2011), which includes i s
connec i i y and he ac i a ion unc ions o each node, has a g ea impac on a ne wo k’s in o ma ion
p ocessing capabili ies (S anley and Miikkulainen, 2002; Yao, 1993; Yao, 1999), as i de ines he
numbe o pa ame e s o be op imized (Pio owski and Napio kowski, 2013).
2.1.1 Rep esen a ion
Rega ding he way we can ep esen ou ne wo k’s a chi ec u e, he e is no dominan me hod ha
ou pe o ms he emaining and he choice o he ep esen a ion elies p ima ily on he applica ion (Yao,
1993). The e o e, o p ac ical easons we a e using an indi ec encoding o he ANN a chi ec u e:
only speci ying in he ch omosome he mos impo an pa ame e s (such as he numbe o laye s o
he numbe o nodes a each laye ), ins ead o speci ying all he de ails, i.e. de ailing e e y node and
i s connec ions wi hin he a chi ec u e. This allows a mo e compac ep esen a ion o he ne wo k’s
connec i i y (S anley and Miikkulainen, 2002; Yao, 1993), enabling us o sea ch a la ge hype space
o pa ame e s, o he de imen o ine- uning a smalle a chi ec u e.
To illus a e he cons uc ion o an ANN connec i i y by speci ying indi ec ly hei componen s, we
ha e d awn an example o a 3-laye ne wo k — wi h 1 inpu laye (3 nodes), 1 hidden laye (5 nodes)
and 1 ou pu laye (1 node) — in Figu e 2.1. Conside ing all nodes a e connec ed h oughou he
laye s, we de ine i as a ully connec ed neu al ne wo k. The hidden laye s enable he neu al ne wo k
o ex ac high o de s a is ics (Moja ad e al., 2011).
The a chi ec u al choice o he ANN will be made by he hype pa ame e op imiza ion sea ch p ocess.
None heless, i is ele an o men ion ha gene aliza ion is mo e cons ic ed due o small ne wo ks
han la ge ones (Ca uana e al., 2000), so bigge ANNs should be na u ally a o ed.
2.1.2 Ac i a ion Func ions
Ac i a ion unc ions ans o m he ac i a ion s a e o each neu on o an ou pu . They in oduce
non-linea i y (Njikam and Zhao, 2016) which gi es ANNs non-linea capabili ies (LeCun e al., 1998)
and can signi ican ly impac he ANN pe omance (Xu e al., 2016; Yao, 1993).
I is hus ele an o no e ac i a ion unc ions ace mul iple p oblems when dealing wi h backp op-
6Chap e 2. Neu al Ne wo ks: An In oduc ion
Inpu
Ou pu
Figu e 2.1: A simple ully connec ed neu al ne wo k wi h one hidden laye .
aga ion: o e ly linea uni s do no compu e in e es ing esul s (Glo o and Bengio, 2010); excessi e
sa u a ion1can cause he g adien s o anish o explode (Xu e al., 2016); and ac i a ion unc ions
no symme ic a ound 0 should be a oided when ini ializing om small andom weigh s, because hey
yield poo lea ning dynamics (Glo o and Bengio, 2010), due o hei p oximi y o he null. On he
posi i e side, symme ic unc ions a e belie ed o yield as e con e gence (LeCun, 1989).
The poo pe o mance o he adi ional ac i a ion unc ions (Njikam and Zhao, 2016) eques s an
in es iga ion o new unc ions and o he po en ia ing echniques, such as he weigh s ini ializa ion (as
discussed on sec ion 2.2).
The e o e, o he choice o he ac i a ion unc ions we es ed commonly used ans o ma ions (na i ely
p esen in he Ke as sou ce code (Cholle e al., 2015)) along wi h some o he unc ions om he e iew
o li e a u e. A comp ehensi e lis is de ailed in able Table 2.1 on page 15. Fo he las ac i a ion
we ha e o choose only he ans o ma ions ha ma ch ou expec ed ou pu — i we a e classi ying
a scena io on a bina y ou pu (0 o 1) and we wan a con inuous alue ha app oxima es wi h some
con idence deg ee o such scena io, i is na u al ha he las ac i a ion unc ions ou pu s pe cen age
1We alk abou unc ion sa u a ion when he a gumen is oo posi i e o nega i e ha causes he unc ion o become
e y la and insensi i e o small changes (Good ellow, Bengio, e al., 2016). Using he logis ic sigmoid unc ion as an
example, a change in he a gumen nea he asymp o es will ha e less e ec han changes nea he o igin x= 0.
2.2. T aining 7
alues be ween [0,1], such as he logis ic sigmoid unc ion.
A complimen a y Py hon code o ac i a ion unc ions no usually ound in he usual neu al ne wo k
lib a ies can be ound in (F. Fonseca, 2017a).
2.2 T aining
2.2.1 Weigh Ini ializa ion
The s a ing alues o he weigh s can ha e a signi ican impac on he aining p ocess (LeCun e al.,
1998). They should be chosen in such a way ha (LeCun e al., 1998):
•The ac i a ion unc ion is ac i a ed on i s linea egion,
•The s anda d de ia ion o he inpu s is close o 1.
We will conside mul iple ini ializa ion me hods, which include he LeCun No mal,LeCun Uni o m,
Glo o Uni o m and Glo o No mal.
The LeCun No mal Ini ialize , p esen ed in (LeCun e al., 1998), conside s he weigh s being
andomly d awn om a ze o mean dis ibu ion, wi h he s anda d de ia ion gi en by:
σw=m−1
2(2.1)
whe e mis he numbe o inpu s o he uni .
The LeCun Uni o m Ini ialize is p esen ed in (LeCun, 1989). In his case, he weigh s a e
uni o mly ini ialized om:
W∼Uh−2.4
Fi
,2.4
Fii,(2.2)
whe e Fiis he numbe o inpu s o he connec ion.
The Glo o Uni o m is p esen ed in (Glo o and Bengio, 2010). The weigh s a each laye a e
ini ialized om:
Wij ∼Uh−1
√n,1
√ni,(2.3)
whe e nis he size o he p e ious laye .
8Chap e 2. Neu al Ne wo ks: An In oduc ion
Glo o and Bengio also p esen a Glo o No malized Ini ializa ion, wi h he p emise o main ain-
ing he ac i a ion and g adien a iances ac oss he ne wo k laye s. To achie e his:
W∼Uh−√6
√nj+nj+1
,√6
√nj+nj+1 i(2.4)
Conside ing ha ing di e en magni udes in he g adien s may slowe he aining p ocess (Glo o
and Bengio, 2010), he no malized ini ializa ions which coun e ac ha p oblem ha e heo e ical
ad an ages.
We will u he s udy he implemen a ion o he O hogonal (Saxe e al., 2013) and he He No mal
and He Uni o m (He e al., 2015) ini ialize s, na i ely p esen in he Ke as (Cholle e al., 2015)
lib a y.
2.2.2 Loss Func ion
Du ing he aining p ocess, he neu al ne wo k is gi en some eedback on i s pe o mance o o ien
he aining scheme o he bes possible scena io. By compa ing he ne wo k’s ou pu wi h he desi ed
ou pu , such cos unc ion (usually deno ed J(θ) whe e θ ep esen he pa ame e s) is minimized wi h
espec o he ne wo k’s pa ame e s (Moja ad e al., 2011) by he op imiza ion algo i hm (see sec ion
2.2.3). A b ie o e iew o he conside ed loss unc ions is de ailed below.
Some wo ks ound he condi ional log-likelihood, o C oss En opy (CE), cos unc ion o wo k
much be e o classi ica ion han he mean squa ed e o (MSE), p esen ing less pla eaus in he
aining c i e ion (Glo o and Bengio, 2010) and o e ing as e con e gence (Golik e al., 2013).
The wo ks o (Janocha and Cza necki, 2017) ound he log loss o be a poo choice o he loss unc ion,
wi h a good pe o mance o he squa ed hinge loss o he su p isingly mean squa ed e o .
In (Golik e al., 2013), he CE ou pe o ms he MSE. This is o en caused by he anish g adien s o
using MSE wi h he so max ac i a ion unc ion and wi h andom weigh ini ializa ion. None heless,
wi h a good ini ializa ion he MSE c i e ion seems o consis en ly imp o e he CE-based solu ion.
The non-dominance o a loss unc ion is s udied by conside ing hem all. By uni ying he sco ing
unc ion in he e olu iona y and andom sea ch p ocesses, we can inpu he loss unc ion as a unable
hype pa ame e o he ne wo k. This implies ha ing wo loss unc ions du ing he aining s age: one
o upda ing he ne wo k’s weigh s and ano he o measu e he ne wo k’s capabili y o p edic ion.
O he wise, he di e en na u al anges o each loss unc ion would block he compa ison be ween
2.2. T aining 15
Ac i a ion Func ion Abb e ia ion Func ion Re e ence
Logis ic Sigmoid LogSig (x) = 1
1+e−x
(LeCun e al., 1998)
(Glo o and Bengio, 2010)
(Good ellow, Bengio, e al., 2016)
(Dugas e al., 2000)
(K izhe sky e al., 2012)
(Io e and Szegey, 2015)
(Janocha and Cza necki, 2017)
( an Laa ho en, 2017)
Hype bolic Tangen anh (x) = anh(x)
(LeCun e al., 1998)
(Glo o and Bengio, 2010)
(K izhe sky e al., 2012)
So sign so (x) = x
1+|x|(Glo o and Bengio, 2010)
So plus so + (x) = log(1 + ex) (Dugas e al., 2000)
Rec i ied Linea elu (x) = max(0, x)(K izhe sky e al., 2012)
( an Laa ho en, 2017)
Scaled Exponen ial Linea selu (x) = (x x > 0
αex−α x ≤0(Klambaue e al., 2017)
LeCun Sigmoid lecun (x)=1.7159 anh(2
3x) + αx (LeCun, 1989)
(LeCun e al., 1998)
ScaledSigmoid scalsg (x) = 4
1+e−x−2 (Xu e al., 2016)
Ha dSigmoid ha dsigm (x) =
0x < −2.5
0.2∗x+ 0.5−2.5≤x≤2.5
1x > 2.5
PenalizedTanh pnl nh (x) = ( anh(x) i x >0
α anh(x) o he wise, α∈[0,1] (Xu e al., 2016)
Rec i ied Hype bolic Secan esech (x) = x∗sech(x) (Njikam and Zhao, 2016)
T unca ed Sin .sin (x) =
0,−π
2> x
sin(x),−π
2≤x≤π
2
1,π
2< x
(Pa ascandolo e al., 2017)
Sin sin (x) = sin(x)
Linea lin (x) = x
AlphaLinea alphlin (x) = α∗x
S ep s ep (x) = (0 i x ≤ h eshold
1 o he wise
Table 2.1: Comp ehensi e lis o ac i a ion unc ions conside ed in he hype pa ame e s op imiza ion
sea ch.
16 Chap e 2. Neu al Ne wo ks: An In oduc ion
Loss Func ion Fo mula
Log Loss / C oss En opy Pjy(j)log σ(o)(j)
L1Loss / Leas Absolu e E o s |y−o|
Mean Squa ed E o (y−o)2
Squa ed Hinge Pjmax(0,1
2−ˆ
y(j)o(j))
Table 2.2: The Cos Func ions conside ed.
Chap e 3
S a e o he A
The mo e ecen heo ies and echniques a e now p esen ed. They include inno a ions such as D opou ,
Ba ch No maliza ion o Sel -No malizing Neu al Ne wo ks, and Py hon lib a ies, such as Ke as and
Sciki -lea n. Conside ing he inno a ions in he ield o Neu al Ne wo ks a e p ima ily modula and
easily compa ible wi h he s a e o he a so a , we oo a e going o p esen hem in such way.
3.1 D opou
D opou is a egula iza ion echnique p esen ed in (S i as a a e al., 2014) which p e en s o e i ing.
By empo a ily emo ing a de e mined pe cen age o he uni s om he laye by se ing o ze o he
ou pu o he neu ons (K izhe sky e al., 2012; Wage e al., 2013; Zhang e al., 2016), he echnique
adds noise o he aining p ocess (Njikam and Zhao, 2016) and p o ides an inexpensi e simple way
o combine an ensemble o models by a e aging hei p edic ions (Good ellow, Wa de-Fa ley, e al.,
2013).
D opou o ces he neu al ne wo k o lea n mo e obus ea u es by educing complex co-adap a ions
o neu ons (K izhe sky e al., 2012) which lowe s he gene aliza ion e o and p e en s o e i ing
wi hou he need o dimensionali y educ ion (S i as a a e al., 2014).
The o iginal pape (S i as a a e al., 2014) e e s some p ac ical ips, which we will s a e and use as
a pendulum o ou hype pa ame e sea ch. Fo he d opou p obabili ies, he ecommended 20% o
he inpu s laye and 50% o he hidden uni s a e conside ed, along wi h he max-no m egula iza ion
— cons aining he maximum no m o he incoming weigh ec o a each hidden uni o a ixed
17
18 Chap e 3. S a e o he A
cons an — be ween 3 and 4. Also, la ge decaying lea ning a es and high momen um ( om 0.95 o
0.99) a e ad ised.
We will conside he d opou p obabili y and he max-no m cons an as addi ional unable hype pa-
ame e s.
Along wi h he classical D opou echnique, we will also conside he Alpha D opou as p esen ed
in (Klambaue e al., 2017), a echnique ha i s well o he SELU ac i a ion unc ion, along wi h
he Gaussian D opou also p esen ed in he o iginal pape (S i as a a e al., 2014). The di e ence
be ween he o iginal and he Gaussian D opou is ha while he i s mul iplies he hidden ac i a ion
unc ions by Be noulli dis ibu ed andom a iables1, he la he adds Gaussian noise wi h ze o mean
and s anda d de ia ion o σ, which becomes ano he hype pa ame e .
An in e es ing applica ion o his concep is he ex ension o he concep o d opou as a gene ic
lea ning me hod ha can be applied o any lea ning algo i hm, as ound in (Wage e al., 2013).
3.2 Ba ch No maliza ion
Th oughou he aining o a neu al ne wo k, he dis ibu ion o each laye ’s inpu s change due o
p eceden change o he p e ious laye . This is known as he in e nal co a ia e shi and has been
know o slow he aining p ocess. Ba ch No maliza ion (BN) (Io e and Szegey, 2015) is a echnique
ha ac s as a egula ize ( o u he de ails on egula ize s see 2.2.4). This echnique wo ks by
whi ening he inpu s a each laye , i.e. by no malizing he means and a iances o ba ches in he
aining da a ( an Laa ho en, 2017). BN has mul iple bene i s: i accele a es he aining p ocess and
makes i mo e esilien o pa ame e scale, p e en s sa u a ion in he ne wo k and educes he need
o d opou . BN is widely adop ed and i is o en ound o imp o e he gene aliza ion pe o mance
(Zhang e al., 2016).
One o he main ad an ages o BN and D opou is ha hey can be coded as laye s we add on o he
neu al ne wo k in a modula way. Ne e heless, i a ises hus he need o pick he igh o de ing o hei
combina ion o a p ope applica ion. The o iginal pape o D opou (S i as a a e al., 2014) e e s
o applying he echnique a e he ac i a ion unc ion. Rega ding he posi ion o he BN hough, i s
o iginal pape (Io e and Szegey, 2015) ad ises o use i a e a ully connec ed laye bu be o e he
1Fo mo e in o ma ion on Be noulli dis ibu ion see A.
3.3. Sel No malizing Neu al Ne wo ks 19
ac i a ion unc ion. Howe e , ecen ye unpublished wo ks seem o sugges using BN a e d opou
migh be a p omising scena io, so we will es hem bo h (wi h he o de as ano he hype pa ame e ).
3.3 Sel No malizing Neu al Ne wo ks
Sel No malizing Neu al Ne wo ks (SNNNs) (Klambaue e al., 2017) a e based on he ”Scaled Ex-
ponen ial Linea Uni s” (SELU) ac i a ion unc ion which induces sel -no malizing p ope ies such as
a iance s abiliza ion, hus a oiding exploding and anishing g adien s. SNN can keep he no mal-
iza ion h oughou mul iple laye s wi h many uni s bo h in he mean and he a iance, which speeds
up he con e gence (LeCun e al., 1998). Con inuing he o iginal D opou echnique (S i as a a e
al., 2014), Klambaue e al. p opose an Alpha D opou which keeps he mean and a iance a e he
d opou o also keep he sel -no malizing p ope y when using SELUs. The o iginal pape ecommends
d opou a es o 5% o 10% o good pe omance.
SNNNs a e able o wo k wi h many laye s because hey do no ace he ac i a ion unc ion sa u a ion
( anishing o exploding g adien s) by en o cing ac i a ions owa ds ze o mean and uni a iance.
3.4 So wa e and Tools
The coding p ocesses we e implemen ed wi h Py hon, wi h speci ic dependency on he Ke as lib a y
(Cholle e al., 2015) o he neu al ne wo ks, Sciki -lea n (Ped egosa e al., 2011) o gene al ma-
chine lea ning pu poses (c oss- alida ion and o he s ope a ions), Pandas (McKinney, 2010) o da a
manipula ion, sklea n-deap (sklea n-deap 2017) o he e olu iona y sea ch using he DEAP (Fo in
e al., 2012) e olu iona y compu ing amewo k, Numpy (Wal e al., 2011) o scien i ic compu -
ing and Ma plo lib (Hun e , 2007) o he g aphical en i onmen s. In pa allel, he e was he need
o de eloping some complemen ing lib a ies, such as No maliza o (F. Fonseca, 2017c) o no -
maliza ion o con inuous a iables, con usion ma ix c (F. Fonseca, 2017b) o c ea ing con usion
ma ixes o c oss- alida ed algo i hms and abno mal ac i a ions (F. Fonseca, 2017a) o unusual
ANNs ac i a ion unc ions.
Chap e 4
Bonds
Fo deepe unde s anding o he compu a ional di icul ies and speci ici ies, i is necessa y o unde -
s and he unde lying heo y su ounding he inancial secu i ies. The simplis ic app oach p o ided
below should allow a ull comp ehension o he hesis.
A bond is a inancial secu i y which en i les he bondholde o ecei e om he issue he p incipal
bo owed amoun plus pe iodic in e es (Hull, 2012; Ma ellini e al., 2003). Fo he issue , he cos o
inancing will be he coupon a e inhe en o he secu i y. Ce e is pa ibus, he la ge he company’s
s abili y, he lowe he coupon i needs o pay o a ac in es o s.
Fo he bondholde , he a e o e u n o ma u i y is gi en by he quo ed yield, which akes in o
accoun bo h he coupon a e and he p ice o he secu i y. The yield o ma u i y is he discoun a e
o e u n ha equals cu en p ice wi h he u u e cash lows (Hull, 2012) and i is he a e o e u n
an in es o ea ns om in es ing in such secu i y i he holds i un il he ma u i y (Ma ellini e al.,
2003).
Classic economic heo y s a es ha a bond p ice can be calcula ed as he sum o he u u e cash-
lows discoun ed by hei app op ia e discoun a e (Hull, 2012). The discoun a e mus be a eal
quan i ica ion o he isk a bondholde incu s on lending he money o he company, i.e. buying he
bond. Such isk can be ei he caused by ma ke mo emen s (an inc ease in he o e all a es causes a
ixed coupon bond o be less a ac i e, hus diminishing i s p ice) o by he c edi isk (a company
migh ail o epay any o he pe iodic coupons o he unde lying p incipal, incu ing in de aul ).
20
4.1. Bond P icing and Li e cycle 21
The bond ai alue is, in ma hema ical o m:
B=
X
=1
CF
(1 + i ) (4.1)
whe e CF is he cash low a pe iod (in e es o p incipal paymen ) and i is he ele an discoun
a e o he bond’s isk a pe iod .
In he case o a ixed coupon a e bond, an in es o knows a incep ion which a e going o be he
u u e cash lows, simply by mul iplying he coupon a e by he p incipal, lea ing he unce ain y o
he p icing o he quan i ica ion o he discoun a e.
When dealing wi h loa ing a es, he coupon a e no mally ollows a ma ke index plus a sp ead o
he company’s isk, making hei p ice close o pa ( he edemp ion alue).
4.1 Bond P icing and Li e cycle
I we conside a cons an in e es a e, he bond’s ai alue will be he sum o each cash- low discoun ed
a he same a e. This implies ha he p ice ends o he pa om di e en di ec ions depending on
whe he i ’s coupon a e s a s abo e o below i ’s app op ia e discoun a e.
Conside wo coupon-paying bonds wi h 3-yea ma u i y on a 5% app op ia e discoun a e. Fo some
eason, Bond A was p iced wi h a 6% coupon a e and Bond B wi h 4%. A incep ion, hei alue is:
BA=6%
(1 + 5%)1+6%
(1 + 5%)2+6% + 100%
(1 + 5%)3= 102.72% (4.2)
BB=4%
(1 + 5%)1+4%
(1 + 5%)2+4% + 100%
(1 + 5%)3= 97.28% (4.3)
Sol ing o he nex yea s, each bond’s ai alue is hus:
Fai Value =0 =1 =2 =3
Bond A 102.72% 101.86% 100.95% 100%
Bond B 97.28% 98.14% 99.05% 100%
Table 4.1: Fai Value wi h cons an in e es a es.
22 Chap e 4. Bonds
0 1 2 3
95
97.5
100
102.5
105
Yea s
P esen Value o he Bond
PV o he bond pe yea
Bond A
Bond A
Figu e 4.1: PV o Bonds A and B h oughou hei li e cycle.
This would imply ha , ega dless he yea , he Bond A p ice would always dec ease and Bond B he
e e se, which we could easily p edic jus by compa ing he ini ial coupon and discoun a es.
When excluding such assump ion on he discoun a e s abili y, we obse e ha he in e es a e cu es
a e non-ho izon al and hey can e en jump on unexpec ed in e es a e changes along he yea s, which
will c ea e some dis up ion on he p e iously explained p ice ends.
As an example, a sudden pa allel inc ease he ma ke in e es a es also inc ease a bond’s discoun
a e, which will ul ima ely dec ease i ’s ai alue. Conside a 1% inc ease in he discoun a e on he
p e ious bonds, jus be o e = 1. This would make Bond A’s discoun a e equal o i s coupon a e
and, as such, i ’s p ice will be always 100%. On he o he hand, Bond B would be a ec ed wi h a
sudden dec ease in i s alue o bo h = 1 and = 2.
Fai Value =0 =1 =2 =3
Bond A 102.72% 100% 100% 100%
Bond B 97.28% 96.33% 98.11% 100%
Table 4.2: Fai Value wi h pa allel inc ease in in e es a es in = 1.
As we can see, Bond’s B ai alue su e s a sha p dec ease and only hen s a s o inc ease. In an
in es o ’s poin -o - iew, and speci ically in Asse Managemen , i is impo an o know wha a e he
expec ed changes in he in e es a es because hey in luence he in es men beha io . I he 1% ise is
4.1. Bond P icing and Li e cycle 23
expec ed a he s a , wai ing o = 1 o buy he bond is an op imal decision because a oids ha ing
he downside on he bond’s alue on he i s yea . No e ha his does no in luence he a e o e u n
i one wai s un il he ma u i y. When measu ing in e ms o yield o ma u i y, bo h bonds pe o m
a he discoun a e, i alued a hei ai p ice. None heless, gi en he ading app oach and he
necessi y o a oid downsides o comme cial in e es , he ele an measu e mus be in e ms o p icing
and no in e ms o yield, because we may wan o sell be o e he ma u i y.
When lea ing he assump ion o a cons an in e es a e scena io, he e olu ion o a bond p ice along
i s li e cycle can be in e es ing o a ading app oach. Conside a 4% coupon- a e, 5-yea bond, wi hin
a posi i e-sloped in e es a e cu e, which is cons an in ime, i.e. despi e he inc ease in he in e es
a e o longe ma u i ies, such beha io will pe sis in he u u e. I we conside he ollowing cu e:
12345
i 0.5% 1.5% 2.5% 3.5% 4.5%
Table 4.3: Posi i ely-sloped In e es Ra e Cu e.
by he discoun ed cash lows me hod we will ha e he ollowing ai alues ( hei p esen alue (PV))
o he bond, a each yea :
012345
PV 98.52% 102.21% 104.44% 104.93% 103.48% 100%
Table 4.4: 5-yea , 4%-coupon bond p esen alue wi h inc easing in e es a es.
I is clea ha , om a ading pe spec i e, i is only in e es ing o go long, i.e. buying he bond,
be ween he yea s 0 and 3, gi en ha he p ice only lowe s he ea e .
This pe iod be ween which is ele an o in es in a ading pe spec i e, which we name he good
ading pe iod, is in luenced bo h by he coupon a e and by he in e es a e cu e. I he bond was
issued wi h a 6% coupon a e, in es ing om yea 2 onwa ds would no be ad an ageous. Also, i
he in e es a e was e e sed, i.e. nega i ely sloped om 4.5% o 0.5%, he bond would ne e ha e a
p ice inc ease.
These examples se e o p esen heo e ical e idence o he possibili y o ading on bonds — i we
can cap u e hese ela ionships and mo emen s, we migh be able o use hem in ou a o .
When dealing wi h bonds, he absence o a cen alized ma ke s, such as S ock Ma ke s, c ea es a he
same ime a p oblem and an oppo uni y. Gi en he ac quo ed p ices a e, by no means, absolu e
24 Chap e 4. Bonds
0 1 2 3 4 5
95
97.5
100
102.5
105
107.5
110
Yea s
P esen Value o he Bond
PV o he bond pe yea
PV
Pa /Redemp ion
Figu e 4.2: P esen Value wi h posi i ely-sloped in e es a e cu e.
— mul iple OTC 1 ade s quo e di e en p ices which will ul ima ely be nego ia ed — such implies
ha , o e e y p ice we conside , we will no know he o al ce i ude o i s alue. This can be ei he
ad an ageous o he algo i hm, i.e. he p ices can swing a ou a o , o he e e se. An ul ima e
possible e ec o ading models based on A i icial In elligence is in oducing ma ke dep h 2which
can e en ually c ea e he condi ions o a mo e igo ous p icing by he ma ke playe s.
4.2 Ca ego ical Cha ac e is ics
Bonds ha e some ca ego ical cha ac e is ics we belie e can be help ul in e alua ion hei ai alue and
hei p ice mo emen s. Fo being inhe en ly p esen in de ining he inancial secu i y, we conside ed
he Coupon Type, Ma u i y Type, Call Op ion and Paymen Rank cha ac e is ics as ca ego ical
a iables in ou da abase. Fu he de ails on hei implemen a ion can be ound in he Ca ego ical
Da a subsec ion (5.3.2).
Bonds can be Callable, i.e., he issue can buy hem back om he bond holds a p e-de ined p ices
on p e-de ined call da es (Ding e al., 2012), which can lead o some unce ain y on i s p icing.
Bonds ha e di e en anking on hei p omp i ude o paymen in he case o de aul , so i he de aul
scena io is included in he bond’s p icing alua ion (which i should), lowe - anking bonds should also
1An O e -The-Coun e (OTC) ma ke is a decen alized exchange whe e he ma ke playe s alk di ec ly be ween
hemsel es, ins ead o placing he o de s on a o mal exchange.
2Ma ke dep h measu ed by he liquidi y: ease a which one in es o can en e (buy) o lea e he ma ke (sell).
5.3. Da a P ep ocessing 31
sa u a ion o weigh s due o he possible p oximi y o he bounda ies o he ac i a ion unc ions. AS
such, we will ans o m o 0.1 and 0.9 ins ead, in acco dance wi h (Fi ko -No is e al., 2012).
The ca ego ical encoding mus be a p ep ocessing s ep echnique, due o he dimension a iabili y i
in oduces when used inside a pipeline. Fo u he de ails see sec ion 6.3.2.
5.3.3 Scaling
By changing he loca ion and scale pa ame e s, he sco es om di e en dis ibu ions a e ans o med
in o a common domain (Jain e al., 2005; La ha and Thangasamy, 2011), which a oids ea u es
in g ea e nume ic anges domina ing o he s in smalle ones (Huang and C. Wang, 2006), educes
compu ing ime by ini ializing he aining p ocess o mul iple ea u es on simila scales (Jayalakshmi
and San hakuma an, 2011) and a oids nume ical di icul ies du ing he calcula ion (Hsu e al., 2010).
Fu he mo e, in he speci ic cases o backp opaga ion, (LeCun e al., 1998) ad ises o a e age each
inpu o ze o mean and scale he a iables so ha hei co a iances a e simila . The i s ick o ces
he weigh s o be upda ed on bo h signs (+ and -), in con as o only using posi i e weigh s, while
he second helps o s abilize he a e a which he weigh s a e upda ed.
An example o mul iple scales can be ound in he coupon a e and he bond minimum piece7 ea u es,
which di e na u ally in hei alues. While he coupon a es ypically a y in small pe cen ages
(be ween 0% and 10%), he minimum pieces can ange om 1 cen (1% o an eu o uni ) and 1 million
eu os.
On a p ac ical no e, i is impo an o sa e he scaling pa ame e s be o e applying he ans o ma ion
o bo h he aining and es da a as a p ep ocessing laye . O he wise, he addi ion o new da a could
change he scaling which would, wi hou e- aining he model, cause he p e- ained model o p edic
on w ong inpu s. This is con i med in (Hsu e al., 2010).
Se e al scales we e conside ed, based on he wo ks o (Jain e al., 2005; Jayalakshmi and San haku-
ma an, 2011; La ha and Thangasamy, 2011).
1. S anda d Scale
Also know as Z-Sco e no maliza ion, his echnique uses he a i hme ic a e age and he s anda d
7Smalles amoun allowed in a ma ke ansac ion.
32 Chap e 5. The Da abase
de ia ion o he da a o scale he da a o ze o mean and uni a iance.
x0
i=xi−µx
σx
(5.1)
Gi en ha bo h he mean and s anda d de ia ion a e sensi i e o ou lie s, he echnique is no
obus . Also, he pa ame e s a e only op imal o a Gaussian dis ibu ion, being only easonable
o o he dis ibu ions.
2. Min-Max Scale
I is bes sui ed o cases whe e he minimum and maximum bounds a e known. O he wise,
scaling wi h he es ima ed pa ame e s i will e u n a non- obus me hod, concen a ing he
emaining da a o a smalle ange in he p esence o ou lie s. Shi s he minimum and maximum
bounds o 0 and 1.
x0
i=x0
i−min(x)
max(x)−min(x)(5.2)
3. Decimal Scaling
Applied in he assump ion ha di e en ea u es a y by a loga i hmic ac o . I is non- obus .
x0
i=xi
10n(5.3)
whe e n= log10 max(xi).
4. Median
By no malizing each sample by he median o he aw inpu s, he scale becomes insensi i e o
ex eme de ia ions.
x0
i=xi
median(x)(5.4)
5. Median-MAD
The median and median absolu e de ia ion scale is insensi i e o poin s in ex eme ails in
he dis ibu ion and ou lie s. I does no p o ide a common nume ical ange and has poo
pe o mance when he dis ibu ion is no Gaussian, because i elies on he median and median
absolu e de ia ion as es ima es o he loca ion and scale pa ame e s o he dis ibu ion.
x0
i=xi−median(x)
MAD (5.5)
5.4. Da abase C ea ion: The P ocess 33
whe e MAD =median(|xi−median(x)|).
6. Max Scale Inspi ed by he good pe o mance on (La ha and Thangasamy, 2011). I is simila
o he Min-Max, wi h he min = 0.
x0
i=xi
max(x)(5.6)
7. Modi ied anh
As p oposed in (La ha and Thangasamy, 2011), i is a simpli ied e sion o he anh-es ima o s
in oduced by Hampel (Hampel e al., 1986). Because i does no need he genuine sco e dis i-
bu ion gi en by he Hampel es ima o s, i s complexi y is educed and i s speed inc eases.
anhx0
i=1
2∗ anh0.01(xi−µx)
σx+ 1(5.7)
We could use he mul iple Scale s as ano he hype pa ame e o he lea ning algo i hm o be op imized,
as a p ep ocessing s ep. None heless, o simplici y and o educe he numbe o combina ions o
hype pa ame e s, we will use he Z-Sco e (S anda d Scale ). No e ha his s ep is only applicable
o he con inuous- alued a iables, e.g. coupon a es o ma ke indexes. The ca ego ical a iables
con e ed o bina y ea u es a e no scaled.
5.4 Da abase C ea ion: The P ocess
One in e es ing aspec o ou wo k is he p ocess o c ea ing a da abase om sc a ch, a opic we
belie e lacks documen a ion and in which we aim o documen he hinking p ocess. Thus, o p o ide
some insigh s in o his a ea, we now desc ibe a dia y-like log on ou p ocedu es.
5.4.1 Ea ly S eps
Be o e any cons uc ion, we need o unde s and wha is he main goal o ou wo k and i s speci ica ions.
Being he aim o he p ojec implemen ing a ading decision-make , he simples way o modelize
such decision is a bina y ou pu — ei he we in es /hold he inancial secu i y o we sell/do no buy.
The de ini ion o he Y a iable labels (0,1) will hus depend on he occu ence o an e en we will
de ine as being a good ading oppo uni y o no . Since we ha e he possibili y o c ea ing an his o ical
34 Chap e 5. The Da abase
egis y, we can incu on supe ised lea ning because on op o c ea ing he Xda ase , which holds
all he ea u es, we can label each ins ance acco dingly on he Y ec o , which holds he a ge .
Rega ding he de ini ion o he a ge , he Yield o Ma u i y was one o he me ics hough as he
c i ical indica o , bu i implies a empo al s abili y, i.e. holding un il ma u i y, we will no achie e
due o he need o a o a ing po olio, o company pu poses 8. The o a ion pe iod was de ined,
in acco dance o he company, wi h a 5 business days pe iod span, o achie e a weekly o a ion.
On a heo e ical side, we do no see any eason o con es he applicabili y o such o a ion pe iod,
despi e ou conce ns on being somewha sho sigh ed. Fo he sake o simplici y, he empo al span
we conside o de ine he Ylabels ec o will be he same o calcula e he empo al changes on he
Xhis o ical ea u es.
The choice o he decision igge hus elies on he Clean P ice o he bond, which does no include
any acc ued in e es . This is no he p e e able choice because he al e na i e, he Di y P ice, may
p o ide be e insigh s in o he ading p o i abili y — a s able clean p ice can s ill p o ide a ading
oppo uni y due o he acc ued in e es we ecei e when holding he bond. None heless, he Clean
P ice is he de ac o s anda d. Also, we can conside as negligible he acc ued in e es o e such a sho
span, by he way we de ine he h eshold o he a ia ion needed o conside a posi i e ou come and,
consequen ly, de ine an ins ance as o label y= 1. As an example, a 10% and 5% coupon bonds ha e
app oxima ely 0.139% and 0.069% o 5-day span acc ued in e es . On op o ha , we ha e o quan i y
ou ansac ion cos s on a 2-way basis — i a bond is bough on one week (label 1) and sold on he
nex , he las ansac ion cos mus be inpu ed on he p io s’ week decision. The example ollows wi h
a 25 basis poin s (bps) 9 ansac ion cos o a 1-way ansac ion. The e o e, he minimum h eshold
we ha e o conside ing a 2-way ansac ion is, when dealing wi h clean p ices and o he wo s case
scena io (highes coupon a e):
min( h eshold) = 10%
360 ∗5+2∗25bps
≈0.64%
≈64bps.
In addi ion, we can conside a con idence ma gin so ha all he algo i hm does lea n will undoub edly,
8No e ha , o comme cial pu poses on he buy side, a company has in e es in ca ying a con inuous ading olume
because a lo o endo s end o apply minimum ansac ion olumes which need o be ul illed, o he wise he line o
ading is closed.
9A basis poin is pe cen age o a pe cen age.
5.4. Da abase C ea ion: The P ocess 35
Figu e 5.1: O e all da ase in ma ix o m.
Figu e 5.2: G aphic in e p e a ion o sampling he da ase .
and wi h a isk-ma gin, be good examples. This ma gin is manually de ined, o simplici y a 16 bps,
so ha he inal h eshold is:
Y h eshold =T imeSpanAcc uedIn e es + 2W ayT ansac ionCos s +Con idenceMa gin
= 80bps
Ha ing heo e ized o e he Y ec o , i is now ime o demons a e he de elopmen o he Xda ase ,
also known as he independen a iables da ase . The o e all layou will be a single da ase which
comp ises bo h Xand Y, as demons a ed in Figu e 5.1, whe e na e he numbe o samples and m
he numbe o ea u es o X.
On applying he machine lea ning algo i hm, we a e going o explo e he ela ionships o X, bo h wi h
Yand in e nally wi hin di e en ea u es, ha allow us o explain up o some ex en he beha iou
o Y. G aphically, we can hink o aining he neu al ne wo k in ela ion wi h he da abase as d awn
on Figu e 5.2.
By ansposing 10 he o iginal da ase and sepa a ing each columns, we can isola e each ins ance and
eed he neu al ne wo k on he inpu side and, a he same ime, p o iding he ou pu answe on he
o he . This is demons a ed on Figu e 5.3.
10We can de ine ansposing as lipping a ma ix o e i s diagonal, changing he ows pe columns.
36 Chap e 5. The Da abase
Figu e 5.3: G aphic in e p e a ion o passing a da ase sample h oughou he neu al ne wo k.
Ha ing app o ed he a chi ec u e o he inal s uc u e o he da abase, we will now desc ibe how o
achie e i .
5.4.2 Inpu s
The inpu s we conside a e 3- old: Suppo ing Fea u es, Bond Fea u es and Bond P ices. These will
be desc ibed u he on.
The i s , Suppo ing Fea u es, encompasses ma ke a iables which come in aw alues, such as he
NASDAQ index alue o he p ice o gold, on a daily basis. The inal s uc u e shall be a da ase
o mul iple ea u e alues pe day. Since we a e dealing wi h absolu e alues bu we in end no o
calcula e a eg ession bu o classi y, we p ep ocess such a iables by calcula ing he pe cen age o
change in ela ion o he same ea u e ndays in o he pas , being nde ined manually, o ailo he
ea u es ep esen a ion o he objec i e (Shen e al., 2012). The ime-index o such calcula ions is
ex emely impo an because we wan o clea ly de ine, a each day in he pas , wha was and wha
was no a ailable in o ma ion a ha ime, so ha we do no in e nally o e i somehow he da a.
Thus, a each day (T), we will ha e he pe cen age change be ween (T-1) and (T-1-n) days, whe e
nis o simplici y equal o he o ecas ing pe iod. This aspec o ime-awa eness o he da a will be
ubiqui ous o he da abase c ea ion p ocess and we will u he e u n o his subjec .
Nex , we ha e he Bond Fea u es da ase , which includes he de ails o each bond (e.g. coupon a e,
ma u i y and so o h) and in ends o p o ide he lea ning algo i hm wi h in insic de ails in pa allel
wi h he ma ke ones. Fo p ac ical easons, some o he en ies will be dele ed o coe ce he da ase o
elemen s o ou in e es , as e e enced on sec ion 5.2. Unlike he Suppo Fea u es da ase , he Bond
Fea u es is no ime dependen . Fo he sake o simplici y, we will conside such de ails a e la gely
5.4. Da abase C ea ion: The P ocess 37
cons an , despi e hey can al e a some poin , e.g. a bond can change some o i s de ails along i s li e
cycle, such as he coupon ype o he coupon a e. Such s abili y allows us o ha e a ma ix o i ows
o bonds, which ha e jcolumns o ea u es, which do no change.
The hi d inpu a e he his o ic Clean Bond P ices, o which we c ea e a ime dependen able o
p ices pe day (icolumns o bonds and j ows o days). Despi e being only capable o buy a he Ask
and sell a he Bid p ice (whe e AskP ice ≥BidP ice), we will conside he Mid p ice as he ele an
(MidP ice =A e age(Bid, Ask)) o he a ge ec o and use he sp ead (Sp ead = Ask - Bid) as
an inpu o he X. The mo i e behind c ea ing he sp ead as a ea u e elies on he lexibili y o
he p ices on he Bond Ma ke s, gi en hey a e g ea ly aded o e - he-coun e , which leads o some
ins abili y on he exac p ice and use he sp ead as a p oxy o he liquidi y11 o he secu i y, in he
hope such p ope y can add in o ma ional alue o he model.
The shi ing ope a ion mus also be done o he sp ead ec o . Unlike using he pe cen age change
on he Suppo Fea u es, we will apply o his ea u e a mo ing a e age o he las ndays, wi h
a minimum o 1 day, and shi o 1 day a e , so ha he da abase includes hese alues only o
he nex day. Simila o he p esen ,i we conside o be in-be ween he ading pe iod o day T
(i.e. in aday pe iod), we only know he mo ing a e age o ndays o yes e day’s close and no as o
oday, o (T-n-1) o (T-1). Simila o he p ecau ions we had wi h he ime dependen da a se ies
o Suppo Fea u es, we oo ha e o be ca e ul on de ining empo a ily he mo emen s. Since we a e
able o de elop wi hin a supe ised lea ning en i onmen , ou c ea ion o he label mus be clea and
hough ul.
Ou a ge ec o will hus be he di e ence o he bonds’ p ices, ollowed by an e alua ion (a con e sion
on an in e es ing mo emen (label 1) i he change is g ea e han a h eshold and 0 o he wise) and
a consequen empo al shi backwa ds o ndays, wi h nbeing he o ecas ing pe iod. This can
in ui i ely unde s andable by he ollowing example.
Imagine ha we ha e a 5-day week o his o ic bond p ices and we wan o de ine hei a ge , i.e. o
classi y hem as wo hy o in es men o no , o a 1-day and a 2-day o a ion ho izon. We a e going
o use he schema ic s a ed be o e o di e encing, applying a a ge unc ion and a empo al shi .
Conside he ollowing p ices on able 5.2. I we conside he minimum h eshold o 0.8 o going long
on he ading decision, we will ha e he op imal decisions, pe o a ion schedule, on able 5.3.
11The liquidi y p ope y e e s o he capaci y o easily and e icien ly en e (buy) o lea e (sell) he secu i y on he
ma ke .
38 Chap e 5. The Da abase
Day T-4 T-3 T-2 T-1 T
P ice 100.0 100.5 101.5 100.0 102.0
Table 5.2: Examples o bond p ices o a ge de ini ion.
Day T-4 T-3 T-2 T-1 T
1-Day 0 1 0 1 nan
2-Day 1 0 0 nan nan
Table 5.3: Op imal decisions o he ading scena io.
We now eplica e, o he 2-day o a ion, he ope a ions desc ibed be o e, on able 5.4, whe e Di
calcula es he absolu e di e ence be ween alues sepa a ed by ndays (Di (T-2) = 101.5 - 100), Tg
Fn con e s he di e ence o 1 i Di ≥ eshold and 0 o he wise, and he shi ope a ion mo es he
o iginal a ge unc ion by npe iods backwa ds. Bea in mind no o o e ide he nan’s wi h ze os
because hey signal an absence o in o ma ion, which we canno in e p e as o label 0. Also, he 2-day
pe iod clea y demons a es he need o shi in ega d o he o ecas ing pe iod.
The combina ion o he da ase s will hus ely on using he day T as a pendulum and using he
pe cen changes o he p e ious ndays o he independen a iables o X, shi ed by one pe iod, and
he pe cen changes o he nex ndays o he a ge Y.
Ha ing he di e en da ase s a ailable, i is now ime o combine hem in o one unique Xand Y.
A c ucial s ep o compu a ional e iciency is selec ing he ele an da a be o e c ea ing he soon- o-
be-la ge da ase which will conca ena e all he in o ma ion. One way o achie e his is o ake in o
conside a ion a minimum numbe o days o da a a ailable o each bond and e ase he ones who do
no mee such condi ion. This minimizes he p obabili y ha , o any sampled bond, we will no
encoun e hem in a speci ic li e cycle o i s p ice. As an example, a bond issued a a highe coupon
a e han i s eal isk a e will inc ease i s p ice on he ea ly days and ice- e sa, which despi e he
po en ial o a good pe o mance, i does no in e es us due o i s sho -sigh edness and possible
Day P ice Goal Di Tg Fn Shi
T-4 100.0 1 nan nan 1
T-3 100.5 0 nan nan 0
T-2 101.5 0 +1.5 1 0
T-1 100.0 nan -0.5 0 nan
T 102.0 nan +0.5 0 nan
Table 5.4: C ea ion o he a ge by di e encing, applying a a ge unc ion and empo al shi .
5.4. Da abase C ea ion: The P ocess 39
o e i ing abili y. This was p io ly demons a ed on sec ion 4.1.
5.4.3 O e all F ame
In p ac ical e ms, he me ge will be made by days and, consequen ly, we will ga he all days’ da ase s
in o one. As such, o each day we ancho he cons uc ion on he da a ame ha is cons an h ough-
ou he en i e ime se ies: he Bond Fea u es. Wi hin i , he only calcula ion is o es ima e, o each
day, wha was he ime o ma u i y o each bond a ha ime, o ha e an inpu which e lec s he
emaining li e ime and i s impac on he ading capabili y.
A e his, we append he Suppo ing Fea u es (which includes s ock indexes, commodi ies p ices and
o he indexes) ele an ow o ha speci ic da e. No e ha i includes he pe cen age changes o he
p e ious ndays we ha e chosen, which implici ly ake in o conside a ion he inal alue o he day we
a e calcula ing upon. This is some hing we wan o a oid because, a each day o he aining da ase ,
all he in o ma ion mus e lec p io e en s, as including any calculus which ega ds he e en s o
ha same day may be implici ly co ela ed wi h he a ge i sel . Making such mis ake would esul
in o e i ing he aining da a and, consequen ly, lowe ing he gene aliza ion capabili y. To a oid his
we shi he suppo ing ea u es ec o o da a by one ading day — emembe his is a ma ix o
ea u es alues pe days, so o each day we ha e a ec o . Shi ing by one day is he lowes alue we
can shi o bo h o e come he o e i ing p oblem (possible di ec co ela ion be ween he dependen
and he independen da ase s), while gi ing us he mos ecen da a possible, a he same ime.
A e wa ds, we append he Y ec o o ha day, which con ains each bonds’ u u e mo emen s ue
p edic ion, al eady shi ed in ega d o he empo al o e - i ing. I e a ing he abo e ope a ions o
each day, we will end wi h a lis o da ase s o me ge and, again, manually clean non- ele an da a.
Fo his ime, we will exclude absu d alues, such as nega i e ime o ma u i ies o coupon a es, as
discussed in 5.2.
A key s ep on inalizing he da ase is he ca ego ical con e sion. We will c ea e dummies on he 1
ou o N-1 echnique; o u he de ails please see sec ion 5.3.2. F om he ull da ase , we a e going
o di ide i in o wo: be o e and a e 2017. We will lea e he las as a empo al-con inuous alida ion
da ase on whe e we a e going o e alua e ou lea ned model, as an ou -o -sample es ing. Fo he
emaining da ase , p io o 2017, we will balance he impo ance each bond has on he aining da ase
by pe o ming a Random Unde Sampling echnique, b ie ly desc ibed on 5.3.1, only his ime i does
no conce n he dis ibu ion o he Ylabels bu he numbe o samples o each bond. This echnique
40 Chap e 5. The Da abase
will, o he whole da ase , andomly selec uunique numbe o samples om each bond, so ha all
ha e he same weigh o e all on he lea ning scheme. We will also d op he Da e ea u e on his
da ase o o ce he algo i hm o lea n in insic ela ionships wi hin he da a wi hou p o iding a key
iden i ie such as he da e ea u e.
On summa y, ou subse ing o aining and es da ase s goes as ollows. A majo spli is pe o med
on he end o he yea 2016: all da a a e Decembe 31s , 2016 is conside ed as ou -o -sample es
se . This will be he da ase on which we will d aw ou conclusions. Be o e 2017, we will spli he
da ase in 4 subse s: weigh aining, encoding/dimensionali y educ ion, hype pa ame e op imiza-
ion and alida ion.The alida ion da ase will show us he expec ed gene aliza ion capabili y o he
p edic o s wi hing he aining ime ame bu ou side he da a poin s on which he lea ning algo-
i hm has ained. Also, he c oss- alida ion ope a o ( u he de ailed below on 6.3.2) will c ea e
empo a ily o e idable es se s wi hin he be o e men ioned subse s o weigh aining, encoding and
hype pa ame e op imiza ion.
6.3. Gene aliza ion, O e - i ing and he Lea ning Cu e 47
Figu e 6.1: Ex a T ees, Linea SVM and Pe cep on ea u es weigh s/ impo ances.
us o alida e ha no ea u e is, in he ligh o his model, specially dominan and ha , as such, no
o e i o he a ge is made. We can also obse e a ela i e s abili y in he s anda d de ia ions o he
ea u es impo ances (black e ical lines) h oughou he mul iple (500) es ima o s used.
Ano he way o measu e a ea u e ele ance is o use Suppo Vec o Machines, an algo i hm which
inds a sepa a ing hype plane wi h he maximal ma gin in he dimension space (Hsu e al., 2010). As
desc ibed in (Y.-W. Chang and Lin, 2008), we ain a Linea SVM on a L-2 loss and so weigh s in
he model. The SVM g aph shows an absence o a dominan ea u e impo ance, wi h an une enly
dis ibu ion h oughou he signal (posi i e o nega i e) and h oughou he absolu e ea u e impo -
ance alue. Such esul s alida e he absence o o e i ing and, hus, he use o he da ase .
On he subjec o Neu al Ne wo ks, we can use a 1-neu on classi ie , capable o ecognizing linea ly
sepa able pa e ns, called he Pe cep on (Rosenbla , 1958). Decoding he alue o he weigh s will
no be ou inal aim, we jus in end o check o a non-ex eme-balanced dis ibu ion. The Pe cep on
g aph shows di e se esul s on he ea u es impo ances, bo h in amoun as in signal. The e is also
an absence o specially dominan ea u es, which could imply a di ec ela ionship o such ea u e(s)
wi h he a ge , wha would ul ima ely lead o o e i ing he model. We ha e ound he esul s o
alida e he use o he da ase .
The weigh s o he SVM, Ex aT ees and Pe cep on linea classi ie s a e depic ed in Figu e 6.1.
Ano he way o es ou da ase alidi y is o calcula e he Pea son’s co ela ion coe icien — which
measu es he linea dependence be ween a iables o each a iable (Guyon and Elissee , 2003; Moja ad
e al., 2011) — o he ea u es se Xagains he a ge Yand be ween hemsel es. The Pea son’s
co ela ion be ween a iables X and Y a ies be ween -1 and +1 and i is calcula ed by:
ρXY =co (X,Y)
σxσy
=E[(X−µx)(Y−µy)]
σxσy
; (6.2)
whe e σis he s anda d de ia ion, µis he a e age and Eis he expec ed alue. The co ela ion o
48 Chap e 6. The Model
Figu e 6.2: Pea son’s co ela ion o he a iables.
he a iables be ween hemsel es and agains he a ge is depic ed Figu e 6.2.
This allows us o check i we a e somehow o e i ing he model by p o iding an implici answe on
he X, which would be no iced by an abno mal co ela ion be ween he a iables. One migh use such
es o educe da a dimensionali y by disca d a iables which ha e nea -ze o co ela ion (no ela ion
be ween bo h a iables) o o which he p- alue o he null hypo hesis is abo e a ce ain h eshold
p > α, being αa ce ain signi icance le el ( ha he ela ionship is no s a is ically signi ican ).
None heless, as we a e dealing wi h neu al ne wo ks capable o cap u ing complex ela ions be ween
he da a, emo ing ea u es which may no seem ele an in a linea way may esul in lowe ing he
ne wo k’s capaci y.
F om he Pea son’s co ela ion g aph be ween Xand Y, we alida e ha he e is no s ong abno mal
ela ion o a speci ic ea u es, wi h he maximum absolu e coe icien close o 0.1 and ha all ea u e’s
co ela ion coe icien s a e s a is ically signi ican a , a leas , 1% ( hei p- alue is less han his
h eshold). The Co ela ion Ma ix be ween he ea u es o Xs udies hei in e nal co ela ion wi hin
he da ase . We obse e an absence o clea pa e ns o co ela ions, despi e obse ing highe alues
o he posi i e han o he nega i e coe icien s. This es alida es ha wi hin ou da ase we do
no ha e abno mal ela ions which p e en i s use.
The Logis ic Reg ession (Cox, 1958) is a linea bina y classi ie which we will use as he es ima o
o a Recu si e Fea u e Elimina ion (RFE) s udy, a backwa ds ea u e elimina o . By i e a i ely
elimina e ea u es and es he pe o mance, we can plo he p edic ing pe o mance pe numbe o
ea u es selec ed which allow us o sense he o e all pe o mance o he classi ie when he da ase
is educed a each s ep. Again, ou in e es elies mo e on he o e all igu e han s udying deeply
his beha io — heo e ically, we can conside he da ase as balanced i no single ea u e has an
ex eme p edic abili y capaci y and he pe o mance inc eases, a na u ally di e en paces, wi h he
numbe o ea u es. S a ing wi h all he ea u es allow us o s udy hei combina ion in ull p o ides
6.3. Gene aliza ion, O e - i ing and he Lea ning Cu e 49
Figu e 6.3: A Recu si e Fea u e Elimina ion s udy using Logis ic Reg ession.
s onge combina ions despi e being mo e slow, when compa ing o wo king he o he way a ound
wi h a Fo wa d Fea u e Selec o (Guyon and Elissee , 2003). The Recu si e Fea u e Elimina ion
pe o mance is shown on image 6.3. In acco dance wi h he expec ed beha io , he RFE has an
adequa e pe o mance, i.e. no ex emely good, o a educed numbe o ea u es, a apidly inc easing
pe o mance ha ends up s agna ing when he e a e s ill se e al ea u es o be selec ed. No e ha
he conside ed e o unc ion is he nega i e c oss en opy, o which highe alues ep esen a lowe
e o and, hus, a be e esul . The o e all beha io illus a es he exis ence o di e en ea u e’s
ele ances o linea models.
The P incipal Componen Analysis (PCA) is a echnique o educing dimensionali y while p ese ing
as much s a is ical in o ma ion as possible (Jolli e and Cadima, 2016). While his can be used as a
p ep ocessing s ep, we in end he e o alida e he da ase . As we a e unning a classi ie , he s udy
o unning he PCA pe class can p o ide an insigh ul and in e es esul . To allow he isualiza ion
o he esul s, we will un he PCA o 2 and 3 componen s, c ea ing wo- and h ee-dimensional
plo s, espec i ely. A e ans o ming he o iginal da ase , we assign each P inciple Componen o
he g aph’s axis and di e en ia e hei class label by colo . This will e u n a dis ibu ion o class
labels on he PCA’s axis. Thus, i he da ase is imbalanced, we a e expec ed o see clea ly sepa able
clus e s. The 2D and 3D PCA a e depic ed on Figu e 6.4. F om he igu e we can obse e gene ally
dis ibu ed alues, bo h o e all and wi h ela ion o he class labels. Despi e he exis ence o some
ou lie s, we conside he da ase o be adequa e acco ding o he PCA by he o e all non-exis ence o
class clus e s.
O e all, he esul s ound in he S a is ical Analysis es s, bo h uni a ia e as mul i a ia e, alida e he
use o he ea u es (X) da ase . Despi e p esen ing some high alues o in e nal co ela ions wi hin
50 Chap e 6. The Model
Figu e 6.4: 2D and 3D PCA pe class label.
ea u es, all o he es s ailed o iden i y a se e e o e i ing o da ase ea u es o he a ge which
could ha e implica ions on he model’s inal capaci y o p edic ion.
6.3.4 Me ics
A e he lea ning algo i hm has ained, i is ele an o measu e i s pe o mance on known da a
o ge a glimpse o he gene aliza ion capaci y — he abili y o pe o m on unseen da a. A me ic
alone is no su icien o de ail all o he capaci ies o he model. As such, a combina ion o measu es
is needed o gi e a balanced e alua ion o he algo i hm’s pe o mance (Sokolo a e al., 2006). The
complimen a y discussion o esul s is a ailable in chap e 8.
Measu ing he accu acy o he model, i.e. he pe cen age o co ec ly p edic ed samples (Fi ko -No is
e al., 2012; Good ellow, Bengio, e al., 2016; Koha i, 1995), is e y simple and easy o implemen bu
is o en a poo choice o e alua e pe o mance Fawce and i can be misleading (Jeni e al., 2013).
Fo example, when leading wi h an unbalanced da a se (Fi ko -No is e al., 2012), he accu acy will
end o co ec ly classi y he majo i y class and igno e he emainde . Despi e i s ad an ages, he
accu acy is a e y common pe o mance measu e (Lu e al., 2001). A common me ic o ca ego ical
classi ica ion accu acy is he con usion ma ix (Fi ko -No is e al., 2012), which eco ds he co ec ly
and inco ec ly ins ances o each class (Koha i, 1995) and allows o cons uc a mul iplici y o me ics.
A bina y classi ica ion p oblem gene a es a 2x2 ma ix, as show on Table 6.1. A comp ehensi e lis o
he Con usion Ma ix used me ics is de ailed on Table 6.2. A Py hon applica ion o he con usion
ma ix can be ound in (F. Fonseca, 2017b; Ped egosa e al., 2011).
O he ele an measu e is he A ea Unde Cu e o he ROC (Recei e Ope a ing Cha ac e is ics)
cu e (Fawce , 2006). The ROC cu e plo s he classi ica ion esul s om he mos posi i e o he
mos nega i e (Sokolo a e al., 2006) wi h he ue posi i e a e ( p ) on he y−axis and he alse
6.3. Gene aliza ion, O e - i ing and he Lea ning Cu e 51
T ue P edic ed 1 0
1 T ue Posi i e False Nega i e
0 False Posi i e T ue Nega i e
Table 6.1: 2x2 con usion ma ix.
posi i e a e ( p ) on he x−axis. By compa ing he ecall ( p ) agains he p , we can s udy ou
lea ning algo i hm’s p edic ion capaci y agains a pu e andom classi ie . The u he we a e om
he diagonal andom line o he no hwes , he be e . Classi ie s ha ou pu disc e e alues c ea e
single poin s on he ROC space. None heless, as we can ou pu a con inuous (p obabilis ic) sco e, we
may s udy he impac o conside ing mul iple h esholds: i he sco e is abo e a ce ain h eshold,
e u n 1; o he wise e u n 0. This gene a es mul iple poin s in he ROC space which e en ually me ge
o o m a line, below which we calcula e he A ea Unde Cu e. The AUC-ROC is in ui i ely ’ he
p obabili y ha he classi ie will ank a andomly chosen posi i e ins ance highe han a andomly
chosen nega i e ins ance’ (Fawce , 2006).
The Disc iminan Powe has a pe o mance o poo when DP < 1, limi ed o 1 ≤DP ≤2, ai o
2≤DP ≤3 and good i DP ≥3.(Sokolo a e al., 2006)
In pa allel wi h he me ics al eady discussed, i is ele an o measu e he pe o mance o he ading
models h oughou he es ing pe iod. As such, we will sample he op decisions o each week,
conside ing he model is designed o a weekly po olio o a ion, and measu e he p o i abili y o
hose ading decisions. The buying decision will be made o he highes p obabili ies p edic ed — as
we a e ading on a bina y decision o 0 o 1, he neu al ne wo k ou pu s a con inuous alue be ween
hose limi s o which highe alues, i.e. close o 1, ha e highe p obabili y o being in e es ing buying
oppo uni ies.
The esul s o he me ics s a ed abo e a e a ailable on chap e 8.
52 Chap e 6. The Model
Me ic Fo mula Ci a ion
Sensi i i y / Recall p
p+ n
(Sokolo a e al., 2006)
(Fawce , 2006)
(Jeni e al., 2013)
(Lu e al., 2001)
(Moja ad e al., 2011)
Speci ici y n
n+ p
(Sokolo a e al., 2006)
(Fawce , 2006)
(Lu e al., 2001)
(Moja ad e al., 2011)
P ecision p
p+ p
(Sokolo a e al., 2006)
(Fawce , 2006)
(Jeni e al., 2013)
False Posi i e a e p
p+ n
(Fawce , 2006)
(Lu e al., 2001)
Accu acy p+ n
p+ n+ p+ n
(Sokolo a e al., 2006)
(Fawce , 2006)
(Jeni e al., 2013)
(Lu e al., 2001)
(Moja ad e al., 2011)
AUCb(Balanced Accu acy) Sensi i i y+Speci ici y
2(Sokolo a e al., 2006)
F-measu e 2 ∗P ecision∗Recall
P ecision+Recall
(Fawce , 2006)
(Jeni e al., 2013)
Youden’s index J=sensi i i y −(1 −speci ici y)(Youden, 1950)
(Sokolo a e al., 2006)
Likelihoods ρ+=Sensi i i y
1−Speci ici y ;ρ−=1−Sensi i i y
Speci ici y (Sokolo a e al., 2006)
Disc iminan Powe DP =√3
π(logX +logY ), X=Sensi i i y
1−Sensi i i y ,Y=Speci ici y
1−Speci ici y (Sokolo a e al., 2006)
Table 6.2: Comp ehensi e lis o me ics om con usion ma ix.
Chap e 7
Deploymen
7.1 Dimensionali y Reduc ion
One o he ocus poin s o ou wo k is he applicabili y o he gene al eade who wan s o implemen
such echniques wi hou ha ing o esou ce o dedica ed se e s (which ha e mo e compu a ional
capaci y han he common lap op). The need o inc ease he pe o mance hus aises he ques ion o
subse he o iginal ea u es space and/o educe hei dimensionali y. Reducing he dimensionali y o
he da ase is hus a necessa y s ep o as en he o e all p ocess and i can be ad an ageous since i
acili a es classi ica ion asks (G. E. Hin on and Salakhu dino , 2006).
Dimensionali y educ ion elies on he assump ion o a lowe dimensional in insic dimensionali y. We
will conside dimensionali y educ ion in h ee ca ego ies (Y. Chang, 2014): subspace, mani old and
ke nel. They di e on he assump ion o he unde lying opology o he da a, wi h subspace lea ning
ocusing on linea i y, mani old lea ning on non-linea i y and he ke nel has an hyb id app oach com-
bining bo h wo lds. Bo h PCA and LDA (Linea Disc iminan Analysis) a e compu a ionally e icien
unde a linea subspace, bu ail when he s uc u e o he da a is no , i.e. when he low-dimensionali y
lies on a non-linea mani old. Mani old echniques o Isomap o Locally Linea Embedding (LLE) su -
e om he cu se o dimensionali y o cha ac e ize a mani old and do no ha e easy ou -o -sample
ex ensions. An in e es ing al e na i e on his ca ego y is -SNE ( an de Maa en and G. Hin on,
2008).
The p oblem wi h such echniques is ha hey exploi ixed ela ionships in o iginal dimension o he
da a o lea n, which may no be alid (W. Wang e al., 2014). As such, we belie e using an Au oen-
53
54 Chap e 7. Deploymen
Figu e 7.1: Rep esen a ion o an au oencode .
code (Rumelha e al., 1986) can be use ul due o i s lexibili y and because i has been ound o be
be e and mo e lexible han PCA and LLE (G. E. Hin on and Salakhu dino , 2006; Kagua a e al.,
2014).
An au oencode is an unsupe ised neu al ne wo k ained o lea n a comp essed ep esen a ion o
i s inpu by minimizing i s econs uc ion e o (W. Wang e al., 2014). The in ui ion behind he
unde comple e au oencode — whose code dimension is less han he inpu dimension (Good ellow,
Bengio, e al., 2016) — is ha i we educe he numbe o nodes ac oss he laye s o expand a e wa ds,
we a e o cing he neu al ne wo k o lea n he in insic s uc u es which will ha e o be p esen on he
smalle laye s o he ne wo k. Ha ing such comple e ne wo k ained wi h p ope esul s, we can spli
i o use only he encode sec ion o educe he da ase dimensionali y. Such echnique is mo e lexible
ha mani old lea ning because i also lea ns spa se, o e comple e ea u e ep esen a ions o he da a
(Ng, 2011). We ha e decided o s a wi h a simple au oencode and es i s esul s. I hose do no
mee ou needs o e icacy, we will explo e u he adap a ions and inno a ions o he echnique. A
g aphic ep esen a ion can be ound in Figu e 7.1.
The au oencode hype pa ame e s we e manually de ined due o he good e iciency o he esul s
ob ained on he ea ly implemen a ion. The layou on which we will cen e ou de elopmen is a
20-nodes-mid-laye wi h a ke nel egula ize (an ex a loss penal y on he weigh s ma ix) o L1=
1E−5 and selu ac i a ion unc ions (Klambaue e al., 2017). Simila o subse ing he o iginal
da ase o op imize he hype pa ame e s combina ions, we oo ha e subse some examples dedica ed
o ain he au oencode . The main idea behind i is o a oid aining he encode and he consequen
weigh s/a chi ec u e op imiza ion o he neu al ne wo k on he same da ase , as i would double adap
he lea ning algo i hm o he speci ic da ase noise.
We ha e es ed he in luence on he numbe o epochs, ac i a ion unc ions and he numbe o he
7.1. Dimensionali y Reduc ion 55
nodes on he mid-laye by measu ing hei combina ions’ MSE e o s on he o iginal da ase . The
main obse a ions o ou wo k a e:
•The au oencode only wo ks a e scaling he con inuous and ’so -bina izing’ he disc e e bina y
ea u es;
•The mos di use PCA esul s a e ob ained on low numbe o epochs (1, 2, 5) and low numbe
o nodes in he mid laye (5, 10);
•High numbe o epochs and numbe o nodes in he mid laye esul in PCAs ei he nea -ze o o
wi h a clea ly de ined shape;
•The e o alues s agna e app oxima ely a 50 epochs;
To gua an ee he s abili y and accu acy o he au oencode dimensionali y educ ion echnique we
ha e plo ed he lea ning cu es, i.e. ain and alida ion loss h oughou he epochs on weigh
aining, measu ing he e o on he mul iple da ase s a ailable: weigh aining, hype pa ame e s
sea ch, c oss- alida ion, encoding and es da ase . The lea ning cu es can ale us o p oblems o
high bias (unde i ing) o high a iance (o e i ing). The quali y es s a e depic ed in Table 7.1 and
Figu e 7.2.
Da ase Recons uc ing E o
Weigh T aining 0.0881
Encoding 0.0887
Hype pa ame e s Sea ch 0.0906
C oss- alida ion 0.0884
Tes ing 0.07707
Table 7.1: Mean Squa ed E o o econs uc ing he mul iple da ase s wi h an Au oencode .
F om Table 7.1, we obse e a simila e o alues o he aining da ase s (Au oencode T aining,
Weigh T aining, Hype pa ame e s Sea ch and C oss- alida ion), so he e is no p oblem o high a i-
ance. The g aph shows an e olu ion owa ds a low aining e o , which signals a su icien capaci y
56 Chap e 7. Deploymen
Figu e 7.2: Lea ning Cu e s o he Au oencode aining.
o he au oencode , along wi h a simila ly low es e o which signals no o e i ing is occu ing. The
applica ion o such echnique is hus alida ed by he obse ance o low bias (low unde i ing) and
low a iance (low o e i ing).
7.2 Hype pa ame e s Op imiza ion S a egies
7.2.1 In oduc ion
Mos machine lea ning algo i hms ha e se ings ha mus be de ined ex e nally, ou side he lea ning
en i onmen (Good ellow, Bengio, e al., 2016). These a e called hype pa ame e s and hey can ha e
a deep impac on he algo i hm pe o mance. Conside ing a majo ad an age o neu al ne wo ks is
hei modula design (Janocha and Cza necki, 2017), he way we sea ch he lexible pa ame e s is
highly ele an . Despi e being de ined ex e nally, we can in e nalize he hype pa ame e s combina-
ion op imiza ion wi hin a lea ning p ocess i sel by applying di e en schemes o sea ches o hose
combina ions. These include he Manual, G id, Random and o ien ed sea ches desc ibed below.
Bo h g id and manual sea ches a e widely used s a egies o hype -pa ame e op imiza ion (Be gs a
and Bengio, 2012). G id-sea ch is a nai e sea ch me hod which ies e e y combina ion possible,
which is compu a ionally expensi e. E en in he cases o a low-dimensional hype space sea ch, some
o ien a ion is ad ised by combining wo g id sea ches: a b oade one o iden i y po en ial good egions
and a ine g id on he ”be e ” egion (Hsu e al., 2010). On he o he hand, he manual sea ch is
63
Accu acy
The accu acy me ic measu es he pe cen age o co ec ly p edic ed ou comes, ei he posi i e (1) o
nega i e (0). Ce e is pa ibus, he highe he accu acy he be e .
The manual sea ch me hod is dominan on he accu acy esul s, ha ing an highe minimum and
han he emainde maximum alues and a mo e concen a ed dis ibu ion o esul s. Be ween he
e olu iona y and he andom sea ch me hod, hey bo h sha e he same ange o alues, app oxima ely,
wi h he gene ic sea ch ha ing sligh less s abili y.
Log Loss
The C oss En opy me ics measu es he closeness o he model’s p edic ions o he a ge . I is ele-
an o know i a ”buy/hold” decision, i.e. p edic ed a ge = 1, is due o a small deg ee o con idence
abo e andom (p edic ed p obabili y = 0.55) o due o a high con idence (close o 1). The e e se
o he ”sell/ do no buy” decision is also ue. The h eshold o a good log loss esul is below
0.6931 = −ln(0.5).
The bes Log Loss (o C oss En opy) esul s a e ound by he manual sea ch me hod, along wi h he
bes s abili y (despi e ha ing a poo ou lie ). Be ween he E olu iona y and he Random sea ch me h-
ods, he second p esen s highe s abili y and be e esul s, bu bo h he au oma ic sea ch me hods
p esen se e al nega i e esul s.
Sensi i i y
The Sensi i i y (o Recall) me ic measu es he pe cen age o he ue buy decisions (posi i e scena ios)
co ec ly p edic ed, i.e., om he ue oppo uni ies, how much did we p edic ed co ec ly.
The bes scena io is ound by he E olu iona y sea ch me hod, ollowed by he Random. Despi e
ha ing he bes esul s, hei dispe sion o esul s is qui e high. The manual p esen s lowe alued
esul s bu wi h highe s abili y.
I is impo an o no e he sho comings o using his me ic isola ed. A classi ie who only p edic s
posi i e scena ios will ha e he highes alue o sensi i i y, despi e no p o iding use ul insigh s. Thus,
we belie e his me ic should be analised in ela ion o i s equi alen o he nega i e examples —
speci ici y — which we will u he discuss combining bo h in he app op ia e me ics: AUC, Youden’s
Index, Likelihoods and Disc iminan Powe .
64 Chap e 8. Resul s
Figu e 8.1: Accu acy, LogLoss and Sensi i i y box plo s.
Speci ici y
The speci ici y measu es he pe cen age o he ue nega i e esul s co ec ly p edic ed. As wi h he
sensi i i y me ic, his measu e can be misleading i we ake i in o conside a ion alone.
Bo h Random and E olu iona y sea ch me hods ha e high dispe sion o esul s, sligh ly highe o he
second me hod, bu bo h wi h e y good and e y bad esul s. On he con a y, he manual sea ch
p o ides a low dispe sion wi hin a good egion o alues.
False Posi i e Ra e
The alse posi i e a e shows he w ongly posi i e p edic ed examples, om all he ue nega i e. The
desi able beha iou o his me ic is ha ing he lowes possible e o . None heless, his can be a
misleading measu e o pe o mance — a model which only p edic s 0’s has a alse posi i e a e o 0
bu i is no o in e es . As such, only he s abili y can be analised and he me ic’s alue mus be
aken in o accoun along wi h o he pe o mance me ics.
The manual sea ch p esen s he highes s abili y, wi h he andom coming in second and he e olu-
iona y sea ch wi h he wo s (highes ) sp ead o esul s. The di ec ion p o ided by he gene ic-based
sea ch me hod does no p o ide a isible ad an age on s abili y o he alse posi i e a e me ic.
P ecision
The p ecision measu es he co ec ly p edic ed examples om all he posi i e p edic ed samples.
The bes esul is ound by he e olu iona y sea ch me hod, which seems o be an ou lie . The manual
sea ch p o ides a ela i e low dispe sion wi hin a good egion o esul s. The andom sea ch has simila
esul s o he e olu iona y, bo h in alue and in dispe sion.
65
Figu e 8.2: Speci ici y, False Posi i e Ra e and P ecision box plo s.
Balanced A ea Unde Cu e
The balanced Accu acy (app oxima ed A ea Unde Cu e) is he a e age be ween he speci ici y and
he sensi i i y me ics. To achie e in e es ing esul s, he alues mus be abo e he 50% h eshold —
his is he alue a model which p edic s he same a ge e e y ime achie es (e.g. a model ha ne e
p edic s a posi i e scena io).
The o e all bes esul s a e ound by he manual sea ch, wi h a easonable s abili y. Be ween he
e olu iona y and he andom, he i s has an equi alen pe o mance o esul s wi h lowe dispe sion
(highe s abili y), so we belie e i o be p e e able. None heless, he wo a e bo h e y nea , and
some imes below, he h eshold.
Youden’s Index
The Youden’s Index o mula is J=sensi i i y +speci iciy −1, wi h he las ope a o o cing i s
alues o oscilla e be ween -1 and 1, since bo h me ics a e be ween [0,1]. The las ope a ion (−1)
also ans o ms he me ic o ha e alue o 0 when he p edic o is no use ul (e.g. p edic s all samples
in o he same class). Thus, a use ul p edic o p esen s a posi i e Youden’s Index and, in e e se, a
wo se- han- andom p edic o p esen s a nega i e index alue. Despi e aking bo h simple me ics
in o accoun , i doesn’ punish all 1’s o all 0’s ype p edic ions. A isualiza ion o he con ou plo is
a ailable in he appendix A.2.
The manual sea ch me hod domina es he emainde , wi h a simila s abili y and a dis ibu ion on
highe alues. The E olu iona y sea ch esul s a e simila o he Random ones, bu hey do p esen an
highe s abili y, which makes hem p e e able. Bo h au oma ic me hods p esen esul s nea h eshold.
66 Chap e 8. Resul s
Disc iminan Powe
The Disc iminan Powe me ic is a ze o-cen ic me ic (i.e., a andom classi ie has DP = 0) which
punishes p edic ions who do no p esen good esul s on he sensi i i y and speci ici y me ics, simul-
aneously. A isualiza ion o he con ou plo is a ailable in he appendix A.2.
The manual sea ch me hod p esen s he be e esul s (highe alues) and a be e s abili y (lowe
dispe sion) han he emaining me hods. The E olu iona y ha e sligh be e esul s and highe dis-
pe sion han he Random me hod. I is ele an o no e ha only he manual sea ch me hod has clea
esul s abo e he andom h eshold (DP = 0) and ha he o he me hods p esen some bad esul s
(DP < 0)
Figu e 8.3: AUC, Youden’s Index and Disc iminan Powe box plo s.
F-Measu e
The F-measu e akes in o accoun he P ecision and Recall (a.k.a. Sensi i i y) me ics, so ha bo h
ha e o p esen in e es ing esul s o ha e a high F-sco e. A isualiza ion o he con ou plo is
a ailable in he appendix A.2.
The Random and E olu iona y ha e simila esul s on he bes pe o mance (highes alue), bu bo h
ha e low s abili y. The manual sea ch me hod p esen s a sligh ly lowe bes esul bu wi h be e
s abili y, being hus p e e able.
P o i
The p o i me ic is he inancial gain made by he models du ing he es pe iod. Na u ally, he
highe he p o i he be e he model. The inancial gain h oughou he es ing pe iod can be ound
in Figu e 8.5.
The bes p o i is achie ed by he manual sea ch me hod, ollowed by he andom and, a las , he
67
E olu iona y. Wi h ega d o he esul s dispe sion, he manual is appea s he bes sea ch me hod,
simila s abili y o he e olu iona y bu wi hin an highe ange o alues. The Random me hod does
no only ha e a low s abili y (high a iance) o esul s, bu also p esen s se e al nega i e esul s.
The e o e, we conclude he manual sea ch me hod o p esen he bes esul o he P o i me ic,
since i displays he bes esul and i s dis ibu ion is he only wi hin posi i e alues.
Figu e 8.4: F-Sco e and P o i box plo s.
Figu e 8.5: Tes ing Pe iod Financial Gain by Sea ch Me hod.
Likelihoods
The likelihoods me ics e alua e he classi ie pe o mance on he posi i e and nega i e classes sep-
a a ely. A highe posi i e likelihood and a lowe nega i e likelihood mean be e pe o mance on
posi i e and nega i e classes, espec i ely.
Rega ding he posi i e likelihood, he e olu iona y and manual sea ch me hods p esen s he bes e-
68 Chap e 8. Resul s
sul s (highe posi i e likelihood alue), despi e he i s seems o be an ou lie . The nega i e likelihood
esul s a e a o able o he manual sea ch me hod, p esen ing he bes esul s and he lowe dispe -
sion.
O e all, he manual sea ch me hod has supe io pe o mance han he Random and E olu iona y
me hods on he likelihood me ics.
Figu e 8.6: Posi i e and Nega i e Likelihoods box plo s.
T ue Posi i e s False Posi i e
By plo ing he esul s o he models’ False Posi i e Ra e and T ue Posi i e (Sensi i i y) a es, along
he x- and y-axis, espec i ely, we can g aphically obse e and compa e he models’ capabili y o
p edic ion wi h he scope o such me ics. Fo his compa ison we used he esul s om he Manual,
E olu iona y and Random sea ch me hods, along wi h a Logis ic Reg ession classi ie o se e as
baseline.
The bes possible classi ie p edic s co ec ly all o he ’posi i e’ scena ios (y= 1) and does no make
commi e o s on p edic ing ’nega i e’ scena ios (y= 0), which loca es himsel in he op-le co ne o
he g aph. The wo s case classi ie does no co ec ly classi y one ’posi i e’ scena io and mis akenly
p edic s all he ’nega i e’, which g a i a es i s esul s owa ds he bo om- igh co ne o he g aph.
I a classi ie is pu ely andom, i will e en ually g a i a e owa ds he diagonal do ed line which
connec s he bo om-le o he op- igh co ne s o he plo . In his linea and simple way, we can
condense he o e all esul o he models: he close o he op-le co ne , he be e .
By obse ing he Figu e 8.7, we can no e an absence o he numbe o expec ed da a poin s o he
E olu iona y me hod, as we only see 13 o he expec ed 15. This is due o an o e lap be ween esul s
loca ed on he lowe -le co ne , speci ically wi h p a e = 0 and p a e = 0.
F om he g aph we obse e ha only he manual sea ch me hod esul s a e clea ly dis an om
he diagonal andom line and ha hey a e he closes o he op-le co ne . While some o he
E olu iona y and Random sea ch me hods esul s a e be e han a andom classi ie , hey a e no
69
s able enough o p esen clea ly good esul s. Also, he baseline logis ic eg ession p esen s a bad
esul , being loca ed below he diagonal line.
The e o e, we conclude o he dominance o he Manual in ela ion o he E olu iona y and Random
sea ch me hods, in hei esul s on he combina ion o T ue Posi i e (a.k.a. Sensi i i y, Recall) and
False Posi i e a es me ics.
Figu e 8.7: T ue Posi i e s False Posi i e Resul s pe Sea ch Me hod.
Chap e 9
Conclusions
9.1 Resea ch Answe s
The esul s abo e de ailed p esen su icien e idence o answe he ini ial esea ch ques ions o which
we p opose o answe :
H1. A e he E olu iona y and Random sea ch p ocedu es e icien , in hei ime-cos o applica ion?
H2. Does he di ec ion p o ided by he E olu iona y Sea ch p ocedu e p o ides an ad an age o e
a pu e Random Sea ch p ocedu e?
F om he 13 me ics discussed on he chap e 8, he manual sea ch me hod is p e e able on he mos
majo i y (11) han he emaining me hods. O e all, he manual me hod p esen s he bes esul s (i
no , hey a e close) and he lowes dispe sion/highe concen a ion o esul s, which indica e highe
s abili y o he me ics esul s. We hus conclude he dominance o he manual sea ch me hod in
ela ion o he e olu iona y and andom me hods, in he con ex o op imizing a la ge numbe o
hype pa ame e s. Conside ing bo h au oma ic sea ch me hods ha e a high ime cos o applica ion,
as p e iously discussed on sec ion 7.2, we conclude such me hods a e no no e icien on op imizing
complex hype pa ame e s combina ions hype spaces. Rega ding he bene i s gi en by he sea ch
o ien a ion p o ided by he e olu iona y sea ch me hod, we ha e no ound a dominance o he gene ic-
based app oach in compa ison o a pu e andom sea ch me hod. By obse ing he me ics esul s
abo e, we ace simila esul s on bo h me hods, wi h no signi ican di e ences. The e o e, we conclude
ha he e olu iona y sea ch me hod p o ides no ad an age o e a pu e andom sea ch me hod in he
70
9.1. Resea ch Answe s 71
con ex o a la ge numbe o hype pa ame e s op imiza ion and a limi ed compu ing capabili y.
I is impo an o no e ha he dominance o he manual sea ch me hod may no be alid o op imizing
a small combina ion o pa ame e s. E e y hing else de ined, we belie e he andom and e olu iona y
sea ch me hods a e ele an o op imize mo e speci ic combina ions o pa ame e s, e.g. only he
numbe o hidden laye s and ac i a ion unc ions. This is also applicable o he non-dominance o he
e olu iona y o e he andom — in he con ex o a smalle hype space o pa ame e s, he di ec ion
gi en by he gene ic-based app oach may p o ide in e es ing esul s o e a pu e andom.
Conside ing we a e op imizing p ac ically all he hype pa ame e s a ailable in he neu al ne wo k,
he inc ease in he lea ning algo i hm complexi y makes i p one o o e i ing he da a, as discussed
p e iously on chap e 2. Such endency mus be coun e ac ed wi h signi ican egula iza ion and/o
aking in o accoun mul iple me ics a he same ime du ing aining, o gua an ee gene aliza ion
capaci y. Despi e es ima ing he gene aliza ion capaci y h ough c oss- alida ion, he au oma ic sea ch
me hods only conside one me ic a a ime which will decide e en ually decide he ’bes ’ model. This
sho sigh edness o he e olu iona y and andom me hods e lec s poo ly on he esul s ob ained abo e,
as we obse ed be e gene aliza ion esul s o he manual sea ch me hod which allows aking in o
conside a ion mul iple me ics and isualiza ion echniques du ing aining. Ou esul s ul ima ely
de end human in e ac ion and he alue o accumula ed knowledge o op imizing a lea ning algo i hm
wi h la ge numbe o hype pa ame e s. I is ele an o no e he possibili y ha all he accumula ed
knowledge ga he ed o p oduce his documen was e en ually bene icial o he good esul s p o ided
by he manual implemen a ion.
A possible eason o he ac ha he ob ained esul s do no indica e a dominance o he E olu iona y
o e he Random me hod is ha he GA condi ions may no be p ope ly adequa e o assess i s
pe o mance. Bo h he numbe o gene a ions and he size o hei popula ion may be oo small
o obse e a p ope e olu ion in he esul s. Also, he size o he ou namen may also be e y
high ( eaching 75% o he popula ion in some cases). This may cause he gene ic sea ch me hod o
unsui ably sea ch he pa ame e s’ hype space. Such possibili y is pa amoun o ake in o conside a ion
he esul s ob ained in he con ex o a limi ed compu ing capabili y.
A po en ial ad an age o he manual sea ch me hod is he human awa eness o he s a e-o - he-a
echniques (e.g. Ba ch No maliza ion, D opou and so o h), which a e heo e ically expec ed o
deli e be e esul s o he algo i hm. As such, he manual sea ch can be ini ia ed on such expec ably
mo e sui able hype pa ame e s, con a ily o he andomly-ini ia ed au oma ic sea ch me hods which
may be ini ia ed a om hei op imal and, in a limi ed capaci y con ex , hey migh no ha e he
72 Chap e 9. Conclusions
condi ions o e ec i ely sea ch he hype space.
I is ele an o no e ha ou conclusions o a dominance o he manual sea ch me hod agains andom
and e olu ion algo i hms a e opposed o he common li e a u e. This may be due o he cons ain s
and con ex we ha e de eloped ou wo k upon.
9.2 Summa y o Thesis Achie emen s
We ound he S a e o he A echniques, o iginally discussed on chap e 3, o be use ul and ele an
o he deploymen o ou applica ion. The egula izing p ope ies o he p esen ed echniques —
D opou , Ba ch No maliza ion and SNNs — in pa allel wi h he a ailable p og amming capabili ies,
we e success ully implemen ed h oughou he p esen s udy wi h ela i e ease and speed.
Some o he good esul s ob ained gi e easibili y o he p o i abili y o a Bond ading algo i hm,
based on a machine lea ning algo i hm. We hus conclude ha , wi hou any p i ileged in o ma ion
besides ma ke da a, one can ake ull ad an age o he a ailable da a o co ec ly p edic , o some
ex en , he u u e beha io o such inancial secu i ies.
The au oencode dimensionali y educ ion echnique e ealed o be use ul o accele a ing he aining
and hype pa ame e s op imiza ion sea ch p ocesses, while main aining su icien da a knowledge which
allowed he classi ica ion lea ning algo i hm o e ec i ely de ine he decision bounda ies wi h good
esul s on he ou -o -sample da ase . We hus conclude on he alidi y and use ulness o cons uc ing
a neu al ne wo k o unsupe ised da a dimensionali y educ ion.
The complex e o hype space aced by mul i-pa ame e ed neu al ne wo ks p o ide dange ous subop-
imal solu ions aps which sho sigh ed op imiza ion echniques, ha only conside one me ic a a
ime in ou con ex , may all in o. Such ulne abili y p o ides an in e es ing oppo uni y o a manual
sea ch, which can ake a he same mul iple me ics and isualiza ion echniques in o accoun .
Despi e he absence o ea u e enginee ing, he neu al ne wo ks lea ning algo i hm e ealed ap i ude
o cap u e he in insic ela ionships unde lying he da a. The posi i e esul s ob ained con i m
he algo i hm’s complex capaci y and demons a e i s po en ial as bo h a dimensionali y educ ion
echnique and a classi ie .
A main acknowledgmen o ou wo k, i no he mos ele an , is he impo ance o human supe i-
sion and guidance h oughou he implemen a ion o a machine lea ning model. A simple choice o
BIBLIOGRAPHY 79
on he causa ion and p edic ion challenge a wcci 2008 (Vol. 3, pp. 53–64). P oceedings o
Machine Lea ning Resea ch. Hong Kong: PMLR. Re ie ed om h p://p oceedings.ml .p ess/
3/chang08a.h ml
Cholle , F. e al. (2015). Ke as. h ps://gi hub.com/ cholle /ke as. Gi Hub.
Cox, D. R. (1958). The eg ession analysis o bina y sequences (wi h discussion). Jou nal o he Royal
S a is ical Socie y. B (Me hodological), 20, 215–242.
Deb, K., P a ap, A., Aga wal, S., & Meya i an, T. (2002, Ap il). A as and eli is mul iobjec i e
gene ic algo i hm: nsga-ii. IEEE T ansac ions on E olu iona y Compu a ion,6(2), 182–197.
doi:10.1109/4235.996017
Ding, D., Fu, Q., & So, J. (2012). P icing callable bonds based on mon e ca lo simula ion echniques.
Technology and In es men ,3, 121–125.
Domingos, P. (2012, Oc obe ). A ew use ul hings o know abou machine lea ning. Commun. ACM,
55(10), 78–87. doi:10.1145/2347736.2347755
Dugas, C., Bengio, Y., B´elisle, F., Nadeau, C., & Ga cia, R. (2000). Inco po a ing second-o de unc-
ional knowledge o be e op ion p icing. In P oceedings o he 13 h in e na ional con e ence
on neu al in o ma ion p ocessing sys ems (pp. 451–457). NIPS’00. Den e , CO: MIT P ess. Re-
ie ed om h p://dl.acm.o g/ci a ion.c m?id=3008751.3008817
Fawce , T. (2006). An in oduc ion o oc analysis. Pa e n Recogni ion Le e s,27(8), 861–874. ROC
Analysis in Pa e n Recogni ion. doi:h ps://doi.o g/10.1016/j.pa ec.2005.10.010
Fayyad, U., Pia e sky-Shapi o, G., & Smy h, P. (1996). F om da a mining o knowledge disco e y in
da abases. AI magazine,17(3), 37.
Fi ko -No is, E., Vahid, S., & Hand, C. (2012). E alua ing he impac o ca ego ical da a encoding
and scaling on neu al ne wo k classi ica ion pe o mance: he case o epea consump ion o
iden ical cul u al goods. In C. Jayne, S. Yue, & L. Iliadis (Eds.), Enginee ing applica ions o
neu al ne wo ks: 13 h in e na ional con e ence, eann 2012, london, uk, sep embe 20-23, 2012.
p oceedings (pp. 343–352). Be lin, Heidelbe g: Sp inge Be lin Heidelbe g. doi:10.1007/978-3-
642-32909-8 35
Fonseca, F. (2017a). Abno mal ac i a ions. h ps://gi hub.com/U banoFonseca/abno mal ac i a ions.
Gi Hub.
Fonseca, F. (2017b). Con usion ma ix c oss alida ion. h ps://gi hub.com/U banoFonseca/con usion
ma ix c . Gi Hub.
80 BIBLIOGRAPHY
Fonseca, F. (2017c). No maliza o : py hon package o no maliza ion o con inuous a iables. h ps:
//gi hub.com/U banoFonseca/No maliza o . Gi Hub.
Fonseca, P. (1995, July). Neu al ne wo ks: a su ey.
Fo in, F., De Rain ille, F., Ga dne , M., Pa izeau, M., & Gagn´e, C. (2012, July). DEAP: e olu iona y
algo i hms made easy. Jou nal o Machine Lea ning Resea ch,13, 2171–2175.
Genue , R., Poggi, J.-M., & Tuleau-Malo , C. (2010, Oc obe ). Va iable selec ion using andom o es s.
Pa e n Recogn. Le . 31(14), 2225–2236. doi:10.1016/j.pa ec.2010.03.014
Geu s, P., E ns , D., & Wehenkel, L. (2006, Ap il). Ex emely andomized ees. Machine Lea ning,
63(1), 3–42. doi:10.1007/s10994-006-6226-1
Glo o , X. & Bengio, Y. (2010, May). Unde s anding he di icul y o aining deep eed o wa d neu al
ne wo ks. In Y. W. Teh & M. Ti e ing on (Eds.), P oceedings o he hi een h in e na ional
con e ence on a i icial in elligence and s a is ics (Vol. 9, pp. 249–256). P oceedings o Ma-
chine Lea ning Resea ch. Chia Laguna Reso , Sa dinia, I aly: PMLR. Re ie ed om h p:
//p oceedings.ml .p ess/ 9/glo o 10a.h ml
Golik, P., Doe sch, P., & Ney, H. (2013). C oss-en opy s. squa ed e o aining: a heo e ical and
expe imen al compa ison. In In e speech.
Good ellow, I. J., Bengio, Y., & Cou ille, A. (2016). Deep lea ning. h p://www.deeplea ningbook.o g.
MIT P ess.
Good ellow, I. J., Wa de-Fa ley, D., Mi za, M., Cou ille, A., & Bengio, Y. (2013, June). Maxou ne -
wo ks. In S. Dasgup a & D. McAlles e (Eds.), P oceedings o he 30 h in e na ional con e ence
on machine lea ning (Vol. 28, 3, pp. 1319–1327). P oceedings o Machine Lea ning Resea ch.
A lan a, Geo gia, USA: PMLR. Re ie ed om h p://p oceedings.ml .p ess/ 28/good ellow13.
h ml
Guyon, I. & Elissee , A. (2003, Ma ch). An in oduc ion o a iable and ea u e selec ion. J. Mach.
Lea n. Res. 3, 1157–1182. Re ie ed om h p://dl.acm.o g/ci a ion.c m?id=944919.944968
Haixiang, G., Yijing, L., Shang, J., Myngyun, G., & Yuanyue, H. (2017). Lea ning om class-imbalanced
da a: e iew o me hods and applica ions. Expe Sys ems wi h Applica ions,73, 220–239.
Hampel, F., Rousseeuw, P., Ronche i, E., & S ahel, W. (1986). Robus s a is ics: he app oach based
on in luence unc ions. Wiley.
He, K., Zhang, X., Ren, S., & Sun, J. (2015). Del ing deep in o ec i ie s: su passing human-le el
pe o mance on imagene classi ica ion. CoRR,abs/1502.01852. a Xi : 1502.01852. Re ie ed
om h p://a xi .o g/abs/1502.01852
BIBLIOGRAPHY 81
Hea on, J. (2017). An empi ical analysis o ea u e enginee ing o p edic i e modeling. Compu ing
Resea ch Reposi o y,abs/1701.07852. Re ie ed om h p://a xi .o g/abs/1701.07852
He z, A. & Koble , D. (2000). A amewo k o he desc ip ion o e olu iona y algo i hms. Eu opean
Jou nal o Ope a ional Resea ch,126(1), 1–12. doi:h ps://doi.o g/10.1016/S0377-2217(99)
00435-X
Hin on, G. E. & Salakhu dino , R. R. (2006). Reducing he dimensionali y o da a wi h neu al
ne wo ks. Science,313(5786), 504–507. doi:10.1126/science.1127647. ep in : h p://science.
sciencemag.o g/con en /313/5786/504. ull.pd
Hsu, C., Chang, C., & Lin, C. (2010). A p ac ical guide o suppo ec o classi ie . Re ie ed om
h p://ci esee x.is .psu.edu/ iewdoc/summa y?doi=10.1.1.224.4115
Huang, C. & Wang, C. (2006). A ga-based ea u e selec ion and pa ame e s op imiza ion o suppo
ec o machines. Expe Sys ems wi h Applica ions,31, 231–240.
Hull, J. (2012). Op ions, u u es, and o he de i a i es (Eigh Edi ion). P en ice Hall.
Hun e , J. D. (2007). Ma plo lib: a 2d g aphics en i onmen . Compu ing In Science & Enginee ing,
9(3), 90–95. doi:10.1109/MCSE.2007.55
Io e, S. & Szegey, C. (2015). Ba ch no maliza ion: accele a ing deep ne wo k aining by educing
in e nal co a iance shi . In P oceedings o he 32nd in e na ional con e ence on machine lea ning
(Vol. 37, pp. 448–456). P oceedings o Machine Lea ning Resea ch.
Jain, A., Nandakuma , K., & Ross, A. (2005). Sco e no maliza ion in mul imodal biome ic sys ems.
Pa e n Recogni ion,38, 2270–2285.
Janocha, K. & Cza necki, W. M. (2017). On loss unc ions o deep neu al ne wo ks in classi ica ion.
CoRR,abs/1702.05659. a Xi : 1702.05659. Re ie ed om h p://a xi .o g/abs/1702.05659
Japkowicz, N. (2000). Lea ning om imbalanced da a se s: a compa ison o a ious s a egies. In
P oceedings o he aaai’2000 wo kshop on lea ning om imbalaced da a se s (pp. 10–15). AAAI
P ess.
Jayalakshmi, T. & San hakuma an, A. (2011). S a is ical no maliza ion and back p opaga ion o
classi ica ion. In e na ional Jou nal o Compu e Theo y and Enginee ing,3(1), 89–93.
Jeni, L. A., Cohn, J. F., & De La To e, F. (2013). Facing imbalanced da a– ecommenda ions o
he use o pe o mance me ics. In P oceedings o he 2013 humaine associa ion con e ence on
a ec i e compu ing and in elligen in e ac ion (pp. 245–251). ACII ’13. Washing on, DC, USA:
IEEE Compu e Socie y. doi:10.1109/ACII.2013.47
82 BIBLIOGRAPHY
Jolli e, I. T. & Cadima, J. (2016). P incipal componen analysis: a e iew and ecen de elop-
men s. Philosophical T ansac ions o he Royal Socie y o London A: Ma hema ical, Physi-
cal and Enginee ing Sciences,374(2065). doi:10.1098 / s a.2015.0202. ep in : h p:// s a .
oyalsocie ypublishing.o g/con en /374/2065/20150202. ull.pd
Kagua a, A., Nam, K. M., & Reddy, S. (2014). A deep neu al ne wo k classi ie o diagnosing sleep
apnea om ecg da a on sma phones and small embedded sys ems. Re ie ed om h ps://doi.
o g/10.13140/2.1.4174.5448
Kim, K. (2006). A i icial neu al ne wo ks wi h e olu iona y ins ance selec ion o inancial o ecas ing.
Expe Sys ems wi h Applica ions.
Klambaue , G., Un e hine , T., A, M., & Hoch ei e , S. (2017). Sel -no malizing neu al ne wo ks.
CoRR,abs/1706.02515. a Xi : 1706.02515. Re ie ed om h p://a xi .o g/abs/1706.02515
Koha i, R. (1995). A s udy o c oss- alida ion and boo s ap o accu acy es ima ion and model
selec ion. In P oceedings o he 14 h in e na ional join con e ence on a i icial in elligence -
olume 2 (pp. 1137–1143). IJCAI’95. Mon eal, Quebec, Canada: Mo gan Kau mann Publishe s
Inc. Re ie ed om h p://dl.acm.o g/ci a ion.c m?id=1643031.1643047
K izhe sky, A., Su ske e , I., & Hin on, G. (2012). Imagene classi ica ion wi h deep con olu ional
neu al ne wo ks. In F. Pe ei a, C. J. C. Bu ges, L. Bo ou, & K. Q. Weinbe ge (Eds.), Ad ances
in neu al in o ma ion p ocessing sys ems 25 (pp. 1097–1105). Cu an Associa es, Inc. Re ie ed
om h p://pape s.nips.cc/pape /4824-imagene -classi ica ion-wi h-deep-con olu ional-neu al-
ne wo ks.pd
Lai, K. K., Yu, L., Wang, S., & Zhou, L. (2006). Neu al ne wo k me alea ning o c edi sco ing.
In D.-S. Huang, K. Li, & G. W. I win (Eds.), In elligen compu ing: in e na ional con e ence
on in elligen compu ing, icic 2006, kunming, china, augus 16-19, 2006. p oceedings, pa i
(pp. 403–408). Be lin, Heidelbe g: Sp inge Be lin Heidelbe g. doi:10.1007/11816157 47
Langley, P. (1994). Selec ion o ele an ea u es in machine lea ning. In In p oceedings o he aaai all
symposium on ele ance (pp. 140–144). AAAI P ess.
La ha, L. & Thangasamy, S. (2011). E icien app oach o no maliza ion o mul imodal biome ic
sco es. In e na ional Jou nal o Compu e Applica ions.
LeCun, Y. (1989). Gene aliza ion and ne wo k design s a egies. In R. P ei e , Z. Sch e e , F. Fogelman,
& L. S eels (Eds.), Connec ionism in pe spec i e. Else ie .
LeCun, Y., Bo ou, L., O , G., & M¨ulle , K. (1998). E icien backp op. In Sp inge (Ed.), Neu al
ne wo ks: icks o he ade (pp. 9–48).
BIBLIOGRAPHY 83
Liu, Y., S a zyk, J. A., & Zhu, Z. (2008, June). Op imized app oxima ion algo i hm in neu al ne wo ks
wi hou o e i ing. IEEE T ansac ions on Neu al Ne wo ks,19(6), 983–995. doi:10.1109/TNN.
2007.915114
Lu, C., De B aban e , J., Van Hu el, S., Ve go e, I., & Timme man, D. (2001). Using a i icial neu al
ne wo ks o p edic malignancy o o a ian umo s. In Enginee ing in medicine and biology soci-
e y, 2001. p oceedings o he 23 d annual in e na ional con e ence o he ieee (Vol. 2, pp. 1637–
1640). IEEE.
Ma ellini, L., P iaule , P., & P iaule , S. (2003). Fixed income secu i ies: Valua ion, isk managemen
and po olio s a egies. John Wiley & Sons L d.
McKinney, W. (2010). Da a s uc u es o s a is ical compu ing in py hon. In S. an de Wal & J.
Millman (Eds.), P oceedings o he 9 h py hon in science con e ence (pp. 51–56).
Moja ad, S. A., Dlay, S. S., & She be , G. V. (2011). C oss alida ion e alua ion o b eas cance
p edic ion using mul ilaye pe cep on neu al ne wo ks. (Vol. 4, pp. 576–585).
Mu phey, Y., Guo, H., & Feldkamp, L. (2004). Neu al lea ning om unbalanced da a. Applied In el-
ligence,21(2), 117–128.
Neelakan an, A., Vilnis, L., Le, Q. V., Su ske e , I., Kaise , L., Ku ach, K., & Ma ens, J. (2015).
Adding g adien noise imp o es lea ning o e y deep ne wo ks. CoRR,abs/1511.06807. a Xi :
1511.06807. Re ie ed om h p://a xi .o g/abs/1511.06807
Ng, A. (2011). Cs294a lec u e no es. Re ie ed om h ps : / / web . s an o d . edu / class / cs294a /
spa seAu oencode 2011new.pd
Njikam, A. & Zhao, H. (2016). A no el ac i a ion unc ion o mul ilaye eed- o wa d neu al ne wo ks.
Applied In elligence,45(1), 75–82. doi:10.1007/s10489-015-0744-0
Pa ascandolo, G., Hu unen, H., & Vi anen, T. (2017). Taming he wa es: sine as ac i a ion unc ion
in deep neu al ne wo ks.
Pascanu, R., Mikolo , T., & Bengio, Y. (2013). On he di icul y o aining ecu en neu al ne wo ks.
In P oceedings o he 30 h in e na ional con e ence on in e na ional con e ence on machine lea n-
ing - olume 28 (pp. III-1310–III-1318). ICML’13. A lan a, GA, USA: JMLR.o g. Re ie ed om
h p://dl.acm.o g/ci a ion.c m?id=3042817.3043083
Ped egosa, F., Va oquaux, G., G am o , A., Michel, V., Thi ion, B., G isel, O., . . . Duchesnay, E.
(2011). Sciki -lea n: machine lea ning in Py hon. Jou nal o Machine Lea ning Resea ch,12,
2825–2830.
84 BIBLIOGRAPHY
Pio owski, A. P. & Napio kowski, J. J. (2013). A compa ison o me hods o a oid o e i ing in
neu al ne wo ks aining in he case o ca chmen uno modelling. Jou nal o Hyd ology,
476(Supplemen C), 97–111. doi:h ps://doi.o g/10.1016/j.jhyd ol.2012.10.019
Po da , K., Pa dawala, T. S., & Pai, C. D. (2017, Oc obe ). A compa a i e s udy o ca ego ical
a iable encoding echniques o neu al ne wo k classi ie s. In e na ional Jou nal o Compu e
Applica ions,175(4), 7–9. doi:10.5120/ijca2017915495
P echel , L. (1998). Ea ly s opping - bu when? In G. B. O & K. M¨ulle (Eds.), Neu al ne wo ks:
icks o he ade (pp. 55–69). Be lin, Heidelbe g: Sp inge Be lin Heidelbe g. doi:10.1007/3-
540-49430-8 3
Rosenbla , F. (1958). The pe cep on: a p obabilis ic model o in o ma ion s o age and o ganiza ion
in he b ain. Psychological Re iew,65(6), 386–408.
Rude , S. (2016). An o e iew o g adien descen op imiza ion algo i hms. CoRR,abs/1609.04747.
a Xi : 1609.04747. Re ie ed om h p://a xi .o g/abs/1609.04747
Rumelha , D. E., Hin on, G. E., & Williams, R. J. (1986). Pa allel dis ibu ed p ocessing: explo-
a ions in he mic os uc u e o cogni ion, ol. 1. In D. E. Rumelha , J. L. McClelland, & C.
PDP Resea ch G oup (Eds.), (Chap. Lea ning In e nal Rep esen a ions by E o P opaga ion,
pp. 318–362). Camb idge, MA, USA: MIT P ess. Re ie ed om h p://dl.acm.o g/ci a ion.
c m?id=104279.104293
Saxe, A. M., McClelland, J. L., & Ganguli, S. (2013). Exac solu ions o he nonlinea dynamics o
lea ning in deep linea neu al ne wo ks. CoRR,abs/1312.6120. a Xi : 1312.6120. Re ie ed om
h p://a xi .o g/abs/1312.6120
Shen, S., Jiang, H., & Zhang, T. (2012). S ock ma ke o ecas ing using machine lea ning algo i hms.
Re ie ed om h p://cs229.s an o d.edu/p oj2012/ShenJiangZhang-S ockMa ke Fo ecas ingusingMachineLea ningAlgo i hms.
pd
sklea n-deap. (2017). h ps://gi hub.com/ s eca/sklea n-deap. Gi Hub.
Sokolo a, M., Japkowicz, N., & Szpakowicz, S. (2006). Beyond accu acy, -sco e and oc: a amily
o disc iminan measu es o pe o mance e alua ion. In A. Sa a & B.-h. Kang (Eds.), Ai
2006: ad ances in a i icial in elligence: 19 h aus alian join con e ence on a i icial in elligence,
hoba , aus alia, decembe 4-8, 2006. p oceedings (pp. 1015–1021). Be lin, Heidelbe g: Sp inge
Be lin Heidelbe g. doi:10.1007/11941439 114
BIBLIOGRAPHY 85
S i as a a, N., Hin on, G., K izhe sky, A., Su ske e , I., & Salakhu dino , R. (2014). D opou : a
simple way o p e en neu al ne wo ks om o e i ing. Jou nal o Machine Lea ning Resea ch,
(15), 1929–1958.
S anley, K. O. & Miikkulainen, R. (2002). E ol ing neu al ne wo ks h ough augmen ing opologies.
E olu iona y Compu a ion,10(2), 99–127. Re ie ed om h p://nn.cs.u exas.edu/?s anley:ec02
Tahi , M., Ki le , J., Mikolajczyk, K., & Yan, F. (2009). A mul iple expe app oach o he class
imbalance p oblem using in e se andom unde sampling. (pp. 82–91).
an de Maa en, L. & Hin on, G. (2008). Visualizing high-dimensional da a using -sne. Jou nal o
Machine Lea ning Resea ch,9, 2579–2605.
an Laa ho en, T. (2017). L2 egula iza ion e sus ba ch and weigh no maliza ion. CoRR,abs/1706.05350.
a Xi : 1706.05350. Re ie ed om h p://a xi .o g/abs/1706.05350
Wage , S., Wang, S., & Liang, P. S. (2013). D opou aining as adap i e egula iza ion. In C. J. C.
Bu ges, L. Bo ou, M. Welling, Z. Ghah amani, & K. Q. Weinbe ge (Eds.), Ad ances in neu al
in o ma ion p ocessing sys ems 26 (pp. 351–359). Cu an Associa es, Inc. Re ie ed om h p:
//pape s.nips.cc/pape /4882-d opou - aining-as-adap i e- egula iza ion.pd
Wal , S., Colbe , S. C., & Va oquaux, G. (2011, Ma ch). The numpy a ay: a s uc u e o e icien
nume ical compu a ion. Compu ing in Science and Engg. 13(2), 22–30. doi:10.1109/MCSE.2011.
37
Wang, W., Huang, Y., Wang, Y., & Wang, L. (2014). Gene alized au oencode : a neu al ne wo k
amewo k o dimensionali y educ ion. In P oceedings o he 2014 ieee con e ence on compu e
ision and pa e n ecogni ion wo kshops (pp. 496–503). CVPRW ’14. Washing on, DC, USA:
IEEE Compu e Socie y. doi:10.1109/CVPRW.2014.79
Xu, B., Huang, R., & Li, M. (2016). Re ise sa u a ed ac i a ion unc ions. CoRR,abs/1602.05980.
a Xi : 1602.05980. Re ie ed om h p://a xi .o g/abs/1602.05980
Yao, X. (1993). A e iew o e olu iona y a i icial neu al ne wo ks. In e na ional Jou nal o In elligen
Sys ems,4, 539–567.
Yao, X. (1999). E ol ing a i icial neu al ne wo ks. P oceedings o he IEEE,87(9), 1423–1447.
Youden, W. J. (1950). Index o a ing diagnos ic es s. Cance ,3(1), 32–35. doi:10 . 1002/1097-
0142(1950)3:1h32::AID-CNCR2820030106i3.0.CO;2-3
Zhang, C., Bengio, S., Ha d , M., Rech , B., & Vinyals, O. (2016). Unde s anding deep lea ning
equi es e hinking gene aliza ion. CoRR,abs/1611.03530. a Xi : 1611.03530. Re ie ed om
h p://a xi .o g/abs/1611.03530
86 BIBLIOGRAPHY
Zu , R. M., Jiang, Y., Pesce, L. L., & D ukke , K. (2009). Noise injec ion o aining a i icial neu al
ne wo ks: a compa ison wi h weigh decay and ea ly s opping. Medical Physics,36(10), 4810–
4818. doi:10.1118/1.3213517
