MACHINE LEARNING
FOR SURVIVAL PREDICTION
IN BREAST CANCER
BY
LEONARDO VANNESCHI
NOVA In o ma ion Managemen School (NOVA IMS)
Uni e sidade No a de Lisboa
Campus de Campolide, 1070-312 Lisboa, Po ugal
Ti le
Machine Lea ning o Su i al P edic ion in B eas Cance
Au ho
Leona do Vanneschi
Publishe
Ins i u o Supe io de Es a ís ica e Ges ão de In o mação da Uni e sidade No a de Lisboa NOVA
In o ma ion Managemen School (NOVA IMS)
ISBN
978-972-8093-18-1
Digi al Ve sion
h p://hdl.handle.ne /10362/110873
© Leona do Vanneschi, Ins i u o Supe io de Es a ís ica e Ges ão de In o mação da Uni e sidade No a de
Lisboa. NOVA In o ma ion Managemen School (NOVA IMS), 2021
This wo k is licensed unde a C ea i e Commons A ibu ion-NonComme cial-NoDe i a i es 4.0
In e na ional License.
This wo k is suppo ed by na ional unds h ough FCT – Fundação pa a a
Ciência e a Tecnologia, I.P., in he con ex o he p ojec BINDER (PTDC/CCI-
INF/29168/2017)
How o ci e
Vanneschi, L. (2021). Machine Lea ning o Su i al P edic ion in B eas Cance . Lisboa: Ins i u o
Supe io de Es a ís ica e Ges ão de In o mação da Uni e sidade No a de Lisboa. NOVA In o ma ion
Managemen School (NOVA IMS). ISBN: 978-972-8093-18-1. Link: h p://hdl.handle.ne /10362/110873
Con en s
1 In oduc ion ................................................... 1
2 Machine Lea ning.............................................. 3
2.1 Me hods o Tes Gene aliza ion Abili y . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.1 Da aSpli ing ....................................... 8
2.1.2 C oss alida ion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1.3 How o Pe o m a Fai Expe imen al S udy. . . . . . . . . . . . . . . 10
2.2 Measu es o Pe o mance o a Classi ie . . . . . . . . . . . . . . . . . . . . . . . 13
2.3 Fea u es .................................................. 18
2.3.1 Fea u e Enginee ing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3 Ma e ials and Me hods ......................................... 23
3.1 Ma e ial .................................................. 23
3.2 Me hods .................................................. 24
4 Expe imen al S udy ............................................ 27
4.1 P edic i e Powe o Machine Lea ning Me hods . . . . . . . . . . . . . . . . . 27
4.2 Compa ison wi h he Sco ing Me hod. . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.3 The Role o Fea u e Selec ion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.4 Pe o mance on Unbalanced Da ase s . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.5 Pe o mance on an Independen Da ase . . . . . . . . . . . . . . . . . . . . . . . . 29
4.6 Assessmen o Sensi i i y . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.7 Maximizing Sensi i i y in GP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
5 Conclusions and Fu u e Wo k ................................... 33
Re e ences..................................................... 37
Chap e 1
In oduc ion
Cu en cance he apies ha e se ious side e ec s: ideally ype and dosage o he
he apy should be ma ched o each indi idual pa ien based on his/he isk o e-
lapse. The e o e he classi ica ion o cance pa ien s in o isk classes is a e y ac i e
ield o esea ch, wi h di ec clinical applica ions. Un il ecen ly, pa ien classi i-
ca ion was based on a se ies o clinical and his ological pa ame e s. The ad en
o high- h oughpu echniques o measu e gene exp ession led in he las decade
o a la ge body o esea ch on gene exp ession in cance , and in pa icula on he
possibili y o using gene exp ession da a o imp o e pa ien classi ica ion. A gene
signa u e is a se o genes whose le els o exp ession can be used o p edic a biolog-
ical s a e (see [Ne ins and Po i, 2007]): in he case o cance , gene signa u es ha e
been de eloped bo h o dis inguish cance ous om non-cance ous condi ions and o
classi y cance pa ien s based on he agg essi eness o he umo , as measu ed o
example by he p obabili y o elapsing wi hin a gi en ime.
While many s udies ha e been de o ed o he iden i ica ion o gene signa u es
in a ious ypes o cance , he ques ion o he algo i hms o be used o maximize
he p edic i e powe o a gene signa u e has ecei ed less a en ion. To in es iga e
his issue sys ema ically, one o he mos es ablished gene signa u es is conside ed
in his wo k, i.e. he 70-gene signa u e o b eas cance [ an ’ Vee e al., 2002].
This wo k p oposes a compa ison o he pe o mance o ou di e en machine
lea ning algo i hms in using his signa u e o p edic he su i al o a coho o
b eas cance pa ien s. The 70-gene signa u e is a se o mic oa ay ea u es se-
lec ed in [ an ’ Vee e al., 2002] based on co ela ion wi h su i al, on which he
molecula p ognos ic es o b eas cance “MammaP in ”TM is based. While se -
e al machine lea ning algo i hms ha e been used o classi y cance samples based on
gene exp ession da a [Chu and Wang, 2005, Deb and Reddy, 2003, Deu sch, 2003,
Langdon and Bux on, 2004, Paul and Iba, 2005, Yu e al., 2007], in his a sys em-
a ic compa ison o he pe o mance o ou machine lea ning algo i hms using he
same ea u es o p edic he same classes is pe o med. In his compa ison, ea u e
selec ion is hus no explici ly pe o med as a p e-p ocessing phase be o e execu ing
he machine lea ning algo i hms. The machine lea ning algo i hms s udied he e a e
Gene ic P og amming (GP), Suppo Vec o Machines, Mul ilaye ed Pe cep ons
1

2 1 In oduc ion
and Random Fo es s. These me hods we e applied o he p oblem o using he 70-
gene signa u e o p edic he su i al o he b eas cance pa ien s included in he
NKI da ase [ an de Vij e e al., 2002]. This is conside ed one o he gold-s anda d
da ase s in he ield, and he p edic i e powe o he 70-gene signa u e on hese pa-
ien s was al eady shown in [ an de Vij e e al., 2002]. In his p elimina y s udy all
he s udied machine lea ning me hods we e used in an “ou -o - he-box” e sion, so
as o ob ain a i s e alua ion, as unbiased as possible, o he pe o mance o he
me hods.
Many di e en machine lea ning me hods [Lu and Han, 2003] ha e al eady been
applied o mic oa ay da a analysis, like k-nea es neighbo s [Michie e al., 1994],
hie a chical clus e ing [Alon e al., 1999], sel -o ganizing maps [Hsu e al., 2003],
Suppo Vec o Machines [Guyon e al., 2002, He nandez e al., 2007] o Bayesian
ne wo ks [F iedman e al., 2000]. Fu he mo e, in he las ew yea s E olu ion-
a y Algo i hms (EA) [Holland, 1975] ha e been used o sol ing bo h p oblems
o ea u e selec ion and classi ica ion in gene exp ession da a analysis. Gene ic
Algo i hms (GAs) [Goldbe g, 1989] ha e been employed o building selec o s
whe e each allele o he ep esen a ion co esponds o one gene and i s s a e
deno es whe he he gene is selec ed o no [Liu e al., 2005]. GP on he o he
hand has been shown o wo k well o ecogni ion o s uc u es in la ge da a
se s [Moo e e al., 2001]. GP was applied o mic oa ay da a o gene a e p og ams
ha eliably p edic he heal h/malignancy s a es o issue, o classi y di e en ypes
o issues. An in insic ad an age o GP is ha i au oma ically selec s a small num-
be o ea u e genes du ing he e olu ion [Rosskop e al., 2007]. The e olu ion o
classi ie s om he ini ial popula ion seamlessly in eg a es he p ocess o gene se-
lec ion and classi ie cons uc ion. In ac , in [Yu e al., 2007] GP was used on can-
ce exp ession p o iling da a o selec po en ially in o ma i e ea u e genes, build
molecula classi ie s by ma hema ical in eg a ion o hese genes and classi y umou
samples. Fu he mo e, GP has been shown a p omising app oach o disco e ing
comp ehensible ule-based classi ie s om medical da a [Boja czuk e al., 2001] as
well as gene exp ession p o iling da a [Hong and Cho, 2006]. The esul s p esen ed
in hose con ibu ions a e encou aging and pa e he way o a u he in es iga ion o
GP o his kind o da ase s, which is epo ed in his manusc ip .
Chap e 2
Machine Lea ning
Machine Lea ning (ML) [Mi chell, 1997, Shale -Shwa z and Ben-Da id, 2014] is
a ield o s udy whose objec i e is o p og am compu e s o au oma ically lea n o
sol e a p oblem, o accomplish a ask. ML is use ul when manually p og amming
a compu e o ca y ou a ask is ei he imp ac ical o in easible. Typical cases a e
ei he p oblems ha a e so complex o be beyond human capabili ies, like he ones
cha ac e ized by as amoun s o da a, o asks ha li ing beings pe o m ou inely,
ye ou in ospec ion on how we do i is no su icien ly elabo a ed o allow us o
ex ac a well de ined algo i hm, like o ins ance d i ing, speech ecogni ion, image
unde s anding o clien ca ego iza ion. O he asks whe e ML is use ul a e he ones
whe e adap a i i y o changes in he en i onmen is a necessa y equi emen , like
o ins ance ime se ies o ecas ing, handw i en ex decoding o spam de ec ion. In
i s mos accep ed de ini ion:
“Machine Lea ning is he s udy o algo i hms ha au oma ically imp o e by
means o expe ience” [Mi chell, 1997].
In his de ini ion, lea ning is in ended as imp o ing by means o expe ience. E en
hough he e m “lea ning” can ha e se e al meanings and in e p e a ions, i is clea
ha “imp o ing by means o expe ience” is one o he mos in ui i e and close o
ou e e yday expe ience. Fo ins ance, i includes he idea o “ ial and e o ”, ha
is e y o en implemen ed by many li ing beings when hey a e abou o lea n how
o sol e new asks: lea ning o en implies nume ous consecu i e a emps (o ials)
o sol ing he ask. I a ial gi es a posi i e esul , i will be ewa ded by simila
u u e ials; on he o he hand, i a ial gi es a nega i e esul , i is cus oma y o
iden i y i wi h an e oneous beha iou , and hus no epea i in he u u e a emp s.
I e a ing he p ocess, he ials should become mo e and mo e e ec i e wi h ime,
un il he ask ge s sol ed. A simple example consis s in he me hod a s use o selec
ood: when a s encoun e ood i ems wi h new look and smell, hey will i s ea
a small amoun o i . Acco ding o he la ou and he physiological e ec o he
ood, he a s will la e decide i ea ing mo e o no . I he ood p oduces an ill
e ec , ha ood will be associa ed wi h illness, and no ea en again. I i as es good
and does no p oduce any nega i e e ec on he heal h o he a , i will p obably be
3
4 2 Machine Lea ning
ea en again. Also human beings use ial and e o se e al imes o lea n asks. Fo
ins ance, when a pe son is lea ning how o play ennis, she will p obably y o hi
he ball by pe o ming pa icula mo emen s o he a ms, shoulde s and legs. Those
mo emen s will be iden i ied as e ec i e o e oneous, acco ding o he esul o he
sho , and his esul will a ec he nex a emp s o hi he ball. As a las example
o how much ial and e o is used by humans o lea ning new asks, he s uden s
ha ha e ecen ly a ended a cou se o in oduc ion o p og amming should ag ee on
how many w ong a emp s, wi h subsequen mis ake iden i ica ions and adap ions,
we e needed be o e becoming able o w i e co ec compu e p og ams.
This lea ning p ocess is wha has inspi ed he in oduc ion o he ield o ML. Bu
wha do we exac ly wan machines o lea n? E en hough i is impossible o gi e
gene al de ini ions o co e such a as ield as ML, i is possible o co e he la ge
majo i y o he si ua ions saying ha one o he mos equen objec i es o ML is
he one o lea ning a unc ion. In la ge pa o he si ua ions, ML is dealing wi h a
p oblem ha can be de ined as ollows. Gi en a se o da a pai s:
D={(x1,y1),(x2,y2),...,(xn,yn)}
he objec i e is o ind (o app oxima e) a unc ion (o ela ion) φ, such ha :
∀i=1,2,...,n:φ(xi) = yi
In he mos gene al de ini ion, xiand yican be any kind o objec (numbe s, ec-
o s, ma ices, exp essions, images, mo ies, sen ences, o he objec s om he eal
wo ld, e c.), howe e he mos ypical si ua ion is he one in which he xia e
m-dimensional ec o s o objec s o any ype (including, bu no necessaily, num-
be s), while he yia e scala alues.
Be o e ha ing a close look a he p oblem o lea ning, i is use ul o ix some
e minology:
•Dis called da ase ;
• {x1,x2,...,xn}a e called inpu da a, inpu ec o s, ins ances o obse a ions;
• {y1,y2,...,yn}a e called expec ed ou pu s, o a ge alues;
• he sough o unc ion φ, i.e. he unc ion ha pe ec ly ma ches all possible da a
in he inpu domain in o he co esponding a ge s, is called a ge unc ion;
•lea ning is a p ocess ha allows us o ob ain a unc ion ha app oxima es he
a ge unc ion φ;
• unc ion , i.e. he unc ion ob ained as a esul o he lea ning p ocess, is called
da a model, o simply model;
• inally, we will alk o supe ised lea ning in case he a ge alues {y1,y2,...,yn}
a e known o each obse a ion, and unsupe ised lea ning o he wise. This
manusc ip ocuses on supe ised lea ning gi en ha , o he used da ase s, a -
ge alues (su i al a es o cance pa ien s) a e known o a se o obse a ions
(cance pa ien s).
Las bu no leas , we will say ha model has a good gene aliza ion abili y i be-
ha es like he a ge unc ion φalso o da a ha do no belong o D. Unde s anding
2 Machine Lea ning 5
i a model has a good gene aliza ion abili y can be a ha d ask, because, o cou se,
in gene al he a ge unc ion φis no known a p io i and canno be ex apola ed
by simply looking a he da a. Ac ually, in some senses, we could e en say ha φ
is no e en an exis ing unc ion, bu mo e he concep , he logic, o he unde ly-
ing knowledge, ha allowed a gi en en i y (a pe son, a de ice, e c.) o gene a e he
da a (Example 2.2 should cla i y his). Fu he mo e, in some cases, se e al di e -
en unc ions can pe ec ly ma ch he known da a, and in such a si ua ion, deciding
which one is he a ge unc ion is impossible, unless u he da a a e p o ided. How-
e e , he concep o gene aliza ion is c ucial o ML. The ollowing examples should
cla i y he issue.
Example 2.1. (A “Toy” Nume ic Example). Le us conside he ollowing simple
nume ic da ase , whe e x1and x2a e he componen s o he bi-dimensional inpu
ec o s (inpu a iables, o ea u es) and yis he a ge :
D=
x1x2y
1 8 9
3 2 5
4 1 5
7 3 10
A ques ion a ises na u al: “Wha is he a ge unc ion?”. Any a emp o answe his
ques ion in a o mal way can only lead o he answe “I don’ know”, gi en ha one
may imagine se e al unc ions ma ching he da a in D, and no in o ma ion is gi en
on how o choose among hem. Howe e , gi en he simplici y o his example, one
could easily hypo hesize ha in his case, he a ge unc ion is he unc ion ha
sums wo numbe s, in o he wo ds: φ(x1,x2) = x1+x2.
Now, le us assume ha a ML sys em is able o ind a model like:
(x1,x2) = x1+x2
I is ob ious ha now we can apply o any pai o numbe s, and no only he
ones in D, and he esul will be he sum o hose numbe s; o ins ance (2,6) = 8.
Le us, ins ead, assume ha ou ML sys em inds a model like:
g(x1,x2) = i ((x1== 1)&(x2== 8)) hen e u n 8;
else i ((x1== 3)&(x2== 2)) hen e u n 5;
else i ((x1== 4)&(x2== 1)) hen e u n 5;
else i ((x1== 7)&(x2== 3)) hen e u n 10;
else e u n a andom alue;
(2.1)
I one looks a hese wo models and g, i is no di icul o con ince onesel
ha bo h o hem wo k pe ec ly on he da a in D, bu (s ill in he hypo hesis
ha φ(x1,x2) = x1+x2is he a ge unc ion) is a pe ec app oxima ion o φ
12 2 Machine Lea ning
Algo i hm 1: Pseudo-code o he nes ed da a spli , o (k∗`)- andom olds
c oss alida ion.
1. o i=1,2,...,kdo
1.1. pa i ion he da ase D in o a lea ning se Jiand a es se D−Ji;
1.2. o j=1,2,..,` do
1.2.1. pa i ion he lea ning se Jiin o a aining se Ui j and a alida ion se Ji−Ui j;
1.2.2. o each pa ame e se ing ha needs o be compa ed do
– ain he ML sys em on Ui j;
– collec he esul s ob ained on Ji−Ui j;
end
end
1.3. Selec he con igu a ion ci ha e u ns he bes a e age esul , calcula ed on all he
alida ion se s Ji−Ui1,Ji−Ui2, ..., Ji−Ui`;
1.4. T ain he ML sys em on he whole lea ning se Ji, using con igu a ion ci. Le ibe he
ob ained model, whose gene aliza ion abili y can be assessed using he es se D−Ji;
end
2. Agg ega e models 1, 2,..., k( o ins ance pe o ming a o ing o classi ica ion p oblems o
an a e age o eg ession), in o de o ob ain he inal model .
o p o ide an unbiased e alua ion o he models on unseen da a. The con igu a-
ion ha ob ained he bes a e age esul s on he alida ion se s is i on he en i e
lea ning se . The pe o mance o hese models is hen e alua ed using he es
se s. Analogously o he di e ence be ween he adi ional c oss alida ion and
he Mon e Ca lo c oss alida ion, he di e ence be ween he (k∗`)- old c oss al-
ida ion and he (k∗`)- old Mon e Ca lo c oss alida ion is ha in he (k∗`)- old
Mon e Ca lo c oss alida ion some obse a ions may ne e be selec ed in he es
and/o alida ion se s, whe eas o he s may be selec ed mo e han once. On he
o he hand, con a ily o he (k∗`)- old c oss alida ion, in he (k∗`)- old Mon e
Ca lo c oss alida ion he p opo ion o he di e en da a spli s is no dependen
on kand `.
•k-Fold C oss alida ion wi h Valida ion and Tes se . This me hod can be seen as
ak∗1- old c oss alida ion. The whole da ase is pa i ioned in o ksubse s. One
by one, a se is selec ed as es se . Then, one by one, one o he emaining se s
a e used as a alida ion se and he o he k−2 se s, joined, a e used as a aining
se . This is epea ed o all possible combina ions. As o he p e ious me hods,
he aining se is used o model i ing and he alida ion se is used o model
e alua ion o each o he pa ame e se ings. Finally, o he con igu a ion ha
ob ained he bes a e age esul on he alida ion se s, he es se is used o assess
he gene aliza ion abili y. Two a ian s a e possible: ei he e alua ing he model
ha was ained on he aining se o e alua ing a new model ha was i on he
combina ion o he aining and he alida ion se .

2.2 Measu es o Pe o mance o a Classi ie 13
2.2 Measu es o Pe o mance o a Classi ie
The discussion o he p e ious sec ion is gene al, in he sense ha i holds bo h o
classi ica ion and eg ession p oblems. In his sec ion, we ocus on classi ica ion.
The simples and mos popula measu e o quan i y he e o made by a classi ica-
ion model is he numbe o co ec ly classi ied ins ances. To calcula e his measu e,
we simply ha e o coun he numbe o ins ances o which he p edic ed class label
is iden ical o he class label ha appea s in he supe ised da ase . The numbe o
co ec ly classi ied ins ances is a measu e ha depends on he numbe o ins ances.
In o de o make he measu e compa able when used on da ase s o di e en sizes,
i is ypical o no malize his measu e, so as o oba in ano he measu e called accu-
acy, de ined as:
Accu acy =numbe o co ec ly classi ied ins ances
o al numbe o ins ances
Using a measu e like he numbe o co ec ly classi ied ins ances, o he accu acy,
can be no su icien o unde s and he beha io o a classi ie . O en, mo e sophis-
ica ed measu es o pe o mance a e needed o classi ie s. Fo ins ance, we may
need measu es ha exp ess he quali y o a classi ica ion o each class, and no
jus one single gene al numbe . To con ince onesel abou he impo ance o ha -
ing a measu e ha exp esses a di e en alue o each class, one may imagine, o
ins ance, a bina y classi ica ion p oblem, whe e he objec i e is o ca ego ize a se
o pa ien s in o one o he wo possible classes: heal hy o sick. I is clea ha , in
some cases, misclassi ying a sick pa ien can ha e mo e se ious consequences han
misclassi ying a heal hy pa ien . S a ing by saying ha bo h misclassi ica ions a e
mis akes, and, as such, bo h o hem ha e se ious consequences, ea ing a heal hy
pa ien as a sick one, among o he consequences, may cause he pa ien o unde ake
unnecessa y ea men s o o ge uselessly wo ied. On he o he hand, ea ing a
sick pa ien as a heal hy one may cause he pa ien o no unde ake ea men s ha
may be c ucial o sa e he li e. In such an applica ion, i is clea ha an in o ma ion
like he numbe o misclassi ica ions made by he sys em is no su icien . On wha
class hose missclassi ica ions happened is also a needed in o ma ion.
Two o he mos known and employed measu es o quan i y he pe o mance o
a classi ie on he single exis ing classes a e p ecision and ecall. Gi en a class Cin
which da a can be pa i ioned, hese measu es a e de ined as ollows:
P ecision(C) = #ins ances belonging o C, classi ied as C
#ins ances classi ied as C(2.4)
Recall(C) = #ins ances belonging o C, classi ied as C
#ins ances belonging o C(2.5)
As we can see, he measu es sha e he same nume a o , consis ing in he in e sec-
ion be ween he obse a ions labelled as Cin he gi en da ase and he obse a ions
ha he model has ca ego ized as belonging o class C(in o he wo ds, he nume -
14 2 Machine Lea ning
a o con ains he numbe o ins ances ha he classi ie has ca ego ized co ec ly
as class C). The wo measu es di e because o he denomina o : o p ecision, he
denomina o ells us abou he wo k made by he classi ie , while o ecall, i ells
us abou he g ound u h. Unde his pe spec i e, one may ha e an in ui ion on he
meaning o p ecision and ecall by compa ing hem o concep s such as co ec ness
and comple eness, espec i ely.
To ha e a be e unde s anding on how o calcula e p ecision and ecall, conside
he example shown in Figu e 2.2. In his example, h ee classes a e gi en: C1,C2
Fig. 2.2 Example used o explain he concep s o p ecision and ecall. The uppe line shows how
da a a e eally pa i ioned in o h ee classes C1,C2and C3in he gi en da ase . The lowe line
shows how a classi ica ion model has ca ego ized he same objec s in he same classes.
and C3. The uppe line o he igu e shows how 15 objec s (obse a ions) a e pa -
i ioned in o hese h ee classes in he gi en da ase . The lowe line, ins ead, shows
how hose objec s ha e been ca ego ized by a ML model. P ecision and ecall o
he di e en classes a e:
P ecision(C1) = 2
3≈0.66, Recall(C1) = 2
6≈0.33
P ecision(C2) = 4
6≈0.66, Recall(C2) = 4
5=0.8
P ecision(C3) = 4
6≈0.66, Recall(C3) = 4
4=1
Bo h p ecision and ecall a e numbe s included in [0,1], whe e 1 ep esen s he bes
alue, while 0 is he wo s one. An ideal model, i.e. he one ha classi ies each obse -
a ion co ec ly, has bo h p ecision and ecall equal o 1. Models ha ha e only one
among p ecision and ecall is equal o 1 dese e a discussion. One may be emp ed
o conside such models as good ones, bu his can be e y misleading. Conside , o
ins ance, he case o C3in he p e ious example. The ecall o C3is equal o 1 be-
2.2 Measu es o Pe o mance o a Classi ie 15
cause he classi ie has ca ego ized as C3all he ins ances ha a e eally in C3, plus
o he s. I we ex emize his si ua ion, e en a “nai e” model ha blindly ca ego izes
all exis ing obse a ions in C3can ha e a ecall equal o 1. Howe e , his model has
clea ly no lea ned any hing. A simila a gumen also holds o p ecision: any model
ha ca ego izes in a class only a subse o he objec s ha ac ually belong o ha
class is equal o 1. Bu his a gumen also holds i many objec s belong o he class
and only one is ca ego ized in i . Fo ins ance, gi en he objec s A,B,C,D,E,F ha
belong o class C1, a model ha ca ego izes objec Aas he only membe o class C1
has a p ecision equal o 1. Howe e , also in his case, he amoun o in o ma ion ha
his model has lea ned is poo . In conclusion, i does no make much sense o s udy
only one among p ecision and ecall, wi hou s udying he o he . These measu es
gi e us wo di e en pieces o in o ma ion, each o which is incomple e wi hou he
o he . Only s udying hem oge he makes sense.
O he popula measu es o pe o mance o a classi ie a e ue posi i es (TP),
ue nega i es (TN), alse posi i es (FP) and alse nega i es (FN). Gi en a class C,
hese measu es a e de ined as ollows:
•TP(C) = # ins ances belonging o C, classi ied as C
•TN(C) = # ins ances ha do no belong o C, ha ha e no been classi ied as C
•FP(C) = # ins ances ha do no belong o C, classi ied as C
•FN(C) = # ins ances belonging o C, ha ha e no been classi ied as C
When he i s wo d is ue, he measu e quan i ies a co ec beha io o he model:
ue posi i es quan i y he numbe o imes an objec o Chas been co ec ly ca e-
go ized, while ue nega i es quan i y he numbe o imes ha an objec has been
co ec ly iden i ied as no belonging o C. Analogously, when he i s wo d is alse,
he measu e quan i ies e o s o he model: alse posi i es coun he numbe o imes
he model has ca ego ized an objec in Ce oneously, while alse nega i es quan i y
he numbe o objec s ha ha e no been classi ied in C, bu hey we e supposed o.
Knowing he alues o TP, FP and FN, i is possible o immedia ely ob ain he
p ecision and he ecall. In ac , di ec ly om he de ini ion o p ecision and ecall,
we ha e ha , o each class C:
P ecision(C) = TP(C)
TP(C)+FP(C)
Recall(C) = TP(C)
TP(C)+FN(C)
Ano he impo an measu e ha joins p ecision and ecall in o one single numbe
o each class is he F-measu e (also called F-sco e o F1-sco e), de ined as:
F-measu e(C) = 2∗P ecision(C)∗Recall(C)
P ecision(C) + Recall(C)
16 2 Machine Lea ning
The F-measu e is o en p e e ed o e he accu acy in case o unbalanced da ase s.
In ac , le us conside , o ins ance, a bina y classi ica ion p oblem, i.e. a p oblem
ha consis s in ca ego izing obse a ions in o one o he wo possible classes C1
and C2. Le us assume, wi hou loss o gene ali y, ha , in ou da ase , numeous ob-
se a ions labelled wi h class C1a e a ailable, while only a negligible numbe o
obse a ions a e labelled wi h class C2. I is clea ha a “nai e” model, ha blindly
ca ego izes all obse a ions as belonging o class C1has an excellen accu acy. I he
ML sys em is guided by accu acy, as a pe o mance measu e o choose among
he candida e solu ions, i is clea ha such a model is likely o be he p e e ed
one, e en hough i has lea ned none o he in o ma ion a ailable in he da a. On
he o he hand, gi en ha such a model has poo p ecision and ecall on class C2,
i s F-measu e is also poo . In o de o ha e a good F-measu e, also in case o unbal-
anced da ase s, he ML sys em is o ced o lea n he in o ma ion ha allows us o
dis inguish be ween he di e en classes.
In some cases, i is use ul o unde s and how be e o wo se a classi ie is, com-
pa ed o a andom classi ie , whe e by andom classi ie , he e, i is mean an algo-
i hm ha , o each possible ins ance, always e u ns a andom class, picked up wi h
uni o m dis ibu ion among all he possible exis ing al e na i es. This can be quan-
i ied, o ins ance, by he K S a is ic o K measu e, ha some ML packages ha e
implemen ed, including Weka [Hall e al., 2009]. This measu e is de ined as:
K=Accu acy −P(E)
1−P(E)
whe e P(E)is he p obabili y o he andom classi ie o co ec ly classi y all ele-
men s in he conside ed da ase . I is clea ha Kge s close o he ideal alue o 1
as much as also he accu acy ge s close o i s ideal alue o 1. When s a i ying he
esul s class by class is no a equi emen , he K measu e can gi e some in e es -
ing in o ma ion, ha may be in eg a ed wi h he in o ma ion gi en by he accu acy,
and/o wi h s a is ics calcula ed o e o he measu es such as p ecision, ecall and
F-measu e.
We conclude his p esen a ion o measu es o pe o mance o classi ie s wi h he
discussion o a measu e ha is e y popula , bu can be used only o bina y classi-
ica ion, and only in case he classi ie wo ks wi h a h eshold mechanism. Imagine,
o ins ance, he wo classes o a bina y classi ica ion p oblem o be ep esen ed
by labels 0 and 1. Gi en an obse a ion, he ML model could wo k by gene a ing
a numbe x, ha is hen ans o med in o ei he 0 o 1. The ypical case is: i xis
smalle han 0.5, he e u n 0, else e u n 1. In his case, a h eshold equal o 0.5
is used. This is he ypical unc ioning, o ins ance, o supe ised a i icial neu-
al ne wo ks. In such a si ua ion, he pe o mance o he model can be ep esen ed
by a plo , called Recei e Ope a ing Cha ac e is ic (ROC) cu e. The plo is c e-
a ed by epo ing he alues o he ue posi i e a e (TPR) agains he alse posi i e
a e (FPR) o a ious di e en alues o he h eshold.
TPR and FPR a e de ined as ollows:
2.2 Measu es o Pe o mance o a Classi ie 17
TPR =T P
T P +FN ,FPR =FP
FP +T N
TPR is iden ical o ecall, and i is also known as sensi i i y o p obabili y o de ec-
ion. FPR is also known as all-ou o p obabili y o alse ala m. In some e e ences,
i is also possible o ind he e m speci ici y, whe e:
speci ici y =1−FPR
Le us conside he equen case in which he ou pu o he model is a numbe
in [0,1]. In his case, o combine he FPR and he TPR in o a single me ic, we i s
compu e he wo o me measu es wi h a se o di e en h eshold alues (like, o
ins ance, 0.00,0.01,0.02,...,1.00). Then we plo hem on a single g aph, wi h he
FPR alues on he abscissa and he TPR alues on he o dina e. The esul ing cu e
is he ROC cu e, and he me ic we conside is he a ea unde he cu e (AUC),
also called AUROC. An example o a ROC cu e is epo ed in Figu e 2.3. In his
Fig. 2.3 Example o a ROC cu e. The AUROC is ep esen ed in ligh blue.
igu e, he blue a ea co esponds o he AUROC. The dashed line is he diagonal; i
ep esen s he ROC cu e o a andom p edic o , and i has an AUROC equal o 0.5.
The andom p edic o is commonly used as a baseline o compa e wi h he model.
The alue o he AUROC is always included in [0,1]. The bes possible p edic ion
me hod would yield a poin in he uppe le co ne (coo dina e (0,1)) o he ROC
space, ep esen ing 100% speci ici y (i.e. no alse posi i es). The poin (0,1)is also
called a pe ec classi ica ion. A comple ely andom guess would gi e a poin along
he diagonal ( he so-called line o no-disc imina ion) om he le bo om o he op
igh co ne .

18 2 Machine Lea ning
2.3 Fea u es
A ea u e is a cha ac e is ic o he objec s ha ha e o be classi ied o , mo e gene -
ally, o which a p edic ion is needed. So, da ase s a e usually a collec ion o alues
(o ins ances) o ea u es. Gi en a da ase o he o m:
D=




x11 x12 ... x1my1
x21 x22 ... x2my2
... ... ... ... ...
xn1xn2... xnm yn




We no mally use he ollowing e mnilogy:
•Fo each j=1,2,...,m, column {x1j,x2j,...,xn j} ep esen s a ea u e, and o
each i=1,2,...,n, elemen xi j is a ea u e alue, o ea u e ins ance.
•Fo each i=1,2,...,n, line {xi1,xi2,...,xim}is a da ase ins ance, obse a ion o
sample, and yiis he co esponding a ge alue.
In case o classi ica ion, ea u es a e app op ia e o use ul i hey allow us o make
a di e ence be ween one class (o mo e) and he o he s. This is why an app op ia e
choice o he ea u es is o en c ucial in supe ised ML. Le us conside , o ins ance,
he ollowing oy da ase , whose objec i e is o classi iy animals in o oos e s and
dogs:
# paws # eyes ha ing a c es body a blood p essu e a ge
animal 1 2 2 T ue 7% 97 Roos e
animal 2 4 2 False 18% 118 Dog
animal 3 4 2 False 22% 126 Dog
animal 4 2 2 T ue 10% 101 Roos e
This da ase con ains ou obse a ions, each one ep esen ing a di e en animal.
Each animal is ep esen ed by i e ea u es: numbe o paws and numbe o eyes,
which a e in ege numbe s, ha ing o no ha ing he c es , a Boolean alue, body
ay a e, which is a pe cen age, and blood p essu e, which is a loa ing poin numbe .
Obse ing his da ase , we can immedia ely no ice ha :
•Numbe o paws and ha ing/no ha ing he c es a e good ea u es: hey clea ly
allows us o ell dogs om oos e s.
•numbe o eyes is a o ally useless ea u e: i s alue is he same o bo h classes,
and so he ea u e is cons an in he whole da ase .
•Body a a e and blood p essu e may help o make he classi ica ion, bu i we
use hese wo ea u es, he classi ica ion may be ha de han i we simply use one
among numbe o paws and ha ing/no ha ing he c es .
2.3 Fea u es 19
Examples o models ha allow us o make a pe ec classi ica ion o each ins ance
in he da ase a e:
i (ha ing c es ) hen Roos e else Dog
o :
i (numbe o paws == 2)
hen Roos e
else i (numbe o paws == 4)
hen Dog
Bo h hese models use a es ic ed numbe o ea u es, compa ed o he o al numbe
o ea u es ha appea in he da ase . Remo ing se e al ea u es om he da ase ,
possibly lea ing only numbe o paws and/o ha ing/no ha ing a c es , may signi -
ican ly help he wo k o a classi ie . The p esence o useless ea u es, o o ea u es
which make he classi ica ion ha de , in ac , inc emen s he sea ch space and makes
he model’s op imiza ion ha de .
2.3.1 Fea u e Enginee ing
Fea u e selec ion is he p ocess o choosing he ea u es ha a e use ul o make he
p edic ion, dis ega ding all he o he s. I is o en a e y ha d and complex ask, and
in can, in p inciple, be based on p e ious knowledge o he p oblem, o on ma he-
ma ical ela ionships be ween da a. Fea u e ex ac ion is he p ocess o combining
one o mo e exis ing ea u es o c ea e a smalle numbe o mo e insigh ul ea u es.
Con a ily o ea u e selec ion, in ea u e ex ac ion ea u es a e ypically no chosen
o dis ega ded, bu only combined. Fea u e selec ion and ea u e ex ac ion can bo h
be used, o only one o hem can be used. Reducing he dimensionali y o he ea u e
space can be a c ucial ask o imp o e he gene aliza ion abili y o a ML sys em, so
choosing o c ea ing he app op ia e ea u es is a undamen al s ep om which he
pe o mance o he whole sys em can depend. They a e usually applied be o e be-
ginning he lea ning p ocess, and o his eason, hey a e usually in eg a ed in a so
called da a p ep ocessing phase, a phase ha usually con ains also a s ep o da a
cleaning, aimed a emo ing mis akes, impe ec ion o noise om he da a.
Mode n da ase s ha e hund eds o ens o housands o a iables o ea u es.
Fea u e selec ion and ea u e ex ac ion ha e h ee main objec i es:
•imp o ing he p edic ion pe o mance o he models,
•p o iding as e and mo e cos -e ec i e p edic o s, and
•p o iding a be e unde s anding o he unde lying p ocess ha gene a ed he
da a.
Besides his, he e a e many o he po en ial bene i s o ea u e selec ion/ex ac ion:
acili a ing da a isualiza ion and da a unde s anding, educing he measu emen
and s o age equi emen s, educing aining and u iliza ion imes, e c.
20 2 Machine Lea ning
Me hods o ea u e selec ion can essen ially be pa i ioned in o:
•Fil e s;
•W appe s;
•Embedded me hods.
W appe s u ilize he lea ning machine o in e es as a black box o sco e subse s o
a iable acco ding o hei p edic i e powe . Fil e s selec subse s o a iables inde-
penden ly o he chosen p edic o . Embedded me hods pe o m a iable selec ion in
he p ocess o aining and a e usually speci ic o gi en lea ning machines (Gene ic
P og amming is one o hese me hods).
The mos popula kinds o il e s (al hough by a no he only ones known) a e:
•Co ela ion based me hods;
•In o ma ion Theo y based me hods;
Bo h hese me hods ha e he objec i e o anking he ea u es acco ding o hei
“use ulnes” in helping p edic ion, so ha only he k op- anked ones can be used o
gene a ing he p edic i e model. The in ui ion is ha i a ea u e is independen om
he a ge , i is unin o ma i e o p edic ing i . O cou se, hese me hods in oduce a
new pa ame e k, ha can ha e a c ucial in luence on he pe o mance o he sys em,
and ha can only be se by means o expe imen al compa isons.
The idea o co ela ion-based ea u e selec ion is simple: calcula e he co ela ion
be ween all ea u es and he a ge , and hen ank he ea u es acco ding o his co e-
la ion alue. One o he mos known measu es is he Pea son co ela ion coe icien .
Fo a pa icula ea u e, gi en he ec o o all he ea u e alues x=x1,x2,...,xn
and he ec o o he a ge alues y=y1,y2,...,yn, he Pea son co ela ion be ween
Xand Yis:
Co =co (x,y)
p a (x) a (y)
whe e co is he co a iance o wo ec o s and a is he a iance o one ec o , so:
Co =∑n
i=1(xi−x)·(yi−y)
p∑n
i=1(xi−x)2·∑n
i=1(yi−y)2
whe e xis he a e age o he elemen s o ec o x. By de ini ion, Co is a alue
in [−1,1]. Usually, he measu e ha is used o pe o m he anking is Co 2, because
a nega i e co ela ion can be use ul (i is enough o conside he ea u e wi h a
nega i e sign in he model). One possible d awback o co ela ion c i e ia is ha
hey can only de ec linea dependencies be ween ea u es and a ge . A simple way
o li ing his es ic ion is o make a non-linea i o he a ge wi h single a iables
and ank acco ding o he goodness o ha i .
2.3 Fea u es 21
Conce ning in o ma ion heo y-based ea u e selec ion, he anking o ea u es is
done using mu ual in o ma ion be ween ea u es and he a ge :
In =∑
xi
∑
yi
P(X=xi,Y=yi)·log P(X=xi,Y=yi)
P(X=xi)·P(Y=yi)
This measu e is app op ia e in case he ea u es a e disc e e a iables. The case
o con inuous a iables (and possibly con inuous a ge s) is ha de and one can con-
side disc e izing he a iables.
Besides co ela ion and in o ma ion heo y, ano he possible measu es o ank
he ea u es is he χ2be ween ea u es and a ge s, which also aims a quan i ying
he depencence be ween ea u es and a ge .
One common c i icism o a iable anking is ha i may lead o he selec ion
o a edundan subse . The same pe o mance could possibly be achie ed wi h a
smalle subse o complemen a y a iables. S ill, one may wonde whe he adding
p esumably edundan a iables can esul in a pe o mance gain. Ac ually, i is an
expe imen al ac ha , in classi ica ion, be e class sepa a ion may be ob ained by
adding a iables ha a e p esumably edundan . Mo e p ecisely, pe ec ly co ela ed
a iables a e uly edundan in he sense ha no addi ional in o ma ion is gained by
adding hem; bu e y high a iable co ela ion (o an i-co ela ion) does no mean
absence o a iable complemen a i y. Fu he mo e, expe imen al e idence ells us
ha a a iable ha is comple ely useless by i sel can p o ide a signi ican pe o -
mance imp o emen when aken wi h o he s, and wo a iables ha a e useless by
hemsel es can be use ul oge he . These wo las obse a ions lead he scien i ic
communi y o he idea ha il e s can ha e impo an limi a ions, and hey can be
o e come by means, o ins ance, o w appe s o he use o embedded me hods.
The ML p ocess epo ed in Figu e 2.1 can be ex ended including da a p ep o-
cessing, leading o he mo e comple e scheme ep esen ed in Figu e 2.4. As we
Fig. 2.4 Ex ension o he schema o Figu e 2.1, o include da a p ep ocessing.
can see, he objec i e o da a p ep ocessing is usually he one o gene a ing a new
28 4 Expe imen al S udy
(limi ed o he 70 genes o he signa u e) and a p e iously compu ed ypical exp es-
sion p o ile o a good p ognosis pa ien . To compa e he pe o mance o he a ious
machine lea ning algo i hms wi h his sco ing sys em, he ollowing p ocess was
implemen ed:
•p ognos ic sco e so he pa ien s (excluding he ones used o ain he signa-
u e in [ an ’ Vee e al., 2002]) was ob ained om he Supplemen a y Ma e-
ial o [ an de Vij e e al., 2002], and classi ied as good p ognosis he pa ien s
wi h s>0.4 and as bad p ognosis he ones wi h s≤0.4. This is he cu o used
in [ an de Vij e e al., 2002].
•50 andom lis s o 44 pa ien s we e gene a ed om his se , o ma ch he s a is ic
used o machine lea ning echniques, and compu ed o each lis he numbe o
alse p edic ions gi en by he sco ing me hod.
The mean numbe o alse p edic ions was 16.24, wi h a SEM o 0.37. The e o e
he sco ing me hod appea s o be supe io o all machine lea ning algo i hm o he
han GP, and sligh ly supe io o GP. The di e ence be ween he pe o mances o GP
and he sco ing me hod a e no s a is ically signi ican (P=0.49, 2- ailed S uden
- es ).
4.3 The Role o Fea u e Selec ion
To de e mine o wha ex en ea u e selec ion is esponsible o he good pe o -
mance o GP, we iden i ied he 10 ea u es mos o en selec ed by GP among he
70 ini ial ea u es and an again bo h GP and SVM wi h quad a ic ke nel using
only hese ea u es. Rema kably, he pe o mance o bo h me hods signi ican ly im-
p o ed: o GP, he numbe o inco ec ly iden i ied ea u es dec eased om 16.40
(SEM 0.30) o 12.86 (0.40); o he SVM i wen om 16.76 (0.18) o 14.96 (0.41).
Using his p elimina y ound o ea u e selec ion he pe o mance o GP becomes
signi ican ly be e han bo h SVM and he o iginal sco ing me hod.
These esul s sugges , on one hand, ha he ea u e selec ion pe o med by GP
has in insic alue, no necessa ily ied o he use o syn ax ees, since he SVM can
ake ad an age o he ea u e selec ion pe o med by GP o imp o e i s pe o mance.
Second, ha a ecu si e use o GP, in which a i s un is used o selec he bes
ea u es o be used in a second un, migh be a p omising way o op imizing he
me hod.
4.4 Pe o mance on Unbalanced Da ase s
To check whe he he pe o mance o he GP is ied o he choice o a balanced
da ase , he analysis was epea ed using di e en ime cu o s (5 and 7.5 yea s) and
he pe o mance o GP was compa ed wi h he SVM using polynomial ke nel wi h

4.6 Assessmen o Sensi i i y 29
deg ee 2, which was he bes pe o ming me hod a e GP in he balanced da ase .
A 7.5 yea s he e is again no signi ican di e ence be ween he pe o mance o he
wo me hods. Howe e , a 5 yea s GP pe o ms signi ican ly be e han he SVM
(P=6.46 ×10−6 om wo-sided - es ). We conclude ha he balancing o he
da ase is no c ucial o ob ain a good pe o mance om GP.
4.5 Pe o mance on an Independen Da ase
An impo an ea u e o any p edic o based on gene exp ession da a is i s obus -
ness wi h espec o he choice o da ase , since gene exp ession da a om cance
pa ien s come om s udies using di e en p o ocols and/o mic oa ay pla o ms.
Thus, he bes p edic o s ound by GP in each o he 50 uns we e applied o an
independen b eas cance da ase [Mille e al., 2005], ob ained on a di e en mi-
c oa ay pla o m. Due o he di e ence in gene con en be ween pla o ms, only 17
o he 50 bes solu ions ound by GP could be applied o he new da ase . All o hem
showed s a is ically signi ican p edic i e powe (P- alues be ween 7.6×10−3and
2.9×10−4 om Fishe exac es ). Since his esul was ob ained wi h no u he
aining, i shows he obus ness o he solu ions ob ained by GP wi h espec o he
choice o da ase and mic oa ay pla o m.
4.6 Assessmen o Sensi i i y
When using gene signa u es o p edic he su i al o a coho o b eas cance
pa ien s, one o he main goal in clinical applica ions is o minimize he numbe
o alse nega i e p edic ions. Table 4.4 summa izes he alse nega i e p edic ions
e u ned by each machine lea ning me hod on he 50 uns. The i s line indica es he
di e en me hods, while he second and he hi d lines show he bes (i.e. lowes ) and
mean pe o mances ( oge he wi h he co esponding SEM) alues o inco ec ly
classi ied ins ances.
The bes solu ions we e ound by GP, and s a is ical analysis indica es ha GP
consis en ly ou pe o ms he o he i e me hods as i can be seen in Table 4.5. The
di e ence be ween he a ious a e age esul s is s a is ically signi ican (P- alue
2.75 ×10−9 o ANOVA es on he 4 samples o solu ions ound by each me hod).
Finally, pai wise 2- ailed S uden - es s compa ing GP wi h each o he me hod
demons a e i s be e pe o mance.
The o iginal sco ing me hod o [ an ’ Vee e al., 2002,
an de Vij e e al., 2002], and in pa icula he sugges ed cu o o 0.4, was
chosen in such a way as o minimize he numbe o alse nega i es. The e o e i is
no su p ising ha in his espec he sco ing me hod is a supe io o all machine
lea ning me hods, including GP. Indeed he a e age numbe o alse nega i es gi en
by he sco ing me hod is 1.78, o be compa ed o he numbe s epo ed in Table 4.5.
30 4 Expe imen al S udy
4.7 Maximizing Sensi i i y in GP
I is well know ha he i ness unc ion d i ing he e olu iona y dynamics in a GP
amewo k can be modi ied in o de o le eme ge solu ions wi h di e en cha ac-
e is ics. The esul s p esen ed and discussed in he p e ious sec ion we e ob ained
wi h he goal o minimizing all inco ec ly classi ied ins ances, summing bo h alse
nega i e and alse posi i e p edic ions ob ained by he solu ions. Howe e , when
using gene signa u es o p edic he su i al o a coho o b eas cance pa ien s,
minimizing he numbe o alse nega i e p edic ions is ecognized as one o he mos
impo an goals.
Fo all hese easons, we modi ied he GP i ness unc ion so ha alse nega i es
(posi i es) a e penalized mo e han e o s o he o he ype, hoping o une he al-
go i hm owa ds be e sensi i i y (sensibili y). In pa icula , solu ions wi h g ea e
sensi i i y can eme ge i la ge weigh s a e assigned o alse nega i es compa ed
o alse posi i es. In gene al, we can ans o m he i ness unc ion in a weigh ed
a e age o he o m:
Fi ness =0.9×FalseNega i e +0.1×FalsePosi i e
Wi h espec o his new o mula ion, he i ness unc ion o he GP algo i hm
whose esul s we e p esen ed in he p e ious sec ion can be exp essed as 0.9×
FalseNega i e +0.1×FalsePosi i e. The esul s o 50 uns o his new e sion
o he GP echnique showed an a e age o 16.04 (wi h SEM =0.44) o o al in-
co ec ly classi ied ins ances. Compa ed wi h he pe o mances o he p e ious GP
algo i hm, no s a is ically signi ican di e ence can be highligh ed (S uden - es
P=0.50). When looking only a he numbe o alse nega i e inco ec ly classi ied
ins ances, he a e age pe o mance o 4.32 (SEM =0.346) is be e han he one
o s anda d GP epo ed in Table 4.5 (S uden - es P=6.62 ×10−16), e en i s ill
wo se han ha o he o iginal sco ing me hod.
Figu es and Tables
(o (and (o ORC6L RFC4) (o UCHL5 PRC1))
(and (o PRC1 AI554061) (o ESM1 AW014921)))
Fig. 4.1 T ee ep esen a ion and he adi ional Lisp ep esen a ion o he model wi h he bes
i ness ound by GP o e he s udied 50 independen uns.
4.7 Maximizing Sensi i i y in GP 31
GP Pa ame e s
popula ion size 500 indi iduals
popula ion ini ializa ion amped hal and hal [Koza, 1992]
selec ion me hod ou namen ( ou namen size =10)
c osso e a e 0.9
mu a ion a e 0.1
maximum numbe o gene a ions 5
algo i hm gene a ional ee based GP wi h no eli ism
SVM Pa ame e s
complexi y pa ame e 0.1
size o he ke nel cache 107
epsilon alue o he ound-o e o 10−12
exponen o he polynomial ke nel 1.0,2.0,3.0
ole ance pa ame e 0.001
Mul ilaye ed Pe cep on Pa ame e s
lea ning algo i hm Back-p opaga ion
lea ning a e 0.03
ac i a ion unc ion o all he neu ons in he ne sigmoid
momen um 0.2 p og essi ely dec easing un il 0.0001
hidden laye s (numbe o a ibu es + numbe o classes) /2
numbe o epochs o aining 500
Random Fo es Pa ame e s
numbe o ees 2500
numbe o a ibu es pe node 1
Table 4.1 Pa ame e s used in he expe imen s.
GP SVM-K1 SVM-K2 SVM-K3 MP RF
bes 10 13 14 15 10 12
a e age (SEM) 16.40 (0.30) 18.32 (0.37) 16.76 (0.18) 17.62 (0.17) 18.08 (0.39) 17.60 (0.35)
Table 4.2 Expe imen al compa ison be ween he numbe o inco ec ly classi ied ins ances ound
on he es se s by he di e en machine lea ning me hods. Each me hod was independen ly un 50
imes using each ime a di e en aining/ es pa i ion o he alida ion da ase (see ex o de ails).
The i s line indica es he me hod: Gene ic P og amming (GP), Suppo Vec o Machine wi h
exponen o he polynomial ke nel 1.0 (SVM-K1), 2.0 (SVM-K2), and 3.0 (SVM-K3), Mul ilaye
Pe cep ons (MP), and Random Fo es (RF). The second line shows he bes alue o he inco ec ly
classi ied ins ances ob ained on he es se o e he 50 uns, and he hi d line epo s he a e age
pe o mances o each g oup o 50 uns on hei es se s (s anda d e o o mean is shown in
pa en heses).
ANOVA
P=3.05 ×10−5
GP s. SVM-K1 GP s. SVM-K2 GP s. SVM-K3 GP s. MP GP s. RF
P=0.0001 P=0.3107 P=0.0008 P=0.0009 P=0.0103
Table 4.3 S a is ical signi icance o he di e ence in pe o mance be ween he me hods. Fi s line
shows ANOVA es on he 6 samples o solu ions ound by each me hod, while second line depic s
pai wise 2- ailed S uden - es s compa ing GP wi h each o he me hod.
32 4 Expe imen al S udy
GP SVM-K1 SVM-K2 SVM-K3 MP RF
bes 2 6 6 6 5 6
a e age (SEM) 9.82 (0.44) 13.26 (0.51) 12.60 (0.35) 14.08 (0.39) 12.88 (0.51) 13.38 (0.49)
Table 4.4 Expe imen al compa ison be ween he numbe o alse nega i es ound on he es se s
by he di e en machine lea ning me hods. Each me hod was independen ly un 50 imes using
each ime a di e en aining/ es pa i ion o he alida ion da ase (see ex o de ails). The i s
line indica es he me hod: Gene ic P og amming (GP), Suppo Vec o Machine (SVM), Mul ilaye
Pe cep ons (MP), and Random Fo es (RF). The second line shows he bes alue o he inco ec ly
classi ied ins ances ob ained on he es se o e he 50 uns, and he hi d line epo s he a e age
pe o mances o each g oup o 50 uns on hei es se s (s anda d e o o mean is shown in
pa en heses).
ANOVA
P=2.75 ×10−9
GP s. SVM-K1 GP s. SVM-K2 GP s. SVM-K3 GP s. MP GP s. RF
P=2.74 ×10−6P=3.32 ×10−6P=1.27 ×10−10 P=8.53 ×10−6P=4.65 ×10−7
Table 4.5 False nega i e p edic ion: s a is ical signi icance o he di e ence in pe o mance be-
ween he me hods. Fi s line shows ANOVA es on he 6 samples o solu ions ound by each
me hod, while second line depic s pai wise 2- ailed S uden - es s compa ing GP wi h each o he
me hod.
Accession ID Gene name Gene desc ip ion Solu ions
NM 003981 PRC1 p o ein egula o o cy okinesis 1 48
NM 002916 RFC4 eplica ion ac o C (ac i a o 1) 4, 37kDa 23
AI992158 - - 16
AI554061 - - 10
NM 006101 NDC80 NDC80 homolog, kine ocho e complex com-
ponen (S. ce e isiae)
9
NM 015984 UCHL5 ubiqui in ca boxyl- e minal hyd olase L5 7
NM 020188 C16o 61 ch omosome 16 open eading ame 61 6
NM 016448 DTL den icleless homolog (D osophila) 6
NM 014791 MELK ma e nal emb yonic leucine zippe kinase 6
NM 004702 - - 6
Table 4.6 The 10 mos ecu ing ea u es in he solu ions ound by GP. The ou columns show:
accession ID, gene name, gene desc ip ion, and numbe o solu ions whe e ha ea u e occu s.
Chap e 5
Conclusions and Fu u e Wo k
The in es iga ion p esen ed in his documen was aimed a e ining he se o c i e ia
ha could lead o be e isk s a i ica ion in b eas cance . To each his goal, he
s a ing poin was he use o he well known “70-genes signa u e” and he applica-
ion o se e al machine lea ning schemes, in o de o pe o m a compa ison be ween
hem. Some simpli ying assump ions we e made, p ep ocessing he da a acco d-
ingly and se e al e alua ion expe imen s we e an. The p esen ed esul s showed
ha while all he s udied machine lea ning algo i hms do ha e p edic i e powe in
classi ying b eas cance pa ien s in o isk classes, GP clea ly ou pe o ms all o he
me hods. The ac ha all me hods o he han GP had e y simila pe o mance
sugges s ha GP is indeed he mos p omising me hod. The imp o emen in pe -
o mance shown by GP compa ed o he o iginal sco ing me hod was a he small
and no s a is ically signi ican . As expec ed, he sco ing me hod was supe io o all
machine lea ning algo i hms in minimizing alse nega i es. In a second phase, GP
was en iched by changing i s i ness unc ion in o a weigh ed a e age be ween alse
nega i es and alse posi i es. I was shown ha , when la ge weigh is gi en o alse
nega i es, i is possible o une he GP algo i hm owa ds g ea e sensi i i y. While
he sensi i i y o GP is s ill less han he o iginal sco ing me hod, he possibili y
o uning he i ness unc ion is ano he in insic ad an age o his echnique wi h
espec o he o he machine lea ning ones conside ed he e. Ne e heless hese e-
sul s wa an u he in es iga ion in o he use o GP in his con ex o a leas h ee
easons:
•The implemen a ion o GP was pu posely no op imized, and we can expec sub-
s an ial imp o emen s in pe o mance om u he wo k aimed a uning he a -
ious GP se ings.
•Maybe mo e impo an ly, GP can po en ially o e biological insigh and gen-
e a e hypo heses o expe imen al wo k (see also [Yu e al., 2007]). Indeed an
impo an esul o he p esen ed analysis is ha he ees p oduced by GP end
o con ain a limi ed numbe o ea u es, and he e o e a e easily in e p e able in
biological e ms. Fo example, he bes -pe o ming ee is shown in Figu e 4.1
and includes 7 genes ( ea u es).
33

34 5 Conclusions and Fu u e Wo k
•Finally wi hin he con ex o GP he e is a na u al way o une he algo i hm
owa ds be e sensi i i y (speci ici y), simply by de ining a i ness unc ion in
which alse nega i es (posi i es) a e penalized mo e han e o s o he o he ype.
Fu u e wo k along hese lines should he e o e ocus on bo h imp o ing he pe -
o mance o GP and in e p e ing he esul s om he biological poin o iew. An
ob ious i s s ep owa ds op imiza ion would be o abandon he bina iza ion o he
da a (which he e was used o p oduce ees ha a e easie o in e p e ) and build a GP
based on con inuous exp ession alues. The biological in e p e a ion migh bene i
om a s a is ical and unc ional analysis o he mos ecu ing sub ees in op imal
GP solu ions. As a pa ial conclusion, i is possible o asse ha GP ou pe o ms
o he machine lea ning me hods as a ool o ex ac p edic ions om an es ablished
b eas cance gene signa u e. Gi en he possibili y o gene a ing biological insigh
and hypo heses ha is in insic o he me hod, i dese es deepe in es iga ion along
he lines desc ibed abo e. Finally, i would be an app op ia e ask o es he GP ap-
p oach on o he ea u es/gene se s, ha accoun o o he cance s o o he diseases,
always wi h he objec i e o p o iding clinicians wi h mo e p ecise and indi idual-
ized diagnosis c i e ia.
Ano he impo an esea ch a enue o explo e in he u u e conce ns he possi-
bili y o inc easing ou da ase s by means o Radiomics [Lambin e al., 2012]. Ra-
diomics is an eme ging ield o medical s udies, aimed a ex ac ing la ge amoun s
o highly in o ma i e ea u es om medical images, hus con e ing images in o
mineable da a, and analysing hose da a o decision suppo . The hypo hesis o Ra-
diomics is ha he dis inc i e imaging ea u es be ween disease o ms may be c u-
cial o p edic ing p ognosis and he apeu ic esponse o a ious condi ions, hus
p o iding aluable in o ma ion o pe sonalised he apy. The Radiomics wo k low
can be o ganized in o dis inc phases, each wi h i s own challenges:
•iden i ica ion o a pa ien coho ;
•op imiza ion o acquisi ion p o ocols;
• umo and o gan segmen a ion;
• ea u e ex ac ion and ea u e selec ion; and
•model de elopmen and alida ion.
An impo an objec i e o u u e esea ch is o imp o e he s a e o he a in
wo o hese phases: umo and o gan segmen a ion and model de elopmen and
alida ion, wi h pa icula e e ence o b eas cance , ha is he ype o disease
ha is discussed in his documen . These phases may be app oached using ex-
is ing and no el machine lea ning and deep lea ning echnologies. These ech-
nologies need o be s udied and compa ed, o disco e he mos app op ia e al-
go i hm, able o ou pe o m he s a e o he a in each speci ic case. Conce n-
ing machine lea ning, also in sigh o he esul s p esen ed in his documen ,
ocus should be gi en o wo ecen ly de ined and ex emely p omising bio-
inspi ed algo i hms, which a e new de elopmen s o GP: Geome ic Seman ic GP
(GSGP) [Vanneschi, 2017], mos ly used o eg ession p oblems, and Mul idimen-
sional Mul iclass GP (M3GP) [Mu˜
noz e al., 2015], mos ly used o (bina y o mul-
iclass) classi ica ion p oblems. These me hods will be compa ed o he s a e o he
5 Conclusions and Fu u e Wo k 35
a me hods in Radiomics, including Random Fo es s, Suppo Vec o Machines,
Bayesian Ne wo ks and Linea , Leas Squa e and Logis ic eg ession. In he las
ew yea s, GSGP has de eloped eno mously, becoming one o he mos popula ho
opics in he GP communi y. Thanks o an e icien and inno a i e implemen a ion
o GSGP [Vanneschi, 2017], i was possible o apply GSGP o a as se o applica-
ions om di e en domains, including p edic ion o pha macokine ic pa ame e s
in d ug disco e y, posi ioning o compu e omog aphy slices, p edic ion o he uni-
ied Pa kinson’s disease a ing scale assessmen , p edic ion o an icoagula ion le el
in pha macogene ics, and also a he di e en applica ion domains like ene gy o e-
cas ing, p edic ion o high pe o mance conc e e s eng h, and p edic ion o essels’
ajec o ies a sea o imp o ing ma i ime sa e y and secu i y.
Conce ning deep lea ning, Con olu ional Neu al Ne wo ks (CNNs), which ep-
esen he s a e o he a in compu e ision and many o he p oblem domains, ep-
esen an in e esy ing s a ing poin o compa ison. Fu u e wo k should include he
delopmen o new me hods o in eg a ing GSGP wi h CNNs, o o in oduce in o
CNNs he concep o seman ics, ha is cha ac e is ic o GSGP. The idea is ha sha -
ing he same p ope ies as GSGP, and ex ending hem o deep lea ning, hese no el
sys ems will induce a unimodal e o su ace (i.e. an e o su ace cha ac e ized
by he absence o locally op imal solu ions) o any supe ised lea ning p oblem.
This ac should bes ow on hese no el sys ems a compe i i e ad an age in e ms o
e ol abili y. A he same ime, as o GSGP, hese sys ems should be able o limi
o e i ing, hus being able o gene a e accu a e and obus p edic i e models.
Besides an ex ensi e pe o mance compa ison o machine lea ning and deep
lea ning me hods, in he u u e impo ance should be gi en o an a en i e e alua ion
o he ela i e p os and cons. Gene ally speaking, we expec he machine lea ning
me hods o equi e mo e e o in he p e-p ocessing phase ( o ea u e ex ac ion
and selec ion), as opposi e o he deep lea ning me hods, ha inco po a e ea u e
ex ac ion and selec ion di ec ly in some in e nal lea ning laye s. On he o he hand,
we expec deep lea ning me hods o ha e a be e pe o mance in p oblems cha ac-
e ized by a as amoun o da a, while hey may be ou pe o med by he machine
lea ning me hods in case o smalle amoun s o da a, a no so in equen e en in he
medical ield. The amoun o a ailable da a is indeed a c ucial heme o any medical
applica ion, and o oncology in pa icula . The s udied applica ions a e gene ally
cha ac e ized by a as amoun o da a, bu supe ising hose da a is gene ally a
e y ha d and ime consuming ask. Fo his eason, cu en ly only a small pa o
he a ailable da a a e supe ised. This leads o he po en ial o he exis ence o e y
la ge da ase s, in which unsupe ised obse a ions a e much mo e nume ous han
he supe ised ones. Fo his eason, a signi ican pa o he u u e s udies should
be dedica ed o ad ancemen s in he a ea o semi-supe ised lea ning. Pa icula ly
p omising seems he idea o ex ending he p ope ies ha cha ac e ize he mos e-
cen e sions o GP (and ha de e mined hei ecen success) o semi-supe ised
lea ning. A signi ican i s s ep has been aken ecen ly o he case o he M3GP
algo i hm.
Re e ences 37
Re e ences
[Alon e al., 1999] Alon, U., Ba kai, N., No e man, D., Gish, K., Yba a, S., Mack, D., and
Le ine, A. J. (1999). B oad pa e ns o gene exp ession e ealed by clus e ing analysis o u-
mou and no mal colon issues p obed by oligonucleo ide a ays. In P oc. Na . Acad. Sci., pages
6745–6750. USA 96.
[A che i e al., 2006] A che i, F., Lanzeni, S., Messina, E., and Vanneschi, L. (2006). Gene ic
p og amming o human o al bioa ailabili y o d ugs. In M. Ca olico e al., edi o , P oceed-
ings o he 8 h annual con e ence on Gene ic and E olu iona y Compu a ion, pages 255 – 262,
Sea le, Washing on, USA.
[A che i e al., 2007a] A che i, F., Lanzeni, S., Messina, E., and Vanneschi, L. (2007a). Gene ic
p og amming o compu a ional pha macokine ics in d ug disco e y and de elopmen . Gene ic
P og amming and E ol able Machines, 8(4):413–432.
[A che i e al., 2007b] A che i, F., Messina, E., Lanzeni, S., and Vanneschi, L. (2007b). Gene ic
p og amming and o he machine lea ning app oaches o p edic median o al le hal dose (LD50)
and plasma p o ein binding le els (%PPB) o d ugs. In E. Ma chio i e al., edi o , E olu iona y
Compu a ion, Machine Lea ning and Da a Mining in Bioin o ma ics. P oceedings o he Fi h
Eu opean Con e ence, E oBIO 2007, Lec u e No es in Compu e Science, LNCS 4447, pages
11–23. Sp inge , Be lin, Heidelbe g, New Yo k.
[A che i e al., 2007c] A che i, F., Messina, E., Lanzeni, S., and Vanneschi, L. (2007c). Gene ic
p og amming o compu a ional pha macokine ics in d ug disco e y and de elopmen . Gene ic
P og amming and E ol able Machines, 8(4):17–26.
[Boja czuk e al., 2001] Boja czuk, C., Lopes, H., and F ei as, A. (2001). Da a mining wi h
cons ained-syn ax gene ic p og amming: applica ions o medical da a se s. P oceedings In-
elligen Da a Analysis in Medicine and Pha macology, 1.
[B eiman, 2001] B eiman, L. (2001). Random o es s. Machine Lea ning, 45(1):5–32.
[B eiman e al., 1984] B eiman, L., F iedman, J., Olshen, R., and S one, C. (1984). Classi ica ion
and Reg ession T ees. Belmon , Cali o nia, Wadswo h In e na ional G oup.
[Chu and Wang, 2005] Chu, F. and Wang, L. (2005). Applica ions o suppo ec o machines o
cance classi ica ion wi h mic oa ay da a. In J Neu al Sys , 15(6):475–484.
[Da win, 1859] Da win, C. (1859). On he O igin o Species by Means o Na u al Selec ion. John
Mu ay.
[Deb and Reddy, 2003] Deb, K. and Reddy, A. R. (2003). Reliable classi ica ion o wo-class
cance da a using e olu iona y algo i hms. Biosys ems, 72(1-2):111–129.
[Deu sch, 2003] Deu sch, J. M. (2003). E olu iona y algo i hms o inding op imal gene se s in
mic oa ay p edic ion. Bioin o ma ics, 19(1):45–52.
[Dubi zky e al., 2006] Dubi zky, W., G anzow, M., and Be a , D. P. (2006). Fundamen als o
Da a Mining in Genomics and P o eomics. Sp inge -Ve lag, Be lin, Heidelbe g.
[F iedman e al., 2000] F iedman, N., Linial, M., Nachmann, I., and Pee , D. (2000). Using
bayesian ne wo ks o analyze exp ession da a. J. Compu a ional Biology, 7:601–620.
[Goldbe g, 1989] Goldbe g, D. E. (1989). Gene ic Algo i hms in Sea ch, Op imiza ion and Ma-
chine Lea ning. Addison-Wesley.
[Guyon e al., 2002] Guyon, I., Wes on, J., Ba nhill, S., and Vapnik, V. (2002). Gene selec ion o
cance classi ica ion using suppo ec o machines. Machine Lea ning, 46:389–422.
[Hall e al., 2009] Hall, M., F ank, E., Holmes, G., P ah inge , B., Reu emann, P., and Wi en,
I. H. (2009). The WEKA da a mining so wa e: an upda e. SIGKDD Explo a ions, 11(1):10–18.
[He nandez e al., 2007] He nandez, J. C. H., Du al, B., and Hao, J. (2007). A gene ic embedded
app oach o gene selec ion and classi ica ion o mic oa ay da a. Lec u e No es in Compu e
Science, 4447:90–101.
[Holland, 1975] Holland, J. H. (1975). Adap a ion in Na u al and A i icial Sys ems. The Uni e -
si y o Michigan P ess, Ann A bo , Michigan.
[Hong and Cho, 2006] Hong, J. and Cho, S. (2006). The classi ica ion o cance based on dna
mic oa ay da a ha uses di e se ensemble gene ic p og amming. A i . In ell. Med, 36:43–58.