FINAL DEGREE PROJECT
Random Fo es s:
P ope ies and applica ions
P esen ed by:
Fá ima del Pila Villalba Piza o
Supe ised by:
DR. EMILIO CARRIZOSA PRIEGO
FACULTY OF MATHEMATICS
S a is ics and Ope a ional Resea ch Depa men
Se ille, Sep embe 2020
Índice gene al
Resumen 5
Abs ac 7
In oducción 9
In oduc ion 13
1. F om CART o andom o es s 15
1.1. CART (Classi ica ion and Reg ession T ees) . . . . . . . . . . . . . . 15
1.1.1. S uc u e o decision ees . . . . . . . . . . . . . . . . . . . 16
1.1.2. Cons uc ion o decision ees . . . . . . . . . . . . . . . . . 17
1.1.2.1. Topology and ype o spli ing . . . . . . . . . . . . 18
1.1.2.2. Spli ing ules . . . . . . . . . . . . . . . . . . . . . 18
1.1.2.3. S op-spli ing ule . . . . . . . . . . . . . . . . . . 22
1.1.2.4. Designa ion o class label a e minal nodes . . . . 25
1.1.3. Con usion ma ix and accu acy . . . . . . . . . . . . . . . . 25
1.1.4. Reg ession T ees . . . . . . . . . . . . . . . . . . . . . . . . 26
1.2. Random o es s............................. 26
1.2.1. Cons uc ion o andom o es s . . . . . . . . . . . . . . . . . 27
1.2.1.1. Bagging(Boo s ap-agg ega ing) . . . . . . . . . . 27
1.2.1.2. Randomness in spli ing . . . . . . . . . . . . . . . 28
1.2.1.3. Unp unning . . . . . . . . . . . . . . . . . . . . . 29
1.2.2. P oximi y ma ix . . . . . . . . . . . . . . . . . . . . . . . . 29
1.2.2.1. Dissimila i y ma ix [13][7] . . . . . . . . . . . . . 30
1.2.2.2. Ou lie s ....................... 31
2. Impo ance o a iables 33
3. Applica ions o andom o es s 35
3.1. Random o es s in R........................... 35
3.2. Expe imen I .............................. 37
3.3. Expe imen II.............................. 45
3
4 Índice gene al
3.3.1. Teca o ............................. 45
3.3.1.1. Classi ica ion o eca o . . . . . . . . . . . . . . . 45
3.3.1.2. Reg ession o eca o . . . . . . . . . . . . . . . . 53
3.3.2. G ow h ............................. 54
4. Conclusions 61
Glossa y 63
Bibliog aphy 64
Resumen
Los bosques alea o ios son una he amien a undamen al en el ámbi o del ap en-
dizaje supe isado, po lo que son empleados en mul i ud de disciplinas, demos ando
g andes esul ados y múl iples en ajas en p oblemas de clasi icación y eg esión.
En es e abajo se exponen los undamen os y bases de los bosques alea o ios,
des acando sus en ajosas ca ac e ís icas, pa a pos e io men e hace di e en es aplica-
ciones con ellos, con las que con as a dichas en ajas, obse a dis in os aspec os que
se ha án pa en es e incluso analiza su compo amien o cuando ex endemos su uso al
caso de los da os uncionales.
5

Abs ac
Random o es s a e conside ed a undamen al ool in supe ised lea ning. Conse-
quen ly, andom o es s a e used in a wide ange o disciplines, yielding g ea esul s
and demons a ing many ad an ages in classi ica ion and eg ession p oblems.
Along his wo k, he bases and ounda ions o andom o es s a e exposed, empha-
sizing hei ad an ageous p ope ies, o subsequen ly make di e en applica ions wi h
hem, demons a ing hei ad an ages, analyzing dis inc aspec s ha become no ice-
able, and e en s udying he way hey beha e when ex ending hei use o unc ional
da a.
7
In oducción
En un mundo en cons an e a ance y desa ollo, es impo an e usa écnicas que mo-
delen la ealidad de mane a e icien e. Muchos p oblemas de eg esión y clasi icación
se pueden esol e median e ap endizaje supe isado, es deci , median e una écnica
en la que, dada una mues a con dis in os da os en o ma de pa es, uno de en ada (no -
malmen e un ec o ) y o o de salida, se les a ibuye un alo numé ico o clase a cada
pa , con el in de pode p edeci la salida de u u os da os. Es e mé odo ha ido cob ando
impo ancia en los úl imos años omando un papel ele an e como puede e se en la
Figu a 1, que nos mues a el in e és susci ado po es e ema en elación con el núme o
de búsquedas del é mino “ap endizaje supe isado" (en ingles “supe ised lea ning")
en Google.
Figu a 1: E olución del núme o de búsquedas del é mino “ap endizaje supe isado"
(supe ised lea ning) en Google desde 2004
Una he amien a pa icula de ap endizaje supe isado es la conocida como bos-
ques alea o ios, o del inglés, andom o es s, desa ollados en el año 2001 po Leo
B eiman [6]. Es a écnica se usa en dis in as á eas como son la econome ía [20], la
quimioin o má ica [18], la bioin o má ica [10] o la medicina. En cuan o a es a úl i-
ma, se iene que los bosques alea o ios son ú iles en di e en es campos como son la
selección de ma cado es gené icos esponsables de en e medades, la mic obiología o
la epidemiología gené ica, e incluso pa a p edeci la eplicación de un i us como el
HIV-1 [17].
La ele ancia de los bosques alea o ios se debe a que, a di e encia de o as écnicas,
los bosques alea o ios son e icaces incluso cuando se abaja con a iables con bas an e
9
16 1.1. CART (Classi ica ion and Reg ession T ees)
Decision ees a e conside ed leade s in hei a ea because o hei easy in e -
p e abili y. The eason he e o e esides in he scheme ha ollows such a ee: an
i - hen ule. [9]
1.1.1. S uc u e o decision ees
Le us explain now how decision ees wo k. We will begin wi h classi ica ion he
ees and hen go on wi h an in dep h e iew o he eg ession ees.
Decision ees can handle any ype o a iables, ca ego ical (i.e., hey ake alues
in a ini e se no ha ing any na u al o de ) o nume ical/con inuous (i.e. i is a eal
numbe ).
Le us see i s which is he s uc u e o such a ee.
A classi ica ion ee is a collec ion o nodes and b anches ha display a pa i ion o he
o iginal se . Le 0be such o iginal se . This se can be spli in o a compound o se s
ha o m a pa i ion o 0. Fo simplici y, le us hink we a e wo king wi h he simples
ype o ee, he bina y ee, which means a each spli wo new se s a e ob ained. I we
name he new se s ob ained om 0, 1and 2i is ul illed 0= 1 2. This p ocess is
epea ed wi h such new se s.
All hese se s a e called nodes. Th ee ypes o nodes a e ound:
The o iginal se 0, called oo node.
The subse s ha a e no spli , called e minal nodes. These o m a pa i ion
o 0. Each e minal node is assigned a class label, and i is possible o ind
mo e han jus one e minal node wi h he same class label.
The emaining nodes, called non e minal nodes.
Such s uc u e o a decision ee can be seen in Figu e 1.1.
Figu e 1.1: S c u e o a decision ee

Chap e 1. F om CART o andom o es s 17
Ma i al s a us Age Wo king
Ma ied 28 Yes
Single 37 No
Di o ced 42 No
Ma ied 41 Yes
Table 1.1: Example o lea ning sample L
1.1.2. Cons uc ion o decision ees
Fo he cons uc ion o he classi ica ion ee, a lea ning sample mus be gi en.
Le L={(Xi, Yi)i=1,..,N }deno e he lea ning sample, whe e Nis he numbe o
obse a ions, Xi∈Xa ec o o M a iables, named measu emen ec o , in he
measu emen space X, and Yi he co esponding obse a ion om among Kpossible
classes C=C1, ..., CK.
Example 1.1.1. An example o wha is explained abo e can be seen in Table 1.1, whe e
i can be iden i ied:
N= 4,M= 2,K= 2
X1is he a iable Ma i als a us ha akes alues in Ma ied, Single, Di o ced
X2is he a iable Age ha akes alues in 18,19,20, . . . 
L=(X1
1=Ma ied, X2
1= 28, Y1=Y es),(X1
2=Single, X2
2=
37, Y2=No),(X1
3=Di o ced, X2
3= 42, Y3=No),(X1
4=Ma ied, X2
4=
41, Y4=Y es)
Acco ding o he gi en lea ning sample he classi ica ion ee can be buil . The
cons uc ion o a ee is based on ou poin s:
1. Topology and ype o spli ing
2. Selec ion o spli s
3. Decla a ion o e minal nodes/S op-spli ing ule
18 1.1. CART (Classi ica ion and Reg ession T ees)
4. Designa ion o class label a e minal nodes
The main goal is cons uc ing he classi ica ion ee om he aining se L.
We now ou line some de ails o he abo e men ioned elemen s.
1.1.2.1. Topology and ype o spli ing
Acco ding o he numbe o new nodes esul ing om he spli ing, wo ypes o
spli ing can be dis inguished: bina y spli ing, i.e., he e a e ob ained wo new nodes,
o mul i-spli ing, ha induces mo e han wo new nodes. Fo con inuous a iables he
second one is no eally use ul. I can also be se a di e ence be ween wo ypes o
spli s acco ding o he numbe o a iables in ol ed in he spli ing. I only one a iable
akes pa in he spli , i is called uni a ia e spli , o he wise, i is named mul i a ia e
spli . In wha ollows, jus bina y ees, wi h uni a ia e spli will be conside ed.
1.1.2.2. Spli ing ules
In his sec ion we will assume we a e wo king wi h a s anda d s uc u e, i.e., all
measu emen ec o s Xia e o ixed dimensionali y.
Many di e en c i e ia o spli ing ha e been adop ed. Ou o hen, Leo B eimann
highligh s wo, namely:
Gini c i e ion
Twoing c i e ion
The Gini c i e ion is he one ha is going o be explained, because i is also he
one ha uses he p og am R ha is going o be used in he las chap e . Be o e going
down o he explana ion o how his c i e ion ope a es, we mus i s in oduce a ew
undamen al concep s in wo ini ial s eps, ha will con e ge in he hi d and las s ep.
1. Explana ion o concep s ela ed wi h p obabili y
2. In oduc ion o se s o all possible spli ing
3. Explana ion o selec ing he bes spli : Gini c i e ion
P obabili y concep s
Beginning wi h he i s s ep: Remembe we said we had he aining sample L=
(Xi, Yi)i=1,..,N
, wi h Xi∈Xand dimension M, and Yi∈C=C1, ..., CK.
Chap e 1. F om CART o andom o es s 19
The p io class p obabili ies π(k),1≤k≤Kis de ined as:
π(k) = P(Yi=Ck).
Le Nkbe he numbe o cases o Lin class Ck. O en he p io p obabili ies can
be assessed like:
π(k) = Nk
N,1≤k≤K.
Suppose now we a e a node . Main aining he same idea as abo e, le N( )be he
o al numbe o obse a ions in L ha ha e eached he node , and le Nk( )deno e
he numbe o obse a ions o class Ckin . So, he quo ien
Nk( )
Nk
is in e p e ed as he p opo ion o obse a ions o class Ck alling in o .
Using bo h de ini ions, we ob ain he p obabili y o an obse a ion in L ul illing
wo aspec s simul aneously: alling in o and being o class Ck, gi en by
p(k, ) = π(k)Nk( )
Nk
.
I we sum in k, we ob ain he p obabili y o an obse a ion o any class Ck eaching
he node
p( ) = X
k
p(k, ),
and we can de ine now he p obabili y he obse a ion is o class Cksubjec o ha ing
eached node
p(k| ) = p(k, )
p( )
sa is ying:
X
k
p(k| ) = 1
Acco ding o he app oach done o π(k)we can also assume ha
p(k| ) = Nk( )
N( ).(1.1)
20 1.1. CART (Classi ica ion and Reg ession T ees)
Figu e 1.2: Pa en node and descendan nodes
Spli ing se s
Le us now con inue wi h he second s ep and alk abou wo necessa y se s Qand
S. Supposed we wan o do uni a ia e bina y spli ing, jus one single a iable Xm,
ha ma ches wi h a coo dina e o he measu emen ec o X, is esponsible o he
spli . In case Xmis ca ego ical aking alues in he se B={b1, . . . , bH}, we can
de ine a se Qlike
Q=Ques ions |Xm
i∈A?, A ⊂B
and in case Xmis a con inuous a iable:
Q=Ques ions |Xm
i< c?.
This way, e e y ques ion o Qsugges s a possible spli , as depic ed in Figu e 1.2.
Wi h ega d o he obse a ions ha ha e eached he node , he ques ion o Qhas
o be o mula ed, and he e a e jus wo possible answe s: YES o NO. Depending on
he answe , he obse a ion eaches now he node o he le , le us call i le = Y ES,
o he one o he igh , namely igh = NO.
Each ques ion gene a es one possible spli . The se o spli s is he one called S. I
he a iable Xm
iis ca ego ical, hen he e a e 2H−1−1possible spli s, because he
numbe o possible spli s is gi en by he numbe o possible ques ions, i.e., by |Q|,
aking in o accoun ha le = Y ES and igh = NO is symme ical o le = NO
and igh = Y ES, in o he wo ds, Xm
i∈Ais symme ical o Xm
i/∈A. I is known ha
Xm
i akes alues in B={b1, . . . , bH}so |B|=H, and hus he numbe o possible
subse s o Bis 2H, bu i mus be conside ed he symme y abo e explained, and i
mus also be conside ed ha ∅(and by symme y also he o al) ha e o be ac o ed
ou . So i ollows ha he numbe o ques ions in Q is |Q|= 2H−1−1, and he e o e
he numbe o possible spli s.
In case he a iable Xm
iis con inuous, con a y o wha may seem, he numbe o
Chap e 1. F om CART o andom o es s 21
dis inc spli s is also ini e. The e a e a mos as many spli s as numbe o obse a ions
N, because his ype o spli s:
1. Conside all he dis inc alues o Xm
i ha appea in L
2. So hem
3. Take cihal way be ween wo o he o de ed alues.
Gini c i e ion
Nex ac ion is al eady he hi d s ep, and i consis s in selec ing he bes spli , among
all he possibili ies. Hence, we need o in oduce wo new concep s: impu i y unc ion
and impu i y measu e.
Le us begin by de ining he impu i y unc ion.
De ini ion 1.1.1. A unc ion φ:P→R, whe e P is de ined as P=(p1, . . . , pK)|
Pkpk= 1, pk≥0, k = 1, .., K, is said o be an impu i y unc ion i i sa is ies he
ollowing p ope ies:
1. φachie es i s maximum only a he poin (1/K, . . . , 1/K)
2. φachie es i s minimum a he poin s o he o m (1,0,...,0),(0,1,0,...,0),
...,(0,...,0,1)
3. φis a symme ic unc ion o p1, . . . , pK.
Le us con inue now explaining he impu i y measu e o a node ,i( ). I is de ined
as ( emembe we de ined p(k| )in (1.1) as he p obabili y o he obse a ion being
o class Cksubjec o ha e eached node ):
i( ) = φ(p(1 | ), . . . , p(K| ))
F om he abo e, he dec ease in impu i y o a spli sin a node is de ined as:
∆i(s, ) = i( )−p igh i( igh )−ple i( le )(1.2)
whe e p igh and ple a e he p opo ion o i ems o he node sen by he spli s o
he node igh and o he node le , espec i ely.
Now, once he impu i y concep s ha e been explained, acco ding o he p obabili y
concep s exposed abo e, we can inally alk abou he c i e ion we men ioned be o e:
Gini C i e ion.

22 1.1. CART (Classi ica ion and Reg ession T ees)
This c i e ion uses he impu i y measu e o a node desc ibed by he ollowing
o mula, called Gini Index:
X
k6=l
p(k| )p(l| )
o wha is he same:
i( ) = (X
k
p(k| ))2−X
k
p2(k| )=1−X
k
p2(k| ).(1.3)
Remembe : p(k| ) e e s o he p obabili y o assigning an i em selec ed a an-
dom om node o class Ck, and p(l| )exp esses he p obabili y o he objec be-
longing o he class Cl.
Using his impu i y measu e, i mus be calcula ed now he dec ease in impu i y
∆i(s, )using (1.2) and (1.3). So,
∆i(s, ) = 1−X
k
p2(k| )−p igh "1−X
k
p2(k| igh )#−ple "1−X
k
p2(k| le )#
∆i(s, ) = −X
k
p2(k| ) + p igh X
k
p2(k| igh ) + ple X
k
p2(k| le )
The aim is o maximize ∆i(s, ). The be e spli is he one ha achie es i .
Any spli s e i ies
∆i(s, )≥0.
Bu , i can be demons a ed ha ∆i(s, )is a conca e unc ion. Hence, ∆i(s, ) = 0 i
and only i
p(k| le ) = p(k| igh ) = p(k| ), k = 1, . . . , K.
1.1.2.3. S op-spli ing ule
Once he spli ing ule has been selec ed, he nex ac ion is o decide he s op-
spli ing ule, i.e, a ule ha indica es when o decla e a node a e minal node.
One o he ini ial s opping ules was based on ixing a h eshold, call i β > 0, o
he maximum dec ease in ee impu i y.
De ini ion 1.1.2. The ee impu i y I:T−→ R, whe e Tis a se o ees, is a unc ion
de ined om he impu i y measu e ias:
I(T) = X
∈T0
I( )
Chap e 1. F om CART o andom o es s 23
whe e T0is a se o e minal nodes achie ed a e some spli ing. Le I( ) = i( )p( )
hen he ee impu i y is:
I(T) = X
∈T0
I( ) = X
∈T0
i( )p( )
So, he dec ease in ee impu i y is gi en by:
∆I(s, ) = I( )−I( igh )−I( le )
Hence, he ea ly s op-spli ing is:
max
s∈S∆I(s, )< β
Ne e heless, his ule was shown no o be ully sa is ac o y and ins ead, he ee
should be p uned, once i has g own much oo la ge.
The e o e is also impo an o explain wha "g own much oo la ge" means. Hence,
le us explain, be o e going on, h ee new concep s: misclassi ica ion cos C(k|l),
esubs i u ion es ima e o he expec ed missclassi ica ion cos o a node , ( ), and
he esubs i u ion es ima e o he missclassi ica ion cos o a ee T,R(T).
We a e going o ackle hese ques ion om a gene al pe spec i e and pa icula ize
o he speci ic case o classi ica ion ees.
De ini ion 1.1.3. Le dbe a unc ion d:X→C, called classi ie . The ue misclas-
si ica ion a e o he unc ion d, deno ed R∗(d), cons uc ed om he lea ning sample
L, indica es how accu a e a classi ie is, i.e., he p obabili y o dmisclassi ying a new
sample d awn om he same dis ibu ion as L, by es ing he classi ie on subsequen
cases whose co ec classi ica ion has been obse ed. This unc ion is gi en by:
R∗(d) = P(d(X)6=Y)
whe e X∈X, Y ∈C.
This means ha a good classi ica o has a low alue o R∗(d). I ollows ha he
p unning c i e ion mus minimize R∗(d).
The ques ion is how can R∗(d)be es ima ed. One o he echniques is using he
esubs i u ion es ima e.
The esubs i u ion es ima e is he p opo ion o cases misclassi ied, and is gi en
by:
R(d) = 1
N
N
X
i=1
X(d(Xi)6=Yi)
24 1.1. CART (Classi ica ion and Reg ession T ees)
whe e Xis he indica o unc ion, which has alue equal o 1 i he a gumen is ue
and 0 i i is alse.
I is used L o cons uc dand also in o de o calcula e R(d). So, i we use he alue
o R(d) o es ima e R∗(d), he esul will be un ealis ic good, e en being possible ha
he alue o R(d) = 0 and so R∗(d) = 0, oo. One way o enhancing he es ima ion,
in case he lea ning sample Lis la ge enough, is o di ide Lin o wo subse s L1and
L2, so ha wi h L1we cons uc dand wi h L2calcula e R(d). The second subse
L2is called es sample.L2can be conside ed independen o L1and om he same
dis ibu ion. Howe e o samples Lo small size, he c oss- alida ion me hod can be
used [16].
Now we can use wha is explained abo e in he case o classi ica ion ees.
De ini ion 1.1.4. C(k|l)is de ined as he cos o misclassi ying, as class Ck, an i em
ha indeed co esponds o class Cl, sa is ying:
C(k|l)≥0i k 6=l
= 0 i k =l
De ini ion 1.1.5. The esubs i u ion es ima e o he expec ed misclassi ica ion cos o
a node , ( )is de ined as:
( ) = min
kX
l
C(k|l)p(l| ),
whe e PlC(k|l)p(l| )is he es ima ed expec ed misclassi ica ion cos o an un-
known class i em ha eaches node and is classi ied as class Ck.
De ini ion 1.1.6. The esubs i u ion es ima e o he misclassi ica ion cos o a ee T,
R(T)is gi en by:
R(T) = X
∈T0
( )p( ) = X
∈T0
R( )
whe e R( ) = ( )p( )
The objec i e is o minimize he alue o R(T). The e o e an equilib ium be ween
wo opposing p ope ies mus be ound:
1. I he numbe o spli s inc eases, hen he alue o R(T)dec eases. In o he
wo ds, i he numbe o e minal nodes inc eases, he alue o R(T)dec eases.
2. A ee ha is g own much oo la ge, so ha jus one i em o he lea ning
sample eaches ha e minal node, is o e i ed, i.e. i classi ies pe ec ly
he gi en lea ning sample, bu i is likely i does no classi y co ec ly a new
sample.
I he numbe o e minal nodes inc eases oo much,R(T)inc eases.
Chap e 1. F om CART o andom o es s 25
Once i has been unde s ood how o measu e "how good" a ee is, we can now
explain he p ocedu e ha mus be ollow in o de o p une he ee p ope ly.
Fi s o all, a ee mus g ow un il all e minal nodes a e pu e: In his i s s ep, he
ee mus g ow un il, o e e y e minal node, i is sa is ied ha only one objec has
eached he e minal node in ques ion. Le us call his ee Tmax. Nex s ep is p uning
upwa d, i.e., cu o nodes successi ely. The inal s ep consis s o choosing om he
collec ion o sub ees o med in he p e ious s ep, he "op imum-sized" ee.
Obse a ion 1.1.1. In ac , a alue Nmin (in gene al i akes he alues 1 o 5) can
be se , and he ee g ows jus un il N( )⩽Nmin is ul illed. (Remembe : N( )is he
o al numbe o obse a ions in L ha ha e eached he node .)
1.1.2.4. Designa ion o class label a e minal nodes
Now i is ime o assign a class label a all he e minal nodes o he ee. The e o e
he class assignmen ule C∗
k( )is used, whe e C∗
k( )deno es ha Ckis he class
gi en o he e minal node . The ule is:
C∗
k( ) = Ck o he k sa is ying p(k| ) = max
lp(l| )
I he maximum is a ained a mo e han one alue o k, any o such kcan be aken.
1.1.3. Con usion ma ix and accu acy
Un il his momen i has been explained how o cons uc a classi ie ype called
classi ica ion ee. The goodness o i o he me hod is de ined h ough a con u-
sion ma ix and he accu acy. The accu acy measu es he p opo ion o well-classi ied
i ems, so a high accu acy indica es ha he classi ie is good. The highe he accu-
acy, he be e he classi ie . Assume he e a e wo possible classes: posi i e and
nega i e. When p edic ing he class o an obse a ion, he e a e ou possible endings:
1. P edic class posi i e, being he u h class posi i e
2. P edic class posi i e, being he u h class nega i e
3. P edic class nega i e, being he u h class nega i e
4. P edic class nega i e, being he u h class posi i e
The ou cases ecei e ollowing names: ue posi i e (TP), alse posi i e (FP), ue
nega i e (TN) and alse nega i e (FN), espec i ely.

Chap e 2
Impo ance o a iables
As al eady poin ed ou , andom o es s lose in e p e abili y compa ed o decision
ees, because he las ones show a di ec in luence o he p edic o a iable a i s
posi ion a he ee. While he decision ees show he in luence ha he p edic o
a iables ha e, di ec ly on hei posi ion a he ee [17], andom o es s a e cons uc ed
in a mo e complex way, and hey do no e eal he dependency di ec ly.
In o de o in e p e andom o es , one o he mos impo an asks in andom
o es s a e o be done: o measu e he impo ance o he a iables[3]. The e a e di e -
en ways o add essing his issue.
1. One o hem, and p obably he simples one, consis s in jus coun ing how
many imes each a iable is selec ed by he ees ha con o m he o es [17].
2. Ano he , mo e complex, measu e o he impo ance o a iables is he Mean
Dec ease Impu i y(MDI). In case he selec ed spli ing c i e ion is he Gini
c i e ion, han he Mean Dec ease Impu i y is called Gini impo ance[17].
This measu e o a iable impo ance is based on he ollowing assump ion:
he Gini Index o a pa en node has a highe alue han he one o i s de-
scendan s [13]. As we explained p e iously, when doing a spli , he "bes "
a iable o he spli mus be selec ed acco ding o he dec ease in impu i y
measu e. So a e aging he dec ease in impu i y o e all he nodes o he ees
in he o es , whe e he a iable is he one selec ed o he spli ing, he MDI is
ob ained. So, he highe he alue o he MDI, he mo e impo an he a iable
[15].
3. A u he way is he pe mu a ion o he alues o he a iable in conside a ion,
called Mean Dec ease Accu acy (MDA) [13]. I consis s in pe mu ing o
i= 1, ..., N he p edic o a iable Xm
i(m− h coo dina e o he measu emen
ec o Xi), dissocia ing his a iable om i s o iginal obse a ion Yi.
33
34
Ma i al s a us Wo king
Ma ied Yes
Single No
Di o ced No
Ma ied Yes
Table 2.1: Zoom on a iable X1
iMa i al s a us o Table 1.1
Ma i al s a us Wo king
Single Yes
Ma ied No
Ma ied No
Di o ced Yes
Table 2.2: Pe mu a ion o he a iable Ma i al s a us up o Table 2.1
Fo a be e unde s anding o he pe mu a ion, le us go back o he example
o Table 1.1. The a iable X1
iwas o he di e en i0s X1
1=Ma ied, X1
2=
Single, X1
3=Di o ced, X1
4=Ma ied, and he co esponden obse a-
ions Y1=Y es, Y2=Y es, Y3=No, Y4=Y es.
Once he pe mu a ion o he a iable is done, he appea ance o Table 2.1
becomes, o ins ance, he one in Table 2.2.
I now he pe mu ed a iable, oge he wi h he non pe mu ed a iables, is
used o p edic he esponse, he e a e wo possible scena ios. The i s con-
sis s in any hing changing wi h espec o he o iginal scena io, whe e no pe -
mu a ion had be done. This would mean ha he pe mu a ion o he a iable
has no in luenced on he esul s, so his a iable is no eally impo an o
he p edic ion. The second possible scena io is ha a e he pe mu a ion he
numbe o obse a ions misclassi ied inc eases, i.e. he accu acy dec eases.
The only jus i ica ion o his ou come is ha he a iable has a signi ican
impo ance o he p edic ion [17]. (I is wo h no ing ha he accu acy is
ob ained by classi ying he OOB sample, so he pe mu a ion o he a iable
akes place jus in he OOB) [19].
I he dec ease in accu acy, by pe mu ing he a iable, is a e aged o e all he
ees in he o es , he MDA is ob ained. So, he highe he MDA, he mo e
impo an is he a iable, because mo e dec eases he accu acy i we pe mu e
he a iable [15].
Chap e 3
Applica ions o andom o es s
Chap e s 1 and 2 ha e been ocused on he heo y o andom o es s. Now, in his
chap e , some applica ions o he abo e explained heo y is shown, unning di e en
expe imen s wi h he so wa e R. The e o e, in Sec ion 3.1 he main ideas o compu ing
andom o es s in Ra e in oduced, in o de o subsequen ly use his so wa e o di -
e en applica ions wi h he da a se Wisconsin b eas cance diagnosis
in Sec ion 3.2, and wi h wo di e en da a se s wi h unc ional da a, namely, eca o
and g ow h, in Sec ion 3.3.
3.1. Random o es s in R
Be o e p esen ing he di e en expe imen s, how andom o es s a e implemen ed
in Ris ou lined in his sec ion.
Fi s s ep is o ins all he package ela ed wi h his i em and hen call such lib a y:
ins all.packages(" andomFo es ")
lib a y( andomFo es )
Once his has been done, i is possible o use he unc ions o such package wi h
he da a a ailable. Hence, i is impo an o unde s and co ec ly how hese unc ions
wo k. Le us s a ocusing on he unc ion andomFo es . The help-en i onmen
o Rp o ides ollowing:
## S3 me hod o class ’ o mula’
andomFo es ( o mula, da a=NULL, ..., subse ,
na.ac ion=na. ail)
## De aul S3 me hod:
35
36 3.1. Random o es s in R
andomFo es (x, y=NULL, x es =NULL, y es =NULL,
n ee=500,
m y=i (!is.null(y) && !is. ac o (y))
max( loo (ncol(x)/3), 1) else
loo (sq (ncol(x))),
eplace=TRUE, classw =NULL, cu o , s a a,
sampsize = i ( eplace) n ow(x)
else ceiling(.632*n ow(x)),
nodesize = i (!is.null(y) &&
!is. ac o (y)) 5 else 1,
maxnodes = NULL,
impo ance=FALSE, localImp=FALSE, nPe m=1,
p oximi y, oob.p ox=p oximi y,
no m. o es=TRUE, do. ace=FALSE,
keep. o es =!is.null(y) && is.null(x es ),
co .bias=FALSE,
keep.inbag=FALSE, ...)
As mos o he a gumen s o he unc ion andomFo es a e explained by hem-
sel es, he mos ele an ones a e p esen ed in Table 3.1.
A gumen Meaning and wo king
o mula da a ame o ma ix o p edic o s, o o mula desc ibing he
model o be i ed
da a da a ame con aining he a iables
n ee Numbe o ees o g ow
m y Numbe o a iables andomly sampled as candida es a each
spli . By de aul o classi ica ion i akes p(p), o eg ession
p
3, wi h p he numbe o a iables
impo ance i i is se TRUE i calcula es he impo ance o he p edic o s
p oximi y i i is se TRUE i calcula es he p oximi y ma ix
Table 3.1: Meaning and wo king o he mos impo an a gumen s o andomFo es
Once he main poin s o he compu ing ha e been p esen ed, wo di e en expe i-
men s a e going o be exposed. Fo he i s one, a da a se con o med by con inuous
p edic o a iables, and a ca ego ical esponse a iable is going o be analyzed, ocus-
ing on he a iable impo ance. Fo he second expe imen he use o andom o es s is
going o be ex ended o unc ional da a.
Chap e 3. Applica ions o andom o es s 37
3.2. Expe imen I
The da a ha is used along his applica ion o andom o es s, has been ob ained
om he UCI Machine Lea ning Reposi o y [11], and co esponds o he Wisconsin
b eas cance diagnosis. This da a is con o med by 31 p edic o a iables, and a ca e-
go ical esponse a iable. F om among he p edic o s a iables we dis inguish he ID
numbe (se in column one) and he en p incipal ones (se om column 3 o 12), ha
ep esen he means:
1)ID numbe
3) adius (mean o dis ances om cen e o poin s
on he pe ime e )
4) ex u e (s anda d de ia ion o g ay-scale alues)
5) pe ime e
6) a ea
7) smoo hness (local a ia ion in adius leng hs)
8) compac ness (pe ime e ^2 / a ea - 1.0)
9) conca i y (se e i y o conca e po ions o
he con ou )
10) conca e poin s (numbe o conca e po ions o he
con ou )
11) symme y
12) ac al dimension ("coas line app oxima ion" - 1)
The emaining a iables (13 o 30) a e he s anda d e o and he wo s case o he
10 al eady exposed p edic o a iables.
The esponse a iable is:
2)Diagnosis (M = malignan , B = benign)
The i s s ep consis s in eading, and se ing a headline o he da a.
headline=c("Id","Classi ica ion","Radius","Tex u e",
"Pe ime e ","A ea","Smoo hness","Compac ness",
"Conca i y","Conca e Poin s","Symme y",
"F ac al dimension","Radius_se","Tex u e_se",
"Pe ime e _se","A ea_se","Smoo hness_se",
"Compac ness_se","Conca i y_se","Conca e
Poin s_se", "Symme y_se","F ac al
dimension_se","Radius_wo s ","Tex u e_wo s ",
"Pe ime e _wo s ","A ea_wo s ",

38 3.2. Expe imen I
"Smoo hness_wo s ","Compac ness_wo s ",
"Conca i y_wo s ","Conca e Poin s_wo s ",
"Symme y_wo s ","F ac al dimension_wo s ")
wisconsin= ead. able("wdbc.da a",sep = ",",
col.names = headline)
and in o de o be able o call he a iables by hei names:
a ach(wisconsin)
Now he da a is co ec ly implemen ed and can be used o he expe imen .
In o de o allow he eplica ion o his expe imen , i is impo an o se a seed, and
" ix" he andomness. Nex , he andomFo es is execu ed like i can be seen, using
i e a iables (√31) a each spli and g owing 1000 ees, and he esul s a e p in ed:
se .seed(1081)
w. = andomFo es (Classi ica ion~.,da a=wisconsin,
n ee=1000,m y=5,impo ance=TRUE)
p in (w. )
So, i is ob ained:
Call:
andomFo es ( o mula = Classi ica ion ~ .,
da a = wisconsin,n ee = 1000,m y = 5,
impo ance = TRUE)
Type o andom o es : classi ica ion
Numbe o ees: 1000
No. o a iables ied a each spli : 5
OOB es ima e o e o a e: 3.34%
Con usion ma ix:
B M class.e o
B 350 7 0.01960784
M 12 200 0.05660377
ha can be in e p e ed as ollows. The OOB e o is 3.34 %, i.e. his is he pe cen age
o i ems o he OOB ha ha e been w ongly classi ied, o , in o he wo ds, he andom
o es classi ies 96.66%co ec ly. Also he con usion ma ix is ep esen ed, so i can
Chap e 3. Applica ions o andom o es s 39
be seen how many i ems ha e been co ec ly classi ied o no , i.e., he alse posi i es
(5.66%) and he alse nega i es (1.96%).
Abo e can be seen, ha in o de o ob ain in o ma ion abou he impo ance o he
a iables, he en y impo ance=TRUE has also been included as an a gumen o
he unc ion andomFo es . Jus using now
impo ance(w. )
a lis wi h he MDA and he MDI o each a iable o he model is ob ained:
MeanDec easeAccu acy MeanDec easeGini
Id 4.918394 1.3460295
Radius 13.210533 10.2696696
Tex u e 17.081849 4.0069453
Pe ime e 14.565036 14.0129864
A ea 14.726296 10.8019264
Smoo hness 10.008769 1.7488165
Compac ness 8.583551 2.3981941
Conca i y 16.030364 10.3178601
Conca e.Poin s 21.449959 29.4106724
Symme y 5.431248 1.1330892
F ac al.dimension 5.452440 0.9137801
Radius_se 14.810328 4.0634558
Tex u e_se 5.709398 1.2645055
Pe ime e _se 13.998440 4.2729203
A ea_se 19.755545 8.5036794
Smoo hness_se 4.534937 1.1640510
Compac ness_se 8.130864 1.3049383
Conca i y_se 8.041339 1.6496968
Conca e.Poin s_se 7.312380 1.1211280
Symme y_se 5.618074 1.0870049
F ac al.dimension_se 4.237092 1.3857821
Radius_wo s 24.541105 31.6980086
Tex u e_wo s 19.268786 5.0469406
Pe ime e _wo s 23.814279 32.1185216
A ea_wo s 25.123753 29.3425631
Smoo hness_wo s 16.132225 3.2753597
Compac ness_wo s 11.639484 4.2820763
Conca i y_wo s 19.114623 8.8452845
Conca e.Poin s_wo s 25.047172 34.6431429
40 3.2. Expe imen I
Symme y_wo s 9.118319 2.2282686
F ac al.dimension_wo s 8.311042 1.8237781
To in e p e his mo e easily, le us show i g aphically in Figu e 3.1 om:
a ImpPlo (w. ,n. a = 31)
Figu e 3.1: Impo ance o a iables
As explained in Chap e 2, he highe he MDA (o he MDI), he mo e impo an
he a iable (acco ding o he espec i e c i e ia). Al hough he esul s o he wo me h-
Chap e 3. Applica ions o andom o es s 41
ods a e, in gene al, no exac ly he same, bo h will iden i y e y impo an o e y su-
pe luous a iables. In his case, acco ding o he MDA he mos impo an a iables, in
descendan o de o impo ance a e: Conca e.Poin s_wo s , A ea_wo s ,
Radius_wo s and Pe ime e _wo s . Fo MDI c i e ion hese ou a e also
he mos impo an a iables, pe mu ing he o de o Conca e.Poin s_wo s and
Pe ime e _wo s . When conside ing a iables ha a e no impo an o he an-
dom o es , i s ands ou ha he MDA conside s as comple ely supe luous a iables
less a iables han MDI. Fo MDA we ha e, i we conside o example MDA un-
de 10 as unimpo an , in inc easing o de o impo ance: Smoo hness_se, Id,
Symme y, F ac al dimension, Symme y_se, Tex u e_se and a se-
ies o no ha impo an a iables as can be seen in he g aphic. Ne e heless, ac-
co ding o MDI, he e a e a lo o supe luous a iables, i su ices o ake a look
a he g aphic and see how many a iables ha e a alue MDI=5 (no e en unde 10
as was done wi h MDA), ou s anding F ac al.dimension,Symme y_se,
Conca e.Poin s_se and Symme y among o he s. As migh be expec ed, in
bo h cases, Id is conside ed an unimpo an a iable, because a numbe o iden i ica-
ion assigned o a pa ien canno be a eason o de ec cance .
Figu e 3.2: Ranking o he a iables acco ding o MDA and MDI
F om he abo e in o ma ion, i can be deduced ha he e exis some simila i ies
be ween he anking o he a iables, when lis ing hem acco ding o he MDA o he
MDI. So i we ake a look a Figu e 3.2 he ela ions be ween he posi ion in a iable
impo ance ha he di e en a iables ake, in conco dance o bo h c i e ia, can be
obse ed. E e y a iable is gi en he posi ion hey ake in he anking o a iable
48 3.3. Expe imen II
Va i1
Va i2
Va i3
Va i4
Va i5
Va i6
Va i7
Va i8
Va i9
Va i10
Va i11
Va i12
Va i13
Va i14
Va i15
Va i16
Va i17
Va i18
Va i19
Va i20
Va i21
Va i22
Va i23
Va i24
Va i25
Va i26
Va i27
Va i28
Va i29
Va i30
Va i31
Va i32
Va i33
Va i34
Va i35
Va i36
Va i37
Va i38
Va i39
Va i40
Va i41
Va i42
Va i43
Va i44
Va i45
Va i46
Va i47
Va i48
Va i49
Va i50
Va i51
Va i52
Va i53
Va i54
Va i55
Va i56
Va i57
Va i58
Va i59
Va i60
Va i61
Va i62
Va i63
Va i64
Va i65
Va i66
Va i67
Va i68
Va i69
Va i70
Va i71
Va i72
Va i73
Va i74
Va i75
Va i76
Va i77
Va i78
Va i79
Va i80
Va i81
Va i82
Va i83
Va i84
Va i85
Va i86
Va i87
Va i88
Va i89
Va i90
Va i91
Va i92
Va i93
Va i94
Va i95
Va i96
Va i97
Va i98
Va i99
Va i100
0 5 10 15
alue
a iables
ype
MDA
MDI
Figu e 3.5: Va iable Impo ance

Chap e 3. Applica ions o andom o es s 49
(a) Cu es o i s de i a i e as unc ions o
he wa eleng h
(b) Cu es o second de i a i e as unc ions o he wa e-
leng h
Figu e 3.6: Fi s wo de i a i es o he abso bances wi h colo as class-dis inc ion
The i s de i a i e is buil as ollows:
De i =ma ix(,n ow=n ow(Explica i ),
ncol=ncol(Explica i )-1)
o (i in 1:n ow(Explica i )){
o (j in 1:ncol(Explica i )-1){
De i [i,j]=Explica i [i,j+1]-Explica i [i,j]
}}
So now i can be compu ed he andom o es wi h he i s de i a i e, he same
way as i was done wi h he o iginal unc ion jus eplacing Explica i o De i
and naming also he new a iables.
Da aDe i 1=da a. ame(De i ,Response)
se .seed(1081)
D1= andomFo es (Res~.,da a=Da aDe i 1,n ee=1000,
impo ance=TRUE,p oximi y=TRUE)
p in ( D1)
In his case a OOB e o o 3,26%is ob ained, i.e. i has been educed conside ably.
When in oducing he second de i a i e, i.e., using:
De i 2=ma ix(,n ow=n ow(De i ),
ncol=ncol(De i )-1)
o (i in 1:n ow(De i )){
o (j in 1:ncol(De i )-1){
De i 2[i,j]=De i [i,j+1]-De i [i,j]
}}
50 3.3. Expe imen II
NamesDe i2=c("De i1",...,"Res")
Da aDe i 2=da a. ame(De i 2,Response)
colnames(Da aDe i 2)<-NamesDe i2
a ach(Da aDe i 2)
se .seed(1081)
D2= andomFo es (Res~.,da a=Da aDe i 2,n ee=1000,
impo ance=TRUE,p oximi y=TRUE)
p in ( D2)
he OOB e o is educed un il 0,93%.
I he esul s o he o he combina ions o unc ion and de i a i es be o e men-
ioned a e o be calcula ed, he ollowing mus be compu ed:
Names oge he 1=c("Va i1",...,"De i1",...,"Res")
Va iablesToge he 1=cbind(Explica i ,De i )
Da osToge he 1=da a. ame(Va iablesToge he 1,Response)
colnames(Da osToge he 1)<-Names oge he 1
se .seed(1081)
T1= andomFo es (Res~.,da a=Da osToge he 1,n ee=1000,
impo ance=TRUE,p oximi y=TRUE)
p in ( T1)
IT1=impo ance( T1)
a ImpPlo ( T1)
#################
Names oge he =c("Va i1",...,"2De i1",...,
"Res")
Va iablesToge he 2=cbind(Explica i ,De i 2)
Da osToge he 2=da a. ame(Va iablesToge he 2,Response)
colnames(Da osToge he 2)<-Names oge he 2
a ach(Da osToge he 2)
se .seed(1081)
T2= andomFo es (Res~.,da a=Da osToge he 2,n ee=1000,
impo ance=TRUE,p oximi y=TRUE)
p in ( T2)
IT2=impo ance( T2)
a ImpPlo ( T2)
##################
Chap e 3. Applica ions o andom o es s 51
Names oge he =c("Va i1",...,"De i1",...,"2De i1",...,
"Res")
Va iablesToge he =cbind(Explica i ,De i ,De i 2)
Da osToge he =da a. ame(Va iablesToge he ,Response)
colnames(Da osToge he )<-Names oge he
a ach(Da osToge he )
se .seed(1081)
T= andomFo es (Res~.,da a=Da osToge he ,n ee=1000,
impo ance=TRUE,p oximi y=TRUE)
p in ( T)
IT=impo ance( T)
a ImpPlo ( T)
##################
Names oge he D=c("De i1",...,"2De i1",...,"Res")
Va iablesToge he D=cbind(De i ,De i 2)
Da osToge he D=da a. ame(Va iablesToge he D,Response)
colnames(Da osToge he D)<-Names oge he D
a ach(Da osToge he D)
se .seed(1081)
TD= andomFo es (Res~.,da a=Da osToge he D,n ee=1000,
impo ance=TRUE,p oximi y=TRUE)
p in ( TD)
ITD=impo ance( TD)
a ImpPlo ( TD)
Le us now see he OOB e o s o he di e en combina ions in Table 3.2.
Func ions used o he explica i e a iables OOB e o (%)
16,28
D1 3,26
D2 0,93
+D1+D2 0,93
+D1 2,79
+D2 0,93
Table 3.2: OOB e o s o he di e en combina ions in he classi ica ion p oblem o
eca o
52 3.3. Expe imen II
The esul ob ained jus wi h he second de i a i e (D2) is exac ly he same esul ha
is ob ained when using he second de i a i e and he unc ion wi h ( +D1+D2) o
wi hou ( +D2) he i s de i a i e, i.e. he a iables o he second de i a i e a e he
ones ha be e he esul s. When using he o iginal unc ion and he i s de i a i e
( +D1) he esul s a e much be e han using only he o iginal unc ion ( ) and a ew
be e han using only he i s de i a i e (D1), bu no as good as when using he second
de i a i e (D2), i.e. he i s de i a i e be e s he esul s ob ained wi h he o iginal
unc ion bu no as much as he second de i a i e does. So, answe ing he ques ion
posed abo e, o his da a he use o he de i a i es is a clea way o imp o ing he
OOB.
(a) MDA o o iginal unc ion, i s de i a i e and second de i a i e
(b) MDI o o iginal unc ion, i s de i a i e and second de i a i e
Figu e 3.7: Va iable impo ance di e en de i a i es
Ano he ques ion ha can be ied o be answe ed is i he a iables ha a e impo -
Chap e 3. Applica ions o andom o es s 53
an o he o iginal unc ion, he i s de i a i e and he second de i a i e co espond
o he same wa eleng h. When ep esen ing he MDA and MDI o he andom o es s
ob ained up o he o iginal unc ion, he i s de i a i e (D1) and he second de i a-
i e (D2), app oxima ely he a iables co esponding o he same wa eleng hs a e
impo an , as can be seen in Figu es3.7(a) and 3.7(b). Ne e heless, i is impo an
o men ion ha , when speaking abou a ange, wha has been jus said is co ec ly,
bu when speaking abou he alues inside ha ange i ollows ha no exac ly he
same wa eleng hs a e impo an o he dis inc unc ions. Taking a look a he ange
be ween 900nm and 950nm app oxima ely, i s ands ou ha his ange con ains he
mos impo an wa eleng hs, independen o speaking o he o iginal unc ion, he i s
o he second de i a i e. Ne e heless, inside he ange i mus be emphasized ha he
o iginal unc ion and he second de i a i e ha e as mos impo an wa eleng hs mo e
o less he same alues a ound 930nm, bu a his poin , he i s de i a i e achie es a
minimum. I ollows ha he i s de i a i e is complemen a y o he o iginal unc ion
and he second de i a i e.
3.3.1.2. Reg ession o eca o
In he beginning i was said ha he da a se eca o o iginally consis s o a com-
pound o obse a ions wi h he a iable esponse a index, a con inuous a iable, ha
we decided o ans o m in o a quali a i e a iable g ouping he a indexes in wo
classes: high and low a index, in o de o add ess his p oblem as a classi ica ion
p oblem. Ne e heless, i is also possible o use andom o es echniques when using
a index as a con inuous a iable, conside ing his ime he p oblem as a eg ession
p oblem. P oceding his way, he only di e ence wi h espec o he classi ica ion p ob-
lem (when compu ing i wi h R) is o keep he o iginal esponse a iable like we can
see o he case wi h he o iginal unc ion (Fo he es o he p og amming jus keep
on (wi h Response coded like he e) he same way as was done wi h he classi ica ion
p oblem):
Explica i = eca o $abso p. da a$da a
Response= eca o $y$Fa
Da a=da a. ame(Explica i ,Response)
Le us see now wha happens when using he di e en combina ions o o iginal
unc ion and he i s wo de i a i es as was done wi h he classi ica ion p oblem.
The e o e, see Table 3.3 (The esul s a e ob ained wi h seed 1081).
In ela ion wi h wha can been obse ed in he Table 3.3 i can be said ha when
using he o iginal unc ion and he wo de i a i es ( +D1+D2) he bes esul is ob-
ained. Ne e heless, he esul s a e e y simila o he ones o jus using he second
de i a i e o he second de i a i e and he o iginal unc ion (D2 o +D2). When jus

54 3.3. Expe imen II
Func ions used o he
explica i e a iables Va iable explained(%) Mean squa ed esiduals
70,41 47,80386
D1 98,68 2,135725
D2 99,05 1,526871
+D1+D2 99,32 1,093536
+D1 98,68 2,134657
+D2 99,01 1,595362
Table 3.3: Di e en combina ions in he eg ession p oblem o eca o
using he i s de i a i e (D1) o he i s de i a i e and he o iginal unc ion ( +D1) he
same esul is ob ained o he pe cen age o he explained a iabili y o he esponse
a iable, p obably because he bes esul s when spli ing a e ob ained o he a iables
o he i s de i a i e (in con as o he a iables o he o iginal unc ion), so using
o he andom o es jus he i s de i a i e (D1) o i s combina ion wi h he o iginal
unc ion ( +D1) does no p esen any di e ence. The wo s esul is ob ained when
using jus he o iginal unc ion ( ).
3.3.2. G ow h
The da ase called g ow h, a ailable wi h he lib a y( da) o Rhas also
been add essed as a classi ica ion p oblem. This da a se ep esen s he unc ion o he
heigh o 39 boys and 54 gi ls be ween he ages o 1 and 18, disc e ized in o 31 poin s,
no equally spaced (1, 1.25, 1.5, 1.75, 2, 3, 4, 5, 6, 7, 8, 8.5, 9, 9.5, 10, 10.5, 11, 11.5,
12, 12.5, 13, 13.5, 14, 14.5, 15, 15.5, 16, 16.5, 17, 17.5, 18). So his way o each
indi idual i he e a e 31 a iables Xm
i, wi h m= 1, ..., 31 ha can be used in o de
o classi y wi h andom o es s he esponse a iable, he sex o he indi idual. The
di e en cu es o heigh o he di e en ages a e d awn in Figu e 3.8, in di e en
colo s depending i i co esponds o a boy o a gi l.
As done wi h he eca o da a se , he wo classes o he esponse a iable o he
g ow h da a a e going o be coded:
• "1" i i is a boy
• "-1" i i is a gi l
So ollowing mus be compu ed:
Chap e 3. Applica ions o andom o es s 55
Figu e 3.8: Heigh a he ages be ween 1 and 18
lib a y( da.usc)
lib a y( andomFo es )
####################################################
da a("g ow h")
names=c("Va i1","Va i2","Va i3","Va i4","Va i5","Va i6",
"Va i7","Va i8","Va i9","Va i10","Va i11",
"Va i12","Va i13","Va i14","Va i15","Va i16",
"Va i17","Va i18","Va i19","Va i20","Va i21",
"Va i22","Va i23","Va i24","Va i25","Va i26",
"Va i27","Va i28","Va i29","Va i30","Va i31",
"Res")
boys= ep(1,39)
gi ls= ep(-1,54)
Explica i = bind( (g ow h$hg m), (g ow h$hg ))
Response1=c(boys,gi ls)
Response= ac o (Response1)
Da a=da a. ame(Explica i ,Response)
colnames(Da a)<-names
a ach(Da a)
So, when compu ing he andom o es wi h he o iginal unc ion
56 3.3. Expe imen II
se .seed(33)
= andomFo es (Res~.,da a=Da a,n ee=1000,
impo ance=TRUE,p oximi y=TRUE)
p in ( )
ha will ha e as ou pu :
Type o andom o es : classi ica ion
Numbe o ees: 1000
No. o a iables ied a each spli : 5
OOB es ima e o e o a e: 8.6%
Con usion ma ix:
-1 1 class.e o
-1 48 6 0.11111111
1 2 37 0.05128205
So, he OOB e o ob ained is o 8,6% ha we will y o imp o e as we did wi h
he eca o da a se . The con usion ma ix is also shown in he ou pu , wi h 48 " ue
posi i e", i.e. in his case, 48 gi ls classi ied as gi ls, and 37 " ue nega i e", i.e. 37
boys classi ied by he andom o es as boys.
Figu e 3.9: Va iable Impo ance
Le us now see he impo ance o a iables. Acco ding o Figu e 3.8, i migh be
expec ed ha he mos impo an a iables a e hose o ages ound 15, because un il
Chap e 3. Applica ions o andom o es s 57
ha momen , he cu es o boys and gi ls a e e y simila , and only up o he men ioned
age, he cu es a e di e en o mos o he indi iduals o bo h classes. So, aking a
look a Figu e 3.9, i can been obse ed ha , he mos impo an a iables, acco ding
o MDI and MDA a e Va i27 un il Va i31, which co espond o he ages 16 o 18,
i.e. he esul ob ained is cohe en wi h wha was expec ed.
Nex s ep is o see wha happens when in oducing he i s and he second de i a-
i es. The e o e, le us see i s he cu es o hese unc ions in Figu es 3.10(a) and
3.10(b). F om Figu e 3.10(a) can i can be deduced ha he mos impo an ages in
o de o dis inguish i he pe son is a boy o a gi l, a e app oxima ely he ones o e
13 yea s, because up o ha momen , he cu es o boys and gi ls a e di e en . When
ocusing on he second de i a i e in Figu e 3.10(b) i canno be done such a deduc ion.
(a) Cu es o i s de i a i e o he heigh as
unc ions o he age
(b) Cu es o second de i a i e o he heigh as
unc ions o he age
Figu e 3.10: Fi s wo de i a i es o he heigh wi h colo as class-dis inc ion
Taking a look a Figu es3.11(a) and 3.11(b) i can be seen now which a e he mos
impo an a iables o he andom o es s when using he i s and he second de i a-
i es. Figu e 3.11(a) shows ha he a iables De i19 un il De i30 a e he mos
impo an ones, i.e. he ages 12.25 un il 17.75 as was p edic ed by Figu e 3.10(a).
Ne e heless, when looking a he impo an ages acco ding o he second de i a i e
di e en " alleys" and "moun ains" can be iden i ied, s anding ou he moun ain be-
ween De i17 and De i20, and he one be ween De i22 and De i27, co e-
sponding o he ages be ween 11.5and 13, and be ween 14 and 16.5, espec i ely. I
can also be seen ha in he co esponding Figu e o he i s as also o he second
de i a i e, some MDA alues a e nega i e. This means ha when pe mu ing he a i-
able alues o e he di e en obse a ions he accu acy o he andom o es s o new
da a is imp o ed. Ne e heless, in his case we a e wo king wi h a small da abase
and addi ionally hese alues o MDA a e e y low in con as o he a iables ha a e
conside ed as impo an , so hey can be conside ed as ze o.
64
a node
TT ee
T0Se o e minal nodes
I(T)T ee Impu i y
R∗(d)T ue misclassi ica ion a e o he
unc ion d
R(d)Resubs i u ion es ima e
C(k|l)Cos o misclassi ying
( )Resubs i u ion es ima e o he
expec ed misclassi ica ion cos
o he node
R(T)Resubs i u ion es ima e o he ee T
C∗
k( )Class assignmen ule
FNumbe o ees o he o es
P ox P oximi y ma ix
P ox(i, j)Cell (i,j) o he p oximi y ma ix
ui,k Measu e o ou lyingness
˜ui,k No malized measu e o ou lyingness
mkMean o measu e o ou lyingness
o e all elemen s o class Ck
JkNumbe o obse a ions o class Ck

Bibliog aphy
[1] Nelson Lee A anado , Agnieszka Smolinska, Thanh N. T an, and Lionel
Blanche . Unsupe ised andom o es : a u o ial wi h case s udies. Chemo-
me ics, 30:232–241, 2016.
[2] Philippe Sain -Pie e Bap is e G ego u i, Be and Michel. Co ela ion and a i-
able impo ance in andom o es s. S a Compu , 27:659–678, 2017.
[3] Gé a d Biau and E wan Sco ne . A andom o es guided ou . Tes , 25(2):197–
227, 2016.
[4] R. Blanque o, E. Ca izosa, A. Jiménez-Co de o, and B. Ma ín-Ba agán.
Func ional-bandwid h ke nel o suppo ec o machine wi h unc ional da a: An
al e na ing op imiza ion algo i hm. Eu opean Jou nal o Ope a ional Resea ch,
275(1):195 – 207, 2019.
[5] Leo B eiman. Random o es s. Machine Lea ning, 45.
[6] Leo B eiman. S a is ical modeling: The wo cul u es. S a is ical Science,
16(3):199–215, 2001.
[7] Leo B eiman. Manual on se ing up, using, and unde s anding andom o es s
4.0. Machine Lea ning. Jou nal, 2003.
[8] Leo B eiman, Je ome H. F iedman, Richa d A. Olshen, and Cha les J. S one.
Classi ica ion And Reg ession T ees. CHAPMAN HALL/CRC, 1984.
[9] Emilio Ca izosa and Dolo es Rome o Mo ales. Supe ised classi ica ion and
ma hema ical op imiza ion. Compu e s Ope a ions Resea ch,40(1),150-165,
2013.
[10] Ramón Díaz U ia e and Sa a Al a ez de And és. Gene selec ion and classi ica-
ion o mic oa ay da a using andom o es . BMC Bioin o ma ics, 7:3, 2006.
[11] Dhee u Dua and Casey G a . UCI machine lea ning eposi o y, 2017.
[12] F. Hillie and G. Liebe man. In oduccion a la in es igacion de ope aciones.
Mexico, D.F: McG aw-Hill/In e ame icana, 2010.
65
66 Bibliog aphy
[13] Alan Julian Izenman. Mode n Mul i a ia e S a is ical Techniques. Reg es-
sion,Classi ica ion, and Mani old Lea ning. Sp inge , 2013.
[14] Robe C. Messenge James N. Mo gan. THAID,a Sequen ial Analysis P og am
o he Analysis o Nominal Scale Dependen Va iables. Su ey Resea ch Cen e ,
Ins i u e o Social Resea ch, Uni e si y o Michigan, 1973.
[15] M.R. O iz-Posadas. Pa e n Recogni ion Techniques Applied o Biomedical
P oblems. Sp inge .
[16] Ma ha E. S one. C oss- alida ion: A e iew. Ma h. Ope a ion o sch. S a is . Se
S a is , 9:127–139, 1977.
[17] Ca olin S obl, Anne-Lau e Boules eix, Achim Zeileis, and To s en Ho ho n.
Bias in andom o es a iable impo ance measu es: Illus a ions, sou ces and
a solu ion.”. BMC bioin o ma ics, 8(1):25, 2007.
[18] Vladimi S e nik, Andy Liaw, Ch is ophe Tong, J. Ch is ophe Culbe son,
Robe P. She idan, and B adley P. Feus on. Random o es : a classi ica ion
and eg ession ool o compound classi ica ion and qsa modeling. Jou nal o
Chemical In o ma ion and Compu e Sciences, 43(6):1947–1958, 2003. PMID:
14632445.
[19] Je ome F iedman T e o Has ie, Robe Tibshi ani. The Elemen s o S a is ical
Lea ning: Da a Mining, In e ence and P edic ion. Sp inge , 2009.
[20] Hal R. Va ian. Big da a: New icks o econome ics. Jou nal o Economic
Pe spec i es, 28(2):3–28, 2014.