Random Forests: Properties and applications

FINAL DEGREE PROJECT Random Fo es s: P ope ies and applica ions P esen ed by: Fá ima del Pila Villalba Piza o Supe ised by: DR. EMILIO CARRIZOSA PRIEGO FACULTY OF MATHEMATICS S a is ics and Ope a ional Resea ch Depa men Se ille, Sep embe 2020 Índice gene al Resumen 5 Abs ac 7 In oducción 9 In oduc ion 13 1. F om CART o andom o es s 15 1.1. CART (Classi ica ion and Reg ession T ees) . . . . . . . . . . . . . . 15 1.1.1. S uc u e o decision ees . . . . . . . . . . . . . . . . . . . 16 1.1.2. Cons uc ion o decision ees . . . . . . . . . . . . . . . . . 17 1.1.2.1. Topology and ype o spli ing . . . . . . . . . . . . 18 1.1.2.2. Spli ing ules . . . . . . . . . . . . . . . . . . . . . 18 1.1.2.3. S op-spli ing ule . . . . . . . . . . . . . . . . . . 22 1.1.2.4. Designa ion o class label a e minal nodes . . . . 25 1.1.3. Con usion ma ix and accu acy . . . . . . . . . . . . . . . . 25 1.1.4. Reg ession T ees . . . . . . . . . . . . . . . . . . . . . . . . 26 1.2. Random o es s............................. 26 1.2.1. Cons uc ion o andom o es s . . . . . . . . . . . . . . . . . 27 1.2.1.1. Bagging(Boo s ap-agg ega ing) . . . . . . . . . . 27 1.2.1.2. Randomness in spli ing . . . . . . . . . . . . . . . 28 1.2.1.3. Unp unning . . . . . . . . . . . . . . . . . . . . . 29 1.2.2. P oximi y ma ix . . . . . . . . . . . . . . . . . . . . . . . . 29 1.2.2.1. Dissimila i y ma ix [13][7] . . . . . . . . . . . . . 30 1.2.2.2. Ou lie s ....................... 31 2. Impo ance o a iables 33 3. Applica ions o andom o es s 35 3.1. Random o es s in R........................... 35 3.2. Expe imen I .............................. 37 3.3. Expe imen II.............................. 45 3 4 Índice gene al 3.3.1. Teca o ............................. 45 3.3.1.1. Classi ica ion o eca o . . . . . . . . . . . . . . . 45 3.3.1.2. Reg ession o eca o . . . . . . . . . . . . . . . . 53 3.3.2. G ow h ............................. 54 4. Conclusions 61 Glossa y 63 Bibliog aphy 64 Resumen Los bosques alea o ios son una he amien a undamen al en el ámbi o del ap en- dizaje supe isado, po lo que son empleados en mul i ud de disciplinas, demos ando g andes esul ados y múl iples en ajas en p oblemas de clasi icación y eg esión. En es e abajo se exponen los undamen os y bases de los bosques alea o ios, des acando sus en ajosas ca ac e ís icas, pa a pos e io men e hace di e en es aplica- ciones con ellos, con las que con as a dichas en ajas, obse a dis in os aspec os que se ha án pa en es e incluso analiza su compo amien o cuando ex endemos su uso al caso de los da os uncionales. 5 Abs ac Random o es s a e conside ed a undamen al ool in supe ised lea ning. Conse- quen ly, andom o es s a e used in a wide ange o disciplines, yielding g ea esul s and demons a ing many ad an ages in classi ica ion and eg ession p oblems. Along his wo k, he bases and ounda ions o andom o es s a e exposed, empha- sizing hei ad an ageous p ope ies, o subsequen ly make di e en applica ions wi h hem, demons a ing hei ad an ages, analyzing dis inc aspec s ha become no ice- able, and e en s udying he way hey beha e when ex ending hei use o unc ional da a. 7 In oducción En un mundo en cons an e a ance y desa ollo, es impo an e usa écnicas que mo- delen la ealidad de mane a e icien e. Muchos p oblemas de eg esión y clasi icación se pueden esol e median e ap endizaje supe isado, es deci , median e una écnica en la que, dada una mues a con dis in os da os en o ma de pa es, uno de en ada (no - malmen e un ec o ) y o o de salida, se les a ibuye un alo numé ico o clase a cada pa , con el in de pode p edeci la salida de u u os da os. Es e mé odo ha ido cob ando impo ancia en los úl imos años omando un papel ele an e como puede e se en la Figu a 1, que nos mues a el in e és susci ado po es e ema en elación con el núme o de búsquedas del é mino “ap endizaje supe isado" (en ingles “supe ised lea ning") en Google. Figu a 1: E olución del núme o de búsquedas del é mino “ap endizaje supe isado" (supe ised lea ning) en Google desde 2004 Una he amien a pa icula de ap endizaje supe isado es la conocida como bos- ques alea o ios, o del inglés, andom o es s, desa ollados en el año 2001 po Leo B eiman [6]. Es a écnica se usa en dis in as á eas como son la econome ía [20], la quimioin o má ica [18], la bioin o má ica [10] o la medicina. En cuan o a es a úl i- ma, se iene que los bosques alea o ios son ú iles en di e en es campos como son la selección de ma cado es gené icos esponsables de en e medades, la mic obiología o la epidemiología gené ica, e incluso pa a p edeci la eplicación de un i us como el HIV-1 [17]. La ele ancia de los bosques alea o ios se debe a que, a di e encia de o as écnicas, los bosques alea o ios son e icaces incluso cuando se abaja con a iables con bas an e 9 16 1.1. CART (Classi ica ion and Reg ession T ees) Decision ees a e conside ed leade s in hei a ea because o hei easy in e - p e abili y. The eason he e o e esides in he scheme ha ollows such a ee: an i - hen ule. [9] 1.1.1. S uc u e o decision ees Le us explain now how decision ees wo k. We will begin wi h classi ica ion he ees and hen go on wi h an in dep h e iew o he eg ession ees. Decision ees can handle any ype o a iables, ca ego ical (i.e., hey ake alues in a ini e se no ha ing any na u al o de ) o nume ical/con inuous (i.e. i is a eal numbe ). Le us see i s which is he s uc u e o such a ee. A classi ica ion ee is a collec ion o nodes and b anches ha display a pa i ion o he o iginal se . Le 0be such o iginal se . This se can be spli in o a compound o se s ha o m a pa i ion o 0. Fo simplici y, le us hink we a e wo king wi h he simples ype o ee, he bina y ee, which means a each spli wo new se s a e ob ained. I we name he new se s ob ained om 0, 1and 2i is ul illed 0= 1 2. This p ocess is epea ed wi h such new se s. All hese se s a e called nodes. Th ee ypes o nodes a e ound: The o iginal se 0, called oo node. The subse s ha a e no spli , called e minal nodes. These o m a pa i ion o 0. Each e minal node is assigned a class label, and i is possible o ind mo e han jus one e minal node wi h he same class label. The emaining nodes, called non e minal nodes. Such s uc u e o a decision ee can be seen in Figu e 1.1. Figu e 1.1: S c u e o a decision ee Chap e 1. F om CART o andom o es s 17 Ma i al s a us Age Wo king Ma ied 28 Yes Single 37 No Di o ced 42 No Ma ied 41 Yes Table 1.1: Example o lea ning sample L 1.1.2. Cons uc ion o decision ees Fo he cons uc ion o he classi ica ion ee, a lea ning sample mus be gi en. Le L={(Xi, Yi)i=1,..,N }deno e he lea ning sample, whe e Nis he numbe o obse a ions, Xi∈Xa ec o o M a iables, named measu emen ec o , in he measu emen space X, and Yi he co esponding obse a ion om among Kpossible classes C=C1, ..., CK. Example 1.1.1. An example o wha is explained abo e can be seen in Table 1.1, whe e i can be iden i ied: N= 4,M= 2,K= 2 X1is he a iable Ma i als a us ha akes alues in Ma ied, Single, Di o ced X2is he a iable Age ha akes alues in 18,19,20, . . .  L=(X1 1=Ma ied, X2 1= 28, Y1=Y es),(X1 2=Single, X2 2= 37, Y2=No),(X1 3=Di o ced, X2 3= 42, Y3=No),(X1 4=Ma ied, X2 4= 41, Y4=Y es) Acco ding o he gi en lea ning sample he classi ica ion ee can be buil . The cons uc ion o a ee is based on ou poin s: 1. Topology and ype o spli ing 2. Selec ion o spli s 3. Decla a ion o e minal nodes/S op-spli ing ule 18 1.1. CART (Classi ica ion and Reg ession T ees) 4. Designa ion o class label a e minal nodes The main goal is cons uc ing he classi ica ion ee om he aining se L. We now ou line some de ails o he abo e men ioned elemen s. 1.1.2.1. Topology and ype o spli ing Acco ding o he numbe o new nodes esul ing om he spli ing, wo ypes o spli ing can be dis inguished: bina y spli ing, i.e., he e a e ob ained wo new nodes, o mul i-spli ing, ha induces mo e han wo new nodes. Fo con inuous a iables he second one is no eally use ul. I can also be se a di e ence be ween wo ypes o spli s acco ding o he numbe o a iables in ol ed in he spli ing. I only one a iable akes pa in he spli , i is called uni a ia e spli , o he wise, i is named mul i a ia e spli . In wha ollows, jus bina y ees, wi h uni a ia e spli will be conside ed. 1.1.2.2. Spli ing ules In his sec ion we will assume we a e wo king wi h a s anda d s uc u e, i.e., all measu emen ec o s Xia e o ixed dimensionali y. Many di e en c i e ia o spli ing ha e been adop ed. Ou o hen, Leo B eimann highligh s wo, namely: Gini c i e ion Twoing c i e ion The Gini c i e ion is he one ha is going o be explained, because i is also he one ha uses he p og am R ha is going o be used in he las chap e . Be o e going down o he explana ion o how his c i e ion ope a es, we mus i s in oduce a ew undamen al concep s in wo ini ial s eps, ha will con e ge in he hi d and las s ep. 1. Explana ion o concep s ela ed wi h p obabili y 2. In oduc ion o se s o all possible spli ing 3. Explana ion o selec ing he bes spli : Gini c i e ion P obabili y concep s Beginning wi h he i s s ep: Remembe we said we had he aining sample L= (Xi, Yi)i=1,..,N , wi h Xi∈Xand dimension M, and Yi∈C=C1, ..., CK. Chap e 1. F om CART o andom o es s 19 The p io class p obabili ies π(k),1≤k≤Kis de ined as: π(k) = P(Yi=Ck). Le Nkbe he numbe o cases o Lin class Ck. O en he p io p obabili ies can be assessed like: π(k) = Nk N,1≤k≤K. Suppose now we a e a node . Main aining he same idea as abo e, le N( )be he o al numbe o obse a ions in L ha ha e eached he node , and le Nk( )deno e he numbe o obse a ions o class Ckin . So, he quo ien Nk( ) Nk is in e p e ed as he p opo ion o obse a ions o class Ck alling in o . Using bo h de ini ions, we ob ain he p obabili y o an obse a ion in L ul illing wo aspec s simul aneously: alling in o and being o class Ck, gi en by p(k, ) = π(k)Nk( ) Nk . I we sum in k, we ob ain he p obabili y o an obse a ion o any class Ck eaching he node p( ) = X k p(k, ), and we can de ine now he p obabili y he obse a ion is o class Cksubjec o ha ing eached node p(k| ) = p(k, ) p( ) sa is ying: X k p(k| ) = 1 Acco ding o he app oach done o π(k)we can also assume ha p(k| ) = Nk( ) N( ).(1.1) 20 1.1. CART (Classi ica ion and Reg ession T ees) Figu e 1.2: Pa en node and descendan nodes Spli ing se s Le us now con inue wi h he second s ep and alk abou wo necessa y se s Qand S. Supposed we wan o do uni a ia e bina y spli ing, jus one single a iable Xm, ha ma ches wi h a coo dina e o he measu emen ec o X, is esponsible o he spli . In case Xmis ca ego ical aking alues in he se B={b1, . . . , bH}, we can de ine a se Qlike Q=Ques ions |Xm i∈A?, A ⊂B and in case Xmis a con inuous a iable: Q=Ques ions |Xm i< c?. This way, e e y ques ion o Qsugges s a possible spli , as depic ed in Figu e 1.2. Wi h ega d o he obse a ions ha ha e eached he node , he ques ion o Qhas o be o mula ed, and he e a e jus wo possible answe s: YES o NO. Depending on he answe , he obse a ion eaches now he node o he le , le us call i le = Y ES, o he one o he igh , namely igh = NO. Each ques ion gene a es one possible spli . The se o spli s is he one called S. I he a iable Xm iis ca ego ical, hen he e a e 2H−1−1possible spli s, because he numbe o possible spli s is gi en by he numbe o possible ques ions, i.e., by |Q|, aking in o accoun ha le = Y ES and igh = NO is symme ical o le = NO and igh = Y ES, in o he wo ds, Xm i∈Ais symme ical o Xm i/∈A. I is known ha Xm i akes alues in B={b1, . . . , bH}so |B|=H, and hus he numbe o possible subse s o Bis 2H, bu i mus be conside ed he symme y abo e explained, and i mus also be conside ed ha ∅(and by symme y also he o al) ha e o be ac o ed ou . So i ollows ha he numbe o ques ions in Q is |Q|= 2H−1−1, and he e o e he numbe o possible spli s. In case he a iable Xm iis con inuous, con a y o wha may seem, he numbe o Chap e 1. F om CART o andom o es s 21 dis inc spli s is also ini e. The e a e a mos as many spli s as numbe o obse a ions N, because his ype o spli s: 1. Conside all he dis inc alues o Xm i ha appea in L 2. So hem 3. Take cihal way be ween wo o he o de ed alues. Gini c i e ion Nex ac ion is al eady he hi d s ep, and i consis s in selec ing he bes spli , among all he possibili ies. Hence, we need o in oduce wo new concep s: impu i y unc ion and impu i y measu e. Le us begin by de ining he impu i y unc ion. De ini ion 1.1.1. A unc ion φ:P→R, whe e P is de ined as P=(p1, . . . , pK)| Pkpk= 1, pk≥0, k = 1, .., K, is said o be an impu i y unc ion i i sa is ies he ollowing p ope ies: 1. φachie es i s maximum only a he poin (1/K, . . . , 1/K) 2. φachie es i s minimum a he poin s o he o m (1,0,...,0),(0,1,0,...,0), ...,(0,...,0,1) 3. φis a symme ic unc ion o p1, . . . , pK. Le us con inue now explaining he impu i y measu e o a node ,i( ). I is de ined as ( emembe we de ined p(k| )in (1.1) as he p obabili y o he obse a ion being o class Cksubjec o ha e eached node ): i( ) = φ(p(1 | ), . . . , p(K| )) F om he abo e, he dec ease in impu i y o a spli sin a node is de ined as: ∆i(s, ) = i( )−p igh i( igh )−ple i( le )(1.2) whe e p igh and ple a e he p opo ion o i ems o he node sen by he spli s o he node igh and o he node le , espec i ely. Now, once he impu i y concep s ha e been explained, acco ding o he p obabili y concep s exposed abo e, we can inally alk abou he c i e ion we men ioned be o e: Gini C i e ion. 22 1.1. CART (Classi ica ion and Reg ession T ees) This c i e ion uses he impu i y measu e o a node desc ibed by he ollowing o mula, called Gini Index: X k6=l p(k| )p(l| ) o wha is he same: i( ) = (X k p(k| ))2−X k p2(k| )=1−X k p2(k| ).(1.3) Remembe : p(k| ) e e s o he p obabili y o assigning an i em selec ed a an- dom om node o class Ck, and p(l| )exp esses he p obabili y o he objec be- longing o he class Cl. Using his impu i y measu e, i mus be calcula ed now he dec ease in impu i y ∆i(s, )using (1.2) and (1.3). So, ∆i(s, ) = 1−X k p2(k| )−p igh "1−X k p2(k| igh )#−ple "1−X k p2(k| le )# ∆i(s, ) = −X k p2(k| ) + p igh X k p2(k| igh ) + ple X k p2(k| le ) The aim is o maximize ∆i(s, ). The be e spli is he one ha achie es i . Any spli s e i ies ∆i(s, )≥0. Bu , i can be demons a ed ha ∆i(s, )is a conca e unc ion. Hence, ∆i(s, ) = 0 i and only i p(k| le ) = p(k| igh ) = p(k| ), k = 1, . . . , K. 1.1.2.3. S op-spli ing ule Once he spli ing ule has been selec ed, he nex ac ion is o decide he s op- spli ing ule, i.e, a ule ha indica es when o decla e a node a e minal node. One o he ini ial s opping ules was based on ixing a h eshold, call i β > 0, o he maximum dec ease in ee impu i y. De ini ion 1.1.2. The ee impu i y I:T−→ R, whe e Tis a se o ees, is a unc ion de ined om he impu i y measu e ias: I(T) = X ∈T0 I( ) Chap e 1. F om CART o andom o es s 23 whe e T0is a se o e minal nodes achie ed a e some spli ing. Le I( ) = i( )p( ) hen he ee impu i y is: I(T) = X ∈T0 I( ) = X ∈T0 i( )p( ) So, he dec ease in ee impu i y is gi en by: ∆I(s, ) = I( )−I( igh )−I( le ) Hence, he ea ly s op-spli ing is: max s∈S∆I(s, )< β Ne e heless, his ule was shown no o be ully sa is ac o y and ins ead, he ee should be p uned, once i has g own much oo la ge. The e o e is also impo an o explain wha "g own much oo la ge" means. Hence, le us explain, be o e going on, h ee new concep s: misclassi ica ion cos C(k|l), esubs i u ion es ima e o he expec ed missclassi ica ion cos o a node , ( ), and he esubs i u ion es ima e o he missclassi ica ion cos o a ee T,R(T). We a e going o ackle hese ques ion om a gene al pe spec i e and pa icula ize o he speci ic case o classi ica ion ees. De ini ion 1.1.3. Le dbe a unc ion d:X→C, called classi ie . The ue misclas- si ica ion a e o he unc ion d, deno ed R∗(d), cons uc ed om he lea ning sample L, indica es how accu a e a classi ie is, i.e., he p obabili y o dmisclassi ying a new sample d awn om he same dis ibu ion as L, by es ing he classi ie on subsequen cases whose co ec classi ica ion has been obse ed. This unc ion is gi en by: R∗(d) = P(d(X)6=Y) whe e X∈X, Y ∈C. This means ha a good classi ica o has a low alue o R∗(d). I ollows ha he p unning c i e ion mus minimize R∗(d). The ques ion is how can R∗(d)be es ima ed. One o he echniques is using he esubs i u ion es ima e. The esubs i u ion es ima e is he p opo ion o cases misclassi ied, and is gi en by: R(d) = 1 N N X i=1 X(d(Xi)6=Yi) 24 1.1. CART (Classi ica ion and Reg ession T ees) whe e Xis he indica o unc ion, which has alue equal o 1 i he a gumen is ue and 0 i i is alse. I is used L o cons uc dand also in o de o calcula e R(d). So, i we use he alue o R(d) o es ima e R∗(d), he esul will be un ealis ic good, e en being possible ha he alue o R(d) = 0 and so R∗(d) = 0, oo. One way o enhancing he es ima ion, in case he lea ning sample Lis la ge enough, is o di ide Lin o wo subse s L1and L2, so ha wi h L1we cons uc dand wi h L2calcula e R(d). The second subse L2is called es sample.L2can be conside ed independen o L1and om he same dis ibu ion. Howe e o samples Lo small size, he c oss- alida ion me hod can be used [16]. Now we can use wha is explained abo e in he case o classi ica ion ees. De ini ion 1.1.4. C(k|l)is de ined as he cos o misclassi ying, as class Ck, an i em ha indeed co esponds o class Cl, sa is ying: C(k|l)≥0i k 6=l = 0 i k =l De ini ion 1.1.5. The esubs i u ion es ima e o he expec ed misclassi ica ion cos o a node , ( )is de ined as: ( ) = min kX l C(k|l)p(l| ), whe e PlC(k|l)p(l| )is he es ima ed expec ed misclassi ica ion cos o an un- known class i em ha eaches node and is classi ied as class Ck. De ini ion 1.1.6. The esubs i u ion es ima e o he misclassi ica ion cos o a ee T, R(T)is gi en by: R(T) = X ∈T0 ( )p( ) = X ∈T0 R( ) whe e R( ) = ( )p( ) The objec i e is o minimize he alue o R(T). The e o e an equilib ium be ween wo opposing p ope ies mus be ound: 1. I he numbe o spli s inc eases, hen he alue o R(T)dec eases. In o he wo ds, i he numbe o e minal nodes inc eases, he alue o R(T)dec eases. 2. A ee ha is g own much oo la ge, so ha jus one i em o he lea ning sample eaches ha e minal node, is o e i ed, i.e. i classi ies pe ec ly he gi en lea ning sample, bu i is likely i does no classi y co ec ly a new sample. I he numbe o e minal nodes inc eases oo much,R(T)inc eases. Chap e 1. F om CART o andom o es s 25 Once i has been unde s ood how o measu e "how good" a ee is, we can now explain he p ocedu e ha mus be ollow in o de o p une he ee p ope ly. Fi s o all, a ee mus g ow un il all e minal nodes a e pu e: In his i s s ep, he ee mus g ow un il, o e e y e minal node, i is sa is ied ha only one objec has eached he e minal node in ques ion. Le us call his ee Tmax. Nex s ep is p uning upwa d, i.e., cu o nodes successi ely. The inal s ep consis s o choosing om he collec ion o sub ees o med in he p e ious s ep, he "op imum-sized" ee. Obse a ion 1.1.1. In ac , a alue Nmin (in gene al i akes he alues 1 o 5) can be se , and he ee g ows jus un il N( )⩽Nmin is ul illed. (Remembe : N( )is he o al numbe o obse a ions in L ha ha e eached he node .) 1.1.2.4. Designa ion o class label a e minal nodes Now i is ime o assign a class label a all he e minal nodes o he ee. The e o e he class assignmen ule C∗ k( )is used, whe e C∗ k( )deno es ha Ckis he class gi en o he e minal node . The ule is: C∗ k( ) = Ck o he k sa is ying p(k| ) = max lp(l| ) I he maximum is a ained a mo e han one alue o k, any o such kcan be aken. 1.1.3. Con usion ma ix and accu acy Un il his momen i has been explained how o cons uc a classi ie ype called classi ica ion ee. The goodness o i o he me hod is de ined h ough a con u- sion ma ix and he accu acy. The accu acy measu es he p opo ion o well-classi ied i ems, so a high accu acy indica es ha he classi ie is good. The highe he accu- acy, he be e he classi ie . Assume he e a e wo possible classes: posi i e and nega i e. When p edic ing he class o an obse a ion, he e a e ou possible endings: 1. P edic class posi i e, being he u h class posi i e 2. P edic class posi i e, being he u h class nega i e 3. P edic class nega i e, being he u h class nega i e 4. P edic class nega i e, being he u h class posi i e The ou cases ecei e ollowing names: ue posi i e (TP), alse posi i e (FP), ue nega i e (TN) and alse nega i e (FN), espec i ely. Chap e 2 Impo ance o a iables As al eady poin ed ou , andom o es s lose in e p e abili y compa ed o decision ees, because he las ones show a di ec in luence o he p edic o a iable a i s posi ion a he ee. While he decision ees show he in luence ha he p edic o a iables ha e, di ec ly on hei posi ion a he ee [17], andom o es s a e cons uc ed in a mo e complex way, and hey do no e eal he dependency di ec ly. In o de o in e p e andom o es , one o he mos impo an asks in andom o es s a e o be done: o measu e he impo ance o he a iables[3]. The e a e di e - en ways o add essing his issue. 1. One o hem, and p obably he simples one, consis s in jus coun ing how many imes each a iable is selec ed by he ees ha con o m he o es [17]. 2. Ano he , mo e complex, measu e o he impo ance o a iables is he Mean Dec ease Impu i y(MDI). In case he selec ed spli ing c i e ion is he Gini c i e ion, han he Mean Dec ease Impu i y is called Gini impo ance[17]. This measu e o a iable impo ance is based on he ollowing assump ion: he Gini Index o a pa en node has a highe alue han he one o i s de- scendan s [13]. As we explained p e iously, when doing a spli , he "bes " a iable o he spli mus be selec ed acco ding o he dec ease in impu i y measu e. So a e aging he dec ease in impu i y o e all he nodes o he ees in he o es , whe e he a iable is he one selec ed o he spli ing, he MDI is ob ained. So, he highe he alue o he MDI, he mo e impo an he a iable [15]. 3. A u he way is he pe mu a ion o he alues o he a iable in conside a ion, called Mean Dec ease Accu acy (MDA) [13]. I consis s in pe mu ing o i= 1, ..., N he p edic o a iable Xm i(m− h coo dina e o he measu emen ec o Xi), dissocia ing his a iable om i s o iginal obse a ion Yi. 33 34 Ma i al s a us Wo king Ma ied Yes Single No Di o ced No Ma ied Yes Table 2.1: Zoom on a iable X1 iMa i al s a us o Table 1.1 Ma i al s a us Wo king Single Yes Ma ied No Ma ied No Di o ced Yes Table 2.2: Pe mu a ion o he a iable Ma i al s a us up o Table 2.1 Fo a be e unde s anding o he pe mu a ion, le us go back o he example o Table 1.1. The a iable X1 iwas o he di e en i0s X1 1=Ma ied, X1 2= Single, X1 3=Di o ced, X1 4=Ma ied, and he co esponden obse a- ions Y1=Y es, Y2=Y es, Y3=No, Y4=Y es. Once he pe mu a ion o he a iable is done, he appea ance o Table 2.1 becomes, o ins ance, he one in Table 2.2. I now he pe mu ed a iable, oge he wi h he non pe mu ed a iables, is used o p edic he esponse, he e a e wo possible scena ios. The i s con- sis s in any hing changing wi h espec o he o iginal scena io, whe e no pe - mu a ion had be done. This would mean ha he pe mu a ion o he a iable has no in luenced on he esul s, so his a iable is no eally impo an o he p edic ion. The second possible scena io is ha a e he pe mu a ion he numbe o obse a ions misclassi ied inc eases, i.e. he accu acy dec eases. The only jus i ica ion o his ou come is ha he a iable has a signi ican impo ance o he p edic ion [17]. (I is wo h no ing ha he accu acy is ob ained by classi ying he OOB sample, so he pe mu a ion o he a iable akes place jus in he OOB) [19]. I he dec ease in accu acy, by pe mu ing he a iable, is a e aged o e all he ees in he o es , he MDA is ob ained. So, he highe he MDA, he mo e impo an is he a iable, because mo e dec eases he accu acy i we pe mu e he a iable [15]. Chap e 3 Applica ions o andom o es s Chap e s 1 and 2 ha e been ocused on he heo y o andom o es s. Now, in his chap e , some applica ions o he abo e explained heo y is shown, unning di e en expe imen s wi h he so wa e R. The e o e, in Sec ion 3.1 he main ideas o compu ing andom o es s in Ra e in oduced, in o de o subsequen ly use his so wa e o di - e en applica ions wi h he da a se Wisconsin b eas cance diagnosis in Sec ion 3.2, and wi h wo di e en da a se s wi h unc ional da a, namely, eca o and g ow h, in Sec ion 3.3. 3.1. Random o es s in R Be o e p esen ing he di e en expe imen s, how andom o es s a e implemen ed in Ris ou lined in his sec ion. Fi s s ep is o ins all he package ela ed wi h his i em and hen call such lib a y: ins all.packages(" andomFo es ") lib a y( andomFo es ) Once his has been done, i is possible o use he unc ions o such package wi h he da a a ailable. Hence, i is impo an o unde s and co ec ly how hese unc ions wo k. Le us s a ocusing on he unc ion andomFo es . The help-en i onmen o Rp o ides ollowing: ## S3 me hod o class ’ o mula’ andomFo es ( o mula, da a=NULL, ..., subse , na.ac ion=na. ail) ## De aul S3 me hod: 35 36 3.1. Random o es s in R andomFo es (x, y=NULL, x es =NULL, y es =NULL, n ee=500, m y=i (!is.null(y) && !is. ac o (y)) max( loo (ncol(x)/3), 1) else loo (sq (ncol(x))), eplace=TRUE, classw =NULL, cu o , s a a, sampsize = i ( eplace) n ow(x) else ceiling(.632*n ow(x)), nodesize = i (!is.null(y) && !is. ac o (y)) 5 else 1, maxnodes = NULL, impo ance=FALSE, localImp=FALSE, nPe m=1, p oximi y, oob.p ox=p oximi y, no m. o es=TRUE, do. ace=FALSE, keep. o es =!is.null(y) && is.null(x es ), co .bias=FALSE, keep.inbag=FALSE, ...) As mos o he a gumen s o he unc ion andomFo es a e explained by hem- sel es, he mos ele an ones a e p esen ed in Table 3.1. A gumen Meaning and wo king o mula da a ame o ma ix o p edic o s, o o mula desc ibing he model o be i ed da a da a ame con aining he a iables n ee Numbe o ees o g ow m y Numbe o a iables andomly sampled as candida es a each spli . By de aul o classi ica ion i akes p(p), o eg ession p 3, wi h p he numbe o a iables impo ance i i is se TRUE i calcula es he impo ance o he p edic o s p oximi y i i is se TRUE i calcula es he p oximi y ma ix Table 3.1: Meaning and wo king o he mos impo an a gumen s o andomFo es Once he main poin s o he compu ing ha e been p esen ed, wo di e en expe i- men s a e going o be exposed. Fo he i s one, a da a se con o med by con inuous p edic o a iables, and a ca ego ical esponse a iable is going o be analyzed, ocus- ing on he a iable impo ance. Fo he second expe imen he use o andom o es s is going o be ex ended o unc ional da a. Chap e 3. Applica ions o andom o es s 37 3.2. Expe imen I The da a ha is used along his applica ion o andom o es s, has been ob ained om he UCI Machine Lea ning Reposi o y [11], and co esponds o he Wisconsin b eas cance diagnosis. This da a is con o med by 31 p edic o a iables, and a ca e- go ical esponse a iable. F om among he p edic o s a iables we dis inguish he ID numbe (se in column one) and he en p incipal ones (se om column 3 o 12), ha ep esen he means: 1)ID numbe 3) adius (mean o dis ances om cen e o poin s on he pe ime e ) 4) ex u e (s anda d de ia ion o g ay-scale alues) 5) pe ime e 6) a ea 7) smoo hness (local a ia ion in adius leng hs) 8) compac ness (pe ime e ^2 / a ea - 1.0) 9) conca i y (se e i y o conca e po ions o he con ou ) 10) conca e poin s (numbe o conca e po ions o he con ou ) 11) symme y 12) ac al dimension ("coas line app oxima ion" - 1) The emaining a iables (13 o 30) a e he s anda d e o and he wo s case o he 10 al eady exposed p edic o a iables. The esponse a iable is: 2)Diagnosis (M = malignan , B = benign) The i s s ep consis s in eading, and se ing a headline o he da a. headline=c("Id","Classi ica ion","Radius","Tex u e", "Pe ime e ","A ea","Smoo hness","Compac ness", "Conca i y","Conca e Poin s","Symme y", "F ac al dimension","Radius_se","Tex u e_se", "Pe ime e _se","A ea_se","Smoo hness_se", "Compac ness_se","Conca i y_se","Conca e Poin s_se", "Symme y_se","F ac al dimension_se","Radius_wo s ","Tex u e_wo s ", "Pe ime e _wo s ","A ea_wo s ", 38 3.2. Expe imen I "Smoo hness_wo s ","Compac ness_wo s ", "Conca i y_wo s ","Conca e Poin s_wo s ", "Symme y_wo s ","F ac al dimension_wo s ") wisconsin= ead. able("wdbc.da a",sep = ",", col.names = headline) and in o de o be able o call he a iables by hei names: a ach(wisconsin) Now he da a is co ec ly implemen ed and can be used o he expe imen . In o de o allow he eplica ion o his expe imen , i is impo an o se a seed, and " ix" he andomness. Nex , he andomFo es is execu ed like i can be seen, using i e a iables (√31) a each spli and g owing 1000 ees, and he esul s a e p in ed: se .seed(1081) w. = andomFo es (Classi ica ion~.,da a=wisconsin, n ee=1000,m y=5,impo ance=TRUE) p in (w. ) So, i is ob ained: Call: andomFo es ( o mula = Classi ica ion ~ ., da a = wisconsin,n ee = 1000,m y = 5, impo ance = TRUE) Type o andom o es : classi ica ion Numbe o ees: 1000 No. o a iables ied a each spli : 5 OOB es ima e o e o a e: 3.34% Con usion ma ix: B M class.e o B 350 7 0.01960784 M 12 200 0.05660377 ha can be in e p e ed as ollows. The OOB e o is 3.34 %, i.e. his is he pe cen age o i ems o he OOB ha ha e been w ongly classi ied, o , in o he wo ds, he andom o es classi ies 96.66%co ec ly. Also he con usion ma ix is ep esen ed, so i can Chap e 3. Applica ions o andom o es s 39 be seen how many i ems ha e been co ec ly classi ied o no , i.e., he alse posi i es (5.66%) and he alse nega i es (1.96%). Abo e can be seen, ha in o de o ob ain in o ma ion abou he impo ance o he a iables, he en y impo ance=TRUE has also been included as an a gumen o he unc ion andomFo es . Jus using now impo ance(w. ) a lis wi h he MDA and he MDI o each a iable o he model is ob ained: MeanDec easeAccu acy MeanDec easeGini Id 4.918394 1.3460295 Radius 13.210533 10.2696696 Tex u e 17.081849 4.0069453 Pe ime e 14.565036 14.0129864 A ea 14.726296 10.8019264 Smoo hness 10.008769 1.7488165 Compac ness 8.583551 2.3981941 Conca i y 16.030364 10.3178601 Conca e.Poin s 21.449959 29.4106724 Symme y 5.431248 1.1330892 F ac al.dimension 5.452440 0.9137801 Radius_se 14.810328 4.0634558 Tex u e_se 5.709398 1.2645055 Pe ime e _se 13.998440 4.2729203 A ea_se 19.755545 8.5036794 Smoo hness_se 4.534937 1.1640510 Compac ness_se 8.130864 1.3049383 Conca i y_se 8.041339 1.6496968 Conca e.Poin s_se 7.312380 1.1211280 Symme y_se 5.618074 1.0870049 F ac al.dimension_se 4.237092 1.3857821 Radius_wo s 24.541105 31.6980086 Tex u e_wo s 19.268786 5.0469406 Pe ime e _wo s 23.814279 32.1185216 A ea_wo s 25.123753 29.3425631 Smoo hness_wo s 16.132225 3.2753597 Compac ness_wo s 11.639484 4.2820763 Conca i y_wo s 19.114623 8.8452845 Conca e.Poin s_wo s 25.047172 34.6431429 40 3.2. Expe imen I Symme y_wo s 9.118319 2.2282686 F ac al.dimension_wo s 8.311042 1.8237781 To in e p e his mo e easily, le us show i g aphically in Figu e 3.1 om: a ImpPlo (w. ,n. a = 31) Figu e 3.1: Impo ance o a iables As explained in Chap e 2, he highe he MDA (o he MDI), he mo e impo an he a iable (acco ding o he espec i e c i e ia). Al hough he esul s o he wo me h- Chap e 3. Applica ions o andom o es s 41 ods a e, in gene al, no exac ly he same, bo h will iden i y e y impo an o e y su- pe luous a iables. In his case, acco ding o he MDA he mos impo an a iables, in descendan o de o impo ance a e: Conca e.Poin s_wo s , A ea_wo s , Radius_wo s and Pe ime e _wo s . Fo MDI c i e ion hese ou a e also he mos impo an a iables, pe mu ing he o de o Conca e.Poin s_wo s and Pe ime e _wo s . When conside ing a iables ha a e no impo an o he an- dom o es , i s ands ou ha he MDA conside s as comple ely supe luous a iables less a iables han MDI. Fo MDA we ha e, i we conside o example MDA un- de 10 as unimpo an , in inc easing o de o impo ance: Smoo hness_se, Id, Symme y, F ac al dimension, Symme y_se, Tex u e_se and a se- ies o no ha impo an a iables as can be seen in he g aphic. Ne e heless, ac- co ding o MDI, he e a e a lo o supe luous a iables, i su ices o ake a look a he g aphic and see how many a iables ha e a alue MDI=5 (no e en unde 10 as was done wi h MDA), ou s anding F ac al.dimension,Symme y_se, Conca e.Poin s_se and Symme y among o he s. As migh be expec ed, in bo h cases, Id is conside ed an unimpo an a iable, because a numbe o iden i ica- ion assigned o a pa ien canno be a eason o de ec cance . Figu e 3.2: Ranking o he a iables acco ding o MDA and MDI F om he abo e in o ma ion, i can be deduced ha he e exis some simila i ies be ween he anking o he a iables, when lis ing hem acco ding o he MDA o he MDI. So i we ake a look a Figu e 3.2 he ela ions be ween he posi ion in a iable impo ance ha he di e en a iables ake, in conco dance o bo h c i e ia, can be obse ed. E e y a iable is gi en he posi ion hey ake in he anking o a iable 48 3.3. Expe imen II Va i1 Va i2 Va i3 Va i4 Va i5 Va i6 Va i7 Va i8 Va i9 Va i10 Va i11 Va i12 Va i13 Va i14 Va i15 Va i16 Va i17 Va i18 Va i19 Va i20 Va i21 Va i22 Va i23 Va i24 Va i25 Va i26 Va i27 Va i28 Va i29 Va i30 Va i31 Va i32 Va i33 Va i34 Va i35 Va i36 Va i37 Va i38 Va i39 Va i40 Va i41 Va i42 Va i43 Va i44 Va i45 Va i46 Va i47 Va i48 Va i49 Va i50 Va i51 Va i52 Va i53 Va i54 Va i55 Va i56 Va i57 Va i58 Va i59 Va i60 Va i61 Va i62 Va i63 Va i64 Va i65 Va i66 Va i67 Va i68 Va i69 Va i70 Va i71 Va i72 Va i73 Va i74 Va i75 Va i76 Va i77 Va i78 Va i79 Va i80 Va i81 Va i82 Va i83 Va i84 Va i85 Va i86 Va i87 Va i88 Va i89 Va i90 Va i91 Va i92 Va i93 Va i94 Va i95 Va i96 Va i97 Va i98 Va i99 Va i100 0 5 10 15 alue a iables ype MDA MDI Figu e 3.5: Va iable Impo ance Chap e 3. Applica ions o andom o es s 49 (a) Cu es o i s de i a i e as unc ions o he wa eleng h (b) Cu es o second de i a i e as unc ions o he wa e- leng h Figu e 3.6: Fi s wo de i a i es o he abso bances wi h colo as class-dis inc ion The i s de i a i e is buil as ollows: De i =ma ix(,n ow=n ow(Explica i ), ncol=ncol(Explica i )-1) o (i in 1:n ow(Explica i )){ o (j in 1:ncol(Explica i )-1){ De i [i,j]=Explica i [i,j+1]-Explica i [i,j] }} So now i can be compu ed he andom o es wi h he i s de i a i e, he same way as i was done wi h he o iginal unc ion jus eplacing Explica i o De i and naming also he new a iables. Da aDe i 1=da a. ame(De i ,Response) se .seed(1081) D1= andomFo es (Res~.,da a=Da aDe i 1,n ee=1000, impo ance=TRUE,p oximi y=TRUE) p in ( D1) In his case a OOB e o o 3,26%is ob ained, i.e. i has been educed conside ably. When in oducing he second de i a i e, i.e., using: De i 2=ma ix(,n ow=n ow(De i ), ncol=ncol(De i )-1) o (i in 1:n ow(De i )){ o (j in 1:ncol(De i )-1){ De i 2[i,j]=De i [i,j+1]-De i [i,j] }} 50 3.3. Expe imen II NamesDe i2=c("De i1",...,"Res") Da aDe i 2=da a. ame(De i 2,Response) colnames(Da aDe i 2)<-NamesDe i2 a ach(Da aDe i 2) se .seed(1081) D2= andomFo es (Res~.,da a=Da aDe i 2,n ee=1000, impo ance=TRUE,p oximi y=TRUE) p in ( D2) he OOB e o is educed un il 0,93%. I he esul s o he o he combina ions o unc ion and de i a i es be o e men- ioned a e o be calcula ed, he ollowing mus be compu ed: Names oge he 1=c("Va i1",...,"De i1",...,"Res") Va iablesToge he 1=cbind(Explica i ,De i ) Da osToge he 1=da a. ame(Va iablesToge he 1,Response) colnames(Da osToge he 1)<-Names oge he 1 se .seed(1081) T1= andomFo es (Res~.,da a=Da osToge he 1,n ee=1000, impo ance=TRUE,p oximi y=TRUE) p in ( T1) IT1=impo ance( T1) a ImpPlo ( T1) ################# Names oge he =c("Va i1",...,"2De i1",..., "Res") Va iablesToge he 2=cbind(Explica i ,De i 2) Da osToge he 2=da a. ame(Va iablesToge he 2,Response) colnames(Da osToge he 2)<-Names oge he 2 a ach(Da osToge he 2) se .seed(1081) T2= andomFo es (Res~.,da a=Da osToge he 2,n ee=1000, impo ance=TRUE,p oximi y=TRUE) p in ( T2) IT2=impo ance( T2) a ImpPlo ( T2) ################## Chap e 3. Applica ions o andom o es s 51 Names oge he =c("Va i1",...,"De i1",...,"2De i1",..., "Res") Va iablesToge he =cbind(Explica i ,De i ,De i 2) Da osToge he =da a. ame(Va iablesToge he ,Response) colnames(Da osToge he )<-Names oge he a ach(Da osToge he ) se .seed(1081) T= andomFo es (Res~.,da a=Da osToge he ,n ee=1000, impo ance=TRUE,p oximi y=TRUE) p in ( T) IT=impo ance( T) a ImpPlo ( T) ################## Names oge he D=c("De i1",...,"2De i1",...,"Res") Va iablesToge he D=cbind(De i ,De i 2) Da osToge he D=da a. ame(Va iablesToge he D,Response) colnames(Da osToge he D)<-Names oge he D a ach(Da osToge he D) se .seed(1081) TD= andomFo es (Res~.,da a=Da osToge he D,n ee=1000, impo ance=TRUE,p oximi y=TRUE) p in ( TD) ITD=impo ance( TD) a ImpPlo ( TD) Le us now see he OOB e o s o he di e en combina ions in Table 3.2. Func ions used o he explica i e a iables OOB e o (%) 16,28 D1 3,26 D2 0,93 +D1+D2 0,93 +D1 2,79 +D2 0,93 Table 3.2: OOB e o s o he di e en combina ions in he classi ica ion p oblem o eca o 52 3.3. Expe imen II The esul ob ained jus wi h he second de i a i e (D2) is exac ly he same esul ha is ob ained when using he second de i a i e and he unc ion wi h ( +D1+D2) o wi hou ( +D2) he i s de i a i e, i.e. he a iables o he second de i a i e a e he ones ha be e he esul s. When using he o iginal unc ion and he i s de i a i e ( +D1) he esul s a e much be e han using only he o iginal unc ion ( ) and a ew be e han using only he i s de i a i e (D1), bu no as good as when using he second de i a i e (D2), i.e. he i s de i a i e be e s he esul s ob ained wi h he o iginal unc ion bu no as much as he second de i a i e does. So, answe ing he ques ion posed abo e, o his da a he use o he de i a i es is a clea way o imp o ing he OOB. (a) MDA o o iginal unc ion, i s de i a i e and second de i a i e (b) MDI o o iginal unc ion, i s de i a i e and second de i a i e Figu e 3.7: Va iable impo ance di e en de i a i es Ano he ques ion ha can be ied o be answe ed is i he a iables ha a e impo - Chap e 3. Applica ions o andom o es s 53 an o he o iginal unc ion, he i s de i a i e and he second de i a i e co espond o he same wa eleng h. When ep esen ing he MDA and MDI o he andom o es s ob ained up o he o iginal unc ion, he i s de i a i e (D1) and he second de i a- i e (D2), app oxima ely he a iables co esponding o he same wa eleng hs a e impo an , as can be seen in Figu es3.7(a) and 3.7(b). Ne e heless, i is impo an o men ion ha , when speaking abou a ange, wha has been jus said is co ec ly, bu when speaking abou he alues inside ha ange i ollows ha no exac ly he same wa eleng hs a e impo an o he dis inc unc ions. Taking a look a he ange be ween 900nm and 950nm app oxima ely, i s ands ou ha his ange con ains he mos impo an wa eleng hs, independen o speaking o he o iginal unc ion, he i s o he second de i a i e. Ne e heless, inside he ange i mus be emphasized ha he o iginal unc ion and he second de i a i e ha e as mos impo an wa eleng hs mo e o less he same alues a ound 930nm, bu a his poin , he i s de i a i e achie es a minimum. I ollows ha he i s de i a i e is complemen a y o he o iginal unc ion and he second de i a i e. 3.3.1.2. Reg ession o eca o In he beginning i was said ha he da a se eca o o iginally consis s o a com- pound o obse a ions wi h he a iable esponse a index, a con inuous a iable, ha we decided o ans o m in o a quali a i e a iable g ouping he a indexes in wo classes: high and low a index, in o de o add ess his p oblem as a classi ica ion p oblem. Ne e heless, i is also possible o use andom o es echniques when using a index as a con inuous a iable, conside ing his ime he p oblem as a eg ession p oblem. P oceding his way, he only di e ence wi h espec o he classi ica ion p ob- lem (when compu ing i wi h R) is o keep he o iginal esponse a iable like we can see o he case wi h he o iginal unc ion (Fo he es o he p og amming jus keep on (wi h Response coded like he e) he same way as was done wi h he classi ica ion p oblem): Explica i = eca o $abso p. da a$da a Response= eca o $y$Fa Da a=da a. ame(Explica i ,Response) Le us see now wha happens when using he di e en combina ions o o iginal unc ion and he i s wo de i a i es as was done wi h he classi ica ion p oblem. The e o e, see Table 3.3 (The esul s a e ob ained wi h seed 1081). In ela ion wi h wha can been obse ed in he Table 3.3 i can be said ha when using he o iginal unc ion and he wo de i a i es ( +D1+D2) he bes esul is ob- ained. Ne e heless, he esul s a e e y simila o he ones o jus using he second de i a i e o he second de i a i e and he o iginal unc ion (D2 o +D2). When jus 54 3.3. Expe imen II Func ions used o he explica i e a iables Va iable explained(%) Mean squa ed esiduals 70,41 47,80386 D1 98,68 2,135725 D2 99,05 1,526871 +D1+D2 99,32 1,093536 +D1 98,68 2,134657 +D2 99,01 1,595362 Table 3.3: Di e en combina ions in he eg ession p oblem o eca o using he i s de i a i e (D1) o he i s de i a i e and he o iginal unc ion ( +D1) he same esul is ob ained o he pe cen age o he explained a iabili y o he esponse a iable, p obably because he bes esul s when spli ing a e ob ained o he a iables o he i s de i a i e (in con as o he a iables o he o iginal unc ion), so using o he andom o es jus he i s de i a i e (D1) o i s combina ion wi h he o iginal unc ion ( +D1) does no p esen any di e ence. The wo s esul is ob ained when using jus he o iginal unc ion ( ). 3.3.2. G ow h The da ase called g ow h, a ailable wi h he lib a y( da) o Rhas also been add essed as a classi ica ion p oblem. This da a se ep esen s he unc ion o he heigh o 39 boys and 54 gi ls be ween he ages o 1 and 18, disc e ized in o 31 poin s, no equally spaced (1, 1.25, 1.5, 1.75, 2, 3, 4, 5, 6, 7, 8, 8.5, 9, 9.5, 10, 10.5, 11, 11.5, 12, 12.5, 13, 13.5, 14, 14.5, 15, 15.5, 16, 16.5, 17, 17.5, 18). So his way o each indi idual i he e a e 31 a iables Xm i, wi h m= 1, ..., 31 ha can be used in o de o classi y wi h andom o es s he esponse a iable, he sex o he indi idual. The di e en cu es o heigh o he di e en ages a e d awn in Figu e 3.8, in di e en colo s depending i i co esponds o a boy o a gi l. As done wi h he eca o da a se , he wo classes o he esponse a iable o he g ow h da a a e going o be coded: • "1" i i is a boy • "-1" i i is a gi l So ollowing mus be compu ed: Chap e 3. Applica ions o andom o es s 55 Figu e 3.8: Heigh a he ages be ween 1 and 18 lib a y( da.usc) lib a y( andomFo es ) #################################################### da a("g ow h") names=c("Va i1","Va i2","Va i3","Va i4","Va i5","Va i6", "Va i7","Va i8","Va i9","Va i10","Va i11", "Va i12","Va i13","Va i14","Va i15","Va i16", "Va i17","Va i18","Va i19","Va i20","Va i21", "Va i22","Va i23","Va i24","Va i25","Va i26", "Va i27","Va i28","Va i29","Va i30","Va i31", "Res") boys= ep(1,39) gi ls= ep(-1,54) Explica i = bind( (g ow h$hg m), (g ow h$hg )) Response1=c(boys,gi ls) Response= ac o (Response1) Da a=da a. ame(Explica i ,Response) colnames(Da a)<-names a ach(Da a) So, when compu ing he andom o es wi h he o iginal unc ion 56 3.3. Expe imen II se .seed(33) = andomFo es (Res~.,da a=Da a,n ee=1000, impo ance=TRUE,p oximi y=TRUE) p in ( ) ha will ha e as ou pu : Type o andom o es : classi ica ion Numbe o ees: 1000 No. o a iables ied a each spli : 5 OOB es ima e o e o a e: 8.6% Con usion ma ix: -1 1 class.e o -1 48 6 0.11111111 1 2 37 0.05128205 So, he OOB e o ob ained is o 8,6% ha we will y o imp o e as we did wi h he eca o da a se . The con usion ma ix is also shown in he ou pu , wi h 48 " ue posi i e", i.e. in his case, 48 gi ls classi ied as gi ls, and 37 " ue nega i e", i.e. 37 boys classi ied by he andom o es as boys. Figu e 3.9: Va iable Impo ance Le us now see he impo ance o a iables. Acco ding o Figu e 3.8, i migh be expec ed ha he mos impo an a iables a e hose o ages ound 15, because un il Chap e 3. Applica ions o andom o es s 57 ha momen , he cu es o boys and gi ls a e e y simila , and only up o he men ioned age, he cu es a e di e en o mos o he indi iduals o bo h classes. So, aking a look a Figu e 3.9, i can been obse ed ha , he mos impo an a iables, acco ding o MDI and MDA a e Va i27 un il Va i31, which co espond o he ages 16 o 18, i.e. he esul ob ained is cohe en wi h wha was expec ed. Nex s ep is o see wha happens when in oducing he i s and he second de i a- i es. The e o e, le us see i s he cu es o hese unc ions in Figu es 3.10(a) and 3.10(b). F om Figu e 3.10(a) can i can be deduced ha he mos impo an ages in o de o dis inguish i he pe son is a boy o a gi l, a e app oxima ely he ones o e 13 yea s, because up o ha momen , he cu es o boys and gi ls a e di e en . When ocusing on he second de i a i e in Figu e 3.10(b) i canno be done such a deduc ion. (a) Cu es o i s de i a i e o he heigh as unc ions o he age (b) Cu es o second de i a i e o he heigh as unc ions o he age Figu e 3.10: Fi s wo de i a i es o he heigh wi h colo as class-dis inc ion Taking a look a Figu es3.11(a) and 3.11(b) i can be seen now which a e he mos impo an a iables o he andom o es s when using he i s and he second de i a- i es. Figu e 3.11(a) shows ha he a iables De i19 un il De i30 a e he mos impo an ones, i.e. he ages 12.25 un il 17.75 as was p edic ed by Figu e 3.10(a). Ne e heless, when looking a he impo an ages acco ding o he second de i a i e di e en " alleys" and "moun ains" can be iden i ied, s anding ou he moun ain be- ween De i17 and De i20, and he one be ween De i22 and De i27, co e- sponding o he ages be ween 11.5and 13, and be ween 14 and 16.5, espec i ely. I can also be seen ha in he co esponding Figu e o he i s as also o he second de i a i e, some MDA alues a e nega i e. This means ha when pe mu ing he a i- able alues o e he di e en obse a ions he accu acy o he andom o es s o new da a is imp o ed. Ne e heless, in his case we a e wo king wi h a small da abase and addi ionally hese alues o MDA a e e y low in con as o he a iables ha a e conside ed as impo an , so hey can be conside ed as ze o. 64 a node TT ee T0Se o e minal nodes I(T)T ee Impu i y R∗(d)T ue misclassi ica ion a e o he unc ion d R(d)Resubs i u ion es ima e C(k|l)Cos o misclassi ying ( )Resubs i u ion es ima e o he expec ed misclassi ica ion cos o he node R(T)Resubs i u ion es ima e o he ee T C∗ k( )Class assignmen ule FNumbe o ees o he o es P ox P oximi y ma ix P ox(i, j)Cell (i,j) o he p oximi y ma ix ui,k Measu e o ou lyingness ˜ui,k No malized measu e o ou lyingness mkMean o measu e o ou lyingness o e all elemen s o class Ck JkNumbe o obse a ions o class Ck Bibliog aphy [1] Nelson Lee A anado , Agnieszka Smolinska, Thanh N. T an, and Lionel Blanche . Unsupe ised andom o es : a u o ial wi h case s udies. Chemo- me ics, 30:232–241, 2016. [2] Philippe Sain -Pie e Bap is e G ego u i, Be and Michel. Co ela ion and a i- able impo ance in andom o es s. S a Compu , 27:659–678, 2017. [3] Gé a d Biau and E wan Sco ne . A andom o es guided ou . Tes , 25(2):197– 227, 2016. [4] R. Blanque o, E. Ca izosa, A. Jiménez-Co de o, and B. Ma ín-Ba agán. Func ional-bandwid h ke nel o suppo ec o machine wi h unc ional da a: An al e na ing op imiza ion algo i hm. Eu opean Jou nal o Ope a ional Resea ch, 275(1):195 – 207, 2019. [5] Leo B eiman. Random o es s. Machine Lea ning, 45. [6] Leo B eiman. S a is ical modeling: The wo cul u es. S a is ical Science, 16(3):199–215, 2001. [7] Leo B eiman. Manual on se ing up, using, and unde s anding andom o es s 4.0. Machine Lea ning. Jou nal, 2003. [8] Leo B eiman, Je ome H. F iedman, Richa d A. Olshen, and Cha les J. S one. Classi ica ion And Reg ession T ees. CHAPMAN HALL/CRC, 1984. [9] Emilio Ca izosa and Dolo es Rome o Mo ales. Supe ised classi ica ion and ma hema ical op imiza ion. Compu e s Ope a ions Resea ch,40(1),150-165, 2013. [10] Ramón Díaz U ia e and Sa a Al a ez de And és. Gene selec ion and classi ica- ion o mic oa ay da a using andom o es . BMC Bioin o ma ics, 7:3, 2006. [11] Dhee u Dua and Casey G a . UCI machine lea ning eposi o y, 2017. [12] F. Hillie and G. Liebe man. In oduccion a la in es igacion de ope aciones. Mexico, D.F: McG aw-Hill/In e ame icana, 2010. 65 66 Bibliog aphy [13] Alan Julian Izenman. Mode n Mul i a ia e S a is ical Techniques. Reg es- sion,Classi ica ion, and Mani old Lea ning. Sp inge , 2013. [14] Robe C. Messenge James N. Mo gan. THAID,a Sequen ial Analysis P og am o he Analysis o Nominal Scale Dependen Va iables. Su ey Resea ch Cen e , Ins i u e o Social Resea ch, Uni e si y o Michigan, 1973. [15] M.R. O iz-Posadas. Pa e n Recogni ion Techniques Applied o Biomedical P oblems. Sp inge . [16] Ma ha E. S one. C oss- alida ion: A e iew. Ma h. Ope a ion o sch. S a is . Se S a is , 9:127–139, 1977. [17] Ca olin S obl, Anne-Lau e Boules eix, Achim Zeileis, and To s en Ho ho n. Bias in andom o es a iable impo ance measu es: Illus a ions, sou ces and a solu ion.”. BMC bioin o ma ics, 8(1):25, 2007. [18] Vladimi S e nik, Andy Liaw, Ch is ophe Tong, J. Ch is ophe Culbe son, Robe P. She idan, and B adley P. Feus on. Random o es : a classi ica ion and eg ession ool o compound classi ica ion and qsa modeling. Jou nal o Chemical In o ma ion and Compu e Sciences, 43(6):1947–1958, 2003. PMID: 14632445. [19] Je ome F iedman T e o Has ie, Robe Tibshi ani. The Elemen s o S a is ical Lea ning: Da a Mining, In e ence and P edic ion. Sp inge , 2009. [20] Hal R. Va ian. Big da a: New icks o econome ics. Jou nal o Economic Pe spec i es, 28(2):3–28, 2014.