Automatic analysis of high dimensional categorical variables in medical databases for the prediction of hospital bacteremia

Au oma ic analysis o high dimensional ca ego ical a iables in medical da abases o he p edic ion o hospi al bac e emia Po Jaime del Rey Ga cía G ado en Ingenie ía In o má ica Facul ad de In o má ica Di ec o es : Ósca Ga nica Alcáza y José Manuel Ruiz Gia dín Análisis au omá ico de a iables ca egó icas de al a dimensionalidad en bases de da os médicas pa a la p edicción de bac e iemias hospi ala ias Mad id, 2020–2021 Au ho iza ion o dissemina ion and use The au ho s o his wo k au ho ize he Complu ense Uni e si y o Mad id o use bo h he code and he epo p oduced, solely o didac ic pu poses and men ioning he au ho s o he same. Jaime del Rey Ga cía iii Acknowledgemen s Fi s o all, hank Ósca and José Manuel o p o iding he da ase , sha ing he idea and con ibu ing o he s udy. A special hanks o Ósca again o his dedica ion and unde s anding h oughou he p ocess, he has made me eel ha he was wo king side by side wi h me and con ibu ed as a u o a a le el beyond academic. To all he people who ha e encou aged me o keep on going: o my a he , my mo he and my b o he o hei pe se e ance; and o my iends who ha e no hesi a ed o suppo me. Sob e T EF LO NX Te lon X(cc0 1.0(documen ación) MIT(código))es una plan illa de L A T EX c eada po Da id Pacios Izquie do con echa de Ene o de 2018. Con a ibuciones de uso CC0. Es a plan illa ue desa ollada pa a acili a la c eación de documen ación p o esional pa a T abajos de Fin de G ado, T abajos de Fin de Más e o Doc o ados. La e sión usada es la X V:X O e lea V2 wi h XeLaTeX, ma gin 1in, bib Con ac o Au o : Da id Pacios Izquie o Co eo: [email p o ec ed] ASCII: [email p o ec ed] Despacho 110 - Facul ad de In o má ica ii Con en s Page 1 In oduc ion 1 1.1 Bac e emia .................................. 1 1.2 Con ex .................................... 3 1.3 Objec i es................................... 3 2 Wo k o ganiza ion 5 2.1 Wo kplan ................................... 5 2.1.1 De elopmen en i onmen . . . . . . . . . . . . . . . . . . . . . . 6 2.1.2 Ve sioncon ol ............................ 6 3 The Da ase 7 3.1 S udyo heda a............................... 7 3.1.1 Di y ca ego ical a iables . . . . . . . . . . . . . . . . . . . . . . 12 3.1.2 Encoding S ing Ca ego ical Va iables . . . . . . . . . . . . . . . 14 3.2 Cleaning heda a............................... 15 3.3 P ocessing heda a.............................. 18 3.3.1 S ing simila i y . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.3.2 K-NN ................................. 22 4 Ca ego ies 25 4.1 Ca ego y ecogni ion............................. 25 4.1.1 Choosing h eshold alue . . . . . . . . . . . . . . . . . . . . . . 25 4.1.2 Noise emo al............................. 26 4.1.3 Iden i ying a ian s o s ings . . . . . . . . . . . . . . . . . . . . 27 4.1.4 De ining he classes . . . . . . . . . . . . . . . . . . . . . . . . . . 29 4.1.5 E alua ing he esul s . . . . . . . . . . . . . . . . . . . . . . . . 30 4.2 I e a ing hep ocess ............................. 34 4.3 Subs i u ing ca ego ies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.3.1 Da ase inse ion . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.4 Da ase p epa a ion ............................. 36 5 Bias and a iance 39 5.1 A oiding Bias and Va iance . . . . . . . . . . . . . . . . . . . . . . . . . 40 5.2 K-Fold C oss Valida ion . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 6 Random Fo es 43 6.1 Fi ing a Random Fo es . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 ix G ado en Ingenie ía In o má ica Facul ad de In o má ica Thus, he a ge o his p ojec is o ind he mos ele an a iables o he de ec ion and diagnosis o bac e emia by means o ML models, including he nominal ca ego ical a iables skipped in he p e ious s udy and imp o ing he esul s o he p edic ions in he mos explainable models. Jaime del Rey Ga cía 4 Chap e 2 Wo k o ganiza ion 2.1 Wo kplan Be o e app oaching he p ojec , a weekly ollow-up was ag eed h ough which Ósca could e alua e he p og ess made, gi ing con inui y o he wo k. In his i s con ac , he de elopmen en i onmen , he ool used o e sion con ol, as well as o he wo k low, we e app o ed. Simila ly, he use o he Goole Mee s ool o communica ion and mee ings, and he use o a Google Cha oom o messages be ween mee ings is se . In he i s ins ance, Ósca explained ha i would be necessa y e iewing he wo k ca ied ou he p e ious yea ; eading o esea ch on machine lea ning esul s wi h di e en coding me hods applied o spa se ma ices and unde s anding he se was gi en equal ele ance o da a wi h which o wo k and he e alua ion o he a iables ma ked as pending om he p e ious s udy. F om he e, he wo k plan was es ablished as ollows: (i) C ea e Gi Hub di ec o y ee. (ii) Ge access o he p e ious yea eposi o y. (iii) Reed he a icles abou Missing Da a [8] om Yu en Ding and Je ey S. Simono , Encoding High Ca dinali y Va iables [9] and Simila i y encoding o lea ning wi h di y ca ego ical a iables [10] om Pa icio Ce da. (i ) Ins all Jupy e No ebook. ( ) Ins all Py hon 3. ( i) Ge amilia wi h he p e ious s udy and he sc ip used o wo k wi h he da a ea u es. ( ii) E alua e he da ase a iables. ( iii) Check wi h he physician he ca ego ical a iables o s udy. (ix) Find in he esul ing Da aF ame objec all dis inc alues o he s udy case a i- ables. (x) Modi y he sc ip acco ding o he needs o he cu en ea u es. 5 G ado en Ingenie ía In o má ica Facul ad de In o má ica (xi) C ea e wo king en i onmen wo k low ile wi h pipen . (xii) Cleaning accen s and s ange cha ac e s om he s udy ea u es. (xiii) Ge all dis inc alues om he s udy a iable O ascomo . (xi ) Dep h sea ch o ex analysis echniques and ools. (x ) Apply s ing simila i y me hod o he esul ing bag o wo ds. (x i) Iden i y each o he di e en en i ies om he bag o wo ds and eplace hem in he Da aF ame o model e alua ion. (x ii) Check bag o wo ds wi h 2 and 3 cha ac e minimum size abb e ia ions. (x iii) C ea e subs i u ion lis o abb e ia ion eplacemen . (xix) Run he sc ip ha ing applied he abb e ia ion subs i u ion. (xx) Comple e he wo k low wi h he ca ego ies ob ained om unning he sc ip . (xxi) Apply bina y codi ica ion o all ca ego ies wi hin he Da aF ame o he s udy ea u e. (xxii) Reo ganize and clean he sc ip . (xxiii) Remo e all p edic o a iables om he Da aF ame be o e unning Random Fo es model. (xxi ) Apply Random Fo es o he Da aF ame wi h bina y codi ica ion. (xx ) Run k- old c oss alida ion wi h 80/20 pa ame e s o aining and alida ion se s. (xx i) Apply Random Fo es o he Da aF ame wi h bina y codi ica ion. (xx ii) Re ie e g aph wi h he weigh ed ele ance o he ea u es. (xx iii) Re ie e ROC g aph om he Random Fo es execu ion. (xxix) W i e he p ojec epo . 2.1.1 De elopmen en i onmen The de elopmen en i onmen ha has been used o his p ojec is Jupy e No ebook, wi h he use o Py hon as p og amming language. Ósca emphasized he equi emen o main ain a ile wi h he la es s able e sion o he so wa e and he lib a ies used necessa y o he execu ion o he code. The use o pipen , a p oduc ion ool ha c ea es i ual en i onmen s and main ains he build used du ing de elopmen , co e s his need. 2.1.2 Ve sion con ol Fo e sion con ol, gi hub was used which, in addi ion o s o ing he code e sions, allows c ea ing a ollow-up by a ge ca ds acco ding o he s a us o hei comple ion, allowing con ol o e all p ojec asks apa om de elopmen asks. Wi hin he p ojec , a olde s uc u e was es ablished o dis inguish be ween documen- a ion, code, pa ien da a and esul s ob ained. Jaime del Rey Ga cía 6 Chap e 3 The Da ase 3.1 S udy o he da a This chap e will de elop all he in o ma ion abou he da ase , he ea men i has ecei ed o i s co ec manipula ion, as well as i s in e p e a ion and subsequen p epa- a ion o i s usage wi h Machine Lea ning ools. The da ase is an anonymized da ase p o ided by José Manuel Ruiz, physician a he Hospi al Uni e si a io de Fuenlab ada, a 350-bed hospi al wi h he ollowing se ices: gene al su ge y, u ology, o hopaedic su ge y, gynaecology and obs e ics, paedia ics, in ensi e ca e uni s (ICUs), haema ology-oncology, in e nal medicine and ca diology. The da abase was ga he ed om 2005 o 2015, and i consis s o 4357 anonymous pa ien eco ds, a.k.a. ins ances, con aining 117 ea u es pe pa ien , 49.3% emale wi h age 65.1 ± 19.7, and 56.1% male wi h age 62.7 ± 20.2. Each ins ance con ains demog aphic and medical da a (medical his o y, clinical analysis, como bidi ies, e c.) and he esul o he blood cul u e, he ea u e o be p edic ed, which can ake one o wo alues: bac e aemia and no bac e aemia. The da abase con ains 2123 bac e aemia (51.3%), which includes ae obic, s ic anae obic and acul a i e anae obic bac e ia, and 2234 no bac e aemia (48.7%), including 1844 con amina ions.The inal classi ica ion o ue bac e aemia was done in p ospec i e ime by an in ec ious disease physician, using all he p e ious da a, including mic obiological, clinical and analy ical da a. The a ge a iables in his p ojec a e hose o nominal ype om he lis o a iables, which esul in he ollowing: desmo uci,uci,o igin,o ascomo . A e a i s e alua ion, he o igin a iable was disca ded due o i s na u e. This a iable ep esen s he o igin o he bac e emia, which is a ea u e om he inal diagnosis. Fea u es ha can only be ob ained a e he d awning o he blood cul u e should no be included because in he eal p ocess he physicians would no be able o coun on ha alues. This would also con amina e he esul s o he Machine Lea ning models. Tables 3.1 and 3.2 display he ea u es in he da ase and he selec ed a iables o he s udy. 7 G ado en Ingenie ía In o má ica Facul ad de In o má ica Table 3.1: Fea u es om he da ase Fea u es om he da ase pe iodo Yea o he s udy case O dinal mes Mon h o he s udy case O dinal dia Day o he s udy case Scala edad Age o he pa ien Scala edada Age pe g oup O dinal edada75 Age goe o 75 Scala edada85 Age goe o 85 Scala Diasde De ec ion ime in days Scala P i pos Fi s cul u e o g ow Scala mic o ga Mic oo ganism Scala iden i Species o he bac e emia Nominal Mic o gag upo G oup o he mic oo ganism Scala anhonpol Anae obes O dinal Anae obio P esence o anae obes agains all o he bac e ia O dinal Hongos P esence o ungs agains all o he bac e ia O dinal S a coag S aphyloccocus coag Scala Polimic Polimic obian O dinal mic obpoli Ge ms o mic obial bac e ia O dinal g am G am s ain O dinal medio T ue posi i e g ow h medium O dinal asae G ow h a leas in ae obes Scala asanae G ow h a leas in anae obic lask Scala asex Bo les o ex ac ed blood cul u es Scala con amin Pollu an g ow h O dinal mediocon G ow h medium o he pollu an mic oo ganism O dinal An ibiog ama Mic oo ganism an ibiog am Nominal An i esis Ca ego ized an ibiog am Scala glucosa Blood glucose Scala U ea Blood u ea in mg/l Scala c ea in C ea inine Scala pc ca e Ca ego ical PCR Scala pc PCR alue Scala leuc Leukocy es Scala hgb Hemoglobin Scala pm n PMN pe cen age Scala Jaime del Rey Ga cía 8 Au oma ic analysis o high dimensional ca ego ical a iables in medical da abases o he p edic ion o hospi al bac e emia UCM Fea u es om he da ase hbca eg Ca ego ical Hb Scala plaqu Pla ele s Scala leucoci Leukocy osis O dinal ombope Th ombopenia O dinal Coagulación Al e ed coagula ion O dinal so U ine sys em analysis Scala sedo ina U ine sedimen Scala diashosp Days in hospi al un il bloodcul u e ex ac ion Scala lnghosp1m Hospi al admissions o e he las mon h Scala lnghosp12m Hospi al admissions o o e 48h in he las yea Scala comen a Commen s Nominal COMORBIL Como bidi y Scala Diabe es Diabe es Scala Ca diopa ía Hea disease Scala En esp Ch onic espi a o y disease Scala Neoplasia Ac i e neoplasia Scala Ins enal Renal insufficiency Scala Hepa opa ia Li e disease Scala Ud p Pa en e al d ug addic ion Scala Alcoholismo Alcoholism Scala O ascomo O he como bidi ies Nominal en basWeins Weins ein’s unde lying disease O dinal es e oid S e oids O dinal d ogadic D ug addic ion O dinal an ibio An ibio ics O dinal inmunosu Immunosupp essan s O dinal neu ope Neu openia O dinal m_geni u Geni ou ina y manipula ions O dinal m_ espi Respi a o y manipula ions O dinal m_diges Diges i e manipula ions O dinal ci ugia P e ious su ge y O dinal diagnos Diagnosis o bac e emia O dinal ca e e Days o las ca he e placemen Scala ca e e 1 Type o ca he e O dinal Especialidad Special y whe e he bac e emia is p oduced Scala se icio Se ice whe e he bac e emia is p oduced Scala 9 G ado en Ingenie ía In o má ica Facul ad de In o má ica Fea u es om he da ase O dinal u gencias Blood cul u es aken in he eme gency oom Scala adquisic Adquisi ion O dinal du ac Days o e e ebo e blood cul u e Scala as Sys olic blood p essu e Scala ad Dias olic blood p essu e Scala c Hea a e Scala p im emp Fi s ER empe a u e wi h which blood cul u es a e d awn Scala empo al Tempe a u e based on o al and axilla y empe a u es Scala ieb e Fe e when blood cul u es a e d awn O dinal ieb ePi O al empe a u e classi ica ion in Pi scale Scala hipo ens Hypo ension O dinal Vasop e Use o asop esso agen s a he ime o bac e emia Scala in ubacion Need o in uba ion a he ime o bac e emia Scala RCP Ca diac esusci a ion a he ime o bac e emia Scala Ale a Consciousness a he ime o bac e emia Scala PITT Pi scale in numbe o ICU pa ien s Scala me as as Me as asis O dinal me as a1 Whe e he me as asis is p odued, i any O dinal e olucio E olu ion O dinal mue e Dea h O dinal o igen O igin o bac e emia Nominal dx inal Final diagnosis Scala o igensos Suspec ed o igin o bac e emia a he ime he blood cul u es a e d awn Scala o igen O igin o bac e emia in he inal diagnosis Scala o igen a Vascula o igin Scala a bempi Empi ical an ibio ic ea men Scala _empi i Empi ical ea men adequa e o inadequa e O dinal oesp Speci ic adequa e ea men Scala _especi Speci ic ea men adequa e o inadequa e O dinal dias o Days o ea men un il s a o inadequa e ea men Scala _qui u Indica ion o no o su gical ea men O dinal uci Bloodcul u es d awn in ICU Nominal ucidiashem Days in ICU o blood cul u e d awn Scala mo uci Reason o ICU admission Scala Jaime del Rey Ga cía 10 Au oma ic analysis o high dimensional ca ego ical a iables in medical da abases o he p edic ion o hospi al bac e emia UCM Fea u es om he da ase desmo uci Reaso o ICU admission as ex Nominal cons ieb Consul o e e Scala sin omas Fe e symp oms Scala s local Loca o synd ome Scala des Des ina ion Scala uel a_a Re u n o ER Scala a amie An ibio ic ea men Scala Table 3.2: Selec ed ea u es in he s udy. Selec ed ea u es desmo uci Reason o admission o ICU uci Blood cul u es om ICU o igin O igin o he bac e emia O ascomo O he como bidi ies A s udy o he a iables is ca ied ou indi idually o iden i y he cha ac e is ics o each a iable. S a ing wi h he a iable o ascomo ha con ains he como bidi ies wi h which each pa ien was admi ed. Como bidi ies a e addi ional diso de s ha pa ien s p esen in addi ion o he disease o which hey a e admi ed. This a iable is cha ac e is ic because i can help o iden i y unde wha condi ions a pa ien is mo e likely o ha e bac e emia. This a iable con ains como bidi ies as nouns sepa a ed by punc ua ion ma ks o o he ex cha ac e s. Como bidi ies a e no w i en in a homogeneous way, ha is, some a e w i en wi h diminu i es, o he s con ac ions o di e en leng hs o cha ac e s on he same wo d. The s uc u e o he a iable is i sel a se ies om he pandas lib a y whe e each index in he lis ep esen s he pa ien e e ed and he con en wi hin he lis is a s ing de ailing he como bidi ies. 11 G ado en Ingenie ía In o má ica Facul ad de In o má ica [1]: _nominales['O ascomo '] [1]: 0de e io o cogni i o 1 2CARDIOPATIA.EPILEPSIA 3 al ulopa ia 4bcno, dm, h a, ci ... 5389 5390 Obesidad 5391 5392 5393 Name: O ascomo , Leng h: 5394, d ype: objec Figu e 3.1: Sample o he ea u e O ascomo . The ypo o he a iable is no homogeneous ei he , coun ing wi h uppe and lowe case indisc imina ely. The sepa a o s, which indica e he end and he beginning o each como bidi y in succession, a e also he e ogeneous in a ange om punc ua ion ma ks such as ’,’ o ’.’ o o he ypes o elemen s such as ’+’ o ’e’. Ano he a ibu e ha can be obse ed is he appea ance o spaces as p e ixes and suffixes, in addi ion o he indexes o he se ies ha ha e no con en . The e a e also s ings h oughou he se ies ha con ain accen s, apos ophes and o he elemen s ha pose difficul ies o he ep esen a ion o he ex acco ding o wha o ma s may be used. 3.1.1 Di y ca ego ical a iables In he con ex o Machine Lea ning wi h na u al language a iables, di y ca ego ies a e he de ini ion o non-cu a ed da a wi h high ca dinali y bu edundancy: se e al ca ego ies e lec he same en i y. One o he main challenges wi h di y ca ego ical a iables is o iden i y all he elemen s ha a e ela ed and e e o he same en i y o class. Wi hou da a cleaning, di e en s ing ep esen a ions o he same ca ego y will lead o comple ely di e en esul s o sub- ca ego ies, no only because he di e en elemen s e e o he same ca ego y hemsel es bu also because o e o s such as ypos ha cause mo phological a ia ions. F om a da a-in eg a ion s andpoin , hese ca ego ies may be seen as a da a cleaning p oblem abou en i y esolu ion. Tasks such as deduplica ion, ha ies o me ge di e en a ian s o he same en i y, seek o ecognize di e en a ian s o he same en i y, which may be he bes case o apply as a p ep ocessing s ep. Howe e , da a cleaning usually equi es human in e en ion and majo cos s in da a analysis. In his da ase almos all he examples a e di y and ca ego ical o he a iables o s udy. I wo examples o he a iable O ascomo a e aken, i can be unde s ood ha wi hin his s udy he e is a di y ca ego ical a iable p oblem. To ca e o di e en ways Jaime del Rey Ga cía 12 Au oma ic analysis o high dimensional ca ego ical a iables in medical da abases o he p edic ion o hospi al bac e emia UCM in o ma ion migh appea , se e al en i ies a e shown in Figu es 3.2 and 3.3 o each o he possible subclasses. [1]: anemi anemia, anemia c onica, anemica c onica e , anemia e openica, anemia hemoli ica, anemia megaloblas ic, anemia mic oci ica, anemia n-n, anemia nn, anemia no moci ica no moc onica, anemia po de ici b12, anemia c onicos Figu e 3.2: Sample o he elemen s in he ea u e O ascomo e e ing o anemia as di y ca ego ies. [1]: alzheime alzheime a anzado, alzheime e olucionado, alzheime e olucionado (ins i ucionalizada, alzheime e minal, alzheime ., alzheime .asma, alzheime .dm, alzheime .en e medad ascula ce eb al, alzheime .i us de epe icion., ca.p os a a.alzheime a anzado.pa kinson, demencia alzheime , dm.alzheime . p.i us de ep., d alzheime a anzado, en e medad de alzheime , epoc.neumonia.alzheime ., e c es adio 3. alzheime ., hemo agia ceeb al; e de alzheime , h a; alzheime Figu e 3.3: Sample o he elemen s in he ea u e O ascomo e e ing o alzheime as di y ca ego ies. In bo h examples, all he a iables con ain he wo d ha we migh conside as he en i y 13 G ado en Ingenie ía In o má ica Facul ad de In o má ica Figu e 3.5: Example o a hea map o e di y ca ego ies o di e en wo ks. The ea u e ma ix in Figu e 3.5 is compu ed using, a i s , he bag o wo ds wi h commas as sepa a o s. The p esence o a la ge numbe o ca ego ies calls o ep esen ing he ela ionships be ween hem so he hea map is used o ha end. The esul s a e p omising a e he e alua ion o he i s example shown in Figu e 3.6, in which he map shows a high ela ionship alue be ween all he chains ha sha e he wo d adenoca cinoma in a he e ogeneous way, such as adnoca cinoma,adenoma o adenocea. Adenoca cinoma is a e m ha e e s o cance . Al hough he di e en ypes o cance a ec he body in di e en ways, he esul se s a clea guideline o es ablish adenoca cinoma as a class. Jaime del Rey Ga cía 20 Au oma ic analysis o high dimensional ca ego ical a iables in medical da abases o he p edic ion o hospi al bac e emia UCM Figu e 3.6: Example 1 o a hea map o e 25 andom di y ca ego ies om he da ase . Howe e , a e e alua ing he second example depic ed in igu e 3.7, a p oblem appea s ha will accompany much o he es o he p ojec : he map does no show a simila - i y alue no e en close o 0.4 be ween chains such as ac ca dioembolico and ac con hemipa esia, bo h being a ype o s oke. One ela ionship ha is s iking is he low sim- ila i y alue o acciden e ce eb o ascula and ac . Bo h chains e e o he exac same concep , being able o o m he same class bu acco ding o he ma ix hey show almos no simila i y. 21 G ado en Ingenie ía In o má ica Facul ad de In o má ica Figu e 3.7: Example 2 o a hea map o e 25 andom di y ca ego ies om he da ase . 3.3.2 K-NN The K-NN algo i hm is a supe ised classi ica ion me hod ha allows da a subse s o be g ouped by dis ance be ween he elemen s o each subse . This algo i hm classi ies each new en y in a class, acco ding o he kneighbo s closes o a ce ain g oup. To do his, i calcula es he dis ance o he new en y o each da a al eady exis ing in he model and o de s hose dis ances om lowes o highes o choose he g oup o which i belongs. This means ha he algo i hm will use he simila i y be ween wo s ings o measu e he dis ances. The g oup chosen will be he one ha ep esen s he sho es dis ance, o ha g oup wi h he g ea es ep esen a ion in a g ea e longi udinal spec um. The numbe o neighbo s o a gi en example allows adjus ing he noise o a classi ica ion, including in each class he poin s o he hype plane ha a e alike he mos . Howe e , signi ican ly educing he numbe o neighbo s causes an e ec ha may no be desi ed: c ea ing classes o elemen s ha a e no necessa ily di e en . Jaime del Rey Ga cía 22 Au oma ic analysis o high dimensional ca ego ical a iables in medical da abases o he p edic ion o hospi al bac e emia UCM Figu e 3.8: Example o he classes g ouped ela i e o he numbe o neighbo s [12] K-NN is qui e sensi i e o: a). The a iable k, so ha wi h di e en alues o kwe can also ob ain e y di e en esul s. b). The simila i y me ic used, since his will s ongly in luence he closeness ela ion- ships ha will be es ablished in he algo i hm cons uc ion p ocess. The dis ance me ic can con ain weigh s ha will help us calib a e he classi ica ion algo i hm, making i , in e ec , a cus om me ic. Thus, he algo i hm is highly bene i ed by he na u e o he da ase wi h which i is going o wo k. The ma ix al eady p o ides a weigh ed me ic be ween each pai o elemen s o he same. Fi ing a K-NN model The nex s ep is o i he model o he da ase and check i he e is an es ima e ha he algo i hm can make based on he alues ob ained in he simila i y me ic. The inpu o he alue kis chosen o be 4. The objec i e is o ha e a glance o whe he he algo i hm is able o g oup s ings ha he hea map did no ma ch bu ha ac ually e e o he same concep , ha is why he e is no in es men o esou ces in inding he op imal numbe o neighbo s. om sklea n.neighbo s impo Nea es Neighbo s nn_c =Nea es Neighbo s(n_neighbo s=4). i ( ans o med_ alues_c) _, indices_ =nn_c.kneighbo s( ans o med_ alues_c[ andom_poin s]) indices =np.unique(indices_.squeeze()) Figu e 3.9 shows o 20 selec ed i ems in he bag o wo ds g ouped. I is obse ed ha chains such as a osis de columna,a i is euma oide o posible a e i is de la em- 23 G ado en Ingenie ía In o má ica Facul ad de In o má ica po al a e loca ed nea by. This esul is p omising since he a h i is class is he mos ep esen a i e elemen in bo h cases and hey ha e been classi ied as simila . On he o he hand, i is obse ed in he same g aph ha elemen s such as polia osis o polinosis appea as simila elemen s and, al hough i is ue ha hey sha e he p e ix poli, hey ha e widely di e en meanings: he i s class, polya h osis is he de ini ion o in lamma ion o se e al join s a he same ime while polinois is an alle gic disease ha a ec s he eyes, nose and lungs p oduced as a eac ion o he immune sys em o pollen. Figu e 3.9: K-NN o e he bag o wo ds using k=4 neighbo s Conclusion o he me hod The in ui ion behind his e alua ion is ha o s ings wi h a low numbe o cha ac e s, he s ing simila i y algo i hm p esen s mo e noise, as he n-g ams ha e g ea e weigh o e he o al s ing. Jaime del Rey Ga cía 24 Chap e 4 Ca ego ies 4.1 Ca ego y ecogni ion Despi e he conclusion shown in he p e ious sec ion, he elemen s o he bag o wo ds a e e alua ed, eaching he conclusion ha possibly a la ge numbe o examples ha e simila i y alues high enough o be able o wo k wi h hem and ob ain a easonable numbe o ca ego ies ha include he highes numbe o como bidi ies exposed in he a iable. In o de o ace he p oblem om a Machine Lea ning poin o iew, he nex s ep is o be able o iden i y all he ca ego ies using he ma ix o alues and o be able o me ge all he simila s ings in o he one ha iden i ies he class by adap ing he da ase ob ained up o now o adjus i o he needs o he s udy. This p ocess o adap ing he da ase o he p oblem is manually coded as he e is no ool o pe o ms hese asks so cus omized as i is needed; he e o e a se ies o unc ions a e de ined and de eloped ha will be shown o each speci ic s ep. Each o he unc ions wen h ough se e al es s on he da ase , so he p ocess in ol ed a g ea in es men in ime, his sec ion being he bulk o he p ojec Based on he da a ob ained so a , he app oach ha is ca ied ou is he ollowing: a). Se a ele ance h eshold o he simila i y alue. b). Ge id o as much noise as possible. c). Iden i y o each oken all hose ha sha e a ele an simila i y. d). Se a class o each se o s ings. e). Replace each elemen in he o iginal a iable wi h i s supe class. 4.1.1 Choosing h eshold alue The idea o es ablishing a h eshold alue o he simila i y o he s ings is o be able o elimina e he la ges numbe o candida es ha do no p esen a ela ionship close o he iden i ica ion as an en i y wi h each o he s ings o he bag o wo ds. 25 G ado en Ingenie ía In o má ica Facul ad de In o má ica This, o begin wi h, makes i possible o elimina e noise om he da ase by making he ca ego ies ob ained ep esen as eliable a e lec ion as possible o how a p o essional in he ield would in e p e and g oup he da ase . The alue aken by his pa ame e is signi ican ly ele an o he es o he p ocess, wi h he unc ions and esul s ob ained being sensi i e o he inc ease o dec ease o he h eshold. Based on he i s esul ob ained om e alua ing he ea u e ma ix om igu e 3.6, a h eshold alue g ea e han 0.5 sugges s ha he esul ing s ings will necessa ily be e y simila o each o he , ensu ing a ela i ely small numbe o ca ego ies and ai h ul o he g oups ha make up each o he subse s o s ings. On he o he hand, looking a he esul s o he second hea map on igu e 3.7, he choice o such a high h eshold alue would ule ou , in his speci ic example, all he s ings ha e e o di e en ypes o ac . A e assessing i wi h Ósca , he decision ha seems he mos app op ia e is o es ablish di e en alues o he h eshold and es he execu ion o he en i e p ocess o each o hem. Las ly, he esul s would be e alua ed and he mos con enien one would be chosen. Se o h eshold alues: 0.2 0.3 0.4 0.5 0.65 (4.1) 4.1.2 Noise emo al The denoising p ocess begins by using he ma ix i sel wi h he simila i y alues as he s uc u e o wo k on. Since he goal is ha all hose s ings ha do no espec he h eshold a e disca ded, he p ocess ca ied ou is simila o applying a bina y encoding o he ma ix o , analogously, applying a one-ho encoding o each simila i y ec o in which he elemen s ha mee he condi ion a e kep . Le 𝑥be he ec o ep esen ing he i s ow o he ma ix, wi h 𝑥𝑖,𝑖 ∈ [0..𝑚] being 𝑚 he o al size o he bag o wo ds. We should emembe ha in each ow o he ma ix he oken o he bag o wo ds ha ep esen s he index o he ow is compa ed wi h each o he o he s ings. We know ha , being he i s oken, he elemen 𝑥0has he alue o 1 because i is compa ed wi h i sel . Taking in o accoun ha he simila i y alues oscilla e in a ange o [0..1] we apply, ha ing 𝑢as he h eshold alue and de ining a unc ion 𝐹on he ec o such ha 𝐹(𝑥𝑖)=0,∀𝑥<𝑢. This is he unc ion shown below. Knowing ha i is going o wo k wi h he elemen s ha mee he condi ion, he unc ion e u ns a lis wi h he indexes o i s ow ha ha e no been a 0. Jaime del Rey Ga cía 26 Au oma ic analysis o high dimensional ca ego ical a iables in medical da abases o he p edic ion o hospi al bac e emia UCM de co e_ alo es(lis a_ca ego ias,ma iz_simili ud,co e): ma ix =np.emp y([len(ma iz_simili ud),len(ma iz_simili ud)]) ””” A gumen os cla e: lis a_ca ego ias -- obje o ipo nda ay, con iene los alo es de la bolsa de palab as ma iz_simili ud -- obje o ipo nda ay, con iene pa a cada alo su alo de simili ud con el es o (0..1) co e -- alo decimal (0..1) pa a il a las palab as Pone a ce o odos los alo es en la ma iz po debajo del co e De uel e: lis a => pa a cada ca ego ía las posiciones de las ca ego ías cuya simili ud es ele an e (según co e) ””” o i, alue in enume a e(ma iz_simili ud): ma ix[i] =lis (map((lambda x: 0i x<co e else x),ma iz_simili ud[i])) ma iz_indices =[] o i, alue in enume a e(ma ix): ma iz_indices.append(lis (np.nonze o(ma ix[i])[0])) lis a_co e =[] o i, alue in enume a e(ma iz_indices): lis a_co e.append([i,0,[j o jin ma iz_indices[i]]]) e u n lis a_co e I u ns ou ha , as each elemen o he a ay has been compa ed wi h he es o he elemen s in he same o de , he alues o he posi ions o each o he s ings sha e he same e e ence, ha is, i he elemen 𝑦has in he ec o o simila okens he index 37, his same index will ep esen he same s ing ega dless o whe he i appea s in he ec o 𝑥o 𝑧, whe e 𝑥,𝑦,𝑧 a e en i ies o he wo d bag and, hus, ha e a ow assigned in he ma ix. This is an ad an age because i a oids ha ing o sa e an objec o each o he s ings in he wo d bag, sa ing esou ces in ime and memo y. 4.1.3 Iden i ying a ian s o s ings The nex s ep is o iden i y all he s ings ha sha e simila i y and es ablish a class o g oup hem all. Ha ing applied he unc ion 𝐹o e he en i e ea u e ma ix, i emains in a s a e ha can be in e p e ed as an adjacency ma ix, gi ing he possibili y o seeing he p oblem om a g aph poin o iew. I we es ablish ha a g aph is a pai o se s 𝐺 = (𝑉,𝐴) whe e 𝑉is he se o e ices and 𝐴is he se o edges as pai s o he o m (𝑢,𝑣) such ha 𝑢,𝑣 ∈ 𝑉, we de ine, hen, ∀𝑢,𝑣 ∈𝑉 exis s a uple o he o m (𝑢,𝑣)∈𝐴 i and only i 𝑢[𝑣]>0 o 𝑣[𝑢]>0. 27 G ado en Ingenie ía In o má ica Facul ad de In o má ica The ollowing illus a ion allows us o see how he ma ix would be in e p e ed, i we conside ha e e y elemen g ea e han 0 can be shown as 1. Figu e 4.1: Example o g aph wi h adjacency ma ix [13] Taking he e ices and edges, we know ha each edge (𝑢,𝑣) ul ills he condi ion ha bo h 𝑢and 𝑣ha e a simila i y alue g ea e han o equal o he h eshold and, he e o e, hey can be iden i ied unde he same ca ego y. The unc ion co e_ alo es e u ned a lis o indices in which he alues we e no 0 a e applying he 𝐹 unc ion on a s ing. Conside ing his lis 𝐴′as an adjacency lis , he p ocess o ollow is o go h ough 𝑣,∀𝑣∈𝐴′and ma k he node as isi ed and changing he alue o he node o ha o he node om which he sea ch s a s, ha is, upda ing he index alue ha heads he adjacency lis by he index alue ha s a ed om. In he example below, an example adjacency lis is conside ed: 0→0 1 4 6 (4.2) I we ollow he algo i hm s a ed abo e, hen we should isi all he nodes un il he nex esul is achie ed. 0→1 7 24 146 89 (4.3) 0→4 3 (4.4) 0→6 11 43 94 159 271 (4.5) Fo each o he nodes in he i s adjacency lis , hey ha e been isi ed and changed he head o 0, which is he head o he s a ing node. I can s ill be associa ed o which elemen he adjacency lis belonged as he i s elemen o each oken is always he oken i sel . Ini ially, an in dep h algo i hm was un o e each o he nodes o he g aph, bu s ing simila i y is no a ansi i e p ope y and i was disca ded. The e o e, i a node has al eady been isi ed, i s adjacency lis is no a e sed since i would ela e elemen s in a ansi i e way and would lead o an inco ec g ouping by Jaime del Rey Ga cía 28 Au oma ic analysis o high dimensional ca ego ical a iables in medical da abases o he p edic ion o hospi al bac e emia UCM classes. The ollowing unc ion shows he p ocess, which e u ns he upda ed adjacency lis s wi h he changed lis head alues. I is no ewo hy o no e ha his p ocess is only execu ed o hose nodes ha ha e mo e han one node in hei adjacency lis , since all will ha e a leas one node: hemsel es. de aplica_ca ego ias(posiciones_equi alencia): ””” De uel e las ca ego ías únicas esul an es de aplica a la bolsa de palab as los alo es de simili ud con umb al ””” o i, lis a_simila es in enume a e(posiciones_equi alencia): posiciones_equi alencia[i][1]= 1 i len(lis a_simila es[2]) > 1: o pos in lis a_simila es[2]: i (posiciones_equi alencia[pos][1]== 0): posiciones_equi alencia[pos][0]=posiciones_equi alencia[i][0] posiciones_equi alencia[pos][1]= 1 #p in ( alo es_cp[pos] + ' ' + s (posiciones_equi alencia[pos][1])) e u n np.a ay(posiciones_equi alencia,d ype='objec ') 4.1.4 De ining he classes Once he connec ions be ween all he okens ha e been es ablished, he ca ego ies ha will g oup he es o he s ings a e de ined, hus dealing wi h he p oblem o di y ca ego ies. Following he p ocess es ablished a he beginning o he sec ion, i is necessa y o de ine a map ha allows iden i ying he ca ego y ha de ines o g oups i o each class. Wi h he da a s uc u es o med so a , he simples op ion o de elop is o es ablish a dic iona y whe e he key is he index o he elemen o be consul ed and he alue is he ca ego y ha g oups i . This p ocess is he one ha is encoded in he ollowing unc ion, e u ning said dic iona y. 29 G ado en Ingenie ía In o má ica Facul ad de In o má ica The hi d op ion, on he o he hand, signi ican ly inc eases he numbe o columns in he Da aF ame bu is much easie o unde s and o bo h models and people. Each ca ego y will o m a new column in which each pa ien su e ing om his como bidi y will ha e he ow a 1. de aniade_ca ego ias_po _columnas(columna_ca ego ias, bolsa_ca ego ias,da F ame): o i, ba 1 in enume a e(bolsa_ca ego ias): col =[0 o xin ange(len(da F ame))] o j, oo1 in enume a e(columna_ca ego ias): o oo2 in oo1: i (ba 1 == oo2): col[j] = 1 da F ame[bolsa_ca ego ias[i]] =col e u n da F ame 4.4 Da ase p epa a ion Finally, be o e ca ying ou he es s wi h he model, he choice o a iables emains. In he s udy ca ied ou las yea , hey es ablished which a iables we e he mos ele an o op imizing he esul s ob ained wi h he models. This includes he elimina ion o unusable a iables, as well as hose ha can se e as p edic o s o he model and, as such, mus be elimina ed. Table 4.1: Remaining ea u es Fea u es selec ed by he algo i hm Clasi ica edad sexo Polimic anhonpol Anae obio Hongos pe iodo mes dia medio mic obpoli asae asanae asex COMORBIL so an ibio diashosp Especialid hgb Inghosp1m plaqu s local pm n leuc Inghosp12m Neoplasia Hepa opa ia En esp Diabe es Ca diopa ia Ins enal Ud p Alcoholismo c ea in sedo ina en basWeins d ogadic inmunosu es e oid ci ugia neu ope glucosa sin omas m_diges m_ espi leucoci ombope m_geni u m_ ascul None heless he e is a modi ica ion o be ca ied ou . The ea u e COMORBIL is a a iable ha indica es whe he he pa ien had any como bidi y a he ime o hospi al- iza ion. This ea u e has al eady no use, each como bidi y will be checked o each o Jaime del Rey Ga cía 36 Au oma ic analysis o high dimensional ca ego ical a iables in medical da abases o he p edic ion o hospi al bac e emia UCM he pa ien s as a bina y a iable, he e o e COMORBIL ⊂O ascomo . The ea u e is no needed, he in o ma ion ha i b ings o he da ase is sp ead and speci ied o e he di e en columns o each o he como bidi ies. Lea ing he a iable COMORBIL in he da ase would only add edundan da a and one mo e dimension o he da ase ea u es, so i is emo ed. 37 Chap e 5 Bias and a iance Supe ised Machine Lea ning algo i hms equi e a la ge olume o da a ha will be used du ing he aining and es ing o alida ion phases. The aining phase in Machine Lea ning algo i hms is whe e he algo i hms ind ela ionships o co ela ions in he da a, depending on he p oblem and model, among he da a o be ween he da a in oduced and he ou pu expec ed. The es ing phase consis s on supplying a smalle se o da a o check whe he he p edic ions a e accu a e o each o he cases. I is used o measu e he accu acy o he model. The p ocedu e desc ibed abo e is called induc i e lea ning. The induc ion capaci y o a model de e mines he le el o p ecision ha an algo i hm has when ying o sol e a p oblem simila o hose p o ided as an example. The goal o any Machine Lea ning algo i hm is o deduc he aining da a well o any domain o he p oblem. The pu pose o he echnique is o p edic u u e ac ions on ne e -be o e-seen da a. The main causes o a model wi h un eliable accu acy a e called o e i ing o high a iance and unde i ing o high bias. As he name says, o e i ing is p oduced when, du ing he lea ning phase, he model adjus s he weighs oo igh o he aining se in oduced. The easies way o ecognize his si ua ion is when a model has e y good accu acy le els wi h he aining da a bu su p isingly poo accu acy a es wi h he es da a. This is usually due o he use o small da ase s. On he con a y, unde i ing does no achie e high accu acy a es nei he wi h he aining no he es da ase s. This can be p oduced due o la ge se s o da a du ing he aining phase and sho pe iods o aining. The algo i hms hen lack o ime o adjus ing p ope ly he weighs and esul s in a gene ic model whe e no esul is ” oo good” no ” oo bad”. 39 G ado en Ingenie ía In o má ica Facul ad de In o má ica Figu e 5.1: Bias and Va iance [14] 5.1 A oiding Bias and Va iance O e i ing and unde i ing di ec ly a ec he accu acy and eliabili y o he models we a e wo king wi h. I is impo an o a oid his si ua ions by es ing he da a we a e going o wo k wi h [15]. • Ensu e ha we ha e a sufficien numbe o samples o bo h ain he model and alida e i . • Subdi ide ou da a se and keep a po ion o i o es he model. This will allow us o e alua e he pe o mance o he algo i hm and will also allow us o easily de ec he e ec s o o e i ing o unde i ing. • Make su e ha he es se is la ge enough o yield s a is ically meaning ul esul s and is ep esen a i e o he da a se as a whole. In o he wo ds, do no pick a es se wi h di e en cha ac e is ics han he aining se . • The excessi e numbe o a ibu es should be a oided, since i would gene a e a la ge numbe o dimensions in ou model. This is because each a ibu e makes up one dimension o he model’s sample space. The numbe o dimensions o he sample space mus be p opo ional o he numbe o cases a ailable o ca y ou he s udy, ha is, he g ea e he numbe o a ibu es, he g ea e he numbe o case s udies, o ice e sa, in he case o ha ing ew case s udies ew a ibu es should be used. In his p ojec , we a e using he da ase le om he s udy ca ied he p e ious yea , which has he ad an ages o no malized da a, missing da a ea men and a ibu e il- e ing. Figu e 5.2 illus a es how each o he si ua ions desc ibed abo e in luence he decisions o e a se o da a and how should a model beha e. Jaime del Rey Ga cía 40 Au oma ic analysis o high dimensional ca ego ical a iables in medical da abases o he p edic ion o hospi al bac e emia UCM Figu e 5.2: Examples o o e i ing and unde i ing [16] 5.2 K-Fold C oss Valida ion C oss- alida ion is a esampling p ocedu e used o e alua e Machine Lea ning models on a limi ed da a sample. I allows gene a ing di e en models om he same da ase . The echnique di ides in o di e en subse s om he o iginal se and gene a es a model so ha each subse o da a is used o bo h he aining pa and he alida ion pa . Mo e speci ically, i andomly mixes he da a ame and subdi ides i in o equal g oups. C oss- alida ion is p ima ily used in applied Machine Lea ning o es ima e he skill o a Machine Lea ning model on unseen da a. Tha is, o use a limi ed sample in o de o es ima e how he model is expec ed o pe o m in gene al when used o make p edic ions on da a no used du ing he aining o he model. I is a popula me hod because i is simple o unde s and and because i gene ally esul s in a less biased o less op imis ic es ima e o he model skill han o he me hods, such as a simple ain/ es spli . The gene al p ocedu e is as ollows: • Shu le he da ase andomly. • Spli he da ase in o 𝑘g oups. • Fo each unique g oup: –Take he g oup as a hold ou o es da ase . –Take he emaining g oups as a aining da ase . –Fi a model on he aining se and e alua e i on he es se –Re ain he e alua ion sco e and disca d he model • Summa ize he skill o he model using he sample o model e alua ion sco es 41 G ado en Ingenie ía In o má ica Facul ad de In o má ica Impo an ly, each obse a ion in he da a sample is assigned o an indi idual g oup and s ays in ha g oup o he du a ion o he p ocedu e. This means ha each sample is gi en he oppo uni y o be used in he hold ou se 1 ime and used o ain he model k-1 imes. K-Fold C oss alida ion is one o many p ocesses used o ha end, bu is wha will be used in his s udy. Figu e 5.3 illus a es he p ocess. Figu e 5.3: Examples o o e i ing and unde i ing [17] Jaime del Rey Ga cía 42 Chap e 6 Random Fo es The Random Fo es algo i hm is a supe ised lea ning echnique ha includes di e en me hods in he aining phase. I is a model equen ly used o deal wi h o e i ing and unde i ing p oblems. This algo i hm is used o sol ing eg ession and classi ica ion p oblems. I has a co ec ope a ion e en wi hou adjus ing i s own pa ame e s and emains s able when new da a is en e ed. On he o he hand, i equi es high p ocessing imes, i is difficul o in e p e and small da a ames a e no p ocessed op imally. A Random Fo es is an ensemble o decision ees combined wi h bagging. When using bagging [18], wha is ac ually happening is ha di e en ees see di e en po ions o he da a. The low co ela ion be ween models ( ees) is he key. The eason o his e ec is ha he ees p o ec each o he om hei indi idual e o s (as long as hey don’ cons an ly all e in he same di ec ion). No ee sees all he aining da a. This causes each ee o be ained wi h di e en da a samples o he same p oblem. In his way, when combining hei esul s, some e o s a e compensa ed o o he s and we ha e a p edic ion ha gene alizes be e . When we using bagging, we also combine a ious Machine Lea ning models. Unlike o he me hods, he way o ge e o s o compensa e o each o he is ha each model is ained wi h subse s o he aining se . These subse s a e o med by andomly choosing samples (wi h epe i ion) om he aining se . 6.1 Fi ing a Random Fo es To adjus he model based on his classi ie , i is necessa y o adjus he n_es ima o s pa ame e . This pa ame e ep esen s he numbe o ees ha will make up he model and on which each case s udy will be e alua ed. In addi ion, he andom_s a e pa ame e is se o be able o eplica e he accu acy alues o he model wi h he same inpu pa ame e s. This pa ame e helps o con ol he andomness o he algo i hm when gene a ing he decision ees. I is impo an while changing be ween da ase s, pa ame e s on he same da ase o ee alua ing models. 43 G ado en Ingenie ía In o má ica Facul ad de In o má ica Fi s , o a oid o e i ing and unde i ing, we will spli he da a using 80% o he da a o he aining p ocess and he emaining 20% o he model alida ion. Taking in o accoun he amoun o samples in he da ase and he amoun o a ibu es o each sample, he 80-20 olding dis ibu ion lea es oom o a co ec i ing p ocess. om sklea n.model_selec ion impo ain_ es _spli X_T ain, X_Tes , Y_T ain, Y_Tes = ain_ es _spli (X, Y, es _size=0.2) Once we ha e applied K-Fold C oss Valida ion, we can execu e he code ha will adjus he numbe o es ima o s o minimize he e o . A ange be ween 20 and 90 es ima o s is ixed. Now he model has o ain using he aining da a om he C oss Valida ion me hod and hen calcula e he accu acy wi h he se s o da a used o he es phase. a ayP ed =[] o ind in ange(20,90): p in (ind, end =' ') RF =RandomFo es Classi ie (ind, andom_s a e=0) RF. i (X_T ain, Y_T ain) p edicciones =RF.p edic (X_Tes ) accu acy =accu acy_sco e(Y_Tes , p edicciones) a ayP ed.append(accu acy) i accu acy >maxi_accu acy: maxi =ind maxi_accu acy =accu acy _max=RF In Figu e 6.1 we can see ha o e all he e a e e y decen accu acy alues, bu he peak is eached o 54 es ima o s ( he highes alue in he eco d is placed in posi ion 34, as he sample goes om 20 o 90 he o al numbe o es ima o s is 34+20), a andom_s a e alue o 0wi h an accu acy o o e 0.940. Fo his execu ion, K-Fold pa i ion wi h 80% o aining and 20% o es ing was used. The esul s a e shown o he aining phase. Jaime del Rey Ga cía 44 Au oma ic analysis o high dimensional ca ego ical a iables in medical da abases o he p edic ion o hospi al bac e emia UCM Figu e 6.1: S udy o e he numbe o es ima o s Once he e u ned alues a e known, we alida e he model and he e u ned accu acy alue in o de o de ec o e i ing o unde i ing p oblems. To do his, he p edic ions o a new da ase no used du ing he aining phase and he accu acy o he model on his da ase a e calcula ed. We use now he alida ion se . # un model wi h op imal pa ame e s RF =RandomFo es Classi ie (54, andom_s a e=0) RF = _max RF. i (X_T ain, Y_T ain) # es p edicciones =RF.p edic (X_Tes ) epo =pd.Da aF ame() epo e_ac ual =classi ica ion_ epo (Y_Tes , p edicciones, ou pu _dic =T ue) epo ['1']= epo e_ac ual['1']. alues() Model accu acy is:0.935 In his case, sligh ly lowe accu acy alues a e ob ained han hose p e iously collec ed, which indica es ha he model does no p esen o e i ing o unde i ing p oblems. 6.2 E alua ion me ics In o de o ully unde s and he deg ee o p ecision o he model in ques ion, he e a e a ious in e p e abili y echniques o Machine Lea ning models in classi ica ion p oblems as he e can be p oduced di e en ou comes. In his example, he p edic ed alue can 45 G ado en Ingenie ía In o má ica Facul ad de In o má ica had al eady been d awn. The ea u es we e asae and asanae and we e also emo ed om he da ase . In Figu e 6.11 we can see he esul s o he aining p ocess o he Random Fo es . The model accu acy alue was 0.871 a e es ing he model wi h he alida ion se . Figu e 6.11: S udy o e he numbe o es ima o s wi h il e ed ea u es This esul is a mo e cohe en aking in o accoun he esul s om he p e ious s udy ha sco ed an accu acy o 0.86 and he lis o mos ele an a iables, now upda ed in Figu e 6.12, whe e none o he new ea u es ep esen a signi ican decision alue and he e o e ou model should beha e almos he same. Jaime del Rey Ga cía 52 Au oma ic analysis o high dimensional ca ego ical a iables in medical da abases o he p edic ion o hospi al bac e emia UCM Figu e 6.12: Mos ele an ea u es in he second execu ion 53 Chap e 7 Conclusions Wo king wi h such a he e ogeneous da ase has been a eal challenge o he en i e da ase p epa a ion p ocess. The i s conclusion is ha he whole s udy and, in gene al, any p ocess ha is epea ed simila o his, would g ea ly bene i om a mo e speci ic desc ip- ion o he ex s ings. Al hough i is a wo k ha emains o he w i ing o doc o s, we ha e seen how he subs i u ion o , o example, s oke o ce eb o ascula acciden , allows he g ouping and iden i ica ion o ca ego ies in a much mo e eliable and exac way. As explained in he sec ion on s ing simila i y, misspellings we e ele an when compa ing sho -leng h okens, so ano he ad an age o his si ua ion would be he dec ease in he ele ance o misspellings in ob aining o he simila i y be ween wo s ings. Using he ile p o ided by Ósca has allowed he con e gence o all he cleaning and da a p epa a ion wo k in o a use ul esul . This ile, e en so, does no collec all he a ian s o abb e ia ions o spelling mis akes ha he se o a iables con ains. This leads o he conclusion ha he ool alone did no p o ide sufficien suppo o ca y ou he s udy. In sho , is no iceable ha na u al language p ocessing is a ask ha s ill has oom o imp o emen , howe e he use o his ool has made i possible o educe a se o mo e han 1000 ca ego ies o app oxima ely one en h. Looking a he inal da ase , he amoun o a ibu es ini ially added by using a bina y encoding o include he ca ego ies in he da ase seemed excessi e, inc easing he o al numbe o a ibu es o 247, since i is a pe ec example o a case ha may su e high bias and gene alize he weigh s associa ed wi h he pa ame e s in he lea ning p ocess. Howe e , since he e we e mo e han 4,300 examples, no only has i no p oduced his e ec , bu i has also con ibu ed o an imp o emen in he success a e compa ed o he s udy ca ied ou las yea . Rega ding he model used, he conclusion d awn om he p e ious s udy showed a be e success a e wi h he applica ion o Random Fo es on he da ase . The ad an age o using his model o e o he Machine Lea ning models is ha i is one o he models wi h mo e explana o y powe , as i is di ec ly o med by decision ees. The model esul ing om his algo i hm ained on he p e iously men ioned da ase has an accu acy o app oxima ely 94%. One o he easons ha he i ness o he i s model can be conside ed success ul is he 55 G ado en Ingenie ía In o má ica Facul ad de In o má ica esul o displaying he ROC cu e (6.7) and he con usion ma ix (6.3). The conclusion o he co ec classi ica ion by he model is suppo ed by he me ics ha allow us o app ecia e he clea di e en ia ion be ween bo h dis ibu ions. We can no igno e ha , a e all, he p ocess o cleaning he da a, compu ing he simila i y be ween all he okens and inally ob aining he di e en ca ego ies om he ea u e O ascomo , none o hem we e in he op 50 o he mos weighed ca ego ies. The conclusion is ha , ega ding he esul o he second model, wi h he da a p o ided and he Machine Lea ning ype o model used, i is a he difficul o make a co ec ea ly diagnosis o bac e emia. Jaime del Rey Ga cía 56 Chap e 8 Fu u e imp o emen s This p ojec began wi h he idea o wo king on all he nominal a iables o he da ase p o ided by Hospi al Uni e si a io de Fuenlab ada. Howe e , he o ganiza ion and wo k ha has led o he ea men o a single a iable has aken up mos o his s udy. Fo his eason, one o he possible b anches in which his s udy can lead is he one in which he es o he a iables a e ea ed and Machine Lea ning me hods a e applied o he complemen ed se . Taking in o accoun he da ase esul ing om his s udy, i is possible ha he inclusion o he o he a iables may o e load he numbe o a ibu es o he se p o ided when coding hem o co ec ea men by Machine Lea ning models, ye i seems he mos u gen s ep o ake ega ding he esul s o he las model. O he possible a ian s a e he applica ion o o he Machine Lea ning models such as Neu al Ne wo ks o Suppo Vec o Machines (SVM). These models ha e he g ea dis- ad an age o lacking explicabili y. The e o e, i would equi e an in-dep h s udy o he a iables and he weigh s associa ed wi h hem. Howe e , hey a e models ha can de ec complex ela ionships be ween a ibu es ha help imp o e p edic ion efficiency. This is specially ele an in he con ex o his p ojec , whe e i would also be in e es ing o ind possible ela ionships be ween he ca ego ies ob ained and he o iginal a iables om he da ase . In addi ion, he s udy ca ied ou las yea and his one ha complemen s i , p o ide conclusions and help o ecognize bac e emia as a bina y class, bu in eali y he e a e a di e si y o ypes o bac e emia and no all sha e symp oms o ea men . The e o e, he conclusions d awn du ing hese wo yea s could be applied o he s udy o he de ec ion o bac e emia as a mul iclass classi ica ion p oblem. This pa icula s udy may be he mos labo ious, since i may equi e di e en da ase s o each ype o bac e aemia, howe e , i he esul s we e success ul, i would be a g ea boos in he p edic ion o bac e emia diagnoses. 57 Lis o Figu es 1.1 Blood cul u e examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 3.1 Sample o he ea u e O ascomo . ..................... 12 3.2 Sample o he elemen s in he ea u e O ascomo e e ing o anemia as di yca ego ies................................. 13 3.3 Sample o he elemen s in he ea u e O ascomo e e ing o alzheime as di yca ego ies................................. 13 3.4 Sample o he elemen s in he bag o wo ds. . . . . . . . . . . . . . . . . 18 3.5 Example o a hea map o e di y ca ego ies o di e en wo ks. . . . . . . 20 3.6 Example 1 o a hea map o e 25 andom di y ca ego ies om he da ase . 21 3.7 Example 2 o a hea map o e 25 andom di y ca ego ies om he da ase . 22 3.8 Example o he classes g ouped ela i e o he numbe o neighbo s . . . 23 3.9 K-NN o e he bag o wo ds using k=4 neighbo s . . . . . . . . . . . . . 24 4.1 Example o g aph wi h adjacency ma ix . . . . . . . . . . . . . . . . . . 28 5.1 BiasandVa iance............................... 40 5.2 Examples o o e i ing and unde i ing . . . . . . . . . . . . . . . . . . 41 5.3 Examples o o e i ing and unde i ing . . . . . . . . . . . . . . . . . . 42 6.1 S udy o e he numbe o es ima o s . . . . . . . . . . . . . . . . . . . . 45 6.2 Hypo hesis es ing .............................. 46 6.3 Con usionma ix............................... 48 6.4 IdealAUCscena io.............................. 49 6.5 Wo s ROCscena io ............................. 49 6.6 IdealROCscena io.............................. 49 6.7 ROC cu e om Random Fo es . . . . . . . . . . . . . . . . . . . . . . 50 6.8 Es ima ed weigh pe ea u e . . . . . . . . . . . . . . . . . . . . . . . . . 50 6.9 Mos weighed ea u es . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 6.10 SHAP alue (impac on model ou pu ) . . . . . . . . . . . . . . . . . . . 51 6.11 S udy o e he numbe o es ima o s wi h il e ed ea u es . . . . . . . . 52 6.12 Mos ele an ea u es in he second execu ion . . . . . . . . . . . . . . . 53 59 Lis o Tables 3.1 Fea u es om he da ase . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.2 Selec ed ea u es in he s udy. . . . . . . . . . . . . . . . . . . . . . . . . 11 4.1 Remaining ea u es.............................. 36 61