Automated Feature Engineering for Classification Problems

FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO Au oma ed Fea u e Enginee ing o Classi ica ion P oblems Guilhe me Felipe do Nascimen o Reis Mes ado em Engenha ia de So wa e Supe iso : Ca los Manuel Milhei o de Oli ei a Pin o Soa es July 29, 2019 Au oma ed Fea u e Enginee ing o Classi ica ion P oblems Guilhe me Felipe do Nascimen o Reis Mes ado em Engenha ia de So wa e July 29, 2019 Abs ac The s udy on ea u e gene a ion has g own o e he las yea s.I is en i ely dependen on domain knowledge and done manually, so i is ime consuming and no scalable. In u n, me a-lea ning helps o lea n h ough di e en domains and can b ing bene i s o his a ea. We p esen wo me a-lea ning app oaches o suppo ea u e selec ion. The i s p edic s whe he a gi en au oma ed ea u e enginee ing app oach will imp o e he esul s. The second p edic s i a single ea u e will imp o e he esul s. We es hese app oaches on 100 da a se s om di e en domains. Ou expe imen s showed ha i is possible o use me a-lea ning in he ea u e selec ion p ocess, and can in o m us whe he o no we should gene a e ea u es wi h a he gi en me hod. On he o he hand, he esul s o he second app oach, which p edic s he use ulness o each indi idual ea u e a e no posi i e. The esul s show ha me a-lea ning can be used o aid he gene a ion and selec ion o ea u es. Howe e , ou app oach can s ill be imp o ed, being mo e p ecise in he p edic ions a he me a-le el and be e esul s a he base le el. Ou code is a ailable a h ps://gi hub.com/gui elipe /au oma ed- ea u e-enginee ing Keywo ds: da a mining, machine lea ning, me alea ning, KDD, ea u e enginee ing, au oma ed ea u e enginee ing i ii Resumo O es udo sob e ge ação de ea u es em aumen ado con o me os anos. To almen e dependen e de conhecimen o de domínio e ei o de o ma manual, po isso consome mui o empo e não é escalá el. Po sua ez, me a-lea ning auxilia o ap endizado a a és di e en es domínios. Nos ap esen amos duas abo dagens de me a-lea ning pa a auxílio na seleção de ea u es. A p imei a p e ê se uma de e minada abo dagem de ge ação au omá ica de ea u es melho a á os esul ados. O segundo p e ê se um único ecu so melho a á os esul ados. Nós es amos essas abo dagens em 100 conjun os de dados de di e en es domínios. Nossos expe imen os mos a am que é possí el usa o me a-lea ning no p ocesso de seleção de ecu sos, e pode nos in o ma se de emos ou não ge a ecu sos com um de e minado mé odo. Po ou o lado, os esul ados da segunda abo dagem, o qual p e ê a u ilidade de cada ca - ac e ís ica indi idual, não são posi i os. Os esul ados mos am que o me a-lea ning pode se usada pa a auxilia na ge ação e seleção de ecu sos. No en an o, nossa abo dagem ainda pode se melho ada, sendo mais p ecisa nas p e isões no me a-le el e melho es esul ados no base le el. Nosso código es a disponí el em h ps://gi hub.com/gui elipe /au oma ed- ea u e-enginee ing. Keywo ds: da a mining, machine lea ning, me alea ning, KDD, ea u e enginee ing, au oma ed ea u e enginee ing iii i Acknowledgemen s Fi s , I would like o hank my amily and wi e o all he suppo and ad ice h oughou my jou ney. De ini ely wi hou hem, I would no be he pe son I am oday. I also wan o hank all he men o s who ha e appea ed in my li e, all o hem helped me o become he p o essional ha I am oday. Mainly, P o . João Rod igues, who unknowingly in oduced me o enginee ing when I was s ill in high school. The P o . Tiago Hen ique, who imp o ed me du ing my enginee ing ca ee when I was s ill in Bachelo , and my P o esso s Ca los Soa es and Rui Quin ino who helped and guided me a lo in his wo k. Fo all hose who helped me and encou aged me in my pa h, I hank you om my hea , and I will ce ainly always emembe you. Guilhe me Reis xii LIST OF TABLES Abb e ia ions Au oML Au oma ion Machine Lea ning DFS Deep Fea u e Syn hesis CRISP-DM C oss Indus y S anda d P ocess o Da a Mining NA No A ailable NHST Null Hypo hesis Signi icance Tes ML Machine Lea ning xiii Chap e 1 In oduc ion One o he ac o s o a success ul applica ion o a machine lea ning app oach is a well p epa ed da a se . The da a needs o be cleaned and p epa ed o he algo i hm o induce a good model. P epa ing a da a se co ec ly usually akes a signi ican amoun o ime, and i equi es solid knowledge abou he domain. One pa icula ly impo an ask is ea u e enginee ing, which con- sis s o c ea ing new a iables ha make i easie o he algo i hm o ob ain a good model. The e a e wo app oaches o ea u e enginee ing: manual and au oma ed. In he manual app oach, one ea u e is buil a a ime. This is ime-consuming, challenging and e o -p one p ocess and depends on domain knowledge. The au oma ed app oach can ex ac s many ea u es om a da a se in a sys ema ic way which can be done wi hou human in e en ion. Addi ionally, i can be applied o any p oblem and he building p ocess is quicke and no dependen on domain knowledge. Se e al au oma ion s a egies ha e been de eloped ecen ly, o dec ease he e o de o ed o ea u ing enginee ing. Howe e , hese me hods commen ed on in his hesis and ou me hod do no gua an ee ha he se o ea u es gene a ed will lead o be e models [6,11,8]. “Au oma ing ea u e enginee ing op imizes he p ocess o building and deploying ac- cu a e machine lea ning models by handling necessa y bu edious asks so da a sci- en is s can ocus mo e on o he impo an s eps.” [6] 1.1 Mo i a ion On ea u e enginee ing, i is possible o au oma e pa o he p ocess which does no depend on domain knowledge. A suba ea o machine lea ning ha deals wi h he p oblem o ex ac ing knowledge abou lea ning p oblems which is independen o he domain a e me a-lea ning. Me a-lea ning consis s o lea ning abou he beha iou o machine lea ning (ML) algo i hms by collec ing (me a)da a abou ins ances o hose p ocesses and applying ML algo i hms o ha me ada a [15]. Howe e , me a-lea ning app oaches ha e ne e been applied o he p oblem o ea u e enginee ing. 1 2In oduc ion 1.2 Objec i es Gi en a me hod o au oma ically gene a e ea u es, ou goal is o use a me a-lea ning app oach o decide whe he i will lead o be e models compa ed o he o iginal da a se . Ou p elimina y esea ch ques ion is he ollowing: •Does he sys ema ic combina ion o ea u es using simple ope a ions lead o be e p edic i e models han he o iginal da a se ? A e , we speci ically add ess he ollowing ques ions: •Can a me a-lea ning app oach p edic when a me hod o he sys ema ic gene a ion o ea- u es leads o be e p edic i e models? •Can a me a-lea ning app oach p edic when a me hod o au oma ic gene a ion o ea u es leads o be e p edic i e models? •Can a me a-lea ning app oach p edic when an indi idual ea u e leads o a be e p edic i e model? The con ibu ions o his wo k a e: •Applica ion o a me a-lea ning app oach o he p oblem o deciding whe he a me hod o au oma ic ea u e gene a ion should be applied and which ea u es should be used o im- p o emen o he model; •De elopmen o a se o me a- ea u es ha a e speci ic o he p oblem add essed he e; •Empi ical e alua ion o he app oach p oposed using a simple me hod o au oma ic ea u e gene a ion. In sec ion 2, we discuss he backg ound and ela ed wo k; In sec ion 3, we explain he sys em- a ic ea u e gene a ion and he explo a o y da a analysis. Fo he sec ions 4and 5we discuss he app oaches p oposed and he expe imen al se up used and esul s. Finally, in sec ion 6, we p esen he conclusion and u u e wo k. Chap e 2 Backg ound and Rela ed Wo k O e he las yea s, he demands o da a mining ha e been inc easing and also he g owing in e es in au oma ed machine lea ning, see he igu e 2.1 1. Da a mining uses a conside able numbe o da a analysis echniques o he p ocess o iden i ying speci ic pa e ns in a massi e amoun o da a. In his chap e , we discuss an analysis o he da a mining p ocess and mining echniques 2.1, he ac ual s a e o au oma ed machine lea ning 2.4, me a-lea ning and au oma ed ea u e engi- nee ing 2.3. 2.1 Da a Mining Da a mining can use machine lea ning echniques, bu i is possible o use da a managemen ech- niques (e.g. ela ional da abase managemen sys em) . Ne e heless, we a e going o alk only abou machine lea ning echniques. The e a e wo p ima y ype o asks o da a mining, p edic i e and desc ip i e.p edic i e has he in en o p edic based on he his o ical da a, and he mos common asks a e, classi ica ion and eg ession. Reg ession is used o p edic a nume ic alue. Fo example, a new ca model is added o he da abase, and he sys em needs o sugges he p ice o a ca wi h i e doo s, elec ic di ec ion, om a speci ic b and, wi h 100.000 Kms. Reg ession should be used o ha case. Classi ica ion is used o de e mine a class o a da a poin . Fo example, a speci ic en y o he lowe s da abase does no ha e a classi ica ion, and he sys em needs o iden i y wha lowe i is o comple e he da a. Classi ica ion should be used o ha case. Desc ip i e da a mining is used o o ganise he da a and unde s and be e he in o ma ion ha he da abase has; i is also di ided in o wo echniques, associa ion and clus e ing. Clus e ing iden i ies g oups o simila da a poin s. Da a poin s in a clus e a e mo e simila among hemsel es han o da a poin s in o he clus e s. Associa ion analyses he associa ion be ween i ems. Fo 1A ailable a h ps:// ends.google.p / ends/explo e?q=Au oML 3 4Backg ound and Rela ed Wo k Figu e 2.1: Google T ends abou Au oML . example, in a comme cial da abase associa ion can iden i y p oduc s ha a e gene ally pu chased oge he , a amous s o y analysed a ma ke da abase and disco e ed ha usually clien s who buy diape s also buy bee . The p ocess o da a mining is in e ac i e and challenging. The mos common p ocess model in ol es six phases, whe e each one has speci ics deli e ies o he nex phase. The p ocess model is also known as C oss Indus y S anda d P ocess o Da a Mining (CRISP-DM), see li e cycle on igu e 2.2. Business unde s anding - The goal o his phase is unde s anding he business pe spec i e and hen ansla ing i o a da a mining p oblem. Da a unde s anding - The ocus o his phase is o collec and ha e ini ial con ac wi h he da a. Analyse da a quali y and iden i y possible p oblems. Da a p epa a ion - The goal o his phase is o clean he da a, gene a e mo e ea u es, selec he ea u es c ea ed o eed he algo i hms o he nex phase. This phase is he cen e o ou wo k. Modelling - The algo i hms a e selec ed, and hei pa ame e s uned o an op imal esul . E alua ion - Be o e he inal deploymen , his s ep will assess whe he he business goals ha e been me . Deploymen - The goal o his phase is o plan deploymen , p oduce a inal epo and e iew he p ojec o e alua e wha wen igh and wha wen w ong. All o his in o ma ion and mo e de ailed p ocess o CRISP-DM can be ound in he book CRISP- DM 1.0 [14]. 2.2 Au oma ed Machine lea ning 5 Figu e 2.2: Phases o CRISP-DM model [14] 2.2 Au oma ed Machine lea ning Machine lea ning is ime-consuming and needs many esou ces o ge he ask done. The e o e, companies like IBM ha e been in es ing in Au oma ed Machine Lea ning esea ch [11]. Au o- ma ed machine lea ning (Au oML) goal is o change his scena io, de eloping accessible machine lea ning. Cu en ly, Au oML ocuses on da a p epa a ion and modelling. In da a p epa a ion, i is o en used o au oma e he p ocess o da a p ep ocessing, ea u e gene a ion, and selec ion. Fo model aining, i is used o au oma e model selec ion and hype -pa ame e op imisa ion. 2.2.1 Da a p ep ocessing The da a collec ed is no always sui able o un a machine lea ning p ocess. The e o e, i is nec- essa y o ans o m he da a. P ep ocessing in ol es many di e en asks such as scaling, missing alue impu a ion, ou lie de ec ion, binning and ca ego ical da a ans o ma ion [2]. Scaling - Scaling echniques ensu e ha he nume ical ea u es a e weigh ed p opo iona ely by he algo i hms. The e ms used o ha echnique a e s anda disa ion and no malisa ion. Missing alues - Missing alues o en occu s in da a se s. This can happen because o a de ice bug, yping e o s o e en he use choosing no o p o ide some in o ma ion. 1. Use a cons an : I is possible o ill all he missing alues wi h a cons an . The cons an can be a disp opo iona e numbe o a no a ailable (NA) o sepa a e hose o he es o he da a se . 2. Use a mean/median alue o ill: Nume ical alues can be eplaced wi h mean o median and ca ego ical alues can be eplaced by he mode. 6Backg ound and Rela ed Wo k Figu e 2.3: Fea u e selec ion p ocess [18] 3. Remo e/Dele e Da a: This echnique is ecommended when only a ew da a a e missing, he elimina ion can be applied o a speci ic ow o column. Howe e , when oo much da a a e missing, i may educe he amoun o a ailable da a oo much, a ec ing he quali y o he models gene a ed. 4. Use da a mining o p edic he missing alue: I is possible o use machine lea ning algo- i hms o p edic he mos p obable alue o missing da a. Howe e , when used o oo much missing da a i can cause o e i ing on he inal model because he da a used o ill he emp y da a was he same o ain and es . 5. Add lag: Indica e he missing da a wi h a lag. Inse a lag (e.g.: ’missing alue’ s ing o ou lie numbe ) in he missing da a can iden i y whe e he da a was missed, aiding o iden i y he lack o in o ma ion and assess he da a. Ou lie de ec ion - Ou lie s a e hose alues " ha a e so di e en om he o he s ha dis o he da a dis ibu ion. Those alues can lead o low pe o mance models. Ou lie s ha e wo ypes, uni a ia e and mul i a ia e. A uni a ia e ou lie , as he name implies, is an ou lie based on a single a iable. A mul i a ia e ou lie is a mix u e o alues on wo o mo e a iables. Binning - Binning goal is o disc e izes con inuous alues, i is a p ocess o g oup a con inuous numbe in smalle in e al, o example, di iding he age o a su ey in o small in e als. This is use ul o algo i hms which ha e di icul ies in analysing nume ical a iables, such as nai e Bayes. Two common binning me hods a e: •Equiwid h binning: The bins a e di ided in o nin e als o equal size. •Equi equency binning: The bins a e di ided in o ng oups, each con aining he same num- be o alues. All o his in o ma ion and mo e de ail can be ound [2]. 2.2.2 Fea u e selec ion Fea u e selec ion is used o emo e unnecessa y and edundan a ibu es om he da a because some a ibu es dec ease pe o mance ins ead o inc easing i . Fu he mo e, he complexi y o he 2.2 Au oma ed Machine lea ning 7 Table 2.1: Example o ea u e gene a ion ex ac ing in o ma ion om a da e a ibu e Use ID Penul ima e login Las Login Di Login Las Login is Weekend? 865 12/12/16 03/01/17 22 FALSE 1954 18/03/16 23/07/16 127 TRUE 1307 25/11/17 11/03/18 106 TRUE 1966 29/03/16 26/04/16 28 FALSE 51 07/01/16 23/02/16 47 FALSE model can inc ease wi h mo e a ibu es, no helping he da a scien is o unde s and o explain he model. So, ea u e selec ion can help o ha e a be e pe o mance wi h ewe ea u es and less complexi y in he model [4] (Figu e 2.3). Th ee ypes o me hods can be used: Fil e me hods, w appe me hods and embedded me hods. Fil e Me hods These me hods analyse each ea u e independen ly and assign a sco e o each. These sco es a e used o ank he ea u es and he bes ones a e selec ed. The sco es can be based on he in o ma ion gain measu e. W appe Me hods These me hods e alua e combina ions o ea u es. Di e en combina ions a e c ea ed and compa ed by lea ning and e alua ing a model. The pe o mance o he model is used o guide he sea ch o he bes se o ea u es. W appe me hods include ecu si e ea u e elimina ion, dele ing and adding ea u es, o using a andom algo i hm o help he sea ch. Embedded Me hods These a e o en egula isa ion mechanisms embedded in he algo i hms. I consis s o adding a penal y o complex models, helping o educe o e i ing and a iance. Howe e , i adds bias o he model. An example o his me hod is Ridge Reg ession. 2.2.3 Fea u e gene a ion The ea u e gene a ion, also known as ea u e cons uc ion, adds mo e ea u es in o he aw da a and i can esul in mo e in o ma ion ha leads o models wi h be e accu acy. Howe e , adding ea u es inc eases he dimensionali y o he da a se and inc eases he chance o o e i ing, known as he cu se o dimensionali y. To minimise he e ec o ea u e c ea ion, one app oach is ea u e selec ion, which was desc ibed ea lie . To illus a e he ea u e enginee ing dilemma, conside Table 2.1. He e we ha e i e columns showing he beha iou o access o some use s, and he i s h ee columns a e he aw da a, he las wo columns a e he new ea u es gene a ed in he p ocess o ea u e enginee ing. We can see ha he use s who access he pla o m on weekends also s ay longe wi hou using i . When s udying only he o iginal da a, we do no see his pa e ns. New ea u es can help o show hose pa e ns and help o build be e models. Nume ical - Many ma hema ical ope a ions can be applied o ans o m single and g oups o a iables. Fo example, log, a io, addi ion, a iance o wo numbe s o any ma hema ical 14 Backg ound and Rela ed Wo k Figu e 2.7: Example o Decision ee aids he decision be ween wo algo i hms ees o p esen a be e p edic i e esul . The Decision ee in mos o he cases ha e a high a iance, and i happens when we ex ac di e en aining and es se s om he same da a se , esul ing in di e en esul s. Sadly, he Decision ee can cause poo pe o mance o a new da a se . Howe e , Random o es is eage o sol e he weakness o Decision ee educing he a iance and bias e o s, by choosing a subsample o he ea u e space o each spli . [3] 2.4.6 Resul s analysis In his sec ion, we summa ise he esul s o he ela ed pape s. They a e discussed sepa a ely because hey a e no all ela able. Explo eKIT [8] - The au ho s buil ou app oaches and he bes app oach was he ML_ ull, ob aining an a e age e o educ ion o 17.4%–29.3%. Deep Fea u e Syn hesis [6] - The Da a Science Machine, also known as Deep Fea u e Syn hesis, was one o he i s p ojec s wi h success ul esul s on au oma ed ea u e gene a ion. The algo i hm has been es ed in Kaggle compe i ions and i achie ed 90% o he bes sco e achie ed by any compe i o in wo ou o h ee compe i ions. The bes pe o mance was on KDD15 challenge bea ing 86% o o he compe i o s. OneBu on Machine [11] - The One Bu on Machine uses wo app oaches. I was es ed in di e - en da abases, one o hem was he KDD 2014 cup o he compa ison wi h he Deep ea u e 2.4 Au oma ed ea u e enginee ing 15 syn hesis p ojec . The One bu on machine compa e hei esul s wi h he esul s o da a scien is s in he KDD 14 Challenge, and hei esul s a e in he op 17% while he DFS was in he op 30%. The esul s show ha hese app oaches a e compe i i e wi h a la ge numbe o expe ienced da a scien is s. Au oLea n [9] - The Au olea n algo i hm aims o use ewe ea u es o be e accu acy. I achie ed 13.28% and 5.87% imp o emen in e ms o accu acy, agains he o iginal ea- u es. I also ob ained compe i i e esul s when compa ed wi h Explo eKi and o he s. The algo i hm was es ed on 25 da a se s. Cogni o [10]- I ob ained imp o emen s be ween 7% - 40% compa ed wi h he o iginal ea u es on di e en da a se s. Au oma ic ea u e gene a ion o machine lea ning based op imising compila ion [12] - The goal o his p ojec is o use Au oML o de ec ing he bes loop un olling in GCC 4.3.1. The no el algo i hm applied in his p ojec was able o achie e 75% o he maximum eloci y o he compile on a e age. The benchma ks show ha he algo i hm imp o ed 35% o he speed. In his hesis will be used one o he Bayesian es s p esen ed in he a icle, called Bayesian co ela ed - es ha will be used o compa e pe o mance be ween he esul s ob ained by 10- old c oss- alida ion and he baseline o each old. 2.4.7 P og amming languages The p og amming languages used in he ela ed wo k we e Py hon, Ja a and C/C++. The e is no mo i a ion o he selec ion o he p og amming language. Thus, he discussion abou he be e language o Au oma ed Machine Lea ning is open. The choice o p og amming language is no ele an o his hesis. Howe e , he choices in he ela ed wo k a e well di ided. Two pape s use Py hon, Deep Fea u e Syn hesis [6] and Au oLea n [9], wo pape s use Ja a, Cogni o [10] and Explo eKi [8], and he only one ha uses C/C++ is Au oma ic Fea u e Gene a ion o Machine Lea ning Based Op imizing Compila ion [12]. The e is ano he one which ha did no iden i y which p og amming language was used [11]. The Vision Mobile company eleased a epo in Ap il 20172, showing he popula i y o each language o machine lea ning. Howe e , i can change depending on he backg ound o he in- di idual in cha ge o he de elopmen p ocess. Fo example, Py hon was he p e e ed language o machine lea ning on 38% o answe s and used mos ly by da a scien is s and junio p o ession- als. F on -end de elope s use ja asc ip wi h 16%, Elec onics enginee s p e e C/C++ (8%), da a analys s and s a is ician p io i ise R (14%). In conclusion, he e is no co ec language. 2A ailable a h ps:// isionmobile.com/ epo s/s a e-de elope -na ion-q1-2017 16 Backg ound and Rela ed Wo k Chap e 3 Sys ema ic ea u e gene a ion In his chap e , we p esen ou app oach o sys ema ic ea u e enginee ing. This me hod will be used as ea u e gene a ion me hod o he me a-lea ning app oaches used in he ollowing chap e s. We also p esen an explo a o y da a analysis o he empi ical esul s ob ained wi h his me hod. The app oach was implemen ed in Py hon1, using some lib a ies o assis he de elopmen , such as I e ools 2, Sciki Lea n 3and Pandas 4. The ML algo i hm used was Random Fo es wi h 100 es ima o s and all de aul Sciki -lea n pa ame e s . The e alua ion me ic used was accu acy and i was es ima ed wi h s a i ied c oss- alida ion. 3.1 Expe imen al Se up We analysed 100 supe ised classi ica ion da a se s collec ed om he OpenML pla o m [5]. We selec ed da a se s wi h 30 o ewe classes (Figu es 3.1 and 3.2 ). Since we ha e decided o ocus on ope a ions on nume ical a iables, we decided o emo e all a iables which a e no nume ical 3.2. Fo example, conside ing we ha e a da a se wi h 20 a ibu es on he o al, bu only 5 o hem a e nume ical a ibu es, ou o iginal da a se will con ain only he 5 nume ical a ibu es. The da a se s we e collec ed om he OpenML pla o m [5]. The ull lis o da a se s can be ound in appendix A. 3.2 App oach Ou app oach can be applied o a iables o any na u e. Howe e , o simplici y, in his p ojec , we will ocus on nume ical a iables, wi hou loss o gene ali y. 1h ps://www.py hon.o g/ 2h ps://docs.py hon.o g/3/lib a y/i e ools.h ml 3h ps://sciki -lea n.o g/s able/ - e sion 0.19.2 4h ps://pandas.pyda a.o g/ 17 18 Sys ema ic ea u e gene a ion Figu e 3.1: Analysis o da a se s wi hou elimina ing highly co ela ed ea u es. Fo mally, gi en a da a se D:= (X,Y), wi h a a ge a iable Y, and a se o a iables X:= (x1,x2,...,xm)⊆Rm, whe e mis he numbe o a iables. Addi ionally, gi en a bina y unc ion τo: R×R→Rou app oach gene a es a se o new ea u es Excon aining all possible combina ions o he o iginal a iables, wi hou loss o gene ali y, using ha unc ion: Ex= (o,i,j)| (o,i,j)=τo(xi,xj),∀i6=j∈{1,...,m},o(3.1) A e he gene a ion o new ea u es, he app oach combine he new ea u es o he o iginal ea u es, X+= (X,Ex), hen o each o iginal da a se we ha e D+= (X+ ,Y). This app oach is expec ed o gene a e a e y la ge numbe o a iables, many o which may be edundan . Fo ins ance, i he unc ion used o combine a iables is commu a i e (e.g. p oduc ), hen Eo,i,j x= o(xi,xj) = o(xj,xi) = Eo,i,j x, which means ha hese wo new ea u es a e edundan . Thus, we apply a il e ea u e selec ion me hod o elimina e edundan ea u es in Ex[7]: Xx, il = il e (X+)(3.2) 3.3 Explo a o y da a analysis Figu e 3.1 shows he dis ibu ion o he numbe o ea u es in he da a se s, be o e and a e he addi ion o new ea u es. The i s plo shows he dis ibu ion o classes, he majo i y o he da a se s has be ween 2 and 5 classes, bu he e is an ou lie , coun ing 26 classes. In he same igu e, i is possible o obse e he di e ence be ween he ea u es on he o iginal and he ex ended da a se s. The ex ac ion o ea u es does no ake oo much ime. The algo i hm o c ea e new ea u es and e alua e he o iginal and he new da a se o he 100 da a se s, ook 4 hou s. The p ocess gene a ed 132.206 ea u es, which would be impossible o a human o pe o m in such a sho 3.3 Explo a o y da a analysis 19 Figu e 3.2: Analysis o he da a se s wi h highly co ela ed ea u es emo ed. ime. Howe e , a la ge numbe o new ea u es is expec ed. Fo each bina y unc ion, in a da a se wi h m a iables, a combina ion o m, wo-by- wo, will be gene a ed. Fo example, he o iginal hill- alley da a se has 100 nume ic ea u es. The ex ension da a se we gene a ed has 19,900 ea u es in o al. Figu e 3.2 shows he dis ibu ion o he numbe o ea u es in he da a se s, be o e and a e he ex ension a e applying a il e -based ea u e selec ion me hod, as desc ibed in Sec ion 2.2.2. In his case, he o iginal hill- alley da a se has 100 nume ic ea u es, and a e he ex ension and il e ing p ocess, i esul s in 9902 ea u es on he o al. The elimina ion o highly co ela ed ea u es can educe he compu a ional cos o ou model. 3.3.1 Base-le el pe o mance We analysed he base-le el pe o mance on he wo e sions o he ex ended da a se s, wi h and wi hou ea u e selec ion. The da a was classi ied wi h wo op ions 1 and 0, which 1 is when he sys ema ic ea u e gene a ion app oaches ha e a be e pe o mance han he o iginal da a se and 0 when he o iginal da a se has a be e pe o mance han he ex ended da a se . Fo all 100 da a se s analysed we ha e a balanced esul s on bo h e sions. Fo he il e ed s a egy, we see 52 imp o emen s om he o iginal da a se (class 1) and 48, which he o iginal is be e han he ex ended ea u es (class 0). Fo he non- il e ed s a egy, we see 51 imp o emen s om he o iginal da a se (class 1) and 49, which he o iginal is be e han he ex ended ea u es (class 0). We see a sligh di e ence be ween hem. Mo e de ails a e a ailable in Appendix A. Howe e , we can see a mo e signi ican dis inc ion in he di e ence be ween he accu acy o he o iginal da a se and ex ended da a se . In he il e ed da a se , we ha e a maximum im- p o emen on he accu acy o 48.5 pe cen age poin s and a maximum educ ion o 9 poin s. Fo he non- il e ed s a egy, we had a maximum imp o emen on he accu acy o 40.5 poin s and a maximum educ ion o 15.5 poin s. 20 Sys ema ic ea u e gene a ion Figu e 3.3: Explo a o y analysis o he del a be ween he o iginal accu acy s ex ended accu acy. Analysing he a e age o bo h s a egies conside ing he 100 da a se s, he sys ema ic ea u e gene a ion esul s in an o e all imp o emen o 0.97 poin s o he il e ed s a egy and 0.68 poin s o imp o emen o he non- il e ed s a egy. The hi d qua ile shows ha we ob ain gains o 0.83 and 0.57 pe cen age poin s in he on he il e ed and non- il e ed s a egies, espec i ely. In spi e o his, he e a e many da a se s in which he ex ension does no lead o imp o emen s, which means ha , gi en a da a se , i is necessa y o decide whe he i is wo hwhile o gene a e new ea u es o no . We add ess his p oblem wi h me a-lea ning app oaches in he ollowing chap e s. The il e ed ex ended da a se leads us o conclude ha il e ing high co ela ed ea u es lead o a be e model. Also, we can conclude ha he sys ema ic combina ion o ea u es using simple ope a ions can lead us o be e p edic i e models. Chap e 4 Me a-lea ning o selec comple e se s o ea u es In his chap e , we p esen ou me a-lea ning app oach o selec comple e se s o ea u es. In he p e ious chap e (Chap e 3), we obse ed ha many da a se s a e he ex ension did no lead o a be e model, leading us o a new p oblem: Can a me a-lea ning app oach p edic when a me hod o au oma ic gene a ion o ea u es leads o be e p edic i e models? 4.1 P oblem de ini ion The p oblem is o an icipa e whe he he gene a ion o au oma ic ea u es will gene a e any gain in he pe o mance o he model o no . Besides aiming o imp o e accu acy, we can educe he compu a ional esou ces equi ed o lea ning he models. 4.2 App oach We a e going o add ess he p oblem using a me a-lea ning app oach, bu he ypical me a-lea ning app oaches, me a- ea u es cha ac e ise a single da a se . In ou case, we ha e wo da a se s: he o iginal and he ex ended da a se s. Mos impo an ly, we expec ha i is he combined a io in he me a- ea u es be ween he o iginal and ex ended me a-da a ha de e mines whe he he algo i hm will be able o ob ain a be e model on he ex ended one o no . Gi en he se s o da a se s {D1,D2,...,Dm}, he se s o combina ion be ween o iginal da a and new ea u es X+ 1,X+ 2,...,X+ m, he se s o ex ended da a se s D+ 1,D+ 2,...,D+ mand o each p a unc ion m i ha compu es a gi en me a- ea u e (e.g., a e age co ela ion), we apply i o he o iginal da a se , XX i,j=m j(Di)∀i,j, o he ex ended one, XX+ i,j=m j(D+ i)∀i,jand hen we combine using he unc ion, X(X,X+) i,j=T(XX i,j,XX+ i,j). The unc ion used o combine he cha ac e is ics o 21 22 Me a-lea ning o selec comple e se s o ea u es he o iginal and de i ed da a se s should quan i y he di e ences be ween hem, so a a io o a sub ac ion a e sui able unc ions, and in his hesis, we choose he a io. Fo all da a se s, all ele an in o ma ion was eco ded, making i possible o compa e he o iginal and he ex ended da a se s, bo h in e ms o hei cha ac e is ics as well as in e ms o he pe o mance o he algo i hm. The me a-le el class a ibu e holds he inal esul s: 1 i i is wo hwhile o do he au oma ic gene a ion o ea u es o 0 i he algo i hm should be applied o he o iginal da a se . We wan o apply he me a-lea ning app oach o p edic when a me hod o he sys ema ic gene a ion o ea u es leads o be e p edic i e models. 4.3 Expe imen al se up Fo he gene a ion o me a- ea u es, we used he Me a-Fea u e Ex ac o (MFE) [15] R package. I p o ides i e dis inc g oups o me a- ea u es: gene al, s a is ical, in o ma ion- heo e ic, model- based, landma king. Howe e , we do no use all he g oups o a iables o his expe imen , and we ha e some es ic ions in he selec ed g oups. The domain o in o ma ion- heo e ic g oup is only o ca ego ical da a [15], and in ou expe - imen , we used only nume ical da a. This lead us o elimina e he whole g oup. The Landma king g oup measu es he pe o mance o a decision ee o each da a se , in ou case i leads o a sig- ni ican compu a ional cos , and, hus, we decided o elimina e hese me a- ea u es as well o his p ojec . Ne e heless, we belie e hey may be use ul and should be conside ed in u u e ex ensions o his wo k. Addi ionally, as we ha e a easonable a ie y o da a se s, some me a- ea u es gene a ed in- alid esul s. On ha case, me a- ea u es wi h mo e han 10% o no a ailable o no appli- cable we e also elimina ed, namely: gene al.num oca , s a is ical.sd a io, s a is ical.gmean.sd, s a is ical.gmean.mean, s a is ical.wlambda, s a is ical.can co .sd, s a is ical.skewness.sd, s a is i- cal.n co a , s a is ical.ku osis.sd, s a is ical.co .sd, s a is ical.co .mean, s a is ical.skewness.mean, s a is ical.ku osis.mean, s a is ical.hmean.sd. Mos o hese a e based on s anda d de ia ion o mean and we e caused by di isions by ze o, as expec ed [15]. Also, in o de o emo e noise, we applied a d op o highly co ela ed ea u es, pe o med wi h a h eshold o 95% o me a- ea u es o he o iginal da a se . Fo each da a p epa a ion ope a ions, we pe o med wo hypo hesis, he i s was pe o med o he base da a se wi hou il e ing a he base le el, and he second hypo hesis we added he il e men ioned abo e, a he base le el da a se . 4.4 Resul s Ou me a-lea ning app oach can p edic om he me ada a o he o iginal da a se and ex ended ea u es, whe he using he new ea u es p oduces a be e model, as we can see om he model 4.4 Resul s 23 Table 4.1: Pe o mance achie ed by applying he me a-lea ning app oach. No il e Fil e ed A e age (%) sd (+/-) A e age (%) sd (+/-) Base 60.94 (0.21) 61.93 (0.30) Base - add a io 62.14 (0.19) 56.38 (0.31) Base - d op NA 60.83 (0.23) 62.91 (0.33) Base - d op co ela ions 58.85 (0.21) 60.04 (0.34) Base - add a io & d op NA 60.92 (0.19) 62.18 (0.33) Base - add a io & d op co ela ions 55.05 (0.24) 61.09 (0.37) Base - add a io & d op NA & d op co ela ions 57.14 (0.25) 61.20 (0.37) pe o mance in able 4.1. Recalling ha ou baseline is 52%, and we a e using 10- old c oss- alida ion, ou app oach ob ained an accu acy o 62.91% ( +/- 0.33) when he da a se o d op he highly co ela ed ea u es. Fo he able 4.1 we can conside he base da a as he o iginal XX i,jand he ex ended XX+ i,j me ada a on he same da a se . F om his base me a-da a, we pe o med di e en da a p epa a ion ope a ions, emo e all no a ailable gene a ed by he MFE package and emo e highly co ela ed ea u es, and mos impo an ly we added he combina ion a io as he equa ion X(X,X+) i,j. We applied hese p epa a ion ope a ions o he wo a ian s o he me a-da ase s il e and wi hou he il e . Howe e , his p ocess s ill needs he gene a ion o ea u es o compu e he me a- ea u es. This leads o addi ional compu a ional cos . E en i his is signi ican ly smalle han he cos o lea ning he models, we need o in es iga e i success ul ecommenda ions can be ob ained wi hou he gene a ion o he new ea u es. In o he wo ds, i he me a-lea ning p ocess uses only he me ada a o he o iginal da a se o p edic whe he o no o add new ea u es. To p edic when o no o add he new ea u es, we apply se e al app oaches o he o iginal da a se , o be able o emo e any noise ha impac s he pe o mance o he model. The e o e, conside ing he o iginal da a se , we ex ac ed all me a-da a as we saw on he p e ious sec ion XX i,j=m j(Di)∀i,j, including only he gene al, s a is ical and model-based g oups o me a- ea u es. Resul s in able 4.2 show ha i is possible, by using only he me a-da a o he o iginal da a se o p edic when he me hod o au oma ic gene a ion o ea u es leads o be e p edic i e models Table 4.2: Pe o mance achie ed using only he me ada a o he o iginal da a se . No il e Fil e ed A e age (%) sd (+/-) A e age (%) sd (+/-) Base 63.14 (0.26) 59.04 (0.27) Base - d op NA 58.94 (0.23) 61.11 (0.33) Base - d op co ela ions 66.96 (0.25) 62.95 (0.24) Base - d op NA and co ela ions 65.05 (0.25) 61.31 (0.31) 30 Me a-lea ning o selec indi idual ea u es se s o ea u es bu a single ea u e and we ha e collec ed he pe o mance o each ea u e o each da a se , so we decided o use only he ea u es ha supposedly lead o an imp o emen in pe o mance. This app oach needs u he esea ch o add ess his issue. Un o una ely, using only he ea u es ha ou app oach sugges s, he a e age accu acy d opped by 0.72 pe cen age poin s compa ed o applying all gene a ed ea u es. Fu he mo e, he same me hod s ill d opping on an a e age o 0.043 pe cen age poin s compa ed o he o iginal da a se . See igu e 5.1 ha illus a es he accu acy di e ence o he wo app oaches. Acco ding o able 5.2, he selec ed ea u es had a maximum loss o 24 pe cen age poin s and a maximum gain o 10.4 pe cen age poin s compa ed o he da a se wi h all he ea u es c ea ed. Howe e , compa ing he selec ed esul s wi h he o iginal da a se esul s, we ha e a maximum loss o 7.3 pe cen age poin s, while we ha e a maximum gain o 36.7 pe cen age poin s. Again, ou bigges imp o emen was in he Hill- alley da a se eaching an accu acy o 93.48%, losing -3.79 pe cen age poin s when compa ed o all gene a ed ea u es, which has an accu acy o 97.27%, and i compa ed o he o iginal da a se i is a gain o 36.7 pe cen age poin s. See de ailed esul s in Appendix A. These esul s indica e ha e en wi h he wo se esul compa ed o using all he ea u es, and we can some imes imp o e he esul s o he o iginal da a se and he ex ended da a se . We do ha e a highe a e age loss wi h his app oach, bu i shows us ha he app oach can be imp o ed and maybe we can o e come some esul s. The o e all esul s indica e ha e en we collec ed p omising esul s on he me a-le el, i leads us o a wo se esul on he base-le el. This is p elimi- na y wo k and se e al challenges we e add essed wi h simple solu ions ha could cause some loss o pe o mance, as he selec ion o he inal se o ea u es and da a cha ac e isa ion. Howe e , i shows us ha he app oach can be imp o ed and i needs u he in es iga ions. 5.4 Resul s 31 Figu e 5.1: Compa ison o he accu acy om selec ed ea u es, comple e se o ea u es and o iginal ea u es 32 Me a-lea ning o selec indi idual ea u es Chap e 6 Conclusions and Fu u e Wo k P epa ing a da a se co ec ly usually akes a signi ican amoun o ime and i equi es subs an ial knowledge abou he domain. An essen ial ask on his p ocess is ea u e enginee ing, which consis s o gene a ing new ea u es om he exis ing ones. This is one o he mos complex and ime-consuming asks oday. The e is an inc ease o esea ch on ea u e enginee ing o e he las yea s, wi h a g owing in e es in au oma ed machine lea ning. Ou esul s show ha au oma ed app oaches can ex ac many ea u es om he aw da a se independen ly o he p oblem and domain knowledge, o en leading o mo e accu a e models. P e ious esea ch has ocused on applying di e en ope a ions on he aw da a se . In his hesis, we p opose a me a-lea ning app oach o au oma e he p ocess o ea u e enginee ing. Thus gi en a da a se we sys ema ically gene a e a se o new ea u es, and as i can gene a e a as numbe o ea u es we use a il e o d op he highly co ela ed ea u es using he il e co ela ion me hod. Ou app oach uses me a- ea u es ha desc ibe he ex ension and he o iginal da a se s, and a combina ion o bo h o quan i y he di e ences be ween hem. In his p ojec , we used a ios o ha pu pose. This app oach was used o answe : "Can a me a-lea ning app oach p edic when a me hod o he sys ema ic gene a ion o ea u es leads o be e p edic i e models?". The empi ical s udy ca ied ou ocuses only on nume ical a iables. Howe e , he app oach can be applied o a iables o any na u e. We de eloped ano he me a-lea ning app oach o he p oblem o p edic ing whe he a new ea- u e will lead o mo e accu a e models. This p oblem has he addi ional challenge o cha ac e ising a single a iable, while ypical me a- ea u es cha ac e ise da a se s. We ex ac ed he in o ma ion o each new ea u e and he ex ension da a se wi hou he speci ic ea u e, also combining hem o collec he di e ence be ween he da a se s. This app oach was used o answe ou ques ion "Can a me a-lea ning app oach p edic when an indi idual ea u e leads o a be e p edic i e model?" 33 34 Conclusions and Fu u e Wo k 6.1 Resul s This hesis p oposed o iden i y he e ec s o au oma ed ea u e gene a ion using me a-lea ning as he app oach. Based on empi ical esul s, and we can assume ha a sys ema ic combina ion o ea u es using simple ope a ions can lead o be e p edic i e models. Applying a simple il e o highly co ela ed ea u es i is possible o imp o e he model pe o mance. Addi ionally, he empi ical esul s show ha he me a-lea ning app oach can p edic when a me hod o he sys ema ic gene a ion o ea u es leads o be e p edic i e models. The esul s also indica e ha he me a-lea ning app oach can p edic when a me hod o au oma ic gene a ion o ea u es leads o be e p edic i e models wi hou applying i , using only he o iginal da a se wi hou he me ada a o he new ea u es. On he o he hand, we did no ge sa is ac o y esul s on he p oblem o p edic ing whe he a ea u e will lead o be e models. E en hough he esul s a he me a-le el we e p omising, he selec ed ea u es did no lead o be e base-le el models. Howe e , his is a oo complex p oblem and we decided o simpli y, causing some limi a ions o he p oblem. Limi a ions such as he cha ac e isa ion o he da a and he compu a ional cos o gene a e a me a-da a o each ea u e. In addi ion, he esul s indica e ha he e is mo e esea ch o be done o he selec ion o ea u es using he me a-lea ning app oach, applying di e en ech- niques o he same app oach. 6.2 Fu u e Wo k Fo u u e wo k, we see he a emp o elimina e some limi a ions in he cu en wo k. Fo he selec ion o indi idual ea u es, we would like o add ess a mo e ealis ic app oach: ins ead o emo ing only a single ea u e, emo e a andom ac ion o ea u es om he da a se , epea ing he p ocess many imes. Thus, a oiding he es ic ion, we had o gene a e me ada a o a single ea u e, as in es iga e i i is possible o build new me a- ea u es o an indi idual ea u e o a new app oach o he selec ion o ea u es. New app oaches should also seek o mi iga e ime consump ion in me ada a gene a ion, mak- ing his app oach mo e p ac ical. Mo eo e , i would be good o implemen o he ea u e gen- e a ion ans o ma ions such as una y and highe -o de ans o ma ions. Addi ionally, we could combine he ea u e gene a ion ools wi h he me a-lea ning app oach ci ed he e. Du ing he p ocess o analysis o his wo k, i was possible o see he impac ha a single da a se gene a es on he inal esul , so an ob ious s ep would be o add new da a se s, mainly da a se s wi h a la ge olume. He e, we only used he Random Fo es algo i hm. Howe e , he e a e many algo i hms ha we can use o his p oblem and should be conside ed o make a compa ison in u u e wo k. Appendix A Base le el accu acy The ables A.1,A.2,A.3 a e ela ed wi h he o iginal and ex ended ea u es wi hou he il e o highly co ela ed ea u es. The ables A.4,A.5,A.6 is abou he o iginal and ex ended ea u es wi h he il e o highly co ela ed ea u es. The ables A.7,A.8,A.6 is abou he accu acy o each da a se using only he indi idual ea u es ha supposedly would lead a be e pe o mance o he model. 35 36 Base le el accu acy Table A.1: Accu acy o each da a se es ed wi hou il e on base-le el. Da a se s Ex ended O iginal Ex ended s O iginal Class abalone 63.97 63.44 0.53 1 aids 63.33 60 3.33 1 ai lines 59.94 60 -0.06 0 analca da a_c edi sco e 98.89 98.89 0 0 au ho ship 98.1 99.17 -1.07 0 au oMpg 87.92 88.67 -0.75 0 au oUni -au7-700 48.61 48.99 -0.38 0 backache 86.28 86.75 -0.47 0 blood- ans usion-se ice-cen e 68.06 66.6 1.46 1 bond a e 67.81 63.57 4.24 1 b eas -w 96.01 96.29 -0.28 0 chu n 96.28 95.74 0.54 1 cmc 52.34 51.79 0.55 1 c edi -g 69.9 70.8 -0.9 0 cyyoung 82.97 82.75 0.22 1 diabe es 76.43 75.78 0.65 1 diabe es130US 35.59 41.25 -5.66 0 ecoli 86.7 87.57 -0.87 0 eeg-eye-s a e 58.64 59.21 -0.57 0 elec ici y 69.31 69.41 -0.1 0 glass 74.61 82.13 -7.52 0 habe man 67.38 64.78 2.6 1 hea -s a log 83.33 84.07 -0.74 0 hepa i is 75.03 75.03 0 0 houses 94.67 94.28 0.39 1 kddcup09_upselling 92.44 92.47 -0.03 0 le e 97.28 96.57 0.71 1 mammog aphy 98.83 98.75 0.08 1 mo phological 99.85 99.85 0 0 page-blocks 96.89 96.75 0.14 1 phoneme 91.17 91.36 -0.19 0 p nn_c abs 99.5 80 19.5 1 p o b 63.4 59.67 3.73 1 m sa_sleepda a 36.76 36.25 0.51 1 sa elli e 99.33 99.29 0.04 1 sa image 91.66 91.85 -0.19 0 sona 69.95 73.72 -3.77 0 ae 81.14 80.43 0.71 1 eachingassis an 63.99 65.2 -1.21 0 i anic 64.4 64.17 0.23 1 ehicle 78.25 75.99 2.26 1 innie 80.47 80.47 0 0 olcanoes 96.37 96.44 -0.07 0 wall- obo -na iga ion 99.58 99.76 -0.18 0 wholesale-cus ome s 92.26 91.82 0.44 1 wine 96.14 96.73 -0.59 0 yeas 58.85 59.05 -0.2 0 Base le el accu acy 37 Table A.2: Accu acy o each da a se es ed wi hou il e on base-le el [con inui y]. Da a se s Ex ended O iginal Ex ended s O iginal Class allbp 97.77 97.67 0.1 1 hy oid-ann 99.79 99.58 0.21 1 au os 69.15 67.83 1.32 1 op digi s 97.68 97.69 -0.01 0 engine 81.44 86.2 -4.76 0 calenda dow 58.88 58.11 0.77 1 led-display-domain 71.51 71.77 -0.26 0 sma phone-based _ ecogni ion_o _ human_ac i i ies 96.67 97.22 -0.55 0 s eel-pla es- aul 97.16 91.52 5.64 1 olcanoes-d4 94.04 94.2 -0.16 0 BNG_b eas -w 98.61 98.61 0 0 BNG_cmc 52.87 52.02 0.85 1 qsa -biodeg 85.4 84.64 0.76 1 kc2 80.19 80.79 -0.6 0 ozone-le el-8h 93.45 92.66 0.79 1 hill- alley 97.27 56.78 40.49 1 wdbc 96.85 96.15 0.7 1 clima e-model -simula ion-c ashes 91.12 91.31 -0.19 0 spambase 93.59 94.09 -0.5 0 ilpd 68.24 68.26 -0.02 0 kc1 83.46 83.13 0.33 1 pc1 93.51 93.7 -0.19 0 pc3 90.08 89.95 0.13 1 pc4 91.29 91.02 0.27 1 bankno e -au hen ica ion 99.85 99.34 0.51 1 mozilla4 84.89 84.55 0.34 1 pendigi s 99.37 99.07 0.3 1 balance-scale 84.5 68.42 16.08 1 ca dio ocog aphy 100 100 0 0 i s -o de - heo em-p o ing 57.71 57.66 0.05 1 wil 98.39 98.16 0.23 1 hy oid-allhype 70.29 70.68 -0.39 0 hy oid-allbp 70.29 70.68 -0.39 0 hy oid-allhypo 70.29 70.68 -0.39 0 hy oid-all ep 70.29 70.68 -0.39 0 hy oid-dis 70.29 70.68 -0.39 0 sa elli e_image 89.9 89.68 0.22 1 seismic-bumps 91.26 92.19 -0.93 0 segmen 98.1 97.88 0.22 1 38 Base le el accu acy Table A.3: Accu acy o each da a se es ed wi hou il e on base-le el [con inui y]. Da a se s Ex ended O iginal Ex ended s O iginal Class b eas - issue 51.01 66.48 -15.47 0 c edi -app o al 75.12 73.94 1.18 1 banana 89 89.68 -0.68 0 pc1_ eq 67.46 66.49 0.97 1 pa kinsons 84.06 83.59 0.47 1 i is 96 96.67 -0.67 0 jungle_chess_2pcs_ aw_endgame _comple e 71.61 68.22 3.39 1 oil_spill 95.19 95.83 -0.64 0 ho acic-su ge y 83.62 84.04 -0.42 0 jm1 79.18 79.05 0.13 1 JapaneseVowels 93.23 92.52 0.71 1 eucalyp us 22.13 20.79 1.34 1 all ep 98.54 97.8 0.74 1 use -knowledge 90.73 90.2 0.53 1 Base le el accu acy 39 Table A.4: Accu acy o each da a se es ed wi h il e on base-le el. Da a se s Ex ended Fil e ed O iginal Fil e ed Ex ended s O iginal Class abalone 64.35 54.66 9.69 1 aids 60.83 60 0.83 1 ai lines 59.81 60 -0.19 0 analca da a_c edi sco e 98.89 98.89 0 0 au ho ship 98.81 99.17 -0.36 0 au oMpg 89.71 86.64 3.07 1 au oUni -au7-700 46.29 49.28 -2.99 0 backache 85.63 86.75 -1.12 0 blood- ans usion -se ice-cen e 67.13 67.53 -0.4 0 bond a e 64.38 63.57 0.81 1 b eas -w 95.87 96.29 -0.42 0 chu n 95.84 94.96 0.88 1 cmc 52.54 51.79 0.75 1 c edi -g 69.9 70.8 -0.9 0 cyyoung 83.72 84.97 -1.25 0 diabe es 76.3 75.78 0.52 1 diabe es130US 44.88 41.25 3.63 1 ecoli 86.93 87.89 -0.96 0 eeg-eye-s a e 57.03 54.57 2.46 1 elec ici y 68.92 69.41 -0.49 0 glass 74.47 82.13 -7.66 0 habe man 69.36 64.78 4.58 1 hea -s a log 82.22 84.07 -1.85 0 hepa i is 75.03 75.03 0 0 houses 94.63 94.26 0.37 1 kddcup09_upselling 92.46 92.47 -0.01 0 le e 97.24 96.57 0.67 1 mammog aphy 98.84 98.75 0.09 1 mo phological 99.85 99.75 0.1 1 page-blocks 97.09 96.77 0.32 1 phoneme 91.52 91.36 0.16 1 p nn_c abs 99.5 75.5 24 1 p o b 62.8 59.67 3.13 1 m sa_sleepda a 36.25 36.25 0 0 sa elli e 99.45 99.29 0.16 1 sa image 90.96 90.08 0.88 1 sona 70.11 73.72 -3.61 0 ae 79.1 80.43 -1.33 0 eachingassis an 60.66 65.2 -4.54 0 i anic 64.55 64.17 0.38 1 ehicle 78.61 75.53 3.08 1 innie 80.98 80.47 0.51 1 olcanoes 96.34 96.44 -0.1 0 46 REFERENCES [12] Hugh Lea he , Edwin Bonilla, and Michael O’Boyle. Au oma ic ea u e gene a ion o ma- chine lea ning based op imizing compila ion. In P oceedings o he 7 h annual IEEE/ACM In e na ional Symposium on Code Gene a ion and Op imiza ion, pages 81–91. IEEE Com- pu e Socie y, 2009. [13] Ca los Soa es Rica do Vilal a Pa el B azdil, Ch is ophe Gi aud-Ca ie . Me alea ning: Ap- plica ions o Da a Mining. Sp inge Be lin Heidelbe g, 2009. [14] Randy Ke be Thomas Khabaza Thomas Reina z Colin Shea e Pe e Chapman, Julian Clin- on and Rüdige Wi h. CRISP-DM 1.0 - S ep-by-s ep da a mining guide. CRISP-DM Con- so ium, Fi s edi ion, 2009. [15] Ad iano Ri olli, Luís PF Ga cia, Ca los Soa es, Joaquin Vanscho en, and And é CPLF de Ca alho. Towa ds ep oducible empi ical esea ch in me a-lea ning. a Xi p ep in a Xi :1808.10406, 2018. [16] Ros isla S íz. Me alea ning o da a mining and kdd. pages 1–21, 2013. [17] Rica do Vilal a, Ch is ophe G Gi aud-Ca ie , Pa el B azdil, and Ca los Soa es. Using me a- lea ning o suppo da a mining. IJCSA, 1(1):31–45, 2004. [18] Mlle Bouaguel Waad. On Fea u e Selec ion Me hods o C edi Sco ing. PhD hesis, Uni- e si é de Tunis - Ins i u Supé ieu de Ges ion, 2015.