Tackling Ant Colony Optimization Meta-Heuristic as Search Method in Feature Subset Selection Based on Correlation or Consistency Measures

Tackling An Colony Op imiza ion Me a-Heu is ic as Sea ch Me hod in Fea u e Subse Selec ion Based on Co ela ion o Consis ency Measu es An onio J. Tall´on-Balles e os and Jos´e C. Riquelme Depa men o Languages and Compu e Sys ems, Uni e si y o Se ille, Spain [email p o ec ed] Abs ac . This pape in oduces he use o an an colony op imiza ion (ACO) algo i hm, called An Sys em, as a sea ch me hod in wo well- known ea u e subse selec ion me hods based on co ela ion o consis- ency measu es such as CFS (Co ela ion-based Fea u e Selec ion) and CNS (Consis ency-based Fea u e Selec ion). ACO guides he sea ch using a heu is ic e alua o . Empi ical esul s on wel e eal-wo ld classi ica ion p oblems a e epo ed. S a is ical es s ha e e ealed ha In oGain is a e y sui able heu is ic o CFS o CNS ea u e subse selec ion me hods wi h ACO ac ing as sea ch me hod. The use o In oGain is shown o be he signi ican ly be e heu is ic o e a ange o classi ie s. The esul s achie ed by means o ACO-based ea u e subse selec ion wi h he sui - able heu is ic e alua o a e be e o mos o he p oblems compa ing wi h hose ob ained wi h CFS o CNS combined wi h Bes Fi s sea ch. Keywo ds: Fea u e selec ion, classi ica ion, an colony op imiza ion, heu is ic e alua o , il e , ea u e subse selec ion. 1 In oduc ion An colony op imiza ion (ACO) me a-heu is ic [6] was p oposed by Do igo e al. and is inspi ed in he beha iou o eal an colonies. Depending on he amoun o phe omone deposi ed by he an in hei walk he e would be some poin s ha a e mo e likely o be isi ed by he nex an s [4]. The (a ificial) an s in ACO define a andomized cons uc ion heu is ic which makes p obabilis ic decisions depending on he s eng h o a ificial phe omone ails and a ailable heu is ic in o ma ion. As such, ACO can be in e p e ed as an ex ension o adi ional cons uc ion heu is ics, which a e eadily a ailable o many combina o ial op- imiza ion p oblems. Ye , an impo an diffe ence wi h cons uc ion heu is ics is he adap a ion o he phe omone ails du ing algo i hm execu ion o ake in o accoun he cumula ed sea ch expe ience. As cons uc ion algo i hms wo k on pa ial solu ions ying o ex end hese in he bes possible way o comple e p oblem solu ions. In essence, ea u e selec ion (FS) is a NP-ha d combina o ial op imiza ion p oblem. O en, FS is ackled wi h classical sea ch me hods o a oiding he p ohibi i e exhaus i e sea ch. Ins ead o looking o he op imal solu ion, ob- aining a good solu ion in a easonable ime migh be p e e able o ce ain p oblems. On one hand, he ypical NP-ha d T a elling Salesman P oblem has been ea ed success ully wi h ACO. On he o he hand, ACO was applied o ea u e selec ion in [9] in he con ex o ough se educ s wi h e y p omising esul s. He e, we apply ACO as a s ochas ic p ocedu e o quickly finding high quali y solu ions, in he scope o o dina y o c isp se s o FS in classifica ion asks. In his con ex , ea u e subse selec ion (FSS) p oblem is o mula ed using a g aph wi h he pu pose o ge ing a subse o a ibu es ha is ele an o he p oblem a hand. FSS needs a sea ch me hod, ha usually is any kind o a ificial in elligence heu is ic echnique. The cu en p oposal shi s he sea ch om an heu is ic non-s ochas ic pe spec i e o a s ochas ic angle. This pape goals o add ess he sui abili y o using he an colony op imiza ion me a-heu is ic as a sea ch me hod buil -in in he CFS and CNS ea u e subse selec o s o classifica ion p oblems. The es o his a icle is o ganized as ollows: Sec . 2 desc ibes some concep s abou ACO me a-heu is ic and ea u e selec ion; Sec . 3 p esen s ou p oposal; Sec . 4 de ails he expe imen a ion; hen Sec . 5 shows and analyzes s a is ically he esul s ob ained; finally, Sec . 6 s a es he concluding ema ks. 2 Backg ound 2.1 An Colony Op imiza ion Me a-Heu is ic A ificial an s used in ACO a e s ochas ic solu ion cons uc ion p ocedu es ha p obabilis ically build a solu ion by i e a i ely adding solu ion componen s o pa ial solu ions by aking in o accoun a) heu is ic in o ma ion abou he p ob- lem ins ance being sol ed, i a ailable, and b) (a ificial) phe omone ails which change dynamically a un ime o eflec he agen s’ acqui ed sea ch expe ience. The p oblem ep esen a ion is a g aph whe e he nodes ep esen he diffe en poin s ha he an s can isi and he edges a e he link be ween poin s. Links a e unidi ec ional and he e a e no cycles, so i is no possible o go back o a poin p e iously isi ed. A he beginning o he algo i hm e e y an is loca ed in a poin and will cons uc a solu ion aking se e al decisions un il he s op condi ion is me . A he end each an has ound a candida e solu ion o he p oblem a hand. The main s eps o he algo i hm a e he ollowing: 1. Ini ialisa ion. The algo i hm s a s and all he phe omone a iables a e ini- ialized o a alue τ0which is a key pa ame e . 2. Cons uc An Solu ions. This ac ion s a s he algo i hm loop and is e- la ed wi h sending an s a ound he cons uc ion g aph. An an in he node ichooses he jone acco ding o a p obabilis ic decision ule, which is a unc ion o he phe omone τand he heu is ic η.Ase o naan s cons uc s solu ions o he conc e e p oblem being ackled. 3. Upda e Phe omones. The pu pose o his pa is o change he alues o he phe omones, by bo h deposi ing and e apo a ing. Among he se e al a ian s o ACO, he eina e we ocus on An Sys em (AS) ha was in oduced in 1991 by Do igo and published a ew yea s la e by Do igo e al. [5]. 2.2 Fea u e Selec ion Fea u e selec ion me hods y o pick a subse o ea u es ha a e ele an o he a ge concep [2]. Acco ding o Langley [10], he diffe en app oaches o ea u e selec ion can be di ided in o wo b oad ca ego ies (i.e., fil e and w ap- pe ) based on hei dependence on he induc i e algo i hm ha will finally use he selec ed subse . Fil e me hods a e independen o he induc i e algo i hm, whe eas w appe me hods use he induc i e algo i hm as he e alua ion unc ion. FS in ol es wo s ages: a) o ob ain a lis o a ibu es acco ding o an a - ibu e e alua o and b) o pe o m a sea ch on he ini ial lis . All candida e lis s would be assessed using a measu e e alua ion and he bes one will be e u ned. Two o he mos widesp ead ea u e subse selec o s a e Co ela ion-based Fea u e Selec ion (CFS) [8] and Consis ency-based ea u e selec ion (CNS) [3] ha wo k in combina ion wi h a sea ch me hod such as G eedy Sea ch, Bes Fi s (BF) o Exhaus i e Sea ch. Gene ally speaking, BF is a powe ul sea ch me hod [7] which is he eason o be used e y equen ly by he machine lea n- ing communi y nowadays. We ha e chosen CFS and CNS as ep esen a i e FSS me hods, because hey a e based on diffe en kind o measu es, ha e ew pa- ame e s and ha e p o ided a good pe o mance inside he supe ised machine lea ning a ea. O en, BF sea ch is he p e e able op ion by he esea che s o bo h FSS algo i hms. CFS is p obably he mos used FSS in da a mining. CNS is also powe ul, howe e he amoun o published wo ks is mo e educed. 3P oposal Fi s ly, he g aph meaning may be e o mula ed in o de o deal wi h a ea u e selec ion p oblem by means o ACO me a-heu is ic. He e, he nodes ep esen ea u es and edges he link be ween nodes and he possibili y o add ano he ea u e o he cu en solu ion. The sea ch o a candida e solu ion is a walk h ough he g aph. Once an an isi s an edge i con ains a weigh indica ing he s eng h o his solu ion componen . In he cu en wo k, ACO, implemen ed ollowing he AS model, is conside ed as sea ch s a egy in he con ex o CFS and CNS me hods a e he a ibu e e alua ion phase. ACO guides he sea ch by means o a heu is ic e alua o . As heu is ic e alua o s, on one hand we ha e conside ed o CFS and CNS app oaches he own a ibu e e alua o , ob aining he pu e e sions o CFS-AS and CNS-AS. On he o he hand, we ha e ied In o ma ion Gain (In oGain as abb e ia ion) [1] as e alua o esul ing an hyb id app oach. Mo eo e , CFS, CNS and In oGain compu e diffe en kinds o measu e o e alua e he ele ance, such as co ela ion, consis ency and in o ma ion, espec i ely. The p obabilis ic ansi ion ule is defined in he same way as in [9] and is hemos widelyusedinAS[5]: pij =τα ij ·[ηij ]β il∈N (x)τα il ·[ηil]β,∀ij ∈N(x).(1) whe e pij ep esen s he p obabili y ha cu en an a ea u e iwould a el o ea u e j,τij is he amoun o phe omone on he ij edge, ηij is he heu is ic desi abili y o he ij ansi ion and N(x) he se o cu en easible componen s. Las ly, αand βa e pa ame e s ha may ake eal posi i e alues –acco ding o he ecommenda ions on pa ame e se ing in [5]– and a e associa ed wi h heu is ic in o ma ion and phe omone ails, espec i ely. All an s upda e phe omone le el wi h an inc ease o small quan i ies, depend- ing di ec ly on he heu is ic desi abili y o he ij ansi ion gi en by he measu e (me i ) o he subse a ibu e e alua o used as heu is ic e alua o and in e sely p opo ional o he subse size. 4 Expe imen a ion Table 1 depic s he ea u e subse selec ion me hods applied in he expe imen al p ocess. We ha e g ouped hem acco ding o he a ibu e e alua o and sea ch me hod. Table 2 summa izes he main pa ame e s along wi h hei symbols and nu- me ical o concep ual alues o all he ea u e subse selec ion me hods used o he expe imen s. On one hand, in ela ion o ACO-based ea u e subse se- lec ion, he naand gen pa ame e s ha e been se o fix alues o ou choice. Fo τ0pa ame e , in [5] he e is a sugges ion o assign a small posi i e cons an and hence we ha e defined a alue o 0.5. The ade-off be ween αand βpa ame e s may influence in he beha iou o he algo i hm hus o hei de e mina ion a p elimina y expe imen al design by means o a fi e- old c oss alida ion on he aining se has been ca ied ou wi h a couple o alues o each one pa ame e (1 and 2). On he o he hand, o BF-based sea ch me hod in he con ex o CFS and CNS, we ha e ollowed he ecommenda ions o he au ho s ([8] and [3] o he numbe o expanded nodes; he sea ch di ec ion has been fix acco ding o ou p e ious expe iences). Table 3 ep esen s he da a se s employed h oughou he expe imen a ion. They come mos ly om bina y and mul i-class classifica ion eal-wo ld p ob- lems (Cl. column specifies he numbe o classes) aken om he public UCI eposi o y. The numbe o ins ances anges om mo e han one hund ed o ap- p oxima ely ou een housands, hus p oblems wi h a medium size, and he dimensionali y a ies be ween nine een and one hund ed and wen y nine. Also, we ha e included he numbe o selec ed ea u es o e e y ea u e subse selec- o ob ained in he aining se . Las ow shows he dimensionali y educ ion (highe is be e ) in mean o e he o iginal da a se s o each fil e . Table 1. Lis o ea u e subse selec o s o he expe imen a ion A ibu e e alua o Sea ch me hod Heu is ic e alua o Abb. Name CFS An Sea ch CFS CFS −AS h1 In oGain CFS −AS h2 Bes Fi s −CFS −BF CNS An Sea ch CNS CNS −AS h1 In oGain CNS −AS h2 Bes Fi s −CNS −BF Table 2. Pa ame e alues o ACO-based ea u e subse selec ion app oaches (CFS- AS and CNS-AS) and Bes Fi s -based ones (CFS-BF and CNS-BF) Sea ch me hod Pa ame e Symbol Value An sea ch Numbe o an s na10 Numbe o gene a ions gen 10 P he omone ail in luence α 1 Heu is ic in o macion alue β 2 P he omone ini ial alue τ00.5 Bes F i s Consecu i e expanded nodes wi hou imp o ing 5 Sea ch di ec ion F o wa d Table 3. Summa y o he da a se s used and selec ed ea u es o each ea u e subse selec o Da a se Size T ain T es F ea . Cl. Selec ed ea u es CFS CNS AS BF AS BF h1h2h1h2 ba ch(gas) 13910 10432 3478 129 6 7 13 20 7 10 6 ca dio oc. 2126 1595 531 22 10 12 12 12 15 15 13 hepa i is 155 117 38 19 2 10 9 10 15 11 11 ionosphe e 351 263 88 33 2 9 10 11 9 9 9 lib as 360 270 90 90 15 8 34 23 47 55 20 lymph. 148 111 37 38 4 11 8 12 19 13 10 p omo e 106 80 26 582101010158 8 sa image 6435 4435 2000 36 6 20 21 23 13 13 12 sona 2081041046024781499 soybean 683 511 172 82 19 22 22 25 39 58 16 SPECTF 267 80 187 44 2 10 8 12 12 10 8 wa e o m 5000 3750 1250 40 3 12 14 14 14 13 12 A e ages 2479.08 1812.33 666.75 54.25 6.08 11.25 14.00 15.00 18.25 18.67 11.17 Dim. educ ion 79.26 74.19 72.35 66.36 65.59 79.42 The expe imen al design ollows a s a ified hold-ou c oss alida ion wi h h ee and one qua e s o he aining and es se s, espec i ely. Some imes, hese p opo ions do no ma ch since he o iginal da a a e p ea anged. Fo he s a is ical analysis be ween wo ea u e subse selec ion me hods we ha e ca ied a Wilcoxon signed- anks es wi h he es accu acy esul s ob ained. 5Resul s Tables 4 and 5 epo he accu acy es esul s o he FSS based on ACO wi h CFSandCNS- ha is,CFS-ASandCNS-AS, espec i ely- wi h wo diffe en heu is ic e alua o s, hei diffe ence (Di .) and i s anking (R.). Ten execu ions wi h diffe en seeds we e un and he mos equen solu ion was conside ed o he assessmen . We ha e ca ied ou expe imen s wi h h ee kind o de e minis- ic classifie s such: a) C4.5, based on decision ees, b) SVM, ounded in suppo ec o s, and c) PART, a ule-based app oach. The eason o he choice o hese classifie s is mo i a ed by he ac ha hei o e all pe o mance is good in he ea u e selec ion scope [11]. The bes esul s, excluding ies, o each pai (clas- sifie , FSS me hod) ha e been highligh ed wi h bold ace. Acco ding o Wilcoxon signed- anks es , since he e a e 12 da a se s, he T alue a α=0.05 should be less o equal han 14 ( he c i ical alue) o ejec he null hypo hesis. On one hand, in ela ion o CFS-AS, o classifie s C4.5 and SVM he h2 heu is ic e alua o is significan ly be e han h1. On he o he hand, o CNS-AS he pe o mance o h2 wi h PART is he significan bes op ion. Table 4. CFS-AS: Accu acy es esul s and s a is ical es s Da a se C4.5SVM PART CFS −AS CF S −AS CF S −AS h1 h2 Di . R. h1 h2 Di . R. h1 h2 Di . R. ba ch(gas)93.99 96.87 2.88 8 72.28 78.21 5.92 9 93.73 96.35 2.62 7 ca dio oc. 61.58 61.58 0.00 2 67.98 67.80 −0.19 2 60.45 62.52 2.07 5 hepa i is 84.21 89.47 5.26 10 86.84 89.47 2.63 6 84.21 86.84 2.63 8 ionosphe e 90.91 87.50 −3.41 9 82.95 89.77 6.82 10 90.91 93.18 2.27 6 lib as 52.22 65.56 13.33 12 50.00 63.33 13.33 12 54.44 65.56 11.11 12 lymph. 81.08 81.08 0.00 2 81.08 83.78 2.70 8 72.97 70.27 −2.70 9 p omo e 73.08 73.08 0.00 2 73.08 73.08 0.00 1 80.77 80.77 0.00 1 sa image 86.05 86.25 0.20 5 83.80 84.50 0.70 3 85.00 83.55 −1.45 2 sona 67.31 74.04 6.73 11 67.31 75.00 7.69 11 68.27 75.96 7.69 11 soybean 88.95 91.28 2.33 6 94.19 95.35 1.16 5 90.70 92.44 1.74 3 SPECTF 66.84 69.52 2.67 7 66.31 63.64 −2.67 7 70.59 76.47 5.88 10 wa e o m 74.32 74.40 0.08 4 85.92 86.88 0.96 4 78.88 77.04 −1.84 4 T=min{66,12}=12 T=min{68.5,9.5}=9.5T=min{62.5,15.5}=15.5 Table 6 ou lines he global accu acy es esul s o he bes CFS-AS and CNS- AS app oaches e sus based-BF CFS o CNS. The bes esul s, in each da a se , a e ma ked in bold. We can asse he ollowing s a emen s in ela ion o he Table 5. CNS-AS: Accu acy es esul s and s a is ical es s Da a se C4.5SVMPART CNS −AS CNS −AS CNS −AS h1 h2 Di . R. h1 h2 Di . R. h1 h2 Di . R. ba ch(gas)96.00 95.54 −0.46 3 64.49 69.15 4.66 10 95.46 95.46 0.00 1.5 ca dio oc. 63.84 66.48 2.64 6 61.39 64.41 3.01 9 58.76 64.22 5.46 8 hepa i is 84.21 86.84 2.63 5 86.84 84.21 −2.63 8 76.32 84.21 7.89 10 ionosphe e 88.64 93.18 4.55 12 81.82 84.09 2.27 7 89.77 95.45 5.68 9 lib as 56.67 61.11 4.44 11 67.78 70.00 2.22 6 55.56 64.44 8.89 11 lymph. 78.38 81.08 2.70 8 91.89 83.78 −8.11 11 78.38 64.86 −13.51 12 p omo e 84.62 80.77 −3.85 10 73.08 84.62 11.54 12 76.92 80.77 3.85 5 sa image 84.70 84.75 0.05 2 84.50 83.70 −0.80 4 85.50 86.25 0.75 4 sona 75.96 73.08 −2.88 9 75.96 75.00 −0.96 5 72.12 75.96 3.85 6 soybean 91.86 91.86 0.00 1 95.35 94.77 −0.58 3 92.44 92.44 0.00 1.5 SPECTF 69.52 66.84 −2.67 7 64.71 64.71 0.00 1 65.24 69.52 4.28 7 wa e o m 76.00 74.88 −1.12 4 85.12 85.36 0.24 2 77.36 77.84 0.48 3 T=min{44.5,33.5}=33.5T=min{51.5,26.5}=26.5T=min{64.5,13.5}=13.5 Table 6. Global accu acy es esul s o CFS-BF and CNS-BF e sus CFS-AS and CNS-AS Da a se C4.5SVMPART CFS −BF CFS −AS CF S −BF CFS −AS CNS −BF CNS −AS h2 h2 h2 ba ch(gas senso )95.92 96.87 83.04 78.21 98.30 95.46 ca dio oc. 61.58 61.58 67.80 67.80 61.02 64.22 hepa i is 84.21 89.47 86.84 89.47 84.21 84.21 ionosphe e 92.05 87.50 88.64 89.77 88.64 95.45 lib as 61.11 65.56 57.78 63.33 55.56 64.44 lymph. 81.08 81.08 81.08 83.78 67.57 64.86 p omo e 73.08 73.08 73.08 73.08 69.23 80.77 sa image 85.60 86.25 83.85 84.50 85.45 86.25 sona 73.08 74.04 75.00 75.00 75.96 75.96 soybean 93.02 91.28 94.77 95.35 93.02 92.44 SPECTF 66.84 69.52 73.26 63.64 66.31 69.52 wa e o m 74.40 74.40 86.88 86.88 76.16 77.84 Wins by pai s 262637 Global wins 033524 A e ages Accu acy 78.50 79.22 79.33 79.23 76.79 79.29 Selec ed ea .(%) 27.65 25.81 27.65 25.81 20.58 34.41 achie ed esul s. Fi s , he compa ison be ween pai s o FSS me hods o each classifie and da a se s poin s ou ha : a) C4.5 wi h CFS-AS ge s be e esul s 6 imes, b) SVM wi h CFS-AS wins in 6 p oblems, and c) PART classifie wi h CNS-AS 7 imes. Second, a global analysis om a quali a i e poin o iew means ha (SVM, CFS-AS) pai eaches he bes esul s 5 imes, ollowed by (PART, CNS-AS) wi h 4 wins. Thi d, he pe cen age o selec ed a ibu es in (SVM, CFS-AS) pai is close o 25, while wi h (PART, CNS-AS) is sligh ly g ea e and akes a alue nea 35. 6 Conclusions CFS-AS and CNS-AS we e p esen ed. Expe imen s e ealed ha ACO-based sea ch ia AS in ea u e subse selec ion wi h In oGain heu is ic is be e , and in some cases wi h significan diffe ences, han he pu e e sions o ACO-based fil e s ( ha is, a conc e e subse a ibu e e alua o wi h he homonymous heu is ic e alua o , e.g. CFS-AS wi h h1). I is e y impo an o s ess ha ACO-based ea u e subse selec o wi h he p ope heu is ic e alua o is be e in mo e han he hal o he p oblems ha he adi ional Bes Fi s sea ch in CFS o CNS. The wo p e e ed classifie -FSS pai s, bea ing in mind he pe - o mance ega ding he accu acy and numbe o selec ed a ibu es a e, in his o de , (SVM, CFS-AS) and (PART, CNS-AS). Acknowledgmen s. This wo k has been pa ially subsidized by TIN2007-68084- C02-02and TIN2011-28956-C02-02p ojec so he Spanish In e -Minis e ialCom- mission o Science and Technology (MICYT), FEDER unds and P11-TIC-7528 p ojec o he ”Jun a de Andaluc´ıa” (Spain). Re e ences 1. Co e , T.M., Thomas, J.A.: Elemen s o In o ma ion Theo y. Wiley (1991) 2. Dash, M., Liu, H.: Fea u e selec ion o classi ica ion. In elligen Da a Analy- sis 1(3), 131–156 (1997) 3. Dash, M., Liu, H.: Consis ency-based sea ch in ea u e selec ion. A i icial In elli- gence 151(1), 155–176 (2003) 4. Do igo, M., Di Ca o, G., Gamba della, L.M.: An algo i hms o disc e e op imiza- ion. A i icial Li e 5(2), 137–172 (1999) 5. Do igo, M., Maniezzo, V., Colo ni, A.: An sys em: op imiza ion by a colony o coope a ing agen s. IEEE T ansac ions on Sys ems, Man, and Cybe ne ics, Pa B: Cybe ne ics 26(1), 29–41 (1996) 6. Do igo, M., S ¨u zle, T.: An colony op imiza ion: o e iew and ecen ad ances. In: Handbook o Me aheu is ics, pp. 227–263. Sp inge (2010) 7. Gaschig, J.: Pe o mance measu emen and analysis o ce ain sea ch algo i hms. Technical epo , DTIC Documen (1979) 8. Hall, M.A.: Co ela ion-based ea u e selec ion o machine lea ning. PhD hesis, The Uni e si y o Waika o (1999) 9. Jensen, R., Shen, Q.: Finding ough se educ s wi h an colony op imiza ion. In: P oceedings o he 2003 UK Wo kshop on Compu a ional In elligence, ol. 1 (2003) 10. Langley, P.: Selec ion o ele an ea u es in machine lea ning. De ense Technical In o ma ion Cen e (1994) 11. Tall´on-Balles e os, A.J., He ´as-Ma ´ınez, C., Riquelme, J.C., Ruiz, R.: Fea u e selec ion o enhance a wo-s age e olu iona y algo i hm in p oduc uni neu al ne wo ks o complex classi ica ion p oblems. Neu ocompu ing 114, 107–117 (2013)