Tackling An Colony Op imiza ion
Me a-Heu is ic as Sea ch Me hod in Fea u e
Subse Selec ion Based on Co ela ion
o Consis ency Measu es
An onio J. Tall´on-Balles e os and Jos´e C. Riquelme
Depa men o Languages and Compu e Sys ems,
Uni e si y o Se ille, Spain
[email p o ec ed]
Abs ac . This pape in oduces he use o an an colony op imiza ion
(ACO) algo i hm, called An Sys em, as a sea ch me hod in wo well-
known ea u e subse selec ion me hods based on co ela ion o consis-
ency measu es such as CFS (Co ela ion-based Fea u e Selec ion) and
CNS (Consis ency-based Fea u e Selec ion). ACO guides he sea ch using
a heu is ic e alua o . Empi ical esul s on wel e eal-wo ld classi ica ion
p oblems a e epo ed. S a is ical es s ha e e ealed ha In oGain is a
e y sui able heu is ic o CFS o CNS ea u e subse selec ion me hods
wi h ACO ac ing as sea ch me hod. The use o In oGain is shown o be
he signi ican ly be e heu is ic o e a ange o classi ie s. The esul s
achie ed by means o ACO-based ea u e subse selec ion wi h he sui -
able heu is ic e alua o a e be e o mos o he p oblems compa ing
wi h hose ob ained wi h CFS o CNS combined wi h Bes Fi s sea ch.
Keywo ds: Fea u e selec ion, classi ica ion, an colony op imiza ion,
heu is ic e alua o , il e , ea u e subse selec ion.
1 In oduc ion
An colony op imiza ion (ACO) me a-heu is ic [6] was p oposed by Do igo e al.
and is inspi ed in he beha iou o eal an colonies. Depending on he amoun
o phe omone deposi ed by he an in hei walk he e would be some poin s ha
a e mo e likely o be isi ed by he nex an s [4]. The (a ificial) an s in ACO
define a andomized cons uc ion heu is ic which makes p obabilis ic decisions
depending on he s eng h o a ificial phe omone ails and a ailable heu is ic
in o ma ion. As such, ACO can be in e p e ed as an ex ension o adi ional
cons uc ion heu is ics, which a e eadily a ailable o many combina o ial op-
imiza ion p oblems. Ye , an impo an diffe ence wi h cons uc ion heu is ics
is he adap a ion o he phe omone ails du ing algo i hm execu ion o ake
in o accoun he cumula ed sea ch expe ience. As cons uc ion algo i hms wo k
on pa ial solu ions ying o ex end hese in he bes possible way o comple e
p oblem solu ions.
In essence, ea u e selec ion (FS) is a NP-ha d combina o ial op imiza ion
p oblem. O en, FS is ackled wi h classical sea ch me hods o a oiding he
p ohibi i e exhaus i e sea ch. Ins ead o looking o he op imal solu ion, ob-
aining a good solu ion in a easonable ime migh be p e e able o ce ain
p oblems. On one hand, he ypical NP-ha d T a elling Salesman P oblem has
been ea ed success ully wi h ACO. On he o he hand, ACO was applied o
ea u e selec ion in [9] in he con ex o ough se educ s wi h e y p omising
esul s. He e, we apply ACO as a s ochas ic p ocedu e o quickly finding high
quali y solu ions, in he scope o o dina y o c isp se s o FS in classifica ion
asks. In his con ex , ea u e subse selec ion (FSS) p oblem is o mula ed using
a g aph wi h he pu pose o ge ing a subse o a ibu es ha is ele an o
he p oblem a hand. FSS needs a sea ch me hod, ha usually is any kind o
a ificial in elligence heu is ic echnique. The cu en p oposal shi s he sea ch
om an heu is ic non-s ochas ic pe spec i e o a s ochas ic angle.
This pape goals o add ess he sui abili y o using he an colony op imiza ion
me a-heu is ic as a sea ch me hod buil -in in he CFS and CNS ea u e subse
selec o s o classifica ion p oblems.
The es o his a icle is o ganized as ollows: Sec . 2 desc ibes some concep s
abou ACO me a-heu is ic and ea u e selec ion; Sec . 3 p esen s ou p oposal;
Sec . 4 de ails he expe imen a ion; hen Sec . 5 shows and analyzes s a is ically
he esul s ob ained; finally, Sec . 6 s a es he concluding ema ks.
2 Backg ound
2.1 An Colony Op imiza ion Me a-Heu is ic
A ificial an s used in ACO a e s ochas ic solu ion cons uc ion p ocedu es ha
p obabilis ically build a solu ion by i e a i ely adding solu ion componen s o
pa ial solu ions by aking in o accoun a) heu is ic in o ma ion abou he p ob-
lem ins ance being sol ed, i a ailable, and b) (a ificial) phe omone ails which
change dynamically a un ime o eflec he agen s’ acqui ed sea ch expe ience.
The p oblem ep esen a ion is a g aph whe e he nodes ep esen he diffe en
poin s ha he an s can isi and he edges a e he link be ween poin s. Links
a e unidi ec ional and he e a e no cycles, so i is no possible o go back o a
poin p e iously isi ed. A he beginning o he algo i hm e e y an is loca ed
in a poin and will cons uc a solu ion aking se e al decisions un il he s op
condi ion is me . A he end each an has ound a candida e solu ion o he
p oblem a hand.
The main s eps o he algo i hm a e he ollowing:
1. Ini ialisa ion. The algo i hm s a s and all he phe omone a iables a e ini-
ialized o a alue τ0which is a key pa ame e .
2. Cons uc An Solu ions. This ac ion s a s he algo i hm loop and is e-
la ed wi h sending an s a ound he cons uc ion g aph. An an in he node
ichooses he jone acco ding o a p obabilis ic decision ule, which is a
unc ion o he phe omone τand he heu is ic η.Ase o naan s cons uc s
solu ions o he conc e e p oblem being ackled.
3. Upda e Phe omones. The pu pose o his pa is o change he alues o he
phe omones, by bo h deposi ing and e apo a ing.
Among he se e al a ian s o ACO, he eina e we ocus on An Sys em (AS)
ha was in oduced in 1991 by Do igo and published a ew yea s la e by Do igo
e al. [5].
2.2 Fea u e Selec ion
Fea u e selec ion me hods y o pick a subse o ea u es ha a e ele an o
he a ge concep [2]. Acco ding o Langley [10], he diffe en app oaches o
ea u e selec ion can be di ided in o wo b oad ca ego ies (i.e., fil e and w ap-
pe ) based on hei dependence on he induc i e algo i hm ha will finally use
he selec ed subse . Fil e me hods a e independen o he induc i e algo i hm,
whe eas w appe me hods use he induc i e algo i hm as he e alua ion unc ion.
FS in ol es wo s ages: a) o ob ain a lis o a ibu es acco ding o an a -
ibu e e alua o and b) o pe o m a sea ch on he ini ial lis . All candida e lis s
would be assessed using a measu e e alua ion and he bes one will be e u ned.
Two o he mos widesp ead ea u e subse selec o s a e Co ela ion-based
Fea u e Selec ion (CFS) [8] and Consis ency-based ea u e selec ion (CNS) [3]
ha wo k in combina ion wi h a sea ch me hod such as G eedy Sea ch, Bes
Fi s (BF) o Exhaus i e Sea ch. Gene ally speaking, BF is a powe ul sea ch
me hod [7] which is he eason o be used e y equen ly by he machine lea n-
ing communi y nowadays. We ha e chosen CFS and CNS as ep esen a i e FSS
me hods, because hey a e based on diffe en kind o measu es, ha e ew pa-
ame e s and ha e p o ided a good pe o mance inside he supe ised machine
lea ning a ea. O en, BF sea ch is he p e e able op ion by he esea che s o
bo h FSS algo i hms. CFS is p obably he mos used FSS in da a mining. CNS
is also powe ul, howe e he amoun o published wo ks is mo e educed.
3P oposal
Fi s ly, he g aph meaning may be e o mula ed in o de o deal wi h a ea u e
selec ion p oblem by means o ACO me a-heu is ic. He e, he nodes ep esen
ea u es and edges he link be ween nodes and he possibili y o add ano he
ea u e o he cu en solu ion. The sea ch o a candida e solu ion is a walk
h ough he g aph. Once an an isi s an edge i con ains a weigh indica ing
he s eng h o his solu ion componen .
In he cu en wo k, ACO, implemen ed ollowing he AS model, is conside ed
as sea ch s a egy in he con ex o CFS and CNS me hods a e he a ibu e
e alua ion phase. ACO guides he sea ch by means o a heu is ic e alua o .
As heu is ic e alua o s, on one hand we ha e conside ed o CFS and CNS
app oaches he own a ibu e e alua o , ob aining he pu e e sions o CFS-AS
and CNS-AS. On he o he hand, we ha e ied In o ma ion Gain (In oGain
as abb e ia ion) [1] as e alua o esul ing an hyb id app oach. Mo eo e , CFS,
CNS and In oGain compu e diffe en kinds o measu e o e alua e he ele ance,
such as co ela ion, consis ency and in o ma ion, espec i ely.
The p obabilis ic ansi ion ule is defined in he same way as in [9] and is
hemos widelyusedinAS[5]:
pij =τα
ij ·[ηij ]β
il∈N (x)τα
il ·[ηil]β,∀ij ∈N(x).(1)
whe e pij ep esen s he p obabili y ha cu en an a ea u e iwould a el
o ea u e j,τij is he amoun o phe omone on he ij edge, ηij is he heu is ic
desi abili y o he ij ansi ion and N(x) he se o cu en easible componen s.
Las ly, αand βa e pa ame e s ha may ake eal posi i e alues –acco ding
o he ecommenda ions on pa ame e se ing in [5]– and a e associa ed wi h
heu is ic in o ma ion and phe omone ails, espec i ely.
All an s upda e phe omone le el wi h an inc ease o small quan i ies, depend-
ing di ec ly on he heu is ic desi abili y o he ij ansi ion gi en by he measu e
(me i ) o he subse a ibu e e alua o used as heu is ic e alua o and in e sely
p opo ional o he subse size.
4 Expe imen a ion
Table 1 depic s he ea u e subse selec ion me hods applied in he expe imen al
p ocess. We ha e g ouped hem acco ding o he a ibu e e alua o and sea ch
me hod.
Table 2 summa izes he main pa ame e s along wi h hei symbols and nu-
me ical o concep ual alues o all he ea u e subse selec ion me hods used
o he expe imen s. On one hand, in ela ion o ACO-based ea u e subse se-
lec ion, he naand gen pa ame e s ha e been se o fix alues o ou choice. Fo
τ0pa ame e , in [5] he e is a sugges ion o assign a small posi i e cons an and
hence we ha e defined a alue o 0.5. The ade-off be ween αand βpa ame e s
may influence in he beha iou o he algo i hm hus o hei de e mina ion a
p elimina y expe imen al design by means o a fi e- old c oss alida ion on he
aining se has been ca ied ou wi h a couple o alues o each one pa ame e
(1 and 2). On he o he hand, o BF-based sea ch me hod in he con ex o CFS
and CNS, we ha e ollowed he ecommenda ions o he au ho s ([8] and [3] o
he numbe o expanded nodes; he sea ch di ec ion has been fix acco ding o
ou p e ious expe iences).
Table 3 ep esen s he da a se s employed h oughou he expe imen a ion.
They come mos ly om bina y and mul i-class classifica ion eal-wo ld p ob-
lems (Cl. column specifies he numbe o classes) aken om he public UCI
eposi o y. The numbe o ins ances anges om mo e han one hund ed o ap-
p oxima ely ou een housands, hus p oblems wi h a medium size, and he
dimensionali y a ies be ween nine een and one hund ed and wen y nine. Also,
we ha e included he numbe o selec ed ea u es o e e y ea u e subse selec-
o ob ained in he aining se . Las ow shows he dimensionali y educ ion
(highe is be e ) in mean o e he o iginal da a se s o each fil e .
Table 1. Lis o ea u e subse selec o s o he expe imen a ion
A ibu e e alua o Sea ch me hod Heu is ic e alua o Abb. Name
CFS An Sea ch CFS CFS −AS h1
In oGain CFS −AS h2
Bes Fi s −CFS −BF
CNS An Sea ch CNS CNS −AS h1
In oGain CNS −AS h2
Bes Fi s −CNS −BF
Table 2. Pa ame e alues o ACO-based ea u e subse selec ion app oaches (CFS-
AS and CNS-AS) and Bes Fi s -based ones (CFS-BF and CNS-BF)
Sea ch me hod Pa ame e Symbol Value
An sea ch Numbe o an s na10
Numbe o gene a ions gen 10
P he omone ail in luence α 1
Heu is ic in o macion alue β 2
P he omone ini ial alue τ00.5
Bes F i s Consecu i e expanded nodes wi hou imp o ing 5
Sea ch di ec ion F o wa d
Table 3. Summa y o he da a se s used and selec ed ea u es o each ea u e subse
selec o
Da a se Size T ain T es F ea . Cl. Selec ed ea u es
CFS CNS
AS BF AS BF
h1h2h1h2
ba ch(gas) 13910 10432 3478 129 6 7 13 20 7 10 6
ca dio oc. 2126 1595 531 22 10 12 12 12 15 15 13
hepa i is 155 117 38 19 2 10 9 10 15 11 11
ionosphe e 351 263 88 33 2 9 10 11 9 9 9
lib as 360 270 90 90 15 8 34 23 47 55 20
lymph. 148 111 37 38 4 11 8 12 19 13 10
p omo e 106 80 26 582101010158 8
sa image 6435 4435 2000 36 6 20 21 23 13 13 12
sona 2081041046024781499
soybean 683 511 172 82 19 22 22 25 39 58 16
SPECTF 267 80 187 44 2 10 8 12 12 10 8
wa e o m 5000 3750 1250 40 3 12 14 14 14 13 12
A e ages 2479.08 1812.33 666.75 54.25 6.08 11.25 14.00 15.00 18.25 18.67 11.17
Dim. educ ion 79.26 74.19 72.35 66.36 65.59 79.42

The expe imen al design ollows a s a ified hold-ou c oss alida ion wi h
h ee and one qua e s o he aining and es se s, espec i ely. Some imes,
hese p opo ions do no ma ch since he o iginal da a a e p ea anged. Fo he
s a is ical analysis be ween wo ea u e subse selec ion me hods we ha e ca ied
a Wilcoxon signed- anks es wi h he es accu acy esul s ob ained.
5Resul s
Tables 4 and 5 epo he accu acy es esul s o he FSS based on ACO wi h
CFSandCNS- ha is,CFS-ASandCNS-AS, espec i ely- wi h wo diffe en
heu is ic e alua o s, hei diffe ence (Di .) and i s anking (R.). Ten execu ions
wi h diffe en seeds we e un and he mos equen solu ion was conside ed o
he assessmen . We ha e ca ied ou expe imen s wi h h ee kind o de e minis-
ic classifie s such: a) C4.5, based on decision ees, b) SVM, ounded in suppo
ec o s, and c) PART, a ule-based app oach. The eason o he choice o hese
classifie s is mo i a ed by he ac ha hei o e all pe o mance is good in he
ea u e selec ion scope [11]. The bes esul s, excluding ies, o each pai (clas-
sifie , FSS me hod) ha e been highligh ed wi h bold ace. Acco ding o Wilcoxon
signed- anks es , since he e a e 12 da a se s, he T alue a α=0.05 should
be less o equal han 14 ( he c i ical alue) o ejec he null hypo hesis. On
one hand, in ela ion o CFS-AS, o classifie s C4.5 and SVM he h2 heu is ic
e alua o is significan ly be e han h1. On he o he hand, o CNS-AS he
pe o mance o h2 wi h PART is he significan bes op ion.
Table 4. CFS-AS: Accu acy es esul s and s a is ical es s
Da a se C4.5SVM PART
CFS −AS CF S −AS CF S −AS
h1 h2 Di . R. h1 h2 Di . R. h1 h2 Di . R.
ba ch(gas)93.99 96.87 2.88 8 72.28 78.21 5.92 9 93.73 96.35 2.62 7
ca dio oc. 61.58 61.58 0.00 2 67.98 67.80 −0.19 2 60.45 62.52 2.07 5
hepa i is 84.21 89.47 5.26 10 86.84 89.47 2.63 6 84.21 86.84 2.63 8
ionosphe e 90.91 87.50 −3.41 9 82.95 89.77 6.82 10 90.91 93.18 2.27 6
lib as 52.22 65.56 13.33 12 50.00 63.33 13.33 12 54.44 65.56 11.11 12
lymph. 81.08 81.08 0.00 2 81.08 83.78 2.70 8 72.97 70.27 −2.70 9
p omo e 73.08 73.08 0.00 2 73.08 73.08 0.00 1 80.77 80.77 0.00 1
sa image 86.05 86.25 0.20 5 83.80 84.50 0.70 3 85.00 83.55 −1.45 2
sona 67.31 74.04 6.73 11 67.31 75.00 7.69 11 68.27 75.96 7.69 11
soybean 88.95 91.28 2.33 6 94.19 95.35 1.16 5 90.70 92.44 1.74 3
SPECTF 66.84 69.52 2.67 7 66.31 63.64 −2.67 7 70.59 76.47 5.88 10
wa e o m 74.32 74.40 0.08 4 85.92 86.88 0.96 4 78.88 77.04 −1.84 4
T=min{66,12}=12 T=min{68.5,9.5}=9.5T=min{62.5,15.5}=15.5
Table 6 ou lines he global accu acy es esul s o he bes CFS-AS and CNS-
AS app oaches e sus based-BF CFS o CNS. The bes esul s, in each da a se ,
a e ma ked in bold. We can asse he ollowing s a emen s in ela ion o he
Table 5. CNS-AS: Accu acy es esul s and s a is ical es s
Da a se C4.5SVMPART
CNS −AS CNS −AS CNS −AS
h1 h2 Di . R. h1 h2 Di . R. h1 h2 Di . R.
ba ch(gas)96.00 95.54 −0.46 3 64.49 69.15 4.66 10 95.46 95.46 0.00 1.5
ca dio oc. 63.84 66.48 2.64 6 61.39 64.41 3.01 9 58.76 64.22 5.46 8
hepa i is 84.21 86.84 2.63 5 86.84 84.21 −2.63 8 76.32 84.21 7.89 10
ionosphe e 88.64 93.18 4.55 12 81.82 84.09 2.27 7 89.77 95.45 5.68 9
lib as 56.67 61.11 4.44 11 67.78 70.00 2.22 6 55.56 64.44 8.89 11
lymph. 78.38 81.08 2.70 8 91.89 83.78 −8.11 11 78.38 64.86 −13.51 12
p omo e 84.62 80.77 −3.85 10 73.08 84.62 11.54 12 76.92 80.77 3.85 5
sa image 84.70 84.75 0.05 2 84.50 83.70 −0.80 4 85.50 86.25 0.75 4
sona 75.96 73.08 −2.88 9 75.96 75.00 −0.96 5 72.12 75.96 3.85 6
soybean 91.86 91.86 0.00 1 95.35 94.77 −0.58 3 92.44 92.44 0.00 1.5
SPECTF 69.52 66.84 −2.67 7 64.71 64.71 0.00 1 65.24 69.52 4.28 7
wa e o m 76.00 74.88 −1.12 4 85.12 85.36 0.24 2 77.36 77.84 0.48 3
T=min{44.5,33.5}=33.5T=min{51.5,26.5}=26.5T=min{64.5,13.5}=13.5
Table 6. Global accu acy es esul s o CFS-BF and CNS-BF e sus CFS-AS and
CNS-AS
Da a se C4.5SVMPART
CFS −BF CFS −AS CF S −BF CFS −AS CNS −BF CNS −AS
h2 h2 h2
ba ch(gas senso )95.92 96.87 83.04 78.21 98.30 95.46
ca dio oc. 61.58 61.58 67.80 67.80 61.02 64.22
hepa i is 84.21 89.47 86.84 89.47 84.21 84.21
ionosphe e 92.05 87.50 88.64 89.77 88.64 95.45
lib as 61.11 65.56 57.78 63.33 55.56 64.44
lymph. 81.08 81.08 81.08 83.78 67.57 64.86
p omo e 73.08 73.08 73.08 73.08 69.23 80.77
sa image 85.60 86.25 83.85 84.50 85.45 86.25
sona 73.08 74.04 75.00 75.00 75.96 75.96
soybean 93.02 91.28 94.77 95.35 93.02 92.44
SPECTF 66.84 69.52 73.26 63.64 66.31 69.52
wa e o m 74.40 74.40 86.88 86.88 76.16 77.84
Wins by pai s 262637
Global wins 033524
A e ages
Accu acy 78.50 79.22 79.33 79.23 76.79 79.29
Selec ed ea .(%) 27.65 25.81 27.65 25.81 20.58 34.41
achie ed esul s. Fi s , he compa ison be ween pai s o FSS me hods o each
classifie and da a se s poin s ou ha : a) C4.5 wi h CFS-AS ge s be e esul s
6 imes, b) SVM wi h CFS-AS wins in 6 p oblems, and c) PART classifie wi h
CNS-AS 7 imes. Second, a global analysis om a quali a i e poin o iew means
ha (SVM, CFS-AS) pai eaches he bes esul s 5 imes, ollowed by (PART,
CNS-AS) wi h 4 wins. Thi d, he pe cen age o selec ed a ibu es in (SVM,
CFS-AS) pai is close o 25, while wi h (PART, CNS-AS) is sligh ly g ea e and
akes a alue nea 35.
6 Conclusions
CFS-AS and CNS-AS we e p esen ed. Expe imen s e ealed ha ACO-based
sea ch ia AS in ea u e subse selec ion wi h In oGain heu is ic is be e , and
in some cases wi h significan diffe ences, han he pu e e sions o ACO-based
fil e s ( ha is, a conc e e subse a ibu e e alua o wi h he homonymous
heu is ic e alua o , e.g. CFS-AS wi h h1). I is e y impo an o s ess ha
ACO-based ea u e subse selec o wi h he p ope heu is ic e alua o is be e
in mo e han he hal o he p oblems ha he adi ional Bes Fi s sea ch in
CFS o CNS. The wo p e e ed classifie -FSS pai s, bea ing in mind he pe -
o mance ega ding he accu acy and numbe o selec ed a ibu es a e, in his
o de , (SVM, CFS-AS) and (PART, CNS-AS).
Acknowledgmen s. This wo k has been pa ially subsidized by TIN2007-68084-
C02-02and TIN2011-28956-C02-02p ojec so he Spanish In e -Minis e ialCom-
mission o Science and Technology (MICYT), FEDER unds and P11-TIC-7528
p ojec o he ”Jun a de Andaluc´ıa” (Spain).
Re e ences
1. Co e , T.M., Thomas, J.A.: Elemen s o In o ma ion Theo y. Wiley (1991)
2. Dash, M., Liu, H.: Fea u e selec ion o classi ica ion. In elligen Da a Analy-
sis 1(3), 131–156 (1997)
3. Dash, M., Liu, H.: Consis ency-based sea ch in ea u e selec ion. A i icial In elli-
gence 151(1), 155–176 (2003)
4. Do igo, M., Di Ca o, G., Gamba della, L.M.: An algo i hms o disc e e op imiza-
ion. A i icial Li e 5(2), 137–172 (1999)
5. Do igo, M., Maniezzo, V., Colo ni, A.: An sys em: op imiza ion by a colony o
coope a ing agen s. IEEE T ansac ions on Sys ems, Man, and Cybe ne ics, Pa
B: Cybe ne ics 26(1), 29–41 (1996)
6. Do igo, M., S ¨u zle, T.: An colony op imiza ion: o e iew and ecen ad ances.
In: Handbook o Me aheu is ics, pp. 227–263. Sp inge (2010)
7. Gaschig, J.: Pe o mance measu emen and analysis o ce ain sea ch algo i hms.
Technical epo , DTIC Documen (1979)
8. Hall, M.A.: Co ela ion-based ea u e selec ion o machine lea ning. PhD hesis,
The Uni e si y o Waika o (1999)
9. Jensen, R., Shen, Q.: Finding ough se educ s wi h an colony op imiza ion. In:
P oceedings o he 2003 UK Wo kshop on Compu a ional In elligence, ol. 1 (2003)
10. Langley, P.: Selec ion o ele an ea u es in machine lea ning. De ense Technical
In o ma ion Cen e (1994)
11. Tall´on-Balles e os, A.J., He ´as-Ma ´ınez, C., Riquelme, J.C., Ruiz, R.: Fea u e
selec ion o enhance a wo-s age e olu iona y algo i hm in p oduc uni neu al
ne wo ks o complex classi ica ion p oblems. Neu ocompu ing 114, 107–117 (2013)