OWA-FRPS: A Prototype Selection method based on Ordered Weighted Average Fuzzy Rough Set Theory

OWA-FRPS: A P o o ype Selec ion me hod based on O de ed Weigh ed A e age Fuzzy Rough Se Theo y Nele Ve bies 1, Ch is Co nelis1,2, and F ancisco He e a2 1Depa men o Applied Ma hema ics, Compu e Science and S a is ics, Ghen Uni e si y, K ijgslaan 281 (S9), B-9000 Gen , Belgium [email p o ec ed] 2Depa men o Compu e Science and A i icial In elligence, Uni e si y o G anada, Calle del Pe iodis a Daniel Saucedo A anda s/n, 18071 G anada, Spain [email p o ec ed] [email p o ec ed] Abs ac . The Nea es Neighbo (NN) algo i hm is a well-known and e ec i e classi ica ion algo i hm. P o o ype Selec ion (PS), which p o- ides NN wi h a good aining se o pick i s neighbo s om, is an impo an opic as NN is highly suscep ible o noisy da a. Accu a e s a e-o - he-a PS me hods a e gene ally slow, which mo i a es us o p opose a new PS me hod, called OWA-FRPS. Based on he O de ed Weigh ed A e age (OWA) uzzy ough se model, we exp ess he quali y o ins ances, and use a w appe app oach o decide which ins ances o se- lec . An expe imen al e alua ion shows ha OWA-FRPS is signi ican ly mo e accu a e han s a e-o - he-a PS me hods wi hou equi ing a high compu a ional cos . Keywo ds: O de ed Weigh ed A e age, Fuzzy Rough Se s, P o o ype Selec ion, KNN 1 In oduc ion One o he mos well-known and mos widely used classi ica ion algo i hms is Nea es Neighbo s (NN,[1]). This me hod classi ies a es ins ance o he class o he nea es neighbo o in he aining se . Al hough NN has been p o en o be e y use ul o many classi ica ion p oblems, i deals wi h some p oblems, among which i s sensi i i y o noise and i s la ge s o age equi emen s a e he mos impo an ones. In his wo k we alle ia e hese p oblems by using P o o ype Selec ion (PS,[2]). This echnique emo es edundan and/o noisy ins ances om he aining se , such ha he aining se equi es less s o age and such ha he NN algo i hm is mo e accu a e. PS echniques ha mainly y o imp o e he classi ica ion accu acy a e called edi ion me hods, hose ha ocus on educing he equi ed s o age a e condensa ion me hods. Hyb id PS echniques y o ackle bo h p ob- lems simul aneously. In his wo k we de elop an edi ing me hod. Many PS me hods ha e been p oposed in he li e a u e, a comp ehensi e o e iew can be ound in [2]. When he algo i hm does no make use o a speci ic clas- si ie o classi y he en i e aining se , he me hod is called a il e me hod. Condensa ion me hods do use a speci ic classi ie , he NN classi ie in ou case, o classi y he en i e aining da a o ob ain a quali y assessmen o a ce ain p o o ype subse . Fil e me hods a e gene ally as e and less accu a e, while w appe me hods a e slowe and mo e accu a e. Many w appe PS algo i hms a e e olu iona y based, like CHC [3], GGA [4, 5] o SSMA [6], while o he s use o he sea ch heu is ics like RMHC [7] o RNG [8]. Mos o he il e me hods a e based on he NN algo i hm i sel , like AllKNN [9] o MENN [10]. The me hod ha we de elop is a w appe . Al hough many esea che s ha e ocused on de eloping uzzy ough ea u e se- lec ion [11] algo i hms, he e is no much li e a u e on uzzy ough PS ye . Ne e heless, uzzy ough se heo y [12] is a good ool o model noisy da a. To he bes o ou knowledge, he only uzzy ough based PS me hod is FRIS [13]. This me hod selec s hose ins ances ha ha e a uzzy posi i e egion highe han a ce ain h eshold. This me hod has some p oblems, he main one being ha he me hod’s pe o mance highly elies on a good h eshold selec ion. In his wo k, we p opose a new uzzy ough based PS me hod ha assesses he quali y o ins ances using O de ed Weigh ed A e age (OWA) uzzy ough se heo y [14], a mo e obus e sion o uzzy ough se heo y, and au oma ically selec s an app op ia e h eshold. The emainde o his wo k is s uc u ed as ollows. In Sec ion 2, we i s discuss h ee OWA uzzy ough quali y measu es ha can be used o assess he quali y o ins ances, and hen show how hese measu es can be used o ca y ou PS. In Sec ion 3, we e alua e ou algo i hm, called OWA Fuzzy Rough P o o ype Selec ion (OWA-FRPS), and we conclude in Sec ion 4. 2 O de ed Weigh ed A e age based Fuzzy Rough P o o ype Selec ion In his sec ion we p esen ou new PS me hod. In he i s subsec ion we de ine h ee measu es o assess he quali y o ins ances, and in he second subsec ion we demons a e how we can use hese measu es o ca y ou PS. 2.1 Assessing he quali y o ins ances using OWA uzzy ough se s Fi s we in oduce some no a ions. We conside a decision sys em (X, A ∪ {d}), consis ing o nins ances X={x1, . . . , xn},ma ibu es A={a1, . . . , am}and a decision a ibu e d /∈ A. We deno e by a(x) he alue o an ins ance x∈X o an a ibu e a∈ A. We assume ha each con inuous a ibu e a∈ A is no malized, ha is, ∀x∈X:a(x)∈[0,1]. The ca ego ical a ibu es can ake alues in a ini e se . The decision a ibu e dis ca ego ical oo and assigns a class d(x) o each ins ance x∈X. We associa e a uzzy indisce nibili y ela ion R:X×X→[0,1] wi h he decision sys em as ollows. Fi s , we calcula e he uzzy indisce nibili y Ra o each ea u e a∈ A sepa a ely. When ais ca ego ical, Ra(x, y) = 1 o x, y ∈Xi a(x) = a(y) and Ra(x, y) = 0 o he wise. When ais con inuous, Ra(x, y) = 1 − |a(x)−a(y)| o all x, y ∈X. Nex , we combine hese sepa a e uzzy indisce nibili y ela ions using a -no m T( he Lukasiewicz -no m3in his pape ): ∀x, y ∈X:R(x, y) = T(Ra(x, y)) | {z } a∈A (1) This uzzy indisce nibili y ela ion is he keys one o uzzy ough se heo y. A uzzy se Scan be app oxima ed by i s uzzy ough lowe app oxima ion ∀x∈X: (R↓S)(x) = min y∈XI(R(x, y), S(y)) (2) wi h I he Lukasiewicz implica o 4in his pape , and by i s uppe app oxima ion ∀x∈X: (R↑S)(x) = max y∈XT(R(x, y), S(y)) (3) The uzzy lowe app oxima ion exp esses o wha ex en ins ances simila o x also belong o S, while he uppe app oxima ion exp esses o wha ex en he e exis ins ances ha a e simila o xand belong o S. These concep s can be used o assess he quali y o ins ances. Fi s , no e ha we can conside he class [x]do an ins ance x∈Xas a uzzy se in X: ∀y∈X: [x]d(y) = 1 i d(x) = d(y) 0 else (4) which can be conside ed as he c isp se ha con ains all ins ances ha ha e he same class as x. I we wan o assess he quali y o an ins ance x, we can use he lowe app oxi- ma ion o [x]d: (R↓[x]d)(x).(5) This alue exp esses o wha ex en ins ances simila o xalso belong o he same class as x. Ano he op ion is o use he uppe app oxima ion o [x]d: (R↑[x]d)(x) (6) which exp esses o wha ex en he e exis ins ances ha a e simila o xand ha belong o he same class as x. Bo h measu es a e pa icula ly meaning ul in he con ex o NN classi ica ion, because hey a e ins ances highly i hey a e su ounded by neighbo s o he same class: he lowe app oxima ion measu e is high o xi he e a e no ins ances 3The Lukasiewicz -no m is he mapping T: [0,1]2→[0,1], such ha ∀a, b ∈ [0,1],T(a, b) = max(0, a +b−1) 4The Lukasiewicz implica o is he mapping I: [0,1]2→[0,1], such ha ∀a, b ∈ [0,1],I(a, b) = min(1 −a+b, 1) om a di e en class ha a e nea (simila ) o x, while he uppe app oxima ion measu e is high i he e exis neighbo s om he same class. In [14] i was no ed ha he adi ional uzzy ough app oxima ions a e highly suscep ible o noise, as hey use he c isp min and max ope a o s, such ha sin- gle ins ances can d as ically in luence he app oxima ion alues. A solu ion o his p oblem is o use OWA uzzy ough se s [14], which eplace hese c isp ope a o s by so e OWA ope a o s [15]. Recall ha , gi en a weigh ec o W=hw1, . . . , wni o which n P i=1 wi= 1 and ∀i∈1, . . . , n, wi∈[0,1], he OWA agg ega ion o n alues s1, . . . , snis gi en by: OWAW(s1, . . . , sn) = n X i=1 wi i,(7) whe e i=sji sjis he i h la ges alue in s1, . . . , sn. When h0,...,0,1iis used as weigh ec o , he minimum ope a o is e ie ed, which is he ope a o ha is used in he adi ional uzzy lowe app oxima ion. We eplace his minimum by a less s ic ope a o ha s ill has he cha ac e is ics o a minimum ope a o , ha is, we conside a weigh ec o wi h ascending weigh s, such ha lowe alues ge highe weigh s, and highe alues ge lowe weigh s. In his wo k we use he weigh ec o Wmin =hw1, . . . , wniwhe e ∀i∈1, . . . , n :wi=i n(n+ 1)/2.(8) Comple ely analogously, we can de ine he OWAWmax ope a o ha so ens he maximum ope a o . I s weigh s Wmax =hw1, . . . , wnia e de ined as ollows in his pape : ∀i∈1, . . . , n :wi=n−i+ 1 n(n+ 1)/2.(9) Replacing he s ic minimum and maximum ope a o s in he adi ional de ini- ions o uzzy lowe and uppe app oxima ion leads o he ollowing mo e obus de ini ions o OWA uzzy ough se s: ∀x∈X: (R↓OW A S)(x) = OWAWmin y∈X I(R(x, y), S(y)) (10) ∀x∈X: (R↑OW A S)(x) = OWAWmax y∈X T(R(x, y), S(y)) (11) We will use his OWA uzzy ough se model, leading o he ollowing h ee quali y measu es: ∀x∈X:γL(x) = (R↓OW A [x]d)(x),(12) ∀x∈X:γU(x)=(R↑OW A [x]d)(x),(13) and ∀x∈X:γLU (x) = (R↓OW A [x]d)(x)+(R↑OW A [x]d)(x) (14) 2.2 OWA-FRPS Based on he quali y measu es γde ined in he p e ious subsec ion, we can o mula e an algo i hm o ind a good subse o ins ances. We ob iously wan o selec he ins ances wi h a high γ alue and emo e hose wi h a low γ alue, bu now he ques ion aises wha h eshold o use. The main idea o ou app oach is o use he γ alues o all ins ances in Xas h eshold. We calcula e he lea e-one-ou aining accu acy o he co esponding educed subse s o ins ances and selec he h eshold ha co esponds o he highes accu acy. Mo e speci ically, we ca y ou he ollowing s eps: 1. Calcula e he γ(x) alues o all ins ances x∈X. 2. Remo e he duplica es among all hese γ alues, he inal se o γ alues, which will all be conside ed as h esholds, is G={τ1, . . . , τp}, p ≤n. 3. Fo each o he h esholds τ∈G, conside he ollowing subse : Sτ={x∈ X|γ(x)≥τ}. 4. Calcula e he aining lea e-one-ou accu acy o each o hese subse s using he LOO p ocedu e in Algo i hm 1. 5. Selec he subse s Sτi1, . . . , Sτiswi h he highes lea e-one-ou accu acy. No e ha mul iple subse s can co espond o he same lea e-one-ou ac- cu acy. 6. Re u n he subse Smedian(τi1,...,τis). Algo i hm 1 LOO, p ocedu e o measu e he aining accu acy o a subse o ins ances using a lea e-one-ou app oach Inpu : Reduced decision sys em (S, A ∪ {d}) (S⊆X). acc ←0 o x∈Xdo i x∈S hen Find he nea es neighbo nn o xin S {x} else Find he nea es neighbo nn o xin S end i i d(nn) = d(x) hen acc →acc + 1 end i end o Ou pu :acc We illus a e he algo i hm wi h an example. Conside he decision sys em in Table 1, wi h en ins ances, wo con inuous ea u es and one ca ego ical ea u e. The alues γLU a e gi en in he las column o each ins ance. The e a e no duplica es, so he se o h esholds consis s o he en alues in he las column o Table 1. In Table 2, we show he co esponding subse s. In o de o calcula e he aining lea e-one-ou accu acy, we need he Euclidean dis ances be ween he ins ances, which a e gi en in Table 3. In he las wo columns o Table 2, he ins ances ha a e co ec ly classi ied using he subse Sτa e gi en, oge he wi h he aining accu acy. The subse s co esponding o he highes LOO aining accu acy a e Sτ1, Sτ3and Sτ9, and subse Sτ3={x1, x3, x5, x9}will be e u ned by he OWA-FRPS algo i hm. Table 1. Decision sys em wi h 2 con inuous ea u es (a1and a2) and one ca ego ical ea u e (a3). The class is gi en in column dand he alue o he γLU measu e is shown in he las column. a1a2a3dγLU x10.2 0.4 A 0 1.02 x20.3 0.3 A 1 1.016 x31 0 B 0 1.16 x40.7 0.9 B 1 1.07 x50.4 0.3 A 0 1.05 x60.3 0.6 A 1 1.01 x70.4 1 B 0 1.06 x80.3 0.2 B 1 1.15 x90.7 0.5 A 0 1.17 x10 0 0.1 A 1 1.14 Table 2. Th esholds τconside ed in he OWA-FRPS algo i hm and co esponding subse s o ins ances Sτ. Th eshold τCo esponding subse SτCo ec ly classi ied ins ances LOO aining accu acy 1.02 {x1, x3, x4, x5, x7, x8, x9, x10} {x1, x5, x6, x9, x10}0.5 1.016 {x1, x2, x3, x4, x5, x7, x8, x9, x10} {x5}0.1 1.16 {x3, x9} {x1, x3, x5, x7, x9}0.5 1.07 {x3, x4, x8, x9, x10} {x2, x4, x5}0.3 1.05 {x3, x4, x5, x7, x8, x9, x10} {x4, x5, x9}0.3 1.01 {x1, x2, x3, x4, x5, x6, x7, x8, x9, x10} {x5, x9}0.2 1.06 {x3, x4, x7, x8, x9, x10} {x2, x5}0.2 1.15 {x3, x9, x10} {x3, x5, x7}0.3 1.17 {x9} {x1, x3, x5, x7, x9}0.5 1.14 {x3, x9, x10} {x3, x5, x7}0.3 3 Expe imen al E alua ion In his sec ion we ca y ou an expe imen al e alua ion o demons a e he bene i s o OWA-FRPS o e o he PS me hods. Table 3. Euclidean dis ance be ween he ins ances x1x2x3x4x5x6x7x8x9x10 x10.000 0.082 0.775 0.707 0.129 0.129 0.683 0.592 0.294 0.208 x20.082 0.000 0.726 0.712 0.058 0.173 0.707 0.580 0.258 0.208 x30.775 0.726 0.000 0.548 0.695 0.785 0.673 0.420 0.668 0.819 x40.707 0.712 0.548 0.000 0.695 0.645 0.183 0.465 0.622 0.843 x50.129 0.058 0.695 0.695 0.000 0.183 0.705 0.583 0.208 0.258 x60.129 0.173 0.785 0.645 0.183 0.000 0.624 0.622 0.238 0.337 x70.683 0.707 0.673 0.183 0.705 0.624 0.000 0.465 0.668 0.810 x80.592 0.580 0.420 0.465 0.583 0.622 0.465 0.000 0.645 0.606 x90.294 0.258 0.668 0.622 0.208 0.238 0.668 0.645 0.000 0.465 x10 0.208 0.208 0.819 0.843 0.258 0.337 0.810 0.606 0.465 0.000 3.1 Expe imen al Se -up We use 28 da ase s om he KEEL da ase eposi o y5. The cha ac e is ics o hese da ase s a e lis ed in Table 4. As ou main ocus is o imp o e he accu acy o NN, we compa e OWA-FRPS wi h 12 PS algo i hms ha a e mos accu a e acco ding o he s udy pe o med in [2]. Addi ionally, we also compa e OWA- FRPS o FRIS [13] wi h pa ame e alue α= 10. In Table 5, we gi e an o e iew o he algo i hms we conside wi h e e ences o he li e a u e. No e ha we use h ee e sions o he new OWA-FRPS algo i hm, depending on which measu e is used o ank he ins ances. Fo each da ase and PS me hod, we ca y ou he ollowing 10 old c oss ali- da ion p ocedu e. Fo each old, we apply he PS me hod o he emaining olds ( he ain da a) and hen le NN ind he nea es neighbo s o he es ins ances in his educed aining se . We epo he a e age classi ica ion accu acy, e- duc ion and unning ime o e he 10 olds. 3.2 Resul s In Table 6, we show he a e age accu acy, educ ion ( he pe cen age o emo ed ins ances) and unning ime (in seconds) o e all da ase s. Fi s , we no e ha on a e age, he OWA-FRPS-LU algo i hm is mo e accu a e han he o he e - sions, which shows ha bo h he lowe and uppe app oxima ion con ibu e o he quali y assessmen o he ins ances. All OWA-FRPS algo i hms ou pe o m he s a e-o - he-a PS algo i hms. F om now on, we con inue he analysis wi h OWA-FRPS-LU, o which we simply e e o as OWA-FRPS. To es i he im- p o emen is signi ican , we ca y ou he s a is ical F iedman es and Holm pos hoc p ocedu e [21]. The F iedman anks and he adjus ed p- alues o he Holm pos hoc p ocedu e a e lis ed in Table 7. The OWA-FRPS algo i hm has he bes (i.e. lowes ) ank. The low adjus ed p- alues con i m ha OWA-FRPS is signi ican ly mo e accu a e han he s a e-o - he-a PS algo i hms. 5www.keel.es Table 4. Da ase s used in he expe imen al e alua ion wi h hei numbe o ins ances (#Ins .), numbe o ea u es (#Fea .) and numbe o classes (#Cl.). Name #Ins . #Fea . #Cl. Name #Ins . #Fea . #Cl. appendici is 106 7 2 house o es 232 16 2 aus alian 690 14 2 i is 150 4 3 au omobile 150 25 6 led7digi 500 7 10 balance 625 4 3 lymphog aphy 148 18 4 bands 365 19 2 mammog aphic 830 5 2 b eas 277 9 2 new hy oid 215 5 3 bupa 345 6 2 pima 768 8 2 c x 653 15 2 sahea 462 9 2 de ma ology 358 34 6 sona 208 60 2 ecoli 336 7 8 ehicle 846 18 4 glass 214 9 7 owel 990 13 11 habe man 306 3 2 wine 178 13 3 hayes o h 160 4 3 wisconsin 683 9 2 hea 270 13 2 zoo 101 16 7 Table 5. O e iew o he algo i hms e alua ed in he expe imen al s udy. Name Desc ip ion Re e ence AllKNN NN based il e me hod [9] CHC E olu iona y based w appe me hod [3] GGA E olu iona y based w appe me hod [4, 5] HMNEI Hi and miss ne wo k based il e me hod [16] MENN NN based il e me hod [10] ModelCS T ee-based il e me hod [17] MSS Spa ial-based il e me hod [18] POP Spa ial-based il e me hod [19] RMHC Random mu a ion hill climbing w appe me hod [7] RNG G aph based w appe me hod [8] RNN NN based il e me hod [20] SSMA E olu iona y w appe me hod [6] FRIS Fuzzy ough based il e me hod [13] OWA-FRPS-LU New OWA-FRPS me hod based on he quali y measu e ha akes in o accoun bo h he lowe and uppe app oxima ion - OWA-FRPS-L New OWA-FRPS me hod based on he quali y measu e ha akes in o accoun he lowe app ox- ima ion - OWA-FRPS-U New OWA-FRPS me hod based on he quali y measu e ha akes in o accoun he uppe app ox- ima ion - The educ ion a e o he OWA-FRPS algo i hms is abou 30 pe cen , which is no as high as some o he e olu iona y PS me hods, bu as he ocus o ou me hod is o imp o e he accu acy a he han educing he s o age needs, his esul is o less impo ance. The unning ime is o mo e in e es o us. OWA-FRPS is slowe han 6 o he me hods, bu hese me hods ha e conside ably lowe accu acy a es. The un- ning ime o OWA-FRPS is sho e han he unning imes o he mos accu a e PS me hods, so al hough OWA-FRPS is a w appe and ob ains excellen accu- acy esul s, i does no come wi h he ex a compu a ional cos ha w appe PS me hods ypically ha e. Table 6. A e age esul s o he PS me hods a e aged o e all da ase s, o de ed ac- co ding o pe o mance. Reduc ion is he a io o emo ed ins ances, unning ime is gi en in seconds. Accu acy Reduc ion Running Time OWA-FRPS-LU 0.8087 CHC 0.9681 POP 0.0083 OWA-FRPS-L 0.8053 GGA 0.9391 MSS 0.0297 OWA-FRPS-U 0.7948 SSMA 0.9356 ModelCS 0.0306 RNG 0.7901 RNN 0.9111 MENN 0.0474 CHC 0.7893 RMHC 0.9015 FRIS 0.0576 ModelCS 0.7892 HMNEI 0.5383 AllKNN 0.0580 GGA 0.7863 MENN 0.4723 HMNEI 0.0714 AllKNN 0.7837 MSS 0.4632 OWA-FRPS-U 0.1834 SSMA 0.7828 OWA-FRPS-U 0.3462 OWA-FRPS-L 0.1880 FRIS 0.7808 AllKNN 0.3377 OWA-FRPS-LU 0.2031 RMHC 0.7799 OWA-FRPS-L 0.3247 RNG 2.6473 HMNEI 0.7785 OWA-FRPS-LU 0.2766 RNN 6.3661 POP 0.7741 RNG 0.2323 SSMA 14.9963 MENN 0.7705 ModelCS 0.1152 CHC 16.3427 MSS 0.7674 FRIS 0.0799 RMHC 18.2093 RNN 0.7614 POP 0.0484 GGA 42.9252 4 Conclusion and Fu u e Wo k In his pape , we p oposed a new PS me hod based on he OWA uzzy ough se model, called OWA-FRPS. In o de o selec a subse o ins ances om he aining se ha imp o es he classi ica ion o he NN classi ie , OWA-FRPS anks he ins ances acco ding o a OWA uzzy ough measu e, and hen au- oma ically selec s a sui able h eshold o selec he inal subse o ins ances. An expe imen al e alua ion on se e al da ase s shows ha ou me hod achie es accu acy a es ha a e be e han hose o s a e-o - he-a PS me hods, and mo eo e , OWA-FRPS is conside ably as e . As u u e di ec ions, we would like o expand he use o OWA-FRPS o o he