On the evolutionary weighting of neighbours and features in the k-nearest neighbour rule

Neurocomputing 326–327 (2019) 54–60 Contents lists available at ScienceDirect Neurocomputing journal homepage: www.elsevier.com/locate/neucom On the evolutionary weighting of neighbours and features in the k-nearest neighbour rule Daniel Mateos-García ∗, Jorge García-Gutiérrez, José C. Riquelme-Santos Department of Computer Science, University of Seville Avda. Reina Mercedes S/N, Seville 41012 Spain a r t i c l e i n f o Article history: Received 2 December 2015 Revised 15 June 2016 Accepted 1 August 2016 Available online 12 September 2017 Keywords: Evolutionary computation Neighbours weighting Feature weighting a b s t r a c t This paper presents an evolutionary method for modifying the behaviour of the k-Nearest-Neighbour classifier (kNN) called Simultaneous Weighting of Attributes and Neighbours (SWAN). Unlike other weighting methods, SWAN presents the ability of adjusting the contribution of the neighbours and the significance of the features of the data. The optimization process focuses on the search of two real-valued vectors. One of them represents the votes of neighbours, and the other one represents the weight of each feature. The synergy between the two sets of weights found in the optimization process helps to improve significantly, the classification accuracy. The results on 35 datasets from the UCI repository suggest that SWAN statistically outperforms the other weighted kNN methods. © 2017 Elsevier B.V. All rights reserved. 1. Introduction Weighting is a common technique used to optimize supervised learning [1,2] . A proper fit of weights in the training step may lead to an improvement in the accuracy of a model. Artificial Neural Networks (ANNs) and Support Vector Machines (SVMs) could be the most evident examples of using weights in learning models. However, weighting is also commonly applied to other supervised learning techniques such as the k-Nearest-Neighbour (kNN) classifier [3] . Most of the proposals on weighting methods were developed for feature or instance selection (the latter have also been known as prototype selection [4] ). For instance, Raymer et al. [5] performed a feature selection through a KNN-based genetic algorithm that optimised a weighting vector. In a later work, they provided an improved hybrid evolutionary algorithm which is based on the Bayesian discriminant function [6] . A similar method using tabu search was equally developed by Tahir et al. [7] . In recent time, many scholars have focused on the techniques that carried out both feature and instance selection which produced better results at the expense of increasing execution time [8] . kNN can be significantly useful to weighting techniques [9] . For that reason, Paredes and Vidal [10] used different similarity functions optimized by weighting. A weight by each feature and instance on training data was considered, resulting in a non-viable ∗Corresponding author. E-mail addresses: [email protected] (D. Mateos-García), jorgar[email protected] (J. García- Gutiérrez), riq[email protected] (J.C. Riquelme-Santos). number of parameters in the optimization process in a first approximation. Then, the authors presented three types of reduction, namely: (1) a weight by class and feature (label dependency), (2) a weight by prototype (prototype dependency) and (3) a combination of the previous ones i.e., 1 and 2. The optimization process was performed by descendant gradient. Additionally, Mateos et al. [11] have recently provided an evolutionary algorithm to find a matrix of weights (a weight by feature and label) beside an optimum number of neighbours in order to better explode label-dependency. A similar idea is expressed in Yoon and Friel [12] where the authors tried to enhance kNN performance by explicitly modelling uncertainty in the classification of each feature vector (regardless label-dependence) and the optimal number of neighbours. Weighting has also been applied to modify neighbours votes in kNN. Hence, the distance-weighted kNN rule (WKNN) was proposed by Dudani [13] and has been known for long. WKNN weighted the votes of the k nearest neighbours ( w i ) according to Eq. (1) where d i is the distance to the ith nearest neighbour (and d 1 to the nearest) regarding an instance to be classified. A similar version using a uniform weighting (UWKNN) was also proposed. In UWKNN, each weight was inversely proportional to the position among the neighbours (i.e., w u i = 1 /i ). More recently, Gou et al. [14] have also investigated both techniques working together as a new kNN version called Dual-Weighted kNN (DWKNN), where each weight was calculated according to Eq. (2) . A later work [15] offered another version of DWKNN where the calculation of the weights was improved according to Eq. (3) . One of the latest Works carried out on the use of new distances in kNN can be seen in Jiao et al. [16] where a class-conditional weighted distance metric http://dx.doi.org/10.1016/j.neucom.2016.08.159 0925-2312/© 2017 Elsevier B.V. All rights reserved. D. Mateos-García et al. / Neurocomputing 326–327 (2019) 54–60 55 was presented beside a multi-hypothesis nearest-neighbour proposal based on that metric. w w i = ⎧ ⎨ ⎩ (d k −d i ) (d k −d 1 ) if d i  = d 1 1 if d i = d 1 (1) w dw 1 i = w w i ∗w u i (2) w dw 2 i = ⎧ ⎨ ⎩ (d k −d i ) (d k −d 1 ) ∗(d k + d 1 ) (d k + d i ) if d i  = d 1 1 if d i = d 1 (3) García-Gutiérrez et al. [17] present another point of view where the authors tried to modulate the influence of the neighbourhood by weights obtained after an evolutionary optimization. More recently, a proposal to improve the original datasets with the combination of the use of individual neighbour structures, to develop new neighbourhood representations was also shown [18] . Other researchers have worked on the locally linear reconstruction of kNN which provides a principled and k-insensitive way to determine the weights of kNN learning [19] . Lately, there has been an increase in the interest in kNN and its performance on Big Data [20] . In this novel context, weighted neighbours can play an important role as can be seen in Xia et al. [21] where a weighted model based on Map-Reduce and called Spatial-Temporal Weighted K-Nearest Neighbour (STW-KNN) was proposed to improve the short-term traffic flow forecasting. Although, there are already a large number of literature on feature and voting system weighting (weighted distances could be included in the category of voting system weighting), yet there exists no proposal up till now (to the best of our knowledge) about a strategy to optimize both the voting system of neighbours and feature selection. Hence, the hypothesis for this work was that, regarding the issue of the improvement of kNN, ”the sum is greater tan the parts” and therefore, we proposed an evolutionary method to improve the kNN rule by optimizing the contribution of the neighbours and the importance of each feature simultaneously. Then, we statistically compared its performance with that of a classic kNN and other weighted variants tested on 35 UCI datasets [22] . The remaining part of this study is organized as follows. Section 2 presents the elements of the evolutionary algorithm designed to calculate the contribution of the k nearest neighbours and the effect of every feature. The results and a number of statistical tests are specified in Section 3 . Finally, Section 4 presents the conclusions and future work. 2. Method In this section, we describe our weighting optimization method called Simultaneous Weighting of Attributes and Neighbours (SWAN). The purpose of this work and how the weighting vectors from the learning process are used have been presented in Section 2.1 . While Section 2.2 exposes the optimization algorithm in detail. 2.1. Purpose and functionality As previously described, the aim of our work is to find a set of weights to optimize the influence of every neighbour when they vote, beside the importance of every feature. Unlike common feature weighting in the literature, ours is conditioned with a weighting voting (two vectors are optimized together, the feature weights and the weights of neighbours) fusing the synergistic ideas shown in previous work [11,17] (although label-dependency was not included in order to avoid performance issues since a much higher number of parameters would have to be optimized). As regards the contribution of the neighbourhood, most of the studies focus on the distance between instances. This means that the nearest neighbour instances are ”heavier” than the furthest ones and therefore, their influence is greater in the final voting. However, in our case, the weights are calculated by an evolutionary algorithm regardless the distance. Obtaining a real-valued vector could transform the influence of every neighbour irrespective of the class to predict in the classification step. This means that a vote of a labelled neighbour is a real value instead of the typical value of 1. An unlabelled instance is then labelled according to Eq. (5) . To show the learning process, we assume that the set of classes (or labels) is represented by the natural numbers from 1 to b , with b being the number of the labels. Therefore, let D = { (e, l) | e ∈ R f and l ∈ { 1 , 2 , . . . , b}} be the dataset under study with f being the number of features and b the number of labels. Let label be an application that assigns to every element e , the class to which it belongs to. Let’s suppose that D is divided in the sets TR and TS with each of them being the training and the testing set, respectively, so that D = T R ∪ T Sand T R ∩ T S = ∅ . In this manner, the instances of TS (testing set) will be used to evaluate the fitness of SWAN and so, they are not considered for the weights calculation. As will be detailed in Section 2.2 , we obtain two vectors v = (v 1 , v 2 , . . . , v k ) and ω = (ω 1 , ω 2 , . . . , ω f ) from the instances of TR exclusively. Let H be a function that transforms every feature of an instance x ∈ D according to a set of weights ω. To classify the instance y from TS , four steps are accomplished: • let x  ∈ TR  be the instances resulting from transform every x ∈ TR according to Eq. (4) • let y  be the instance resulting from transform y according to H ( y, ω) • calculate the k nearest instances to y  from TR  • if x  i , i : 1 ..k is each neighbour from previous step, the assigned label to the instance y  is given by Eq. (5) H(x, ω) = x  = (x 1 ∗ω 1 , x 2 ∗ω 2 , ..., x f ∗ω f ) (4) l abel (y  , v ) = arg max l∈{ 1 ..b} k  i =1 v i δ(l , l abel (x  i )) where δ(l , l abel (x  i )) = 1 if l abel (x  i ) = l 0 otherwise (5) 2.2. Evolutionary optimization This subsection details the search algorithm to calculate the optimum contribution of every nearest neighbours and features. As mentioned above, this task is done by an evolutionary algorithm and therefore, it is necessary to define its main features i.e., individual encoding, genetic operators, fitness function and generational replacement policy. 2.2.1. Individual encoding The population for the study consisted of a set of individuals that are represented by two real-valued vectors. The first one symbolizes the relative contribution of every neighbour in the voting stage of the kNN rule, and the second one represents the importance of every data feature (see Fig. 1 ). Although the weights of the votes are independent of the distances of the neighbours, yet the closest neighbour is usually the most important i.e., v 1 ≥v 2 ≥ ···≥v k , and as a result, we constrained the encoding of each individual. Regarding the initial population, the votes are k sorted values between 0 and 1. To include the classic kNN, we populate with 56 D. Mateos-García et al. / Neurocomputing 326–327 (2019) 54–60 2.5 2.0 …0.1 12…k 1.6 0.0 2.3 1.1 3.0 … 2.7 12345…f 2 . 0 … Fig. 1. Individual. Fig. 2. BLX- α. several vectors with the first k values set to 1 and the remaining set to 0 in the initial population e.g., (1 . 0 , 0 . 0 , . . . , 0 . 0) for k = 1, (1 . 0 , 1 . 0 , . . . , 0 . 0) for k = 2, and so on. Note that, during the evolutionary process, the maximum value of 1 for a weight may be surpassed to highlight the importance of a concrete neighbour regarding the rest. The vector of weights for the features does not have any constraint. It consists of random values between 0 and 1 in the initial population. The maximum value of 1 can also be exceeded. If a local minimum could be reached during the evolution process, i.e. the error function returns the same value n times, all individual, except the best are substituted by a new initial population. 2.2.2. Crossover and mutation After a trial-and-error procedure, we selected two concrete operators for crossover and mutation as trade-off between simplicity and quality of the results in comparison with the rest of candidates: BLX- αcrossover and generation-dependent mutation. The main goal of the crossover operator is building a new individual ( offspring ) from the genotypic features of two parents ( parent 1 and parent 2). Considering that there is a constraint in the order of the genes in the voting, the crossover operator for the vector of votes in the i-th gene has been described as follows (see Fig. 3 ). offspring (i ) = BLX −αif i = 1 (max −min ) ∗γ+ min otherwise (6) where - BLX- αis the crossover operator defined in Eshelman and Schaffer [23] and calculated from parent 1( i ) and parent 2( i ) - γis a random value between 0 and 1 - max = offspring ( i −1 ) - min = minimum ( parent1(i) , parent2(i), offspring(i-1)) The weights for the features do not have any constraint in their order, so the crossover operator can be simpler (see Fig. 2 ): offspring (i ) = BLX −αfrom parent1(i ) and parent2(i ) (7) Regarding the mutation operator, the ith gene of the individual could change according to Eq. (8) for the votes. δis a random value in the interval [0, z ] with z being 1 initially. To find a better fit, the z value is decreased every ten generations inversely proportional to the current generation. For example, for an evolution of 100 generations, z is initially 1 and decreases by a factor of 0.1 every ten generations. Therefore, in the first ten generations z = 1 , in the next ten z = 0 . 9 , then z = 0 . 8 and so on. The idea here is to 1.6 1.4 … 0.0 12…k 2.5 2.0 … 0.1 1.51.0…0.0 [min(1.6,2.0,1.0) <= 1.4 <= 1.6] 12…k 1.6=BLX- (2.5,1.5) 1 . 6 1 . 4 … 0 . 0 2. 0 … 0.1 1 . 5 1. 0 … 1.6=B LX- (2.5,1.5) ) Fig. 3. Crossover of neighbours. 1: errorFunction(k,ω,v,TR):error 2: error =0 3: for 1tomdo 4: Divide TR randomly in nsubsets: TR =S1∪S2... ∪Sn 5: for i=1tondo 6: wTrain =buildModel(TR−Si,ω) 7: error =error +evaluate(k, wTrain, Si,ω,v) 8: end for 9: end for 10: error =error/(m∗n) 11: return error 12: buildModel(T rain, ω):wTrain 13: wTrain =∅ 14: for each xin Train do 15: x=H(x, ω) according to the Eq. 4 16: add xto wTrain 17: end for 18: return wTrain 19: evaluate(k, Train, T est, ω, v):error 20: error =0 21: for each yin Test do 22: y=H(y,ω) according to the Eq. 4 23: predLabel =classify(y,Train,k,v) according to the Eq. 5 24: if predLabel =label(y)then 25: error=error+1 26: end if 27: end for 28: error =error/ |Test| 29: return error Fig. 4. Error function. reduce the influence of the mutation operation since the individuals are closer to the end of the evolution. This is as a result of the fact the mutation operator for feature weights is free of constraints, and its implementation is also simpler (Eq. (9)) . indi v  (i ) = ⎧ ⎪ ⎨ ⎪ ⎩ indi v (i ) + indi v (i ) ∗δif i = 1 indi v (i ) −indi v (i ) ∗δif i = k (indi v (i −1) −indi v (i + 1)) ∗δ + indi v (i + 1) otherwise (8) indi v  (i ) = indi v (i ) ±indi v (i ) ∗δ(9) 2.2.3. Error function The evolutionary algorithm uses TR ⊂D exclusively to obtain the contributions of the neighbours in the training step. The fitness function is based on the cross-validation error rate by using a kNN- based classifier and the weighting vectors. The Fig. 4 shows the error calculation of m ×n cross validations, where m stands for the number of iterations of the validation D. Mateos-García et al. / Neurocomputing 326–327 (2019) 54–60 57 process (line 3) and n does for the number of partitions of training data TR (line 4). Set TR is therefore divided in subsets S 1 , S 2 ... S n for each validation. Every subset S i is evaluated through a classification process by using T R −S i as a training set. This evaluation is driven by the function evaluation which we will be described later. The classification error on every S i is accumulated by an error in every validation (line 7). Finally, the error value is the mean of all validations (line 10). The method buildModel receives two parameters: the training data and the feature weighting (line 12). It creates and returns a weighted training set with the use of Eq. (4) (lines 13–18). When instances are transformed, the function evaluate carries out the testing. The result of the evaluate function is the error rate on S i taking wTrain as reference to calculate the neighbours (line 7). The input parameters of evaluate are the number of neighbours, the transformed training data, the current testing set and the weighting vectors w and v (line 19). And so, every single instance y from the set used to measure the error (line 21), is transformed into y  according to Eq. (4) (line 22). The classify function returns the majority label according to the relative contribution of each neighbour expressed by the vector v and applying Eq. (5) (line 23). If a returned label does not correspond with the true label of a testing instance, the error is increased by 1 (line 25). Finally, the resulting error is normalized according to the size of the set used as testing the data (line 28). The value returned by evaluation is then a real number between 0 (all instances are well-classified) and 1 (all instances are misclassified). 2.2.4. Generational policy As regards the transition between generations, we chose an elitist design, where the best individual is part of the next offspring (no mutation is applied). If N is the number of individuals, the remaining population is built as follows: C −1 individuals are created by cloning the best individual from the previous generation. The next N −Cindividuals result from the crossover operation. The selection of the individuals to cross is carried out by the tournament method. All individuals except the first one are affected by the mutation operator with a probability of p . 3. Results In the experiments, we have used Java language and 35 datasets from the repository UCI [22] with different types of features and number of classes (see Table 1 ). All the data were preprocessed with the same techniques i.e., binarization of nominal features, replacement of missing values and normalization to avoid the Hughes effect. Through a trial-and-error process, the evolutionary algorithm was setup with a population of 100 individuals, 200 generations, 10% of elitism and a mutation probability of 0.1. Regarding the parameters α(crossover), and k (number of neighbours) their values were set at 0.5 and 5, respectively. Finally, the z value in mutation operator is decreasing at a rate of 0.1 in every generation. To measure the precision of our approach, we established a comparison among IBk (implementation of kNN in the framework WEKA [24] )), EVoN [17] , WKNN, UKNN, DWKNNv1 [14] and DWKNNv2 [15] . All algorithms have been tested with k = 1, k = 3 and k = 5, with the latter showing the best performance. Table 2 shows the mean accuracy obtained by the analyzed algorithms using 10- fold cross-validation with 5 different seeds (50 runs in the aggregate). We can see that the performance of our algorithm is the best Table 1 Datasets from UCI. #dataset #instances #features #classes %minority %majority Anneal 898 38 6 28 72 Arrhythmia 452 279 16 0 36 Audiology 226 69 24 42 58 Australian 690 14 2 8 46 Autos 205 25 7 34 66 Balance-scale 625 4 3 0 33 Breast-cancer 286 9 2 0 76 Bridges_version1 105 12 6 44 56 Bridges_version2 105 12 6 10 10 Car 1728 6 4 42 58 cmc 1473 9 3 0 54 Colic 368 22 2 10 42 Credit-a 690 15 2 37 63 Credit-g 10 0 0 20 2 36 64 Dermatology 366 34 6 0 64 Diabetes 768 8 2 4 70 Ecoli 336 7 8 10 10 Flags 194 29 8 50 50 Glass 214 9 7 23 43 Haberman 306 3 2 10 10 Hayes-roth_train 132 4 4 10 10 Heart-c 303 13 5 5 31 Heart-h 294 13 5 1 36 Heart-statlog 270 13 2 44 56 Ionosphere 351 34 2 0 54 Labor 57 16 2 0 25 Liver-disorders 345 6 2 10 10 Lung-cancer 32 56 2 35 65 Lymph 148 18 4 10 42 mfeat-karhunen 20 0 0 64 10 1 55 mfeat-morphological 20 0 0 6 10 10 10 mfeat-zernike 20 0 0 47 10 26 74 Monks-problems-1 124 6 2 0 92 Monks-problems-2 169 6 2 9 24 Monks-problems-3 122 6 2 48 52 58 D. Mateos-García et al. / Neurocomputing 326–327 (2019) 54–60 Table 2 Accuracy of every studied algorithm throughout 35 datasets from UCI. Data/classifier SWAN EVoN kNN DWKNN1 DWKNN2 UWKNN WKNN Anneal 98,931 98,976 96,971 99,065 98,441 98,241 98,151 Arrhythmia 58,761 58,142 58,451 53,85 55,796 56,903 55,929 Audiology 71,327 63,186 60,708 66,726 68,142 68,584 6823 Australian 85,478 83,478 83,478 80,116 81,478 8313 81,623 Autos 72,098 69,659 57,756 73,171 73,463 72,683 71,024 Balance-scale 83,232 82,592 87,872 81,824 86,592 82,528 86,592 Breast-cancer 71,608 72,238 73,077 68,811 70,839 7021 70,839 Bridges_version1 59,813 59,626 58,318 60,748 61,308 61,121 61,308 Bridges_version2 63,551 62,617 59,626 59,813 59,813 61,869 59,813 Car 95,602 87,986 92,847 88,16 92,847 8816 92,847 cmc 48,364 44,616 45,418 44,073 4391 44,929 44,318 Colic 80,435 79,837 79,511 72,228 75 78,261 75,109 Credit-a 8542 83,826 83,768 79,594 81,362 83,275 81,449 Credit-g 724 728 7276 7104 72 73,04 72 Dermatology 96,448 96,339 96,612 94,262 95,355 95,191 95,355 Diabetes 7349 73,906 73,984 70,417 72,474 72,708 72,396 Ecoli 85,417 8619 8625 8131 83,214 84,464 8369 Flags 5567 54,021 53,196 55,773 5732 58,144 5732 Glass 72,617 68,505 65,327 68,411 69,626 68,037 69,907 Haberman 69,281 70,327 7098 66,863 70,131 68,954 7085 Hayes-roth_train 83,182 52,727 26,515 76,667 6803 67,879 6803 Heart-c 80,594 81,848 82,112 76,832 79,802 80,264 79,736 Heart-h 80,816 78,707 78,707 76,327 78,571 78,503 78,707 Heart-statlog 78,074 79,704 79,926 74,593 76,222 78,4 4 4 76,222 Ionosphere 88,604 88,376 84,786 86,724 86,895 85,869 86,952 Labor 87,018 8386 81,053 86,316 84,561 85,263 84,912 Liver-disorders 63,478 62,145 61,043 62,493 6342 6371 63,362 Lung-cancer 73,125 73,75 76,875 6375 65,625 7375 65,625 Lymph 8473 81,081 78,378 82,027 83,378 85,135 83,649 mfeat-karhunen 9685 9677 9614 9636 9689 9687 9688 mfeat-morphological 71 7113 7088 6563 6748 6812 6785 mfeat-zernike 8101 8053 8052 789 7894 7939 7896 Monks-problems-1_train 43,548 45 49,194 37,258 33,387 40 33,387 Monks-problems-2_train 5574 51,716 53,136 36,095 36,805 38,107 36,805 Monks-problems-3_train 41,311 37,377 40,164 30,164 30,492 32,131 30,492 in 16 out of the 35 datasets, and the second one in 4 out of the remaining experiments. Additionally, as we can see in Table 1 , there are imbalanced datasets. The columns ”%minority” and ”%majority” present the percentage of instances under the minority label and majority label, respectively. However, although our evolutionary weighting methods optimize the accuracy, we can observe in Table 3 an improvement of kappa statistics too. In fact, the Pearson’s r between the accuracies and kappa values is 0.81, and so, it seems that improving the global accuracy with evolutionary weighting causes an improvement of the classification performance by class. In spite of the overall good performance of our method in direct comparison with the rest, the results must be statistically validated. For this reason, we carried out a non-parametric Friedman’s test and a Holm’s post-hoc procedure. The reason for using nonparametric tests lies in the high vulnerability of the necessary conditions to apply parametric tests, specially for the sphericity condition [25,26] . The first step for the Friedman’s test is the calculation of the average rankings reached by each technique (a ranking of 1 is the best). Table 4 shows the rankings. After the Friedman’s test was applied, the resulting statistic was 48.58, distributed according to a chi-square with 6 degrees of freedom. The p-value for Friedman was around 9.072E-9, and so, the null hypothesis (no statistical difference among the compared techniques) could be rejected with an α= 0 . 05 . The Holm’s Post-hoc Procedure allows us to compare a control algorithm (in this case SWAN, the best approach candidate) with the rest avoiding problems related to family-wise error [26] . The results of the procedure can be seen in Table 5 (Friedman’s statistic, p-value and adjusted αfor Holm’s procedure). In this case, every test rejected the hypothesis of no pairwise difference (pvalues were lower than adjusted α), so we could state that our algorithm was significantly better than its competitors from a statistical point of view. 4. Conclusions This work presented a method to improve the kNN rule. We unified two classic paradigms of weighting by evolutionary computation. On the one hand, we adjusted the contribution of every neighbour used in the classification step, but nevertheless, the significance of the data features was modified simultaneously in order to achieve a better result in the recognition of new instances. In spite of the complexity of the solution to optimize and the increase of the search space in comparison with single-vector based methods, the experiments showed a successful behaviour of our approach. In future researches, we will focus on adapt evolutionary weighting algorithms, to distributed programming models such as MapReduce. By so doing, it could be possible to speed up the performance of our methods on massive data. Moreover, we will explore whether SWAN could confirm similar suitability for regression, where a change of the voting system could lead to higher changes in the model output. The use of deep learning techniques is also a target to reach. Therefore, preprocessing training data in a hierarchical structure of multiple layers could produce good results in our research projects on specific domains, such as image recognition or natural language processing. D. Mateos-García et al. / Neurocomputing 326–327 (2019) 54–60 59 Table 3 Kappa of every studied algorithm (Cohen’s kappa for binary classification and Fleiss’s kappa for the multiclass case). Datas/classifier SWAN EVoN KNN DWKNN1 DWKNN2 UWKNN WKNN Anneal 1 1 097 1 097 1 0955 Arrhythmia 0383 04 0156 0251 0235 0248 0235 Audiology 0744 0684 0707 0665 0642 0738 0666 Australian 0638 0589 0589 0626 0638 0609 0638 autos 0662 0694 0531 0664 0662 0694 0662 Balance-scale 0702 0728 0783 0704 0742 0717 0742 Breast-cancer 0185 0131 0288 0225 0178 0385 0178 Bridges_version1 0414 0329 0292 042 042 0329 042 Bridges_version2 0476 0317 0265 0434 0476 0476 0476 Car 0878 0718 074 0704 074 0704 074 cmc 0128 0132 0132 0104 007 0102 0081 Colic 0703 0631 0596 0517 0566 0676 0566 Credit-a 0762 0702 0687 0629 0702 0702 0702 Credit-g 024 0257 0257 0186 0252 0246 0284 Dermatology 0966 0983 0983 0949 0949 0949 0949 Diabetes 029 029 029 0202 0242 0225 029 Ecoli 0825 0846 0823 0716 0782 0803 0782 Flags 0415 0286 0356 0359 0359 0366 0359 Glass 07 0666 0534 0607 0666 0635 0666 Haberman 0 0 0 0 07 0242 027 0241 0323 Hayes-roth_train 0715 041 0 065 0275 0594 0275 Heart-c 0665 0629 0629 0656 0633 0696 0633 Heart-h 0538 04 04 04 0454 0486 0486 Heart-statlog 0662 0662 0662 0439 0441 0588 0515 Ionosphere 0772 0772 0685 0657 0685 0627 0685 Labor 0799 0307 0307 0571 0571 0571 0571 Liver-disorders 0254 0254 0254 0047 0079 0075 0106 Lung-cancer 0086 03 0086 0086 03 03 0695 Lymph 0448 0377 0514 0462 051 0578 051 mfeat-karhunen 0938 0938 0941 0947 0952 0955 0952 mfeat-morphological 0705 0705 0657 0626 0626 0659 064 mfeat-zernike 0766 0771 0768 0757 0763 0768 0763 Monks-problems-1 0 009 0029 0 0 0 0 Monks-problems-2 0242 0074 0033 0 0 0 0 Monks-problems-3 0 0 0 0 0 0 0 Table 4 Average ranking reached by every compared technique. Method Ranking SWAN 2.314 EvoN 3.514 kNN 3.785 UWKNN 3.857 WKNN 4.285 DWKNN2 4.514 DWKNN1 5.728 Table 5 Results for Holm’s post-hoc procedure. Dataset z p Holm’s adjusted α DWKNNv1 6.612 3.798E-11 0.008 DWKNNv2 4.260 2.042E-5 0.010 WKNN 3.817 1.347E-4 0.012 UWKNN 2.987 0.003 0.016 kNN 2.849 0.004 0.025 EVoN 2.324 0.020 0.050 References [1] E. Corchado , M. Wozniak , A. Abraham , A. de Carvalho , V. Snásel , Recent trends in intelligent data analysis., Neurocomputing 126 (2014) 1–2 . [2] A. Abraham , Special issue: hybrid approaches for approximate reasoning., J. Intell. Fuzzy Syst. 23 (2–3) (2012) 41–42 . [3] D. Wettschereck , D.W. Aha , T. Mohri , A review and empirical evaluation of feature weighting methods for a class of lazy learning algorithms, Artif. Intell. Rev. 11 (1) (1997) 273–314 . [4] J.J. Valero-Mas , J. Calvo-Zaragoza , J.R. Rico-Juan , On the suitability of prototype selection methods for knn classification with distributed data, Neurocomputing 203 (2016) 150–160 . [5] M. Raymer , W. Punch , E. Goodman , L. Kuhn , A. Jain , Dimensionality reduction using genetic algorithms, IEEE Trans. Evolut. Comput. 4 (2) (20 0 0) 164–171 . [6] M. Raymer , T. Doom , L. Kuhn , W. Punch , Knowledge discovery in medical and biological datasets using a hybrid Bayes classifier/evolutionary algorithm, IEEE Transactions on Syst. Man Cybern. Part B: Cybern. 33 (5) (2003) 802–813 . [7] M.A . Tahir , A . Bouridane , F. Kurugollu , Simultaneous feature selection and feature weighting using hybrid tabu search/k-nearest neighbor classifier, Pattern Recognit. Lett. 28 (4) (2007) 438–446 . [8] J. Pérez-Rodríguez , A.G. A.-P. na , N. García-Pedrajas , Simultaneous instance and feature selection and weighting using evolutionary computation: Proposal and study, Appl. Soft Comput. 37 (2015) 416–443 . [9] J. Hocke , T. Martinetz , Maximum distance minimization for feature weighting, Pattern Recognit. Lett. 52 (2015) 48–52 . [10] R. Paredes , E. Vidal , Learning weighted metrics to minimize nearest-neigh- bor classification error, IEEE Trans. Pattern Anal. Mach. Intell. 28 (7) (2006) 110 0–1110 . [11] D. Mateos-García , J. García-Gutiérrez , J.C. Riquelme-Santos , On the evolutionary optimization of knn by label-dependent feature weighting, Pattern Recognit. Lett. 33 (16) (2012) 2232–2238 . [12] J.W. Yoon , N. Friel , Efficient model selection for probabilistic k nearest neighbour classification, Neurocomputing 149 (B) (2015) 1098–1108 . [13] S.A. Dudani , The distance-weighted k-nearest-neighbor rule, IEEE Trans. Syst. Man Cybern. SMC-6 (4) (1976) 325–327 . [14] J. Gou , T. Xiong , J. Kuang , A novel weighted voting for k-nearest neighbor rule, J. Comput. 6 (5) (2011) 833–840 . [15] J. Gou , L. Du , Y. Zhang , T. Xiong , A new distance-weighted k-nearest neighbor classifier, J. Inf. Comput. Sci. 9 (6) (2012) 1429–1436 . [16] L. Jiao , Q. Pan , X. Feng , Multi-hypothesis nearest-neighbor classifier based on class-conditional weighted distance metric, Neurocomputing 151 (2015) 1468–1476 . [17] J. García-Gutiérrez , D. Mateos-García , J.C. Riquelme-Santos , Improving the k-n- earest neighbour rule by an evolutionary voting approach, in: Proceedings of the 9th International Conference on Hybrid Artificial Intelligence Systems - Volume 8480, in: HAIS, Springer-Verlag New York, Inc., New York, NY, USA, 2014, pp. 296–305 . [18] X. Gao , T. Mu , M. Wang , Local voting based multi-view embedding, Neurocomputing 171 (2016) 901–909 . 60 D. Mateos-García et al. / Neurocomputing 326–327 (2019) 54–60 [19] S. kyung Lee , P. Kang , S. Cho , Probabilistic local reconstruction for k-nn regression and its application to virtual metrology in semiconductor manufacturing, Neurocomputing 131 (2014) 427–439 . [20] Z. Deng , X. Zhu , D. Cheng , M. Zong , S. Zhang , Efficient knn classification algorithm for big data, Neurocomputing 195 (2016) 143–148 . [21] D. Xia , B. Wang , H. Li , Y. Li , Z. Zhang , A distributed spatial-temporal weighted model on mapreduce for short-term traffic flow forecasting, Neurocomputing 179 (C) (2016) 246–263 . [22] M. Lichman, UCI machine learning repository, 2013, [ http://archive.ics.uci.edu/ ml ]. Irvine, CA: University of California, School of Information and Computer Science. [23] L.J. Eshelman , J.D. Schaffer , Real-coded genetic algorithms and interval-schemata, in: D.L. Whitley (Ed.), Foundation of Genetic Algorithms 2, Morgan Kaufmann, San Mateo, CA, 1993, pp. 187–202 . [24] M. Hall , E. Frank , G. Holmes , B. Pfahringer , P. Reutemann , I.H. Witten , The WEKA data mining software: an update, SIGKDD Explor. 11 (1) (2009) . [25] J. Demšar , Statistical comparisons of classifiers over multiple data sets, J. Mach. Learn. Res. 7 (2006) 1–30 . [26] S. García , F. Herrera , An extension on statistical comparisons of classifiers over multiple data sets for all pairwise comparisons, J. Mach. Learn. Res. 9 (2008) 2677–2694 . Daniel Mateos-García received the Ph.D. degree in Computer Engineering from the University of Seville, Spain, in 2013. Since 2003 he has been with the Department of Computer Science, University of Seville, where he is currently Assistant Professor. His primary areas of interest are machine Learning and evolutionary computation. Jorge García Gutiérrez received the Ph.D. degree in Computer Engineering from the University of Seville, Spain, in 2012. He has been working for the Department of Computer Science of the University of Seville since 2008 where he is currently Lecturer Professor. His primary areas of interest are machine learning techniques, big data, remote sensing, data fusion and evolutionary computation. José C. Riquelme received the M.Sc. degree in Mathematics and the Ph.D. degree in Computer Science from the University of Seville, Spain. Since 1987 he has been with the Department of Computer Science, University of Seville, where he is currently Full Professor. His primary areas of interest are data mining, machine learning techniques and evolutionary computation.