Merging subsets of attributes to improve a hybrid consistency-based filter: a case of study in product unit neural networks

CONNECTION SCIENCE, 2016 VOL. 28, NO. 3, 242–257 http://dx.doi.org/10.1080/09540091.2016.1149146 Merging subsets of attributes to improve a hybrid consistency-based filter: a case of study in product unit neural networks A. J. Tallón-Ballesterosa,J.C.Riquelme aand R. Ruizb aDepartment of Languages and Computer Systems. University of Seville, Seville, Spain; bArea of Computer Science, Pablo de Olavide University, Seville, Spain ABSTRACT This paper presents a quality enhancement of the selected features by a hybrid filter-based jointly on feature ranking and feature subset selection (FR-FSS) using a consistency-based measure via merging new features which are obtained applying other FR-FSS evaluated with a correlation metric. The goal is to overcome the accuracy of a neural network classifier containing product units as hidden nodes combined with a feature selection pre-processing step by means of a single consistency-based FR-FSS filter. Neural models are trained with a refined evolutionary programming approach called two-stage evolutionary algorithm. The experimentation has been carried out in eight complex classification problems, seven out of them from UCI (University of California at Irvine) repository and one real-world problem, with high test error rates (around 20%) with powerful classifiers such as 1-nearest neighbour or C4.5. Non-parametric statistical tests revealed that the new proposal significantly improves the accuracy of the neural models. ARTICLE HISTORY Received 23 May 2015 Accepted 24 January 2016 KEYWORDS Artificial neural networks; feature selection; classification; product units; filters; feature subset selection 1. Introduction A good number of approaches has been proposed to tackle supervised machine learning problems (Dougherty, 2012). Classifiers can be grouped in different categories such as decisiontrees,classifiersbasedonnearestneighbours,artificialneuralnetworks,rule-basedand Bayes classifiers. It is straightforward that feature selection (FS) is becoming an important and common activity in most of the current classification problems. Its motivation is treble: simplifying the classifier, improving the accuracy of the classifier; and reducing the dimensionality for the classifier. The last issue is particularly relevant in the context of classifiers based on artificial neural networks. Our goal is to improve the accuracy of the classification models obtained with product unit neural networks (PUNN) trained with a two-stage evolutionary algorithm. Recently, we have experimented with this classifier by means of a prior pre-processing step with FS (Tallón-Ballesteros, Hervás-Martínez, Riquelme, & Ruiz, 2013) reaching satisfactory results. CONTACT A. J. Tallón-Ballesteros [email protected] © 2016 Informa UK Limited, trading as Taylor & Francis Group CONNECTION SCIENCE 243 From this starting point we deepen into new possibilities to be applied to our neural classifier. This paper is organised as follows: Section 2describes the methodology including FS andevolutionaryproductunitneuralnetworks;Section3introducestheproposal; Section 4 details the experimental process; Section 5shows and analyses the results obtained; finally, Section 6states the concluding remarks. 2. Methodology 2.1. Feature selection Pattern recognition is inherently tied to an information reduction in order to extract the present knowledge inside data. Features can be useful, redundant or irrelevant. An irrelevant feature does not influence on the underlying structure of the data in any way. A redundant feature does not provide anything new in explaining that structure (Guyon & Elisseeff, 2003). FS can be defined as the problem of choosing a small subset of features that is ideally necessary and sufficient to describe the target concept. It can be formulated as a task of searching for an optimal subset of features from all available features (Dash & Liu, 1997). An optimal subset is relative to a certain evaluation function. Typically, an evaluation function tries to measure the discriminating ability of a feature or a subset to distinguish the different class labels. FS methods can be divided into two broad groups (filter Liu & Setiono, 1996and wrapper models) based on their use of an inductive algorithm in FS or not (Blum & Langley, 1997). More recently, Liu and Yu (2005) provides a common taxonomy where a large list of FS methods are categorised. Evaluation functions can be grouped into five categories: distance, information (or uncertainty), dependence, consistency and classifier error rate. Depending on the generation procedure, FS can be divided into individual feature ranking (FR) and feature subset selection (FSS) (Blum & Langley, 1997; Guyon & Elisseeff, 2003). FR measures the feature-class relevance, then rank features by their scores and select the top-ranked ones. These methods are widely used due to its simplicity, scalability, and good empirical success (Golub et al., 1999; Guyon & Elisseeff, 2003). However, FR is criticised because they can only capture the relevance of features to the target concept, but redundancy and basic interactions between features are not discovered, besides, the number of retained features is difficult to determine, a threshold is required. In contrast, FSS attempts to find a set of features with good performance. They integrate the metric to measure the feature-class relevance and the feature–feature interactions. Different algorithms address these issues distinctively. We found different search strategies, namely exhaustive,heuristic and random search, combined with several types of measures to form different algorithms (Dash, Liu, & Motoda, 2000). The time complexity is exponential in terms of data dimensionality for exhaustive search and quadratic for heuristic search. The complexity can be linear to the number of iterations in a random search, but experiments show that in order to find the best feature subset, the number of iterations required is usually at least quadratic to the number of features (Liu & Setiono, 1998). As well as the categorisation of the previous paragraph, a hybrid model was proposed to handle large data sets to take advantage of the above two approaches (FR, FSS). These methods (Ruiz, Riquelme, & Aguilar-Ruiz, 2006;Yu&Liu,2004) decouple relevance analysis 244 A. J. TALLÓN-BALLESTEROS ET AL. and redundancy analysis, and they have been proved to be more effective than ranking methods and more efficient than subset evaluation methods on many traditional highdimensional data sets. In the context of hybrid FS, BIRS (Best Incremental Ranked Subset) (Ruiz et al., 2006) was the proposed method to obtain relevant features and to remove redundancy. Figure 1overviews BIRS in the context of the FS taxonomy. On the one hand, the purpose of a feature subset algorithm is to identify relevant features according to a definition of relevance. However, the notion of relevance in machine learning has not yet been rigorously defined in common agreement (Bell & Wang, 2000). Kohavi and John (1997) include three disjointed categories of feature relevance: strong relevance, weak relevance and irrelevance. Bell and Wang (2000) make use of Information Theory concepts to define the entropic or variable relevance of a feature with respect to the class. Blum and Langley (1997) collect several relevance definitions. On the other hand, notions of feature redundancy are usually defined in terms of feature correlation.Itiswidelyacceptedthattwofeaturesareredundanttoeachotheriftheirvalues are completely correlated. There are two widely used types of measures for the correlation between two variables: linear and nonlinear. In the linear case, the Pearson correlation coefficient is used, and in the nonlinear case, many measures are based on the concept of entropy, or measure of the uncertainty of a random variable. However, it may not be as straightforward in determining feature redundancy when one is correlated with a set of features. Koller and Sahami (1996) apply a technique based on cross-entropy, named Markov blanket filtering, to eliminate redundant features. This idea was formalised using the notion of conditionally independent attribute, which can be defined by several approaches (Xing, Jordan, & Karp, 2001;Yu&Liu,2004). The minimal-redundancy-maximal-relevance (mRMR) (Peng, Long, & Ding, 2005) metric, which is based on mutual information (Yao, 2003), is also another interesting strategy that is very widespread. Between filter and wrapper model, there is an intermediate type of methods that are called embedded FS methods (Lal, Chapelle, Weston, & Elisseeff, 2006). It is particularly frequent in the context of support vector machine (SVM) classifier (Vapnik, 2013); a remarkable approach among them is the recursive feature elimination (RFE) (Guyon, Weston, Barnhill, &Vapnik,2002) selection method that uses SVM classifier to evaluate the goodness of the obtained subset removing at every step a single feature. Feature selection Measure of attribute evaluation Generation procedure Filter Wrapper Feature ranking (FR) Feature Subset Selection (FSS) + BIRS (Best Incremental Ranking Subset) Figure 1. BIRS within FS taxonomy. CONNECTION SCIENCE 245 The selection process in BIRS is composed of two stages. In stage one, features are evaluated individually, providing a ranking based on a criterion. In stage two, a feature subset evaluator is applied to a certain number of features in the previous ranking following a search strategy. BIRS can include any kind of evaluator in the two phases. In the current work, BIRS utilises as a subset evaluator consistency and correlation concepts, CNS (consistency-based measure (Liu & Setiono, 1996)) and CFS (Correlation-based Feature Selection (Hall, 2000)), respectively, at the second phase, and SOAP (Selecion Of Attributes by Projection (Ruiz, Riquelme, & Aguilar-Ruiz, 2002)) measure and the own subset evaluator at the first phase as a ranking evaluator in the case of CFS. To guide the search, CFS evaluates the quality of a feature subset taking into account the hypothesis that good feature subsets contain features highly correlated to the class. CNS is based on the consistency measure, which estimates, for a given subset of features, the number of sources that match all but their class labels. The inconsistency rate is then used to assess its quality. SOAP is a deterministic attribute selection criterion based on this basic principle: to count the label changes of examples projected onto each feature. If the attributes are in ascending order according to the number of label changes we will have a list that defines the priority of selection, from greater to smaller importance. The main advantages are its speed and simplicity in the evaluation of the attributes. BIRS considers that the relevance and the redundancy concepts are included in the following “incremental usefulness” (Caruana & Freitag, 1994) definition: Given a sample of data, an evaluation measure L, a feature space Fand a feature subset S(S⊆F), the feature Fiis incrementally useful to Lwith respect to Sif the evaluation of the hypothesis that Lproduces using the group of features {Fi}∪Sis better than the evaluation achieved using just the subset of features S. That is, if Fiis not incrementally useful to Lwith respect to S, then the evaluation value given the subset Sis equal or better than the subset evaluation result known {Fi}∪S.ItsuggeststhatFigives no information beyond what is already in S, therefore, Ficould be removed safely, or in this case, Fiwould not be added to S. However, since the computational complexity to determine all possible interactions between features is very high (mainly in high-dimensional domains), BIRS operates using a guided search over an ordered list of attributes. BIRS deals with the incremental ranked usefulness in order to devise an approach to explicitly identify relevant features and do not bear in mind redundant features. The idea is to choose the feature Fifrom a ranked list one by one in the following way: firstly, the features are ranked according to some evaluation measure (SOAP, CFS, CNS); and secondly, BIRS deals with the list of features once, crossing the ranking from the beginning to the last ranked feature. It is obtained the evaluation result using CFS or CNS with the first feature in thelist and it ismarked as selected. Again,it is obtained theresult, but now withthe first and second features. The second feature will be marked as selected depending on whether the evaluation obtained is statistically significant better. The process is repeated until the last feature on the ranked list is reached. Finally, the algorithm returns the best subset found, and it can be stated that it will not contain irrelevant or redundant features. Therefore, in the experiments, spBIRS indicates that SOAP will be used as an individual measure in the first part of BIRS and the indicated subset selection as a subset evaluator in the second part. In the same way, cnBIRS denotes that CNS evaluator will be used in both part of the BIRS algorithm. 246 A. J. TALLÓN-BALLESTEROS ET AL. 2.2. Evolutionary product unit neural networks We have considered a single-hidden-layer feed-forward network architecture, that is, a neural structure with one input layer, one hidden layer and one output layer. We focus on feed-forward neural networks (Bishop, 1995) containing in the hidden layer product units that are neurons with their output is based in the multiplication of terms instead of the sum. Particularly, these terms are expressed in the form of a value pow to a real number. The training of the aforementioned networks has been carried out by a refined evolutionary programming approach (Yao & Liu, 1997) which was named two-stage evolutionary algorithm and introduced in Tallón-Ballesteros and Hervás-Martínez (2011) . Such as an evolutionary programming model, the population is only subjected to replication and mutation operators (Moriarty & Miikkulainen, 1995). Parametric and structural mutations have been used and follow the expressions and details given in the aforementioned paper. The main characteristic of this algorithm is to be based on the use of two populations that are evolved at the beginning of the evolution shortly. Then, the best half of both populations is merged into a new population. On the new intermediate population a full evolutionary cycle is carried out. Table 1summarises the main TSEA parameters. In a recent work the TSEAFS (TSEA with FS) framework was proposed (Tallón-Ballesteros et al., 2013). In summary, at the beginning there is a pre-processing step on the training set to obtain a list of selected features. The training of the neural network is performed with the reduced training set via TSEA. Then, the list of selected features is projected into the test set and the performance of the neural models is evaluated in the resulting reduced test set. TSEA does not carry out any kind of FS. Now, we briefly summarise the different configurations and properties of TSEA and TSEAFS. There are two different configurations in TSEA, named 1∗and 2∗. The TSEAFS features are the following: a) PUNN have been employed, with a number of neurons in the input layer equal to the number of variables in the problem after FS; a hidden layer with a number of nodes that depends on the data set to be classified and the number of selected features; and the number of nodes in the output layer equal to the number of classes minus one because a softmax-type probabilistic approach has been used; b) two experiments have been performed for each problem with two different values for α2, that is associated with the residual of the updating expression of the output-layer coefficients; this parameter controls the diversity of the individuals and has a great impact over the performance (Tallón-Ballesteros & Hervás-Martínez, 2011); c) two different configurations (1 ∗and 2 ∗) are applied to subsets obtained with each one of the selectors, for each data set. The parameters of each configuration are neu,genand α2. The first two ones take specific values depending on the data set and the last one depends Table 1. TSEA general parameters. Parameter Value Population size (N) 1000 Gen-without-improving 20 Interval for the exponents wji/coefficientsβl j[−5, 5] Initial value of α10.5 Initial value of α21 Normalisation of the input data [1, 2] Number of nodes in node addition and node deletion operators [1, 2] CONNECTION SCIENCE 247 Table 2. Description of the TSEA/TSEAFS configurations. Methodology Config. Num. of neurons in each pop. Size of each pop. Num. Gener. in each pop. α2 TSEA 1∗neu and neu +1 1000 0.1 ∗gen 1 TSEA 2∗neu and neu +1 1000 0.1 ∗gen 1.5 TSEAFS 1 ∗neuand neu+1 1000 0.1 ∗gen1 TSEAFS 2 ∗neuand neu+1 1000 0.1 ∗gen1.5 on the configuration number (1 ∗,...). Table 2shows the main aspects of TSEA/TSEAFS configurations. 3. Proposal On real-world problems, the attributes may have any kind of main relationship but it is not a fact that this is the single option. So far, the consistency and correlation measures have beenconsideredin an isolated way. The cooperation between different groups ofattributes is the key idea of the current paper. According to the literature, the most common measure in the context of FS is the correlation analysis (Hair, 2010). On the other hand, consistency is not so frequent nowadays, basically because in the majority of applications the data are correlated and the detection of inconsistencies (Aggarwal, 2015) is not always the easiest and promising path to classify new patterns in the future. Now, we describe a motivating example. Lymphography problem (Kononenko, Bratko, & Roskar, 1984) contains data that were obtained from the Institute of Oncology of the University Medical Center in Ljubljana, Yugoslavia about lymph nodes and lymphatic vessels labelled in four classes: normal find,metastases,malign lymph and fibrosis (Bramer, 1999). At first sight it seems an interesting data set because there are four classes. The problem was prepared using a hold-out cross validation in two sets: one to train and another to test the classifier performance. We applied a filter based on consistency and the results improved only around a 1% compared to the case without data pre-processing. After that, we used a filter based on correlation and the results were even worse than with all the feature set. In that point, we felt that the analysis of the selected features was necessary. We detected that five attributes were selected by both methods. We taught that the union operator may be a good action because the number of selected attributes would be increased a bit, but the number of selected attributes will be below 15 and it would be worth evaluating our classifier based on product unit neural networks. Surprisingly, the combination succeeded and the performance was a trade-off between increasing the number of features and classifying better. It has been shown that TSEA performance can be improved by means of FS. Two relevant measures to assess the quality of the features in terms of redundancy and relevance are correlation and consistency. Particularly, the hybrid filter BIRS has an exceptional behaviour in conjunction with TSEA. This paper proposes to extend the list of selected features by BIRS based on CNS. Among the good number of approaches that can be considered we have chosen CFS using SOAP measure (Tallón-Ballesteros, Hervás-Martínez, Riquelme, & Ruiz, 2011) in order to put in a small number of features. It is extremely important to take into account that the base list of features is obtained by the FR-FSS BIRS using a consistency-based measure. After that some new features may be added if different characteristics are selected with the correlation 248 A. J. TALLÓN-BALLESTEROS ET AL. Evaluation of features (cnBI) Feature subset selection (CNS) Evaluation of features (spBI) Feature subset selection (CFS) List of selected attributes Union Figure 2. BIRS(CNS+)framework. measure. In other words, the proposal is the combination of two FR-FSS methods with a good performance in order to get an extended list of attributes. Concretely, we add more attributes to the obtained list by CNS-based FR-FSS BIRS (henceforth, BIRS(CNS)). We have entitled the new proposal BIRS(CNS+) which means that the features selected are those obtainedby BIRS using CNS with extrafeatures.Figure2depictstheBIRS(CNS+) framework. Generally speaking, the increase of the number of features would be very small due to the small size of the list obtained by CFS. The new methodology could be applied to any kind of classification problem with any number of features and classes, however the single constraint is that there would not be total overlapping between the lists obtained by each one of the two FR-RSS methods to be combined. In order to meet this condition we recommend that the number of features would be in the original problem at least more than one or two tens. The justification of the proposal is based on the fact that once we choose a evaluation function for a FS method a possible lost of information may happen due the focus is related with the own criterion of the function. Thus, the idea is try to save at least a few more attributes and let the classifier to decide which attributes from this extended subset are included in the classification model. To the best of our knowledge, there are not contributions that just mix several FR-FSS methods or simply broaden the list of characteristics provided by a FR-FSS filter in any way. A related work belonging to a wrapper model was proposed by Somol, Grim, Novovičová, and Pudil (2011) ; concretely, a k-Nearest Neighbour (k-NN) classifier with four values for kwas used to determine the inclusion or a removal of single features with a stride value of one in relation with the current feature set. The authors of the aforementioned paper concluded that the actual gain is problem dependent and can not be guaranteed, although the improvement on some data sets is substantial. 4. Experimentation Table 3describes the data sets employed together with the parameter values for the base configuration of TSEA and TSEAFS (last two columns). Most of them are publicly available at the UCI repository (Bache & Lichman, 2013) from the University of California. They belong to different domains of application such as Finances, Molecular Biology, Social Sciences, Environment, Oncology, Hand Movement Recognition and Analysis of Olive Oils. The following eight have been used: Statlog (German credit),Labor Relations, Libras Movement, Lymphography,OliveOil(Olitos), Molecular Biology (Promoter Gene Sequences), Statlog (Satellite) CONNECTION SCIENCE 249 Table 3. Summary of the data sets used and parameter values for TSEA and TSEAFS methodologies. Data set Size Features Inputs Classes Neu; Gen Neu;Gen German 1000 22 61 2 6; 500 6; 500 Labor 57 16 29 2 6; 300 5; 300 Libras 360 90 90 15 8; 1000 8; 1000 Lymphography 148 18 38 4 6; 500 6; 100 Olitos 120 25 25 4 6; 300 6; 300 Promoter 106 58 114 2 11; 500 6; 300 Satellite 6435 36 36 6 8; 1000 8; 1000 Water 527 37 37 3 7; 300 7; 300 and Water Treatment Plant. Olitos data set is related with olive oil processing and is a complex real-world problem representing different kind of olive oils whose properties can be read in Armanino, Leardi, Lanteri, and Modi (1989) . The size of the data sets ranges from over fifty-five to more than six thousands. The number of features depends on the problem and varies between 16 and 90, while the number of classes is between 2 (for only 3 cases) and 15. As account of we are using neural networks, all nominal variables have been converted to binary ones; due to this, often the number of inputs is greater than the number of features. Regards the number of inputs it ranges between 26 and 114. Also, the missing values have been replaced in the case of nominal variables by the mode or, when concerning continuous variables, by the mean, taking into account the full data set. These data sets have in common that present important error rates in test phase around 20% or above with reference and robust classifiers such as 1NN or C4.5. The values of the parameters were chosen with a previous experimental design via a five-fold cross validation with five repetitions using the training set for each data set. For the number of generations, four kind of values were defined (100, 300, 500 and 1000) and in regard to the maximum number of neurons in the hidden layer the range [4–12] were considered. In relation to the experimental design we have followed a three-fold stratified cross validation (Hjorth, 1993), whereby data set is divided into three parts and subsequently a partition is the test set and the two remaining ones are pooled as the training data. For stochastic algorithms, for each cross validation fold we perform 30 iterations and since we have three folds the results are averaged from 90 runs in order to get a good reliability level. On the other hand, to evaluate the classification models we have chosen the accuracy measure (Kohavi 1995) that can be defined as the probability of correctly classifying a randomly select pattern. Sometimes, it is called as the number of successful hits (Witten, Frank, & Mark, 2011). Table 4depicts the FR-FSS methods utilised in the experimentation. There are two ones with and one without FS that belong respectively to TSEAFS and TSEA methodologies. As Table 4. List of methods employed in the experimentation with and without FS based on FR-FSS. FR-FSS filter Ranking method Subset evaluation Methodology Abb.name – None None TSEA F0 BIRS(CNS) cnBIRS CNS TSEAFS F1 BIRS(CNS+)cnBIRS+spBIRS CNS+CFS TSEAFS F2 Note: cn and sp stand for consistency and SOAP measures, respectively 250 A. J. TALLÓN-BALLESTEROS ET AL. stated before, feature selectors are FR-FSS filters. Last column defines an abbreviated name for each of them that is employed in next sections. 5. Results This section details the results obtained, measured in Accuracy or Correct Classification Ratio in the test set or in the test subset depending on that FS has been considered or not. First of all, we present the results obtained, including also the average number of inputs for the three train folds of cross validation, with TSEA and TSEAFS by means of F1 and F2. After that, a statistical analysis compares them to determine whether there are significant differences between applying a combination of two FR-FSS methods and a single FR-FSS. Lastly, we present the results with other additional classifiers together with a global plot of them. 5.1. Results applying TSEA and TSEAFS with F1 and F2 Theresultsobtained by applying the initial TSEAmethodology (Tallón-Ballesteros& Hervás- Martínez, 2011), including none FS, are presented, along with those obtained with TSEAFS for the methods F1 and F2. Table 5shows the average number of inputs, the mean and standard deviation (SD) of the test accuracies for each data set for a total of 90 runs. The best results without and with FS appear in boldface for each data set. It is important to highlight that every data fold has the same number of inputs without any kind of pre-processing via Table 5. Results obtained in eight data sets applying TSEA and TSEAFS with F1 and F2. Mean ±SD Data set Method Inputs Config 1∗/1∗Config 2∗/2∗ German F0 61.00 72.24 ±2.29 71.30 ±2.44 F1 16.00 72.29 ±1.59 72.57 ±1.30 F2 17.33 73.41 ±1.61 72.47 ±1.36 Labor F0 29.00 79.77 ±8.71 81.58 ±8.54 F1 3.00 90.58 ±3.39 88.83 ±5.40 F2 4.67 91.40 ±4.35 90.47 ±3.94 Libras F0 90.00 36.67 ±10.41 40.25 ±10.95 F1 29.33 45.68 ±7.06 44.98 ±7.52 F2 40.00 47.32 ±8.53 47.03 ±8.83 Lymphography F0 38.00 77.09 ±6.13 77.02 ±5.98 F1 9.33 76.61 ±7.46 75.51 ±4.70 F2 11.33 80.91 ±7.35 78.11 ±5.47 Olitos F0 25.00 67.81 ±6.21 67.75 ±7.26 F1 10.67 68.81 ±5.36 68.67 ±6.98 F2 13.00 69.47 ±6.78 69.86 ±4.82 Promoter F0 114.00 57.49 ±8.99 63.74 ±9.17 F1 7.33 77.69 ±5.86 77.11 ±3.94 F2 9.00 76.77 ±5.32 77.23 ±5.02 Satellite F0 36.00 83.48 ±2.50 82.08 ±4.49 F1 14.67 83.53 ±2.62 83.42 ±2.40 F2 25.00 83.14 ±1.80 83.74 ±1.88 Water F0 37.00 83.90 ±3.53 83.37 ±3.15 F1 12.33 84.26 ±3.54 83.31 ±2.42 F2 14.33 83.85 ±3.10 84.76 ±3.80 CONNECTION SCIENCE 257 Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. Proceedings of the fourteenth International Joint Conference on Artificial Intelligence (IJCAI 1995), Vol. 2, Montreal, pp. 1137–1145. Kohavi, R., & John, G. (1997). Wrappers for feature subset selection. Artificial Intelligence,1–2, 273–324. Koller, D., & Sahami, M. (1996). Toward optimal feature selection. In L. Saitta (Ed.), 13th International conference on machine learning (pp. 284–292). Bari: Morgan Kaufmann. Kononenko, I., Bratko, I., & Roskar, E. (1984). Experiments in automatic learning of medical diagnostic rules. Technical Report, Jozef Stefan Institute, Ljubljana, Yugoslavia. Lal, T. N., Chapelle, O., Weston, J., & Elisseeff, A. (2006). Embedded methods. In I. Gyon, M. Nikravesh, S. Gunn, & L. A. Zadeh (Eds.), Feature extraction (pp. 137–165). Berlin: Springer. Liu, H., & Setiono, R. (1996). A probabilistic approach to feature selection – a filter solution. In L. Saitta (Ed.), Proceedings of the thirteenth International Conference on Machine Learning (ICML 1996) (pp. 319–327). Italy: Morgan Kaufmann. Liu, H., & Setiono, R. (1998). Incremental feature selection. Applied Intelligence,9(3), 217–230. Liu,H.,&Yu,L.(2005).Towardintegratingfeatureselectionalgorithmsforclassificationandclustering. IEEE Transactions on Knowledge and Data Engineering,17(3), 1–12. Moriarty, D., & Miikkulainen, R. (1995). Discovering complex othello strategies through evolutionary neural networks. Connection Science,7(3–4), 195–210. Peng, H., Long, F., & Ding, C. (2005). Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on Pattern Analysis and Machine Intelligence,27(8), 1226–1238. Ruiz, R., Riquelme, J., & Aguilar-Ruiz, J. (2002). Projection-based measure for efficient feature selection. Journal of Intelligent and Fuzzy Systems,12(3–4), 175–183. Ruiz, R., Riquelme, J., & Aguilar-Ruiz, J. (2006). Incremental wrapper-based gene selection from microarray expression data for cancer classification. Pattern Recognition,39(12), 2383–2392. Somol, P., Grim, J., Novovičová, J., & Pudil, P. (2011). Improving feature selection process resistance to failures caused by curse-of-dimensionality effects. Kybernetika,47(3), 401–425. Tallón-Ballesteros, A. J., & Hervás-Martínez, C. (2011). A two-stage algorithm in evolutionary product unit neural networks for classification. Expert Systems with Applications,38(1), 743–754. Tallón-Ballesteros, A., Hervás-Martínez, C., Riquelme, J., & Ruiz, R. (2011). Improving the accuracy of a two-stage algorithm in evolutionary product unit neural networks for classification by means of feature selection. In J. M. Ferrández, J. R. Álvarez Sánchez, F. de la Paz, & F. J. Toledo (Eds.), New challenges on bioinspired applications (pp. 381–390). Heidelberg: Springer. Tallón-Ballesteros, A. J., Hervás-Martínez, C., Riquelme, J., & Ruiz, R. (2013). Feature selection to enhance a two-stage evolutionary algorithm in product unit neural networks for complex classification problems. Neurocomputing,114, 107–117. Vapnik,V. (2013). The natureofstatisticallearning theory. New York: Springer Science & Business Media. Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics,1(6), 80–83. Witten, I., Frank, E., & Mark, A. (2011). Data mining: Practical machine learning tools and techniques. San Francisco: Morgan Kaufmann. Xing, E., Jordan, M., & Karp, R. (2001). Feature selection for high-dimensional genomic microarray data. In C. E. Brodley & A. P. Danyluk (Eds.), Proceedings of 18th International Conference on Machine Learning, (pp. 601–608). San Francisco, CA: Morgan Kaufmann. Yao, X., & Liu, Y. (1997). A new evolutionary system for evolving artificial neural networks. IEEE Transactions on Neural Networks,8(3), 694–713. Yao, Y. (2003). Information-theoretic measures for knowledge discovery and data mining. In Entropy measures, maximum entropy principle and emerging applications (pp. 115–136). Heidelberg: Springer. Yu, L., & Liu, H. (2004). Efficient feature selection via analysis of relevance and redundancy. Journal of Machine Learning Research,5, 1205–1224.