Multi-Objective Lagged Feature Selection Based on Dependence Coefficient for Time-Series Forecasting

Multi-Objective Lagged Feature Selection Based on Dependence Coefficient for Time-Series Forecasting Mar´ıa Lourdes Linares-Barrera , Manuel J. Jim´enez Navarro , Jos´e C. Riquelme ,andMar´ıa Mart´ınez-Ballesteros(B) Department of Computer Languages and Systems, University of Seville, 41012 Seville, Spain {mlinares,mjimenez3,riquelme,mariamartinez}@us.es Abstract. In the fast-evolving field of machine learning, the process of feature selection is essential for reducing model complexity and enhancing interpretability. Within this context, filter methods have gained recognition for their effectiveness in assessing features through statistical metrics. A recently introduced metric, the Conditional Dependence Coefficient, aims to assess the dependence between subsets of features and a target variable, enhancing our understanding of feature relevance. This paper presents a novel feature selection approach that integrates this statistical metric with a multi-objective evolutionary algorithm. This strategy leverages the flexibility of evolutionary algorithms to efficiently explore the feature space and employs an intuitive metric for identifying pertinent features. Unlike many filter-based approaches, our method does not require thresholds or percentiles related to the number of selected features and evaluates the collective merit of feature subsets instead of the significance of individual features. To address the forecasting challenge of identifying the appropriate time lags and features, we performed experiments on eight distinct datasets containing multivariate time-series data. Comparing our method against a baseline with no feature selection, our results show solid performance in efficacy and a notable reduction in model complexity. Keywords: Feature Selection ·Multi-objective Optimization · Genetic Algorithm ·Neural Network ·Time-Series Forecasting c  1 Introduction In today’s data-driven landscape, where massive and real-time information becomes the norm, datasets often contain a vast number of attributes. This situation demands advanced storage and processing capabilities to handle increased data volumes and rapid and interpretable responses. In this context, feature selection [3] allows the identification of the most relevant subset of features, enabling a focus on a reduced set of variables while maintaining or enhancing efficacy. The Author(s) A. Alonso-Betanzos et al. (Eds.): CAEPIA 2024, LNAI 14640, pp. 81–90, 2024. https://doi.org/10.1007/978-3-031-62799-6_9 82 M. L. Linares-Barrera et al. This process offers several benefits. Firstly, it enhances model interpretability [7], enabling experts to focus on meaningful insights. Secondly, it addresses the curse of dimensionality [17], reducing computational complexity and overfitting. Finally, eliminating irrelevant features prevents the need to sample data requiring extensive resources. Feature selection remains a dynamic area of research. Despite the diverse range of approaches, it is possible to distinguish three distinct groups: filter, wrapper and embedded methods [4,10]. Within this domain, filter methods are gaining significant attention for their ability to quickly assess feature importance through statistical measures, but may overlook feature interactions. This paper introduces MOLS (Multi-Objective Lag Selection), a novel filter feature selection algorithm that combines the Conditional Dependence Coefficient (CODEC) [1] with a multi-objective genetic algorithm. The CODEC metric evaluates the collective importance of a feature subset. This metric addresses the limitations of numerous filter techniques that primarily emphasize individual feature rankings. The multi-objective genetic algorithm approach aims to balance model simplicity and interpretability by selecting a concise set of highly relevant features. Our proposal is based on the Pareto-optimal set and the NSGA-II algorithm [6], overcoming the limitations of traditional genetic algorithms that rely on weighted functions, leading to suboptimal trade-offs and sensitivity issues. Given the complex relationships between variables and the significance of identifying key lags, time series forecasting provides an optimal scenario to evaluate our approach. Therefore, the main contributions of this paper are: – Introduction of a multi-objective filter method for feature selection that evaluates feature subsets collectively. Unlike several filter approaches, this method eliminates the need for hyperparameters that dictate the number of features to be selected, such as thresholds, percentiles, or a specific number of features. – The proposal is applied over a neural network, leveraging the significant benefits that feature selection can provide. This model is further optimized via the Bayesian Optimization. – Evaluation of the proposal effectiveness across eight diverse time-series datasets, covering various domains such as air pollution, electricity, and traffic forecasting. The results demonstrate competitive performance and a reduction in model complexity through feature selection. The paper is organized into several sections. Section 2reviews previous research on feature selection and Sect. 3describes MOLS proposal. Section 4outlines the experimental methodology applied and Sect. 5presents the obtained results. Finally, Sect. 6summarizes the main conclusions and suggests future research directions. MOLS: Multi-Objective Lag Selection 83 2 Related Works As the demand for more interpretable and efficient models grows, there is a noticeable shift towards integrating feature selection methods into neural network architectures. A notable example of this trend is the work of Liu et al. [14], who combined a mutual information-based feature selection method with a hybrid deep neural network model to accurately forecast wind speed. Similarly, Kilincer et al. [12] have made a significant contribution by using recursive feature elimination with multilayer perceptron optimization to improve the detection of cybersecurity threats in healthcare systems. Among feature selection strategies integrated on machine learning models, filter methods stand out for their versatility and computational efficiency. The CODEC metric, introduced in [1], marks a significant advancement in filter methods by efficiently detecting both linear and nonlinear dependencies. Despite the potential of feature selection, optimizing search techniques within a vast feature space presents a considerable challenge. To overcome this, the adoption of bio-inspired algorithmic techniques has been suggested. Zhou et al. [19] proposed a genetic algorithm guided by a correlation matrix as a feature selection strategy. In a similar vein, Espinosa et al. [8] have incorporated a multiobjective evolutionary feature selection approach into an LSTM model, achieving notable improvements in air quality prediction in Italy. Our work similarly leverages a multi-objective evolutionary approach, combined with the CODEC metric, to enhance feature selection. The importance of feature selection is underscored in various domains, particularly in the analysis of time-series data. The need to pinpoint relevant temporal points and tackle issues like temporal dependencies and seasonality is paramount. Addressing this need, in [15] is developed a two-stage feature selection process (correlation analysis and wrapper method utilizing a shallow neural network) for financial time series forecasting. Our research parallels these initiatives, concentrating on rigorously assessing our methodologies within the time-series domain. 3 Description of the Proposed MOLS Algorithm We propose an evolutionary strategy-based method to identify the optimal feature subset. The main goal is to identify the subset of features maximizing CODEC value while keeping the number of features as small as possible. The Conditional Dependence Coefficient (CODEC) [1](seeEq.1) assesses the relationship between a target variable Yand a feature set Zin a dataset of N instances. We focus on the unconditional version of CODEC, which evaluates the direct relationship without conditioning on additional variables. The formulation is as follows: CODEC(Y,Z):=N i=1 Nmin Ri,R M(i)−L2 i N i=1 Li(N−Li)(1) 84 M. L. Linares-Barrera et al. In this formula, for each instance iwe consider the following terms. Rirepresents the rank of the Yvalue for the i-th instance. Lirepresents the inverse rank of the Yvalue for the i-th instance. M(i) represents the index of the nearest data point to the i-th instance, computed using only the dimensions defined by Zand Euclidean distance. The metric quantifies the strength of the relationship between Yand Zby assessing how Zinfluences ranks and positioning. In [1], the authors demonstrate that CODEC is a nonlinear generalization of partial R2. Our proposal is based on a multi-objective genetic algorithm based on Pareto front inspired by the NSGA-II algorithm. 1. Initialization. We start by generating an initial population of μindividuals at random, where each is codified by a binary vector (“1” for inclusion, “0” for exclusion of each feature). 2. Evolutionary process. The whole evolutionary process is repeated until the desired number of generations (ngens) is reached. At each iteration, we generate a new population through a series of steps: (a) Offspring generation. At each generation’s outset, λoffspring are produced through the following evolutionary operations (only one at each time): crossover (chance cxpb), mutation (chance mutpb), or reproduction (chance 1 −mutpb −cxpb). In crossover, two parents are randomly selected from the population, producing a single offspring using a two-point crossover mechanism. During mutation, each gene from a randomly selected individual have a probability (gmutpb) of flipping from 0 to 1 or vice versa, introducing variability. In reproduction, a random individual is cloned and the clone is added to the offspring, preserving genetic information. (b) Evaluation and selection of subsequent population. The offspring, combined with the current population, undergo evaluation. The top μ performers are selected for the next generation. Each individual is evaluated based on two criteria: its CODEC value (reflecting the dependence between the selected features and the target variable) and the number of features it represents. Individuals are sorted into layers called Pareto fronts. The first front contains individuals that are not dominated. Subsequent fronts are filled with individuals that are only dominated by those in the previous front(s). For the next generation, the top μindividuals are selected starting from the first front and moving on to the next ones as needed. If including an entire front would exceed the desired population size, the crowding distance within that front is used to choose individuals, ensuring the selection prioritizes diversity. 3. Best solution selection. Once the evolutionary process is completed, individuals within the population are classified into different Pareto fronts. The first front comprises non-dominated solutions, characterized by achieving an optimal balance between high CODEC values and a reduced number of features. The best individual on the first front is chosen based on the criterion of MOLS: Multi-Objective Lag Selection 85 Table 1. MOLS Parameter Settings. Parameter Default Value Population size (μ) 50 Offspring size (λ) 50 Number of Generations (ngens) 20 Crossover Probability (cxpb)0.5 Mutation Probability (mutpb)0.5 Gene Mutation Probability (gmutpb)0.2 having the minimum number of features. This selection provides an optimal balance between maintaining high dependency and ensuring model simplicity. The parameters of the multiobjective genetic algorithm have been fixed (Table 1). Optionally, these parameters could be tuned, but our results (Sect. 5) suggest that the parameters are good enough to obtain competitive results. 4 Methodology and Experimentation This section outlines the experimental methodology, details the learning model in Sect. 4.1 and the main phases of the approach in Sect. 4.2. 4.1 Model As a base model, we selected a fully connected neural network architecture (MLP). It features one or two hidden layers, each containing 50 to 100 neurons. The Rectified Linear Unit (ReLU) activation function is used for its efficiency in speeding up convergence and reducing the vanishing gradient issue. The Adam optimizer is used for network optimization, with a learning rate set between 0.0001 and 0.01. This learning rate spectrum allows for a balanced approach to learning, adaptable to the unique demands of the problem at hand. Finally, to prevent overfitting, an early stopping mechanism is applied, stopping training if the validation loss fails to improve for 10 consecutive epochs. 4.2 Main Phases 1. Preprocessing: The initial dataset is segmented into three subsets: training (70%), validation (15%), and test sets (15%); ensuring unbiased model evaluation. As the algorithms require the lagged information, each time series is segmented into fixed-size, non-overlapping windows. Data from all channels are flattened into a one-dimensional vector. Standardization is also applied to address issues that arise from different scales and distributions. 86 M. L. Linares-Barrera et al. 2. MLP hyperparameters tuning: First, we perform MOLS feature selection on the training set and apply the selection over train and validation sets. To identify the optimal hyperparameters for the MLP, Bayesian optimization [16] is employed. This involves training models with a variety of configurations on the training set and assessing their performance on the validation set. The configuration that results in the lowest Mean Absolute Error (MAE) is considered optimal. 3. Best model training and evaluation: In this step, MOLS feature selection is applied. Then MLP is trained with the optimal hyperparameters and MAE is evaluated over the test set. The experiment is carried out with and without feature selection and is repeated using 5 different random seeds. 5 Results and Discussion This section presents the results, introducing the datasets in Sect. 5.1 and analyzing the effectiveness of the MOLS approach in Sect. 5.2. 5.1 Datasets The proposal is evaluated in eight time series datasets. Preprocessing techniques are detailed in Sect. 4.2, focusing on windowing and channel flattening. –The Torneo dataset [11] consists of four separate datasets, each focusing on a specific pollutant (CO,NO2,O3or PM 10), gathered by hourly sensors in the Torneo region (Seville). The dataset consists of 7 variables (4 pollutants and 3 meteorological attributes) and a past history of 24 h (thereby resulting in a total of 168 features). The target variable is the level of pollutant for the next hour. Each dataset contains 2,798 instances and 168 features. –The Electricity dataset [9] contains measurements of electricity consumption in Portugal for 321 different clients. A past history of 24 h is taken into account. The target is to predict the next hour for the first sampled client. It comprises 1,095 instances and 7,704 features. –The Traffic dataset [5] details the road occupancy rates captured between 2015 and 2016 by sensors placed on highways in the San Francisco Bay Area at 10-minute intervals. It includes data from 862 sensors and 24 h of past history. The target consists of predicting the occupancy for the first sensor in the next hour. The dataset includes 730 instances and 20,688 features. –The ExchangeRate dataset [13] covers daily exchange rate data for the period between 1990 and 2016 in eight countries: Australia, the United Kingdom, Canada, Switzerland, China, Japan, New Zealand, and Singapore. This dataset considers 8 countries, and a historical record of the 7 previous days is considered. The target of this dataset is to predict the value of the next hour for the first country. It is made up of 1080 instances and 56 features. MOLS: Multi-Objective Lag Selection 87 –The ETTh2 dataset [18] comprises data collected from an electrical transformer located in China, with hourly measurements spanning from July 2016 to July 2018. The dataset includes 7 variables (High Useful Load, High Useless Load, Middle Useful Load, Middle Useless Load, Low Useful Load, Low Useless Load, and Oil Temperature) and a historical record of 24 h is considered. The target is to predict the transformer oil temperature for the next hour. It contains 725 instances and 168 features. 5.2 Result and Analysis Table 2presents the effectiveness of our proposal compared to the results without using feature selection, based on the performance in terms of MAE and selected features. The entire experimentation process was repeated using 5 distinct random seeds to ensure the robustness and reliability of the results. The average MAE (Mean Absolute Error) and the average number of features from the 5 executions were recorded and are shown in Table 2. Table 2. Average efficacy Results in Terms of MAE and Number of Features. MAE # Features Dataset MOLS No Selection MOLS Total ExchangeRate 0.007 0.006 9.8 56 TorneoCO 1.415 1.291 50.4 168 TorneoNO2 1.432 1.386 46.8 168 TorneoPM10 1.225 1.274 49.6 168 TorneoO3 1.362 1.375 48.6 168 ETTh2 1.875 1.897 50.2 168 Electricity 204.573 213.243 3611.6 7704 Traffic 0.007 0.009 9925.8 20688 In terms of MAE, our proposal exhibits superior performance compared to the approach without feature selection in five datasets, while obtaining a MAE close to the baseline method in the remaining three datasets. This underscores the strength of the CODEC metric in accurately capturing the dependence between subsets of features and the target variable. In particular, a significant enhancement was observed in the Electricity dataset. However, it should be mentioned that the results for TorneoCO were slightly worse. Furthermore, our proposal stands out by substantially simplifying the model, selecting fewer than half of the total features available in the datasets, and most cases selecting approximately 20–30% of the total features. Figure 1illustrates the progression of the MOLS algorithm throughout the feature selection process for one of the 5 random seeds executions considered, using the TorneoPM10 dataset. The first Pareto front fitness points is highlighted 88 M. L. Linares-Barrera et al. Fig. 1. MOLS selection on TorneoPM10 in one of the 5 random seeds executions. in red, while the rest of individuals fitness is represented in black. It is evident that, across generations, individuals tend to converge towards the bottom-right corner of the graph. This indicates a tendency towards achieving a balance of higher CODEC values with a reduced number of features. At the end of the process, the MOLS algorithm successfully achieves a robust balance, thus ensuring effective and efficient feature selection. The observed well-balanced relationship between the Mean Absolute Error (MAE) and the number of features underscores the potential of the multiobjective evolutionary approach to maintain or enhance effectiveness while reducing complexity. To ensure the significance of the conclusions obtained, a statistical test [2] was applied to the results. Figure 2presents the results of the Bayesian analysis comparing our methodology with the NoSelection (NS) method. With a 77% confidence level, we found that the NoSelection method results in a higher error. This results allow us to identify that there are significant differences between the methods compared. MOLS: Multi-Objective Lag Selection 89 Fig. 2. Bayesian MAE comparison result. 6 Conclusions and Future Works The paper presents MOLS, a novel approach that integrates the CODEC metric into a filter feature selection algorithm for time-series forecasting. MOLS aims to balance the feature dimension with the efficacy, addressing the challenges posed by datasets with numerous features. The key contributions include proposing a multi-objective filter selection method, applying a neural network model optimized via Bayesian Optimization, and evaluating the proposal across diverse time-series datasets. Results indicate that MOLS outperforms the approach without feature selection in terms of MAE in several datasets, achieving a significant reduction in the number of features while maintaining or improving model efficacy. Statistical analysis supports these findings. However, more research is needed to establish the significance conclusively. Future work could focus on refining the MOLS algorithm, exploring additional datasets and models, and investigating other statistical metrics or optimization techniques. Additionally, extending the application of MOLS to different and more complex time-series forecasting problems could provide valuable insights into its effectiveness and versatility. Acknowledgements. This research has been supported by the grant PID2020117954RB-C22 funded by MICIU/AEI/10.13039/501100011033. This work has also been supported by TED2021-131311B-C21 funded by MICIU/AEI/10.13039/ 501100011033 and by the European Union NextGenerationEU/PRTR. References 1. Azadkia, M., Chatterjee, S.: A simple measure of conditional dependence. Ann. Stat. 49(6), 3070–3102 (2021) 2. Benavoli, A., Corani, G., Demsar, J., Zaffalon, M.: Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis. J. Mach. Learn. Res. 18, 2653–2688 (2017)