An Approach to Enhance Time Series Forecasting by Fast Fourier Transform

An Approach to Enhance Time Series Forecasting by Fast Fourier Transform F. Javier Gal´an-Sales, Pablo Reina-Jim´enez , Manuel Carranza-Garc´ıa, and Jos´eMar´ıa Luna-Romera(B) University of Seville, 41012 Seville, Spain [email protected] Abstract. Feature engineering is a decisive step in time series forecasting, as it directly influences the performance of predictive models. In recent years, the Fast Fourier Transform (FFT) has gained popularity as an algorithm for extracting frequency-domain features from time series data. In this paper, we investigate the potential of using FFT as feature engineering to improve the accuracy and efficiency of time-series forecasting models. We performed a comparative analysis of the performance of models trained with FFT-based features versus traditional time domain features on two datasets. Our results demonstrate that FFT-based feature engineering outperforms traditional feature engineering methods in terms of forecast accuracy and computational efficiency. Additionally, we provide insights into the interpretability of the frequency domain features and their relationship with the underlying time series patterns. Overall, our study suggests that FFT-based feature engineering is a promising approach to enhance the performance of time-series forecasting models. Keywords: Time Series Forecasting ·Fast Fourier Transform · Feature Engineering ·Interpretability 1 Introduction Time-series forecasting has become an integral part of decision-making processes in various domains. Its significance lies in its ability to provide valuable insights into future trends and patterns in time-dependent data, which can help organizations optimize resource allocation, improve operational efficiency, and improve customer satisfaction. In finance, for example, accurate forecasting of stock prices, interest rates, and foreign exchange rates is essential to make informed investment decisions [1]. In healthcare, forecasting patient demand and resource utilization can help hospitals allocate resources effectively and improve patient care [2]. Energy management companies rely on time series forecasting to predict energy demand and optimize energy production, distribution, and consumption [3]. Similarly, time series forecasting is used in forecasting weather [4], traffic management [5], and retail sales forecasting [6]. The accuracy and efficiency of time-series forecasting models can have a significant impact on the performance and decision-making processes of organizations, making it a critical tool in various industries. As the amount and complexity of time series data c The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 P. Garc´ıa Bringas et al. (Eds.): SOCO 2023, LNNS 749, pp. 259–268, 2023. https://doi.org/10.1007/978-3-031-42529-5_25 260 F. J. Gal´an-Sales et al. continue to grow, the importance of accurate and efficient time series forecasting will continue to increase. Deep learning models have revolutionized various domains due to their ability to learn features and patterns automatically from data through multiple layers of processing. They offer higher accuracy and efficiency in data analysis compared to traditional machine learning models that rely on manual feature selection and classification algorithms. However, they can be computationally expensive and require a significant amount of time and resources to train and tune the models. As more layers are added to a deep neural network, the number of parameters increases exponentially, making training the network slower and requiring specialized hardware. Furthermore, selecting hyperparameters can be a time-consuming process and often requires experimentation. Therefore, it is crucial to consider the time and resources required before implementing a deep learning model on a specific problem. Feature engineering plays a critical task in enhancing the accuracy and efficiency of time-series forecasting models. This process involves selecting and transforming input variables to create informative features that can improve the performance of the model. Feature engineering is driven by domain knowledge and the characteristics of the time series data. While traditional time-domain features like lagged values and moving averages are widely used, more sophisticated methods are required to capture hidden patterns and relationships in modern data. Deep learning models require a careful selection of relevant attributes, and the Fast Fourier Transform (FFT), which is a powerful mathematical tool that can be used in a wide range of applications, has proven to be a valuable technique for selecting attributes in time series. The FFT allows identifying patterns at different frequencies and reducing the dimensionality of the data without losing important information. Applying the Fourier transform in attribute selection can significantly improve the performance of the deep learning model, thereby increasing its ability to solve complex problems. This paper presents an approach to attribute selection in deep learning models and, more specifically, in time series prediction. Our objective is to make use of the results of the FFT to select those attributes (time instants) that can help to improve the forecasting of deep learning models. Taking into account the training and inference times of deep learning models when working with time series, the objective is to make use of FFT to indicate which of the time instants in the past are more significant and therefore should be taken into consideration for the training of our prediction model. Thus, for example, if we try to predict the 24 h of electricity consumption of a house, instead of using a time lag of the last 3 days, we will choose only those 3 days in the past whose importance is more relevant to the prediction. The remainder of the paper is structured as follows. Section 2will expand the context of attribute selection in time series and place it within a theoretical and conceptual framework. Section 3describes the methodology followed as well as the experimental setup, including the data used and the prediction models used. An Approach to Enhance Time Series Forecasting by Fast Fourier Transform 261 Section 4presents the results of the work. Finally, in Sect. 5, the final conclusions are expressed and accompanied by future works. 2 Background Time-series prediction is a challenging problem in the field of artificial intelligence. In real-life applications, the precision of the prediction can have a significant impact on business decision making and resource planning. Traditionally, time series models have relied on statistical techniques such as ARIMA, which require manual feature selection and parameter settings. However, with increasing data availability and computational power, deep learning models have proven to be a powerful tool for time-series forecasting. Deep learning models, such as Recurrent Neural Networks (RNNs), can automatically learn relevant patterns and features from time-series data, without the need for manual feature selection. Furthermore, proper attribute selection is crucial for the accuracy of deep learning models, and in this regard, the Fourier transform may be effective in identifying patterns and reducing the dimensionality of the data. The Fast Fourier transform (FFT) is a mathematical tool that is used to decompose a signal into its frequency components. It allows a signal to be represented as the sum of sinusoidal signals of different frequencies, amplitudes, and phases [7]. The FFT is applied to a signal that is continuous in time and periodic and decomposes it into discrete frequency components. This transformation is achieved by integrating the input signal with sinusoidal functions of varying frequency. FFT results in a frequency spectrum that represents the components of the signal in the frequency domain. The frequency spectrum shows the amplitude and phase of each frequency component of the signal. This situation improves the analysis and understanding of the signal, such as the dominant frequency of the signal, the high- and low-frequency components, and the relationship between the different frequencies present in the signal can be identified. Given a sequence of N complex numbers xn, the FFT computes a sequence of N complex numbers Xkas follows: Xk= N−1  n=0 xne−2πikn/N where i is the imaginary unit, and k = 0, 1, ..., N-1 is the index of the frequency component in the output sequence. The input sequence xnis assumed to represent a discrete-time signal sampled at a fixed interval. The output sequence Xk represents the signal in the frequency domain. The FFT algorithm computes Xk in the O(N log N) time, which is much faster than the O(N2) time required by the direct computation of the DFT. The FFT has a wide range of applications in various areas such as signal processing [8], and engineering [9]. It is used to analyze signals in different domains, such as the audio signal in the time domain and the image signal in the frequency 262 F. J. Gal´an-Sales et al. domain. It is also used for data compression, denoising, and pattern identification in complex signals. FFT has also been used in numerous papers in the literature using machine learning techniques. Szymkowski et al. [10] proposed a novel approach to irisbased human identity recognition using discrete FFT components. The components were selected with the principal component analysis algorithm and classified with three algorithms which achieved satisfactory results. The authors of [11] developed a speaker recognition platform based on machine learning using a flexible piezoelectric acoustic sensor with a multiresonant frequency band. The authors used a FFT and a short-time Fourier transform (STFT) to convert the dataset into frequency components. Fatimah et al. [12] used FFT to improve the classification of the process electroencephalogram signals (EEG) of different stages of sleep. The study highlights that the proposed method can be implemented in real time using FFT for continuous patient monitoring. On the other hand, Fourier was also used in Deep Learning approaches such as [13], where a new system was developed to diagnose and categorize cardiac disease using a CNN model. In this study, FFT is applied to phonocardiogram signals, and the results are promising. Shih et al. [14] developed a deep learning network to classify the pilling level of knitted fabrics using an image preprocessing technology that applies the FFT and Gaussian filter to fabric images. We can also find a study [15] that developed fault diagnosis neural network models to identify faulty states in geared transmission systems based on measured gear fault vibration signals and deep learning theory. Models include an FFT network model, a wavelet transform-convolutional neural network model, and a comprehensive deep neural network model. This work showed that the comprehensive deep neural network model was the most effective in gear failure recognition. In time series prediction, applying FFT can identify periodic or cyclic patterns in the data and select the most relevant features for prediction. This reduces data dimensionality and improves the accuracy of deep learning models. Therefore, FFT is a powerful tool for time series prediction, improving model efficiency and accuracy, reducing data collection costs, simplifying data analysis, reducing overfitting, and improving model generalization. 3 Methodology 3.1 Dataset The experiments will be conducted on univariate time series datasets. Therefore, we have selected the ETT (Electricity Transformer Temperature) datasets {ETTh1, ETTh2}[16], which includes two years of data from two electric transformers at two separate stations. The target value for this study will be the Oil Temperature variable, where there is a one-hour gap between each instance of data. Recently, this dataset has been used to compare several state-of-the-art architectures, including Informer [16], Autoformer and CLFormer [17], which use a transformer-based approach [18]. Additionally, LTSF-Linear solutions [19] have been compared, which are linear multiperceptron models. An Approach to Enhance Time Series Forecasting by Fast Fourier Transform 263 Let the univariate temporal data slice be denoted by a sequence X= x1,x 2,...,x n, where nis the length of the sequence. We transform Xinto singledimensional standard input and output vectors, denoted by Y=(y1,y 2), where y1is the input and y2is the output. The output vector y2has a horizon size H such that H∈{6,24}. The length of the input vector y1is a factor of H, such that |y1|=H∗f, where f∈{2,3}is a factor of H. This division results in the separation of the data set into N−|Y|inputoutput combinations, where Nrepresents the length of the time series. 3.2 Feature Engineering To extract the temporal-rich information of the time series, we decompose each sequence into its trend and seasonal component [20]. Next, after obtaining the seasonal component of both time series, applying the Fast Fourier Transform (FFT) can help identify the most significant time cycles in the data. Once these cycles are identified, the next step is to select past time intervals that align with those cycles. Once the time cycle has been identified, two different scenarios will be created for comparison. Firstly, predictions will be made with the standard variant that includes the moments leading up to the prediction horizon. The second scenario is a Fourier-shifted variant, which includes the hours leading up to the prediction point, as well as timestamps of the same duration starting from the time instant identified by the FFT cycle search. It should be noted that, intervals in the Fourier-shifted input have the same length as the horizon. 3.3 Models to Use This section presents the deep learning models that have been used to carry out the experimentation. Two types of networks have been used in this work: Multilayer perceptron (MLP) and convolutional neural network (CNN). On the one hand, Multilayer Perceptron (MLP) [21] has been used, which is a type of artificial neural network inspired by the structure of the human brain. Instead of having a single layer of neurons, as in a simple neural network, an MLP has several layers of interconnected neurons. Each layer is composed of multiple neurons that receive an input, process it, and generate an output that is sent to the next layer. This network works so that, during training, the model adjusts the weights and biases of the neurons to minimize the difference between the generated output and the desired output. Once the model is trained, it can be used to make predictions on new and previously unseen data. This model has been used in the literature in numerous related works [22,23]. On the other hand, Convolutional Neural Networks (CNNs) [24]areatypeof neural network used for image analysis, which extract local and global features using convolutional filters. They consist of input, convolutional, pooling, and fully connected layers, where the latter classify the image. CNNs are effective due to their ability to learn complex features from images [25]. For time series 264 F. J. Gal´an-Sales et al. analysis, CNNs can treat each value as a one-dimensional image, using convolutional layers as filters to extract features. This enables the detection of complex patterns in time series data [26]. We have proposed two different approaches for each type of network. The first approach uses the raw input, while the second approach uses the seasonal and trend components as separate inputs. Specifically, we have used the multilayer perceptron (MLP) and convolutional neural network (CNN) models for the raw input approach, and the decomposed multilayer perceptron (DMLP) and decomposed convolutional neural network (DCNN) models for the separate input approach. The configurations for hidden layers of the MLP models were [8, 16, 32, 16, 8], [16, 32, 64, 32, 16], [8, 32, 8], [16, 64, 16], and [72, 72]. The configuration for convolutional blocks (layer, kernel size) were [64,3], [64,3], [32,3], [32,3], [128,3], [128,3], [64,3], [64,3], [64,3], [32,3], [32,3], [32,3], and [128,3], [128,3], [128,3]. To ensure optimal performance, we have conducted a thorough hyperparameter optimization for each of these models. This included testing various values for hyperparameters, such as the learning rate and batch size. Ultimately, we settled on a batch size of 32 for all models and tested two different learning rates of 0.01 and 0.001. The performance of the models are going to be measured in terms of Weighted Absolute Percentage Error (WAPE) as is done in previous works from the literature [27]. Fig. 1. Frequency domain analysis of datasets using FFT 4 Results and Discussion Figure 1are the FFT representations (frequency in the x-axis, by FFT value in the y-axis) for both datasets. As can be observed, the highest frequency values is reached at 30.250, so it can be assumed that, the monthly cycle is particularly important in these data so those dates should be taken into account for a better forecast. Figure 2illustrates a time-series sample for the standard and the Fouriershifted scenarios respectively. The colored stripes on the graphs represent the attributes (days) selected for predictions. As can be seen, the upper figure has as selected days the days immediately prior to the day to be predicted, while An Approach to Enhance Time Series Forecasting by Fast Fourier Transform 265 in the lower figure, the days to be taken into account are the previous day, and days 30 and 60, as indicated by the Fourier results. Fig. 2. Input selection for ETTh1 with factor of 3 and FH of 24 Tables 1and 2contain the results of experiments on different machine learning models for forecasting on the two datasets (ETTh1 and ETTh2), and the performance of the models (MLP, DMLP, CNN and DCNN) with a PH factor of 2 and 3 respectively. In the tables, we can observe both the results of the models without applying feature engineering and using FFT for the selection of days in the past. The tables include the mean WAPE of each model and configuration, as well as the standard deviation and the training and testing times. As can be seen, the results show that models trained using FFT as feature engineering do not always obtain better results. For the ETTh1 dataset, we can see that with a forecast horizon (FH) of 6, the standard model has obtained better results. However, if we extend the FH to 24, we find the opposite situation, in which it is the models with FFT that have obtained a better performance. On the other hand, in the second dataset, we find the opposite situation, in which the models that include FFT in their process obtain better results with an FH of 6. Table 2shows more consistent results. Here, we can see how the standard models achieve better results in terms of WAPE when working with an FH of 6, while the models that include FFT improve the results with an FH of 24. In summary, the FFT models demonstrate better performance than standard-input models when considering a larger forecast horizon. Therefore, it cannot be stated that FFT improves in all situations the performance of the models. Given the tables of results, it can be seen that, under certain circumstances, including in the learning process those days of seasonality that are more representative of the period to be predicted, improves the results. However, this situation does not occur in all configurations, so it can be thought that this preliminary study may serve as a basis for further experimentation in future work. 266 F. J. Gal´an-Sales et al. Table 1. Comparison of error and computing times using a PH factor of 2. The results of the models are expressed in terms of WAPE. The best results are shown in bold. Dataset FH Model Mean Std Mean(FFT) Std(FFT) Train(s) Test(ms) ETTh1 6MLP 9.920 0.197 10.006 0.312 45.934 11.063 DMLP 9.734 0.010 9.791 0.007 65.080 35.253 CNN 9.478 0.056 9.868 0.052 45.569 17.379 DCNN 9.960 0.781 9.793 0.144 81.831 30.056 ETTh1 24 MLP 15.394 0.437 15.263 0.547 52.219 10.973 DMLP 14.882 0.025 14.693 0.025 77.527 34.267 CNN 14.779 0.129 14.716 0.144 38.498 34.893 DCNN 15.020 0.223 14.953 0.236 60.838 29.814 ETTh2 6MLP 7.194 0.479 6.547 0.526 45.250 10.633 DMLP 6.732 0.042 6.139 0.049 59.662 32.518 CNN 6.030 0.279 5.849 0.206 49.587 17.155 DCNN 5.318 0.114 5.594 0.124 80.639 30.211 ETTh2 24 MLP 10.443 0.559 10.472 0.492 42.674 10.820 DMLP 9.883 0.042 9.920 0.040 54.256 34.679 CNN 9.807 0.098 9.899 0.150 42.260 35.529 DCNN 9.795 0.378 10.073 0.450 74.976 30.522 Table 2. Comparison of error and computing times using a PH factor of 3. The results of the models are expressed in terms of WAPE. The best results are shown in bold. Dataset FH Model Mean Std Mean(FFT) Std(FFT) Train(s) Test(ms) ETTh1 6MLP 9.782 0.397 10.045 0.501 55.512 10.765 DMLP 9.491 0.047 9.670 0.008 70.282 38.894 CNN 9.454 0.052 9.835 0.113 57.932 23.552 DCNN 9.537 0.667 9.899 0.339 65.581 30.584 ETTh1 24 MLP 15.602 0.437 15.184 0.514 49.494 11.033 DMLP 14.995 0.028 14.562 0.045 59.012 34.033 CNN 14.982 0.268 14.892 0.586 44.608 18.258 DCNN 15.365 0.274 15.192 0.228 55.587 30.906 ETTh2 6MLP 6.045 0.324 6.438 0.410 47.607 11.002 DMLP 5.639 0.032 5.919 0.052 68.243 34.424 CNN 5.363 0.168 5.732 0.240 71.924 23.578 DCNN 5.036 0.089 5.475 0.153 81.835 30.535 ETTh2 24 MLP 10.677 0.102 10.577 0.635 48.797 10.8982 DMLP 9.942 0.052 9.865 0.048 65.769 37.176 CNN 9.892 0.102 9.766 0.116 48.054 19.252 DCNN 10.434 1.472 10.203 0.437 73.630 30.447 An Approach to Enhance Time Series Forecasting by Fast Fourier Transform 267 5 Conclusions In this study, different deep learning models have been applied to time series prediction. Although the use of deep learning models in this field is relatively new, their popularity has been increasing due to their ability to capture complex patterns in the data. In this regard, the Fast Fourier Transform has been used to improve model prediction. The results obtained in this study are very promising. The application of Fourier has helped to select the time instants that are most relevant to predict the data being worked with. In addition, an improvement in the accuracy of models that include Fourier compared to those that do not has been observed when considering a large forecast horizon. This indicates that proper attribute selection is essential to improve the accuracy of deep learning models and that Fourier is a marked improvement in terms of selecting the most significant time instants for prediction. This study is an approximation and needs future work to deepen this field. For example, more deep learning methods, such as generative neural networks (GANs), can be used to further improve prediction accuracy. In addition, different datasets can be tested to explore the effectiveness of Fourier in different contexts. Other transforms already used in the literature, such as the wavelet transform, can also be applied to compare their effectiveness with the Fourier transform. In conclusion, attribute selection is essential in time series prediction using deep learning, and the application of transforms such as Fourier offers great potential to improve the accuracy of the models. References 1. Matuozzo, A., Yoo, P.D., Provetti, A.: A right kind of wrong: European equity market forecasting with custom feature engineering and loss functions. Expert Syst. Appl. 223, 119854 (2023) 2. Zhou, Z., Zhao, Z., Zhang, X., Zhang, X., Jiao, P.: Improvement of accuracy and resilience in fhr classification via double trend accumulation encoding and attention mechanism. Biomed. Signal Process. Control 85, 104929 (2023) 3. Zhang, Z., Wang, J., Wei, D., Xia, Y.: An improved temporal convolutional network with attention mechanism for photovoltaic generation forecasting. Eng. Appl. Artif. Intell. 123, 106273 (2023) 4. Wang, J., Li, Z.: Wind speed interval prediction based on multidimensional time series of convolutional neural networks. Eng. Appl. Artif. Intell. 121, 105987 (2023) 5. Qin, Y., Luo, H., Zhao, F., Fang, Y., Tao, X., Wang, C.: Spatio-temporal hierarchical mlp network for traffic forecasting. Inf. Sci. 632, 543–554 (2023) 6. Zhang, X., Kim, T.: A hybrid attention and time series network for enterprise sales forecasting under digital management and edge computing. J. Cloud Comput. 12(1), 1–21 (2023) 7. Cooley, J.W., Tukey, J.W.: An algorithm for the machine calculation of complex fourier series. Math. Comput. 19(90), 249–259 (1965) 8. V¨alim¨aki, V., Bilbao, S.: Giant ffts for sample-rate conversion. AES J. Audio Eng. Soc. 71(3), 88–99 (2023)