HLNet: a novel hierarchical deep neural network for time series forecasting

HLNet: A Novel Hierarchical Deep Neural Network for Time Series Forecasting M. J. Jim´enez-Navarro, F. Mart´ınez-Alvarez, A. Troncoso, and G. Asencio-Cort´es(B) Data Science and Big Data Lab, Pablo de Olavide University, 41013 Seville, Spain {mjjimnav,fmaralv,atrolor,guaasecor}@upo.es Abstract. Time series forecasting is a well-known application area for deep learning, in which the historical data are used to predict the future behavior of the series. Several deep learning methods have been proposed in this context, but they usually try to generate the output from the input, with no data transformation. In this paper, we introduce a novel method to improve the performance of deep learning models in time series forecasting. This method divides the model into hierarchies or levels, from more simple ones to more complex. Simpler levels handle smoothed versions of the input, while the most complex level processes the original time series. This method follows the human learning process, that is, general tasks are firstly performed and, afterwards, more precise ones are accomplished. Our proposed method has been applied with LSTM architectures, showing remarkable performance in a variety of time series. Moreover, comparisons to standard LSTM are reported. Keywords: Time series ·Forecasting ·Deep learning 1 Introduction Time series forecasting is a crucial task in several areas like electricity demand, weather, pollution, etc. Time series consist in a set of ordered data with a time component. In order to forecast the future behavior, we use the past known information because, usually, this information helps to determine the future. When several time series are jointly considered the problem is known as multivariate, whereas a single time series leads to a univariate problem. We can predict either only one-step ahead or multiple steps, being such case called multi-step forecasting [5]. In this paper, we address both univariate and multivariate multi-step time series forecasting due to the importance in many practical scenarios like traffic management, electricity consumption or store sales [14]. In many of these scenarios, we need to look at the evolution over multiple time steps to perform decisions and strategies in order to improve the performance [17]. https://doi.org/10.1007/978-3-030-87869-6_68 718 M. J. Jim´enez-Navarro et al. Deep learning has been used increasingly in multi-step time series forecasting [19], demonstrating great improvements over the traditional Box-Jenkins and machine learning. Previous deep learning models try to perform the forecasting considering the raw input and learning the features automatically, in these cases the model learn some unknown relationship between the inputs and the outputs converting them into black boxes. In this paper we try to throw some light to the models giving a individual task to the stacked layers in the neural network. For that purpose, we make the assumption that, like humans, neural nets can learn faster and better beginning from simpler tasks and increasing the difficulty. For example, if you wanted to learn to ride a motorbike it would result easiest if you already know to ride a bike, as you has learned concepts as equilibrium, brake, circulation, etc. Some concepts learned from the simpler tasks, like riding the bike, can are used in the more complex ones like riding a motorbike. This scenario would be better, faster and more robust than learn it from the complex task first. In this context, we propose a novel method to improve the neural networks performance in multi-step time series forecasting, hereinafter called Hierarchical Learning Network (HLNet). For that purpose, we divide the neural network into a hierarchy and specify a simpler version of the input time series to the firsts layers and more complex input to the layers below. The simplification is obtained by applying a smoothing process to the input of the model controlled by some parameter. Then the firsts layers transfer their knowledge to the layers below. Thanks to this transfer, the next layers can take advantage of this knowledge and learn faster and easily the more complex outcome. This method is applied to a LSTM neural network and then compared the original with the modified version through several forecasting problems. The major contributions are the definition of a novel method that: improves the performance of deep neural networks in multi-step problems, improve the interpretability of the model giving a sense to the layers in the neural network, reduces the number of parameters of the neural network making it faster to train and the model in the final level does not need to be too deep to reach good results so it is more robust to overfitting. The rest of the paper is structured as follows. In the Sect. 1we describe the related work to this paper focusing in the main differences. Then, we describe the methodology applied to this paper. After this, we show the experimentation with a wide set of datasets. Finally, we talk about the conclusions of this work and future directions. 2 Related Works Previous related work consider the use of multi smoothing or multiresolution time series in order to improve the performance of the models. Usually these methods use some model to learn each representation and then join each one. These methods are in some cases inefficient because they need to train as many models as representations and consider the learning of each representation independently. Another type of methods change the training algorithm using some heuristic in order to learn before from simpler instances. HLNet: A Novel Hierarchical Deep Neural Network 719 Kourentzes and Petropoulos [10] propose an extension of MAPA [9]and tested stock keeping unit demand. Their method uses several non overlapping representations of the data and fit an arbitrary model for each one, then the models predictions are averaged by an unweighted mean or median. While this works makes use of several smoothing methods to make more accurate and robust predictions, our proposed method use them differently. In our method, we do not isolate the models and let them leverage automatically their knowledge to improve the predictions and all the representations are trained jointly improving the training efficiency. Afolabi et al. [1] propose a method in which a noise elimination process based on empirical mode decomposition is used in original time series generating what they call meta-info, then the original and the meta-info is fed into a model to get the predictions. The authors tested their method in NARX networks [12]and LSTM with good results. Our proposed method do not focus in noise elimination, in the simplification process we do not worry about losing information but obtain some insights about the time series. Furthermore, the simplification lines up with the model architecture and we do not only input the simplified versions as an attribute of the input. Wu et al. [20] proposed the RESTful framework in order to improve the forecasting performance using the multi-resolution patterns inside the time series. The authors proposed a process of two steps, in the first step the model uses an GRU network to extract the hidden state of each time series and each resolution, then at the second step a convolutional layer is used to extract the pattern of each hidden state and obtain the predictions. Our method do not just lies in the multi-resolution representations but also in the smooth representation of the time series to extract the useful patterns. Furthermore, the proposed method is an one-step process in which the extraction of features and the final predictions are produced in a single model. Finally, the multiple representations of the time series are not isolated as we previously mentioned. Bahrpeyma et al. [3] used multiple representations of the time series and train an recurrent neural network for each one. The recurrent neural networks predict the multiple horizons and finally use a weighted average based in the model performance. As in the previous works, this method exhibits the issue of performance and model isolation. A similar idea to our proposal is the curriculum learning [8]. In this approach, the complexity of the task is increased during the training ordering the data by some criteria. Koenecke and Gajewar [8] applied the curriculum learning to financial time series and obtained good results. They used the baby steps algorithm and the seasonal decomposition as the complexity ordering criteria. In our method, we focus on changing the architecture more than the training algorithm without making assumptions about the complexity into segments of the time series. 720 M. J. Jim´enez-Navarro et al. 3 Methodology Traditional deep learning networks stack a set of layers. First layers learn more abstract low-level representations, while the following layers use this information to create higher-level representations to, finally, generate the prediction. In this paper, we introduce a novel architecture for neural networks which requires a new learning paradigm. Instead of letting the neural network learn the representation as aforementioned, we group the layers and assign a different objective to each of these groups. The first group of layers has the easiest objective assigned and, as we process the following layers, more complex objectives are assigned. In order to understand the proposed architecture, we must specify the concepts of simple and complex objectives. First, we define an input sequence as a multivariate time series, Sm,w(t), which stands for munivariate time series of length w.Thatis,Sm,w(t)={S1(t),S 2(t),...,S m(t)}, where every Si(t) identifies a single time series of length w. Figure 1illustrates an input multivariate time series with w= 1000 and m= 10, S10,1000(t). Time Fig. 1. Input sequence composed of 10 time series of 1000 samples each, S10,1000(t). The model simplifies the input sequence for each group generating an input ˆ Sm,w(t), which is simpler than Sm,w(t). The definition of simpler input sequence in this paper corresponds to a smoothed version of the input sequence. Smoothing helps us to identify patterns in time series like trend and obtain a input sequence with less noise and variation. We have selected the moving average as the smoothing method. The smoothing method receives a parameter (σ) to control the smoothness of the group that HLNet: A Novel Hierarchical Deep Neural Network 721 we must specify. The parameter σi∈N, called smooth factor, specifies the rolling window size in the moving average for each group i. The larger values of σithe greater the smooth and also the simplification degree. Therefore, the first groups have higher values of σthan the following groups. Note that σ= 1 for the last group of layers, thus keeping the input unmodified. . . . Smoothed outputs Final Prediction . . . Fig. 2. Proposed HLNet architecture. Figure 2shows the complete proposed architecture. Each group (Gi) starts with a SmoothLayer with their respective σigenerating ˆ Sm,w(t), where w means that the length of the time series may vary after the smoothing process. The simplified version of the input sequence is fed to the Embedding layer of group i. This layer encodes ˆ Sm,w(t) into a new sequence of latent vectors ˆ Sd,w(t), using unidimensional convolutions in order to extract time patterns from the series. The LSTM layer receives ˆ Sd,w(t) and outputs the hidden state Hi, which encodes the whole sequence. Then the hidden state is concatenated with all the previous hidden states groups. This is a critical step of the proposed algorithm, because, at this point, the previous groups shares the knowledge learned from simpler representations with subsequent groups. Note that the arrow that goes from the concatenation of the hidden states to the next group in Fig. 2is dashed because this connection will not be considered in the backpropagation optimization step. Not considering this connection, allows us to isolate the learning process for each group, preventing other groups to modify it. Finally, the concatenated hidden state is fed to the FeedForward layer of the group iwhich outputs the predictions for each horizon of size h [6]. Each group igenerates its own output, Oi 1,...,h so, at the end, we have as many outputs as groups. However, just the final group have the real unsmoothed prediction. 722 M. J. Jim´enez-Navarro et al. During the training we calculate the output loss for each group with its respective smoothed target and sum it up to generate the final loss. Note that the target prediction must have the same smoothing that each output using the corresponding σ. 4 Experimentation 4.1 Datasets In this work we have selected 12 publicly available datasets of different nature, each of them with multiple related time series. This collection represents a wide variety of time series forecasting problems, covering different sizes, domains and forecasting horizons. Table 1presents the characteristics of each dataset in detail. In total, the experiments involve more than 50000 time series among all the selected datasets. Furthermore, most of these datasets have been used in forecasting competitions and other TSF reviews [7]. Table 1. Number of time series (N), forecasting horizon (FH), maximum (M) and minimum (m) sequence sizes for each dataset used. Dataset NFH M m CIF2016o12 57 12 108 48 CIF2016o6 15 6 69 22 ExchangeRate 8 6 7588 7588 M3 1428 18 126 48 M4 48000 18 2794 42 NN5 111 56 735 735 SolarEnergy 137 6 52560 52560 Tourism 336 24 309 67 The CIF 2016 competition [16] dataset contains a total of 72 monthly time series, from which 12 of them have a 6-month forecasting horizon while the remaining 57 series have a 12-month forecasting horizon. Some of these time series are real bank risk analysis indicators while others are artificially generated. The ExchangeRate dataset [11] records the daily exchange rates of 8 countries including Australia, British, Canada, Switzerland, China, Japan, New Zealand, and Singapore from 1990 to 2016 (a total of 7588 days for each country). The goal of this dataset is to predict values for the next 6 days. Both M3 and M4 competitions [13,15], datasets include time series of different domains and observation frequencies. However, due to computational time constraints, we have limited the experimentation to the monthly time series as they are longer than the yearly, quarterly, weekly and daily time series. Both competitions ask for an 18 months prediction and contain time series of different HLNet: A Novel Hierarchical Deep Neural Network 723 categories such as industry, finance, or demography. The number of instances belonging to each category are evenly distributed according to their presence in the real world, leading novel studies to representative conclusions. Considering only the monthly time series, the M4 dataset has 48000 time series, while M3 is considerably smaller with 1428 time series. The NN5 competition [4] dataset is formed by 111 time series with a length of 735 values. This dataset represents 2 years of daily cash withdrawals at automatic teller machines (ATMs) from England. The competition established a forecasting horizon of 56 days ahead. The Tourism dataset91 is composed of 336 monthly time series of different length, requiring a 24 months prediction. The data was provided by tourism bodies from Australia, Hong Kong, and New Zealand, representing total tourism numbers at a country level. The SolarEnergy [21] dataset contains the solar power production records in the year of 2006, which is sampled every 10 min from 137 solar photovoltaic power plants in Alabama State. The forecasting horizon has been established to one hour, which comprises 6 observations. The Tourism dataset [2] is composed of 336 monthly time series of different length, requiring a 24 months prediction. The data was provided by tourism bodies from Australia, Hong Kong, and New Zealand, representing total tourism numbers at a country level. 4.2 Experimental Settings In order to make fair comparisons, we set up a experimentation environment. A grid search has been used for hyperparameter search and the best model with the best configuration has been selected to obtain the validation results [18]. The hyperparameters space consist on hidden size, number of layers, sequence size and, just in our modified version, smooth factor σ. The hidden size has been selected to have 32, 64 or 128 neurons in it, the number of layers can be in the range [1, 4] and the sequence size has been selected from a range depending of the forecasting horizon size using a factor the 125%, 200% or 300% of the horizon size. Moreover, we used Adam optimizer for both models and stopped the training if there was no improvement in 5 consecutive epochs. Both versions has been built using the same hyperparameters space except for one consideration in our modified version. As we need to simplify the task we need to have at least two layers so the model can use it, we limited the number of layers in our version to have two or four layers. In the two layers version we use the first to the simpler task and the last to the original, in the four layers version we used the first two to the simpler task and the last two to the original task. This environment uses the same instances as input to train the two versions of the model, the usual LSTM and our modified version of LSTM in order to compare if our method improves or not the performance. 724 M. J. Jim´enez-Navarro et al. 4.3 Evaluation Metrics The comparison between our method and the baseline we compared the results on the test instances with three forecasting metrics and one efficiency metric. The first two metrics are the well known Mean Absolute Error and the MSE with the following formulas: MAE =1 h h  t=1 |yt−ˆyt| MSE =1 h h  t=1 (yt−ˆyt)2 We used the Weighted Average Percentage Error as its better than Mean Absolute Percentage Error for diverse datasets which values can be between 0 and 1 and the metrics can not represent the reality. WAPE =h t=1 |yt−ˆyt| h t=1 yt Finally, we measured the training time in order to show the efficiency of the model. 4.4 Results and Discussion In this section we analyze the results of the best models found using a grid search of the parameters. Table 2shows the best parameters found for each model. In this table we show the parameters in terms of layers, units for each layer and, just in our method, the σparameter used for the SmoothLayer. We observed that a value of 6 in σis usually a good value to set by default in any dataset. Table 2. Best models parameters for each dataset. Dataset LSTM HLNet Layers Units Layers Units σ CIF2016o12 4 64 2 128 6 CIF2016o6 4 32 4 32 6 ExchangeRate 2 64 2 128 3 M3 2 128 2 32 6 M4 1 32 2 64 8 NN5 1 128 4 64 4 SolarEnergy 2 32 4 64 6 Tourism 1 128 2 32 6 HLNet: A Novel Hierarchical Deep Neural Network 725 In terms of layers and units we observe a great difference between our method and the baseline LSTM. In Table 3we can observe a difference in terms of training time obtaining similar or better results. This time reduction is achieved thanks to the isolation of the different groups in our method that could allow us to improve the parallelization during the training. The results show an increase in the performance in terms of MAE, MSE and WAPE comparing our method with the baseline. We observe that in terms of MAE the LSTM is better in four datasets than our method, but in terms of MSE and WAPE, HLNet obtains better results in almost all the datasets tested. We can conclude that our method improves the results of an LSTM improving their efficiency. Table 3. MAE, MSE, WAPE and training time for each dataset and model. Dataset LSTM HLNet MAE MSE WAPE Time MAE MSE WAPE Time CIF2016o12 1.26e03 7.52e09 0.17e00 0’ 30” 1.12e03 5.53e09 0.15e00 0’ 11” CIF2016o6 2.34e06 2.56e13 0.32e00 0’ 5” 2.14e06 2.37e13 0.18e00 0’ 4” ExchangeRate 1.77e−36.00e-6 2.82e−32’ 5” 1.83e−38.00e−62.77e−30’ 52” M3 6.91e02 1.55e06 0.15e00 7’ 22” 6.83e02 1.50e06 0.15e00 2’ 49” M4 5.92e02 1.94e06 0.17e00 9’ 15” 5.98e02 1.94e06 0.17e00 12’ 18” NN5 3.47e00 3.05e01 0.18e00 20’ 9” 3.49e00 2.95e01 0.18e00 23’ 23” SolarEnergy 1.96e00 9.54e00 2.29e00 9’ 55” 2.16e00 1.24e01 2.55e00 7’ 21” Tourism 2.34e03 1.21e08 0.18e00 3’ 41” 2.22e03 9.61e07 0.18e00 1’ 40” 5 Conclusions and Future Works In this paper we have introduced HLNet, a novel method for time series multistep forecasting. Thus, we have added an structural change in LSTM networks which allows learning from simpler to more complex time series, in a hierarchical way. This is done by smoothing the original input and transferring the extracted knowledge to the subsequent levels. The results has shown a great improvement both in training time and in performance, which leads to the use of deeper models in a reduced execution time. This work proposes a new general-purpose paradigm for time series forecasting that assigns specific tasks to different groups of layers, each of them processing the input data according to a hierarchical structure. As future works, other smoothing strategies rather than MA could be followed. Additionally, the development of a custom loss function with different weights for each level and evolving with the number of epochs is proposed. Acknowledgements. The authors would like to thank the Spanish Ministry of Economy and Competitiveness for the support under projects TIN2017-88209-C2-1-R and PID2020-117954RB-C2.