From simple to complex: a sequential method for enhancing time series forecasting with deep learning

From simple to complex: A sequential method for enhancing time series forecasting with deep learning M. J. JIMÉNEZ-NAVARRO∗, Department of Computer Science, University of Seville, 41012 ES-Seville, Spain. M. MARTÍNEZ-BALLESTEROS∗∗, Department of Computer Science, University of Seville, 41012 ES-Seville, Spain. F. MARTÍNEZ-ÁLVAREZ†,Data Science and Big Data Lab, Pablo de Olavide University, 41013 ES-Seville, Spain. A. TRONCOSO††,Data Science and Big Data Lab, Pablo de Olavide University, 41013 ES-Seville, Spain. G. ASENCIO-CORTÉS§,Data Science and Big Data Lab, Pablo de Olavide University, 41013 ES-Seville, Spain. Abstract Time series forecasting is a well-known deep learning application field in which previous data are used to predict the future behavior of the series. Recently, several deep learning approaches have been proposed in which several nonlinear functions are applied to the input to obtain the output. In this paper, we introduce a novel method to improve the performance of deep learning models in time series forecasting. This method divides the model into hierarchies or levels from simpler to more complex ones. Simpler levels handle smoothed versions of the input, whereas the most complex level processes the original time series. This method follows the human learning process where general/simpler tasks are performed first, and afterward, more precise/harder ones are accomplished. Our proposed methodology has been applied to the LSTM architecture, showing remarkable performance in various time series. In addition, a comparison is reported including a standard LSTM and novel methods such as DeepAR, Temporal Fusion Transformer, NBEATS and Echo State Network. Keywords:time series, forecasting, deep learning; univariate; multihorizon ∗E-mail: [email protected] ∗∗E-mail: [email protected] †E-mail: [email protected] ††E-mail: [email protected] §E-mail: [email protected] This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact [email protected] for reprints and translation rights for reprints. All other permissions can be obtained through our RightsLink service via the Permissions link on the article page on our site-for further information please contact [email protected]. Advance Access published 14 May 2024 https://doi.org/10.1093/jigpal/jzae030 Hierarchical Method for Enhancing Time Series Forecasting 987 1 Introduction Time series forecasting is a crucial task with several applications like electricity demand, weather, pollution, etc. Time series is defined as a set of data ordered over time. To forecast future behavior, known information from the past is used because, due to the causal nature of the time series, this information helps to determine the future. When several time series including a set of features are considered, the problem is known as multivariate, whereas a single time series including one feature leads to a univariate problem. We can predict either only one-step ahead or multiple steps, known as multistep forecasting [7]. In this paper, we address multistep forecasting in univariate time series due to its importance in many practical scenarios such as traffic management, electricity consumption or store sales [23]. In many of these scenarios, we need to look at the evolution over multiple time steps to perform decisions and strategies to make decisions [28]. Deep learning has been used increasingly in multistep time series forecasting [30], demonstrating great improvements over the traditional Box–Jenkins and machine learning models. Deep learning models perform the forecast feeding past events, learning the features, and obtaining future behavior from these features. In these cases, the model learns some unknown relationship between the inputs and the outputs, converting them into black boxes. In this paper, we try to shed some light on models that assign an additional task to each layer, or group of layers, in the neural network. For that purpose, we make the assumption that, like humans, neural nets can learn faster, starting from simpler tasks and increasing the difficulty incrementally. For example, if you wanted to learn to ride a motorcycle, it would be faster if you already knew how to ride a bike since you have learned concepts such as equilibrium, brake, circulation, etc. Learning these concepts directly from a more complex task can be time-consuming as there are more things to consider compared with the easiest task. Even if the concepts are learned from just the complex task, it could be biased to that task and suboptimal. Following the analogy with the motorbike, if you can ride the motorbike and the bike, then probably you will have a less unbiased sense of equilibrium. This unbiased sense is because, e.g. you cannot associate it with just the weight or shape of the motorcycle. In this context, we propose a novel method to improve the neural networks performance for time series forecasting, hereinafter called Hierarchical Learning Network (HLNet). For that purpose, we divide the neural network into a hierarchy specifying a simpler input to the firsts layers and a more complex input to the following layers. Note that we consider a simple hierarchy of tasks formed as a sequence. Simplification is obtained by applying a smoothing process to the input. The first layers transfer their knowledge to all the following layers, with which the following layers can take advantage of this knowledge and learn complex outcomes faster. Our proposal uses LSTM layers as basic blocks as recent literature shows the best results in most of datasets [17]. The main contributions of this paper are summarized as follows: −A novel methodology applied to a LSTM keeping a similar or better efficacy among several datasets while improving the efficiency. −The proposal of an interpretable mechanism applied to a model that includes some keys of the learned behavior to each layer. −The comparison of our proposal with LSTM [11], NBEATS [24], DeepAR [26], Temporal Fusion Transformers (TFT) [20] and Echo State Network (ESN) [12] using 13 different datasets from different application fields. DeepAR and ESN were selected due to it uses 988 Hierarchical Method for Enhancing Time Series Forecasting recurrent layers, while NBEATS and TFT were selected because it obtained remarkable results in several datasets. The remainder of this paper is structured as follows. In Section 2, we describe the work related to this paper focusing on the main differences. Section 3 we describe the methodology applied to this paper. Section 4 shows the experimentation carried out with a wide set of datasets and a scalability analysis comparing HLNet with the best model with best results in the experimentation. Finally, we discuss the conclusions of this work and future directions. 2 Related works Previous related work consider the use of multi smoothing or multiresolution time series in order to improve the performance of the models. Usually these methods use some model to learn each representation and then join each one. In some cases, these methods are inefficient because they need to train as many models as representations and consider the learning of each representation independently. Another type of methods change the training algorithm using some heuristic in order to learn before from simpler instances. Kourentzes and Petropoulos [15] propose an extension of MAPA [14] and tested stock keeping unit demand. Their method uses several non overlapping representations of the data and fit an arbitrary model for each one, then the models predictions are averaged by an unweighted mean or median. Although, this work makes use of several smoothing methods to make more accurate and robust predictions, our proposed method uses them differently. In our method, we do not isolate the models and let them leverage automatically their knowledge to improve the predictions and all the representations are trained jointly improving the training performance. Afolabi et al. [1] propose a method in which a noise elimination process based on empirical mode decomposition is used in original time series generating what they call meta-info, then the original and the meta-info is fed into a model to get the predictions. The authors tested their method on the NARX networks [21] and LSTM with good results. Our proposed method do not focus in noise elimination, in the simplification process we do not worry about losing information but obtain some insights about the time series. Furthermore, the simplification lines up with the model architecture, and we do not only input the simplified versions as an attribute of the input. Wu et al. [33] proposed the RESTful framework in order to improve the forecasting performance using the multi-resolution patterns inside the time series. The authors proposed a process of two steps, in the first step the model uses an GRU network to extract the hidden state of each time series and each resolution, then at the second step a convolutional layer is used to extract the pattern of each hidden state and obtain the predictions. Our method do not just lies in the multi-resolution representations but also in the smooth representation of the time series to extract the useful patterns. Furthermore, the proposed method is an one-step process in which the extraction of features and the final predictions are produced in a single model. Finally, the multiple representations of the time series are not isolated as we previously mentioned. Bahrpeyma et al. [3] used multiple representations of time series and trained a recurrent neural network for each one. The recurrent neural networks predict the multiple horizons and finally use a weighted average based in the model performance. As in previous work, this method presents the issue of model isolation and performance. Salinat et al. [26] improved an LSTM model by adding a probabilistic forecasting in which the forecasting corresponds to an range instead of a single value. The range is obtained using a likelihood model that determines the parameters of the predicted distribution, such as a normal Hierarchical Method for Enhancing Time Series Forecasting 989 distribution. Although this model is effective, presents the same problems as an usual LSTM in terms of efficiency. Additionally, the probabilistic forecasting is out of the scope of this work. B. N. Oreshkin et al. [24] propose a novel interpretable architecture called NBEATS. The key idea is the inclusion of backward and forward residual links. The backwards reduce the complexity of the input time series, while the forwards are a partial prediction. The final forecasting sums up all the partial predictions. Although the architecture is very efficient, its structure is based on fully connected layers, which may perform poorly in time series datasets. Additionally, interpretability is based on a trend-seasonality decomposition that imposes a strong inductive bias on the model. A similar idea to our proposal is the curriculum learning [13]. In this approach, the complexity of the task is increased during the training ordering the data by some criteria. Koenecke and Gajewar [13] applied the curriculum learning to financial time series and obtained good results. They used the baby steps algorithm and the seasonal decomposition as the complexity ordering criteria. In our method, we focus on changing the architecture more than the training algorithm, without making assumptions about the complexity into segments of the time series. Another idea similar to our proposal is multitask learning. In this type of learning, the models try predicting several outputs for each task using a shared model for all tasks. Jinliang Deng et al. [5], propose a multitask time series forecasting method that adds a Multi-View Multi-Task (MVMT) block to residual blocks in the WaveNet architecture [32]. Although similar, the learning process is not the same. HLNet would be more similar to auxiliary learning [19] as we are interested in optimizing the final prediction task using other tasks. However, to our knowledge, there is no multistep time series forecasting architecture for hierarchical auxiliary tasks. Recent architectures use the novel proposed self-attention mechanism [18,38,39]. However, this mechanism use a sparse approach to compensate the issues presented by the self-attention mechanism like quadratic time complexity or high memory consumption. Additionally, recent studies [36] discuss the effectiveness of this type of methods being far from substitute previous architectures. 3 Methodology Traditional deep learning networks stack a set of layers. Some studies seems to indicate that some type of layers, like convolutional[35], learn more abstract low-level representations. Then, the following layers use this information to create higher-level representations and, finally, generate the prediction. In this paper, we introduce a novel methodology applied to neural networks that proposes a new architectural design. In our proposal, every group has an input and two outputs. The input is a simplified version of the original input, the first output is the knowledge learned in the group and the second output includes a simplified prediction. Each group of layers has a harder input/output than the previous groups. Thus, the first group of layers has the easiest objective and, as we process the following layers, more complex objectives are assigned. To understand the proposed architecture, we must specify the concepts of simple and complex objectives. First, we define an input sequence as a multivariate time series Sm,w(t), which stands for munivariate time series of length w. That is, Sm,w(t)={S1(t),S2(t),...,Sm(t)}, where every Si(t) identifies a single time series of length w. Figure 1illustrates an input multivariate time series S10,1000(t)with w=1000 and m=10. The model simplifies the input sequence for each group igenerating a sequence ˆ Si m,w(t), which is simpler than Sm,w(t). The definition of a simpler input sequence in this paper corresponds to a smoothed version of the input sequence. Smoothing helps us identify patterns in time series, such as trend, and obtain an input sequence with less noise and variation. 990 Hierarchical Method for Enhancing Time Series Forecasting FIGURE 1. Input sequence S10,1000(t)composed of 10 time series with 1000 samples each. The moving average has been used as the smoothing method. This method receives an hyperparameter (σ) that control the smoothness applied to the input of each group. The parameter σi∈N, hereinafter smooth factor, specifies the window size on the moving average for each group i.The larger values of σi, the greater the smoothness, and also the degree of simplification. Therefore, the first groups have higher values of σthan the following groups. Note that σ=1 for the last group of layers, keeping the input unmodified. Figure 2shows the HLNet architecture. Each group (Gi) starts with the SmoothLayer providing their respective σigenerating ˆ Si m,w(t), where wmeans that the time series length can vary after the smoothing process. The LSTM layer receives ˆ Si m,w(t)and obtains the first output: the hidden state Hi, which encodes the learned representation of the entire sequence. Then, the current hidden state is concatenated with previous hidden states. This is a critical step as, at this point, the previous groups share the knowledge learned from simpler representations with subsequent groups. Being Githe function for the group ithat outputs its hidden state Hiand fithe FeedForward layer from group i, the output for each output is defined as a function of the current group concatenated with all previous hidden states: Oi 1,...,h=fi(Gi(Sm,w(t)) +...+G1(Sm,w(t))) Note that the arrow that goes from the concatenation of the hidden states to the next group in Figure 2is dashed as the gradients of this connection are not considered in the backpropagation algorithm. Not considering this connection isolates the learning process for each group, which prevents other groups from modifying the current group at backpropagation. Finally, the concatenated hidden state is fed into the FeedForward layerofgroupi, which outputs the predictions for each horizon of size h[8]. Each group igenerates the second output Oi 1,...,hso, at the end, we have as many outputs as groups. However, only the final group has the real prediction. Hierarchical Method for Enhancing Time Series Forecasting 991 FIGURE 2. Proposed HLNet architecture. During the training, the loss for each group is calculated using the second output and added to generate the final loss. Note that the prediction of each group must have the same smoothing process than the input using their corresponding σ. 4 Experimentation 4.1 Datasets To validate our proposal, 13 publicly available datasets from different application fields have been selected. This collection of datasets represents a wide variety of time series forecasting problems, covering different domains, sequence/window sizes and forecasting horizons. Table 1presents the characteristics of each dataset in detail. In total, the experiments involve more than 50,000 time series among all selected datasets. Furthermore, most of the selected datasets have been used in forecast competitions and other TSF reviews [10]. The Demand dataset [6] records the electricity demand from January 2015 to December 2015, which was included and obtained from the Spanish electricity network website. This dataset contains only one time series sampled from the same building every 10 minutes. We divided the time series into several non-overlapping segments of size 576 time steps, giving us 72 time series. The goal is to predict the values for the next 24 hours. The CIF 2016 competition [27] dataset contains 72 monthly time series, of which 12 of them have a 6-month forecast horizon (CIF16o6) while the remaining 57 series have a 12-month forecast horizon (CIF16o12). Some of these time series are real bank risk analysis indicators while others are artificially generated. 992 Hierarchical Method for Enhancing Time Series Forecasting TABLE 1. Number of time series (N), forecast horizon (FH), maximum (M) and minimum (m) sequence sizes for each dataset used. Dataset NFH M m Demand 72 144 432 144 CIF16o12 57 12 36 15 CIF16o6 15 618 7 ExchangeRate 8 6 18 7 M3 1428 18 36 22 M4 48000 18 36 22 NN5 111 56 168 70 SolarEnergy 137 618 7 Tourism 336 24 48 30 Traffic 862 24 72 30 Traffic-metr-la 207 12 36 15 Traffic-perms-bay 325 12 24 15 WikiWebTraffic 997 59 118 73 The ExchangeRate (ER) dataset [16] records the daily exchange rates of 8 countries including Australia, UK, Canada, Switzerland, China, Japan, New Zealand and Singapore from 1990 to 2016 (7588 days for each country). The goal of this dataset is to predict values for the next 6 days. M3 [22] and M4 [25] competitions include time series of different domains and observation frequencies. However, due to computational time constraints, we have limited the experimentation to the monthly time series as they are longer than the yearly, quarterly, weekly and daily time series. An 18-month prediction is required for both competitions and contains time series of different categories, such as industry, finance or demography. The number of instances belonging to each category is evenly distributed according to their presence in the real world, leading to representative conclusions. Taking into account only the monthly time series, the M4 dataset has 48000 time series, while M3 is considerably smaller with 1428 time series. The NN5 competition dataset [4] is made up of 111 time series with a length of 735 values. This dataset represents 2 years of daily cash withdrawals at automatic teller machines (ATM) in England. The competition established a forecast horizon of 56 days. The SolarEnergy (SE) [37] dataset contains solar power production records in 2006, which are sampled every 10 minutes from 137 solar photovoltaic power plants in the state of Alabama. The goal is to predict an hour. The Tourism dataset [2] is made up of 336 monthly time series of different lengths, requiring a 24 months prediction. The data contain tourism bodies from Australia, Hong Kong and New Zealand, representing total tourism numbers at the country level. Traffic-metr-la (TML) and Traffic-perms-bay (TPB) are two public traffic speed datasets [34]. Traffic-metr-la contains traffic speed information from 207 sensors on the highways of Los Angeles for 4 months. Similarly, Traffic-perms-bay records 6 months of statistics on traffic speed from 325 sensors in the Bay area. All data were aggregated every 5 minutes and the goal is to predict one hour in the future. The Traffic dataset [31] is a collection of hourly time series from the California Department of Transportation collected during 2015 and 2016. The data describes the road occupancy rates Hierarchical Method for Enhancing Time Series Forecasting 993 (between 0 and 1) measured by different sensors on San Francisco Bay area freeways sampled every 10 minutes. The goal is to predict 4 hours in the future. The WikiWebTraffic dataset (WWT) belongs to a Kaggle competition [9] with the goal of predicting future web traffic of a given set of Wikipedia pages. The web traffic is measured in the daily number of hits and the forecasting horizon is 59 days. 4.2 Experimental settings The selected datasets contain different time series measured independently in different times and locations. For each time series in the datasets, the data is splitted into train, validation and test using a fixed origin way [10]. If each time series has length L,the window size is Wand the horizon size is H. The size of the valid and test splits is W+H, while the train split is L−W−H. Note that the validation strategy selected evaluates a division for each time series and average the metrics for each one. This validation strategy provide a way of know the general performance of the model in all time series. We set up a homogeneous experimentation environment for all models and datasets to ensure fair comparisons. A grid search has been used for hyperparameter search and the model with best results on the test set has been selected [29]. The hyperparameter space consists of hidden size, number of layers, sequence size and, just in our modified version, smooth factor σ. The hidden size has been selected to have 32, 64 or 128 neurons, the sequence size has been selected from a range depending of the forecasting horizon size using a 125%, 200% or 300% of the horizon size. The number of layers can be 2 or 4. Moreover, Adam optimizer was used for both models, and the training was stopped if there was no improvement in 5 consecutive epochs. LSTM, DeepAR, NBEATS, TFT and ESN models has been selected to be compared with our proposal due to its remarkable results in several scenarios. LSTM [11] model employs the original layer and uses the last hidden state to produce a multihorizon prediction. DeepAR [26] uses a single LSTM layer to produce a normal distribution of a single-horizon forecasting. The mean of the distribution and an autoregressive approach has been selected to produce the final multihorizon forecasting and using an iterative approach. NBEATS [24] model has been selected using the generic block proposed in the original paper, which does not rely on a seasonal-trend decomposition. TFT [20] has been selected using a direct approach for the multi-horizon forecasting. As the model was not compatible with the libraries used during the experimentation, only the CPU was used for training and the number of layers was limited to 2. ESN [12] employs another type of recurrent neural network obtaining competitive results compared with LSTM or GRU. As in LSTM, the last state is used to make the multihorizon forecasting. All experiments have been executed under the same machine with the same hardware configuration. The hardware configuration is composed of an NVIDIA Geforce RTX 3070 GPU, an AMD Ryzen 7 5800X 3.8GHz and 32 GB RAM memory. 4.3 Evaluation metrics We compared the results in the test instances between LSTM and HLNet with three forecasting metrics and one performance metric. The weighted average percentage error (WAPE) was selected because it is more robust than the mean absolute percentage error for diverse datasets, when there exists values between 0 and 1. Additionally, in order to avoid a division by zero a constant =1e−10 has been added into the 994 Hierarchical Method for Enhancing Time Series Forecasting TABLE 2. Best model parameters found by grid search. LSTM DeepAR NBEATS TFT ESN HLNet Layers Units Layers Units Layers Units Layers Units Layers Units Layers Units GF Demand 232 164 2128 264 2128 464 2 CIF16o12 464 1128 464 264 432 464 6 CIF16o6 432 1128 264 264 464 264 2 ER 264 132 232 2128 2128 432 2 M3 2128 164 232 264 432 232 6 M4 232 132 232 232 2128 432 8 NN5 2128 164 432 264 432 464 4 SE 232 1128 464 2128 464 432 2 Tourism 2 128 164 2128 264 432 232 6 Traffic 2128 1128 464 2128 232 264 4 TML 264 164 432 2128 432 432 2 TPB 464 132 4128 264 4128 4128 4 WWT 4 32 164 4128 232 432 4128 6 denominator. WAPE =h t=1|yt−ˆyt|∗100 +h t=1yt Finally, we measured the training time in order to show the performance of the model. 4.4 Results and discussion Table 2shows the best parameters by the grid search. In this table, we show the parameters in terms of layers, units for each layer and, just in our method, the σparameter used for the SmoothLayer. In general, there is no a clear pattern which relates the units and layers of the evaluated models. It seems that HLNet does not need more or less parameters than other models evaluated, not requiring more parameters in most of datasets. Table 3shows the efficacy of each model for all the datasets. In terms of efficacy, we may conclude that DeepAR and NBEATS performs poorly in most of datasets. We hypothesize that the autoregressive forecasting strategy in the DeepAR model and the decomposition learned by the generic block in NBEATS may be the cause of the low efficacy. The autoregressive may accumulate too much error that causes the error increases considerably and some datasets may not show a good seasonal or trend pattern. TFT and ESN models obtained remarkable results, but were not good enough to compare with LSTM and HLNet as they obtained better results in more datasets. LSTM and HLNet show a similar efficacy in most of datasets. There are five datasets in which LSTM has better metrics and six datasets in which HLNet has better metrics. There are no notable differences in most datasets, but the Demand dataset shows at least a 10% improvement of HLNet over LSTM in all metrics. 1In terms of efficiency, Table 4shows that DeepAR and NBEATS obtain the best results. However, as observed in Table 3this improvement in efficiency comes with a considerable decrease 1Note than the training was performed by using a CPU. Hierarchical Method for Enhancing Time Series Forecasting 1001 by creating general representations of the series and transfer their knowledge to the next layers, the following layers can learn a more detailed representation faster. On the contrary, LSTM needs extra time to learn the general representation of the series. These results emphasize the idea that recurrent neural networks organize their knowledge in a hierarchical way, from more general representations to more specific ones. 5Conclusions and future works In this paper we have introduced HLNet, a novel method for time series multistep forecasting. Thus, we have added a structural change in LSTM networks that allows learning from simpler to more complex time series in a hierarchical way. This is done by smoothing the original input and transferring the extracted knowledge to the subsequent levels. The results have shown a great improvement in terms of training time keeping the efficacy, which leads to the use of deeper models with less execution time. As future works, other smoothing strategies rather than MA could be followed. Additionally, the development of a custom loss function with different weights for each level and evolving with the number of epochs is proposed. Acknowledgements This research has been supported by the grant PID2020-117954RB-C22 and PID2020-117954RB- C21 funded by MICIU/AEI/ 10.13039/501100011033. This work has also been supported by TED2021-131311B-C21 and TED2021-131311B-C22 funded by MICIU/AEI/10.13039/501100011033 and by the European Union NextGenerationEU/PRTR and Junta de Andalucía for projects PYC20 RE 078 US, PY20-00870 and UPO-138516. Funding for open access publishing: Universidad de Sevilla’s library. References [1] D. Afolabi, S. Guan, K. Man, P. W. H. Wong and X. Zhao. Hierarchical meta-learning in time series forecasting for improved interference-less machine learning. Symmetry,9, 283, 2017. [2] G. Athanasopoulos, R. J. Hyndman, H. Song and D. C. Wu. The tourism forecasting competition. International Journal of Forecasting,27, 822–844, 2011. [3] F. Bahrpeyma, A. Mccarren and M. Roantree. Multi-resolution forecast aggregation for time series in Agri datasets. In Proceedings of the Irish Conference on Artificial Intelligence and Cognitive, vol. 25, pp. 1–12, 2017. [4] S. F. Crone. NN5 Time Series Forecasting Competition for Neural Networks,http://www.neuralforecasting-competition.com/, 2008. [5] D. Jinliang, C. Xiusi, J. Renhe, S. Xuan and W. T. Ivor. A Multi-View Multi-Task Learning Framework for Multi-Variate Time Series Forecasting. In IEEE Transactions on Knowledge and Data Engineering, vol. 35, pp. 7665–7680, 2023. [6] Red Eléctrica de España www.ree.es, 2015. [7] A. Galicia, R. Talavera-Llames, A. Troncoso, I. Koprinska and F. Martínez-Álvarez. Multi-step forecasting for big data time series based on ensemble learning. Knowledge-Based Systems, 163, 830–841, 2019. [8] A. Galicia, J. F. Torres, F. Martínez-Álvarez and A. Troncoso. A novel spark-based multi-step forecasting algorithm for big data time series. Information Sciences,467, 800–818, 2018. 1002 Hierarchical Method for Enhancing Time Series Forecasting [9] Google Web Traffic Time Series Forecasting Competition, 2017. https://www.kaggle.com/c/ web-traffic-time-series-forecasting [10] C. Bergmeir, H. Hewamalage and K. Bandara. Recurrent neural networks for time series forecasting: current status and future directions. International Journal of Forecasting,37, 388–427, 2021. [11] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation,9, 1735–1780, 1997. [12] H. Jaeger. The Echo State Approach to Analysing and Training Recurrent Neural Networkswith an Erratum Note, vol. 148. German National Research Center for Information Technology GMD. Technical Report, Bonn, Germany, 2001. [13] A. Koenecke and A. Gajewar. Curriculum learning in deep neural networks for financial forecasting. In Lecture Notes in Computer Science, p. 11985, 2020. [14] N. Kourentze, F. Petropoulos and J. R. Trapero. Improving forecasting by estimating time series structural components across multiple frequencies. International Journal of Forecasting,30, 291–302, 2014. [15] N. Kourentzes and F. Petropoulos. Forecasting with multivariate temporal aggregation: The case of promotional modelling. International Journal of Production Economics,181, 145–153, 2016. [16] G. Lai, W. Chang, Y. Yang and H. Liu. Modeling long- and short-term temporal patterns with deep neural networks. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, vol. 41, pp. 95–104, 2018. [17] P. Lara-Benítez, M. Carranza-García and J. C. Riquelme. An experimental review on deep learning for time series forecasting. International Journal of Neural Systems,31, 2130001, 2021. [18] S. Li, X. Jin, Y. Xuan, X. Zhou, W. Chen, Y. Wang and X. Yan. Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting, 2019. [19] L. Liebel and M. Körner. Auxiliary Tasks in Multi-Task Learning, arXiv, 2018. [20] B. Lim, S. O. Arik, N. Loeff and T. Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting,37, 1748–1764, 2021. [21] T. Lin, B. G. Horne, P. Tino and C. L. Giles. Learning long-term dependencies in NARX recurrent neural networks. IEEE Transactions on Neural Networks,7, 1329–1338, 1996. [22] S. Makridakis and M. Hibon. The M3-competition: results, conclusions and implications. International Journal of Forecasting,16, 451–476, 2000. [23] F. Martínez-Álvarez, A. Troncoso, G. Asencio-Cortés and J. C. Riquelme. A survey on data mining techniques applied to electricity-related time series forecasting. Energies,8, 13162–13193, 2015. [24] B. N. Oreshkin, D. Carpov, N. Chapados and Y. Bengio. N-BEATS: Neural Basis Expansion Analysis for Interpretable Time Series Forecasting. arXiv, 2019. [25] E. Spiliotis, S. Makridakis and V. Assimakopoulos. The M4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting,36, 54–74, 2020. [26] D. Salinas, V. Flunkert, J. Gasthaus and T. Januschowski. DeepAR: probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting,36, 1181–1191, 2020. [27] M. Stepnicka and M. Burda. Computational Intelligence in Forecasting (CIF),https://irafm. osu.cz/cif/main.php, 2016. Hierarchical Method for Enhancing Time Series Forecasting 1003 [28] R. Talavera-Llames, R. Pérez-Chacón, A. Troncoso and F. Martínez-Álvarez. MV-kWNN: a novel multivariate and multi-output weighted nearest neighbours algorithm for big data time series forecasting. Neurocomputing,353, 56–73, 2019. [29] J. F. Torres, A. Galicia, A. Troncoso and F. Martínez-Álvarez. A scalable approach based on deep learning for big data time series forecasting. Integrated Computer-Aided Engineering,25, 335–348, 2018. [30] J. F. Torres, D. Hadjout, A. Sebaa, F. Martínez-Álvarez and A. Troncoso. Deep learning for time series forecasting: a survey. Big Data,9, 3–21, 2021. [31] https://pems.dot.ca.gov/ California Department of Transportation, 2015. [32] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior and K. Kavukcuoglu. WaveNet: A Generative Model for Raw Audio. arXiv, 2016. [33] X. Wu, B. Shi, Y. Dong, C. Huang, L. Faust and N. V. Chawla. RESTFul: resolution-aware forecasting of behavioral time series data. In Proceedings of the ACM International Conference on Information and Knowledge Management, pp. 1073–1082, 2018. [34] C. Shahabi, Y. Li, R. Yu and Y. Liu. Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting. In Proceedings of the International Conference on Learning Representations, vol. 18, 2018. [35] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In Proceeddings of the European Conference on Computer Vision, vol. 8689, pp. 818–833, 2014. [36] A. Zeng, M. Chen, L. Zhang and Q. Xu. Are Transformers Effective for Time Series Forecasting? In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 23, 2022. [37] Y. Zhang. Solar Power Data for Integration Studies (NREL),https://www.nrel.gov/grid/solarintegration-data.html, 2007. [38] C. Zheng, X. Fan, C. Wang and J. Qi. GMAN: a graph multi-attention network for traffic prediction. In Gman: A Graph Multi-Attention Network for Traffic Prediction, vol. 34, pp. 1234–1241, 2019. [39] H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong and W. Zhang. Informer: beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 11106–11115, 2021. Received 20 May 2022