An Extensive Comparative Between Univariate and Multivariate Deep Learning Models in Day-Ahead Electricity Price Forecasting

An Extensive Comparative Between Univariate and Multivariate Deep Learning Models in Day-Ahead Electricity Price Forecasting Bel´en Vega-M´arquez(B), Javier Sol´ıs-Garc´ıa, Isabel A. Nepomuceno-Chamorro, and Cristina Rubio-Escudero Department of Computer Languages and Systems, University of Sevilla, Sevilla, Spain {bvega,inepomuceno,crubioescudero}@us.es Abstract. Electricity is a product that has greatly changed the way we think about the world. In fact, it is one of the indicators that show the progress of a civilisation. With the deregulation of the electricity market in the 1990s, electricity price forecasting became a fundamental task in all countries. This task was a challenge for both producers and consumers, as many factors had to be taken into account. In this paper, given the large number of factors that influence the electricity market price we have conducted several experiments to see how the use of multiple variables can improve the effectiveness of price prediction. Tests have been performed with different time periods using data from Spain and different Deep Learning models, showing that the use of these variables represents an improvement with respect to the univariate model, in which only the price was considered. Keywords: Univariate vs multivariate ·Deep Learning ·Day-ahead electricity price forecasting ·Time series forecasting 1 Introduction The deregulation that took place in the electricity market worldwide in the late 1990s brought a major change in the way the market was viewed. This significant change meant the beginning of a very competitive market in which prices were constantly changing [10]. This change challenged all players in the market, both consumers and producers, as both seek to maximise their profits and optimise their services respectively. These interests have made price one of the target variables within this market, with special interest being paid to the prediction of the day-ahead electricity price. This day-ahead prediction is an advantage for all market participants, since on the one hand, having a reliable prediction of it, electricity producers can make their bids before the closing time in a strategic way and, and on the https://doi.org/10.1007/978-3-030-87869-6_64 676 B. Vega-M´arquez et al. other hand, the system operator can control that there are no anomalies in the supply and demand of this valuable commodity [21]. Unlike other markets, the main characteristic of the electricity market is that it works with a commodity that cannot be stored, which implies a continous effort to maintain a constant balance between consumption and production [24]. Moreover, very recently, with the rise of renewable energies [22], the price of electricity has become a very volatile commodity that is strongly influenced by seasonality. All these characteristics make electricity price forecasting a difficult task and one in which many different aspects should be considered. In recent decades, numerous Deep Learning forecasting models have been proposed for solving the aforementioned challenge in the field of electricity price forecasting. In [9] SEPNET is developed, a hybrid model composed by a Variational Mode Decomposition (VMD), a Convolutional Neural NEtwork (CNN) and Gated Recurrent Unit (GRU) for short-term electricity prediction. Zhang et al. [24] applied a new adaptive hybrid model based on variational mode decomposition (VMD), self-adaptive particle swarm optimization (SAPSO), seasonal autoregressive integrated moving average (SARIMA) and deep belief network (DBN) for short term electricity price forecasting using data from Australia, Pennsylvania and Spain. In [25] an optimized heterogeneous structure LSTM was proposed. It was constructed from electricity price decomposition in combination with a sequence model-based optimization (SMBO) to optimize its hyperparameters. A Deep Learning based hybrid framework for day-ahead electricity price forecasting was proposed by [23] using PJM market data. Lastly, in [7] a Deep Convolutional Neural Network in combination with Label Distribution Learning Forests is used to forecast electricity price in Singapore proving that this method provides superior forecasts over existing approaches. It has also been shown that Machine Learning models are effective tools for electricity price forecasting, as proposed in [2], where a comparative study using Deep Learning and Support Vector Regression for electricity price forecasting in smart grids was performed. These papers mentioned above make the prediction from an univariate point of view, i.e. they do not use variables exogenous to the price (multivariate methodology). Several studies have used this methodology, obtaining quite promising results [11,19]. From a multivariate point of view, it is possible to obtain external information that directly influences the price and that can help to model the price behaviour. In this paper, we have proposed a methodology that extracts the characteristics that are most correlated with price, depending on the input dataset, to serve as input to Data Science models known in the literature. We have made a comparison with three different datasets of electricity data from the Spanish market and seven different models, showing that the results are improved by adding more exogenous variables as input to the models. The rest of this paper is organized as follows: Sect. 2introduces our methodology defined in detail (forecasting models, experimental setup and the evaluation procedure). In Sect. 3results are explained. Finally, Conclusions are provided in Sect. 4. Comparative Between Univariate and Multivariate Models in EPF 677 2 Methodology The aim of this paper is to provide a fair comparison between the use or not of exogenous information for day-ahead price prediction in the electricity market. The first step in our framework consisted on extracting the features that have a strong influence on the target variable, i.e., the price. We used the Pearson’s correlation coefficient as a metric to show the influence among variables. Next, we analysed several well-known algorithms, from Deep Learning approaches to standard Machine Learning methods, to obtain the best results. 2.1 Hyperparameter Optimization and Training Relying on the review performed in [12], where many different architectures were compared, we have decided to use the ones with best results: Long Short-Term Memory (LSTM), Convolutional Neural Networks (CNN) and Temporal Convolutional Networks (TCN). Moreover, we have chosen the standard Multilayer Perceptron (MLP), and a new recurrent model, as it is Gated Recurrents Unit (GRU). We have also chosen another two popular Machine Learning models in the literature: Regression Trees (REG. TREE) and Random Forest (RF). All models are listed below along with a brief description: –LSTM: Introduced in 1997, they can model temporal dependencies in larger horizons without forgetting the short-term patterns [8]. –GRU: They were introduced in 1997 to learn long-term dependencies and further improved over the following years [16]. –CNN: They were mainly designed for computer vision tasks; however, they had groundbreaking results over the past decade in a variety of fields related to pattern recognition; from image processing to voice recognition [1]. –TCN: It is a new architecture based on CNN, but more specialized in temporal series, which is inspired by the Wavenet autoregressive model [13], which was designed for audio generation problems. –MLP: It is the most basic type of feed-forward artificial neural network, with a variable architecture, but, in general, consists of one input layer, several hidden layers, and one output layer [5]. –REG. TREE: It consists of a particular type of decision trees, which is characterized as quite interpretable, and has a reasonably high accuracy, dealing with a continuous goal variable, as electricity price [17]. –RF: It builds a large number of regression trees in order to solve regression tasks. These trees, which act as regression functions, are combined using bootstrap or ensemble techniques in order to come to a final decision [15]. 678 B. Vega-M´arquez et al. To train these models, we have divided our datasets into two different sets: train and validation. The train set has been used to find the best combination of hyperparameters for each model, and for measuring the precision, we used the validation set. In both subsets the Multi-Input Multi-Output (MIMO) technique has been used [12] to avoid error accumulation in the predictions. A grid search methodology [18] has been used to cover as many parameters as possible. All combinations of hyperparameters used and the implemented code can be found at [20]. Both train and validation sets are composed of windows that contain our features’ values at each hour. With one window as input, the model generates another window with the forecasted values for each of the consecutive 24 h of electricity price. 2.2 Validation To evaluate the performance and precision of the proposed models, we used two different metrics: MAE and WAPE. Mean absolute error (MAE), shown in Eq. 1, measures the average absolute error between the original time series and the forecasted one. The closer to 0, the better the result, because it means that the error is low. Also, MAE gives us the error on the same scale as the measured data. MAE(y, o)=mean(|y−o|)= 1 n n  i=1 |yi−oi|(1) Weighted average percentage error (WAPE), shown in Eq. 2, is similar to MAE, except because it is divided by the mean of the time series, thus allowing to compare the error throughout a time series with different scales. These reasons make WAPE a scale-independent measure with great interpretability. As WAPE is an error measurement, the closer it is to 0, the better the results will be. WAPE(y,o)=MAE(y, o) mean(y)=mean(|y−o|) mean(y)(2) 3 Results and Discussion This section will explain the results obtained. First, the chosen datasets will be presented. Afther this, taking into account the correlation coefficient, the variables selected for each dataset will be shown. Finally, the best results for each dataset and each model will be presented, according to the parameterization of each one of them. 3.1 Datasets In this study, we have focused our attention on the electricity price forecasting. The main objective of this study is to see how the choice of multiple variables Comparative Between Univariate and Multivariate Models in EPF 679 can improve the effectiveness of the prediction results. For each of the chosen datasets, three months have been selected: two months have been used to set the parameters of the models and the remaining month to validate them. Attempts have been made to choose periods with some particularity. All data used in the study have been retrieved from [4] and will be explained below. –Normal period: In order to have a reference period, a stage has been chosen in which no anomalies were detected, unlike the two following data sets. This stage covers the months from September to December 2019. –Quarantine Period: The year 2020 marked a change in many daily habits due to the global pandemic suffered. One of the most notable changes was the increase in electricity consumption, as people spent more time at home due to quarantine requirements. The three months from March 15th to June 15th 2020 have been chosen to represent this stage. –Fraud Period: In 2019, the National Commission for Markets and Competition (CNMC) fined Endesa and Naturgy for altering electricity prices between October 2016 and January 2017 [14], the months chosen for this study. This period will help to test how sensitive the models are to periods that may present tax irregularities. 3.2 Feature Selection To improve the benchmark results, further characteristics have been sought to enhance our price forecast. Based on the references [3] and [6], we focus on variables associated to renewable energies and the ones used in our initial experiment. The features under consideration have been Percentage of CO2-free generation (Percentage), Generation of eolic energy (Eolic), Generation of solar energy (Solar), Generation of hydraulic energy (Hydraulic) and Demand estimate (Demand). After obtaining the Pearson’s correlation coefficient of each feature with the price, it can be observed that correlation value changes depending on the period. Because of that, the two features more correlated (those with a higher amount in absolute value) with price in each period were selected. Table 1presents which variables have been chosen for each dataset, these variables are the ones marked in bold. Table 1. Correlations with price Demand Percentage Eolic Hydraulic Solar Quarantine 0.395 −0.508 −0.467 0.385 −0.056 Normal 0.624 −0.656 −0.513 0.5 −0.085 Fraud 0.744 0.184 −0.047 0.787 −0.146 680 B. Vega-M´arquez et al. 3.3 Experimental Results As previously mentioned, in order to see if the use of exogenous variables increases the prediction accuracy, a first study was made on the performance of the models with only one exogenous variable regardless of the correlation coefficient, to take it as a baseline. This main experimentation has been made with demand estimates and price; also, thinner models have been used. Table 2shows MAE results. The best values are obtained with models based on conventional Machine Learning methods, as Tree Regressor. Besides, good results in MLP models can be seen, which are the most simple neural network used in the study. Table 2. MAE Multivariate baseline Dataset LSTM CNN TCN GRU MLP TREE RF Quarantine 4.12 3.811 3.855 3.81 3.775 4.12 4.447 Normal 4.544 4.338 4.907 4.38 4.457 4.087 4.889 Fraud 4.542 4.754 4.611 4.3 4.755 4.025 4.528 Table 3shows WAPE results. It may be observed that they are very similar to MAE, except for the LSTM network, as it is the best model for the fraud period. Table 3. WAPE Multivariate baseline Dataset LSTM CNN TCN GRU MLP TREE RF Quarantine 0.132 0.122 0.123 0.12 0.12 0.132 0.143 Normal 0.092 0.088 0.1 0.09 0.091 0.06 0.072 Fraud 0.067 0.07 0.068 0.06 0.072 0.08 0.092 Once the new variables were added to each of the datasets an enhancement in comparison with the baseline could be observed. New models and new features have proved that they can extract more efficiently the potential of Deep learning, especially the convolutional and recurrent networks, as both type of models improve very much in terms of WAPE and MAE, while Machine Learning models have more difficulties in exploiting the correlations of the new features in order to obtain betters predictions. Table 4shows a comparison of MAE in each dataset after adding the two more correlated features, where the best result of the dataset are marked in bold. Quarantine period is the frame with the most significant enhancement, the best result was obtained with the GRU model, with an average error of 3.191e. This result makes an error reduction of −0.584econcerning the baseline, where Comparative Between Univariate and Multivariate Models in EPF 681 the best model was MLP. Concerning the normal period, the period with the minor enhancement, the best result was obtained with the MLP model, with an average error of 3.932e. This result makes an error reduction of −0.093e compared to the baseline, where the best model was Tree Regressor. Lastly, in fraud period, the best result was obtained with the CNN model, with an average error of 3.83e. This result makes an error reduction of −0.195ecompared to the baseline, where the best model was also Tree Regressor. Table 4. MAE results Dataset LSTM CNN TCN GRU MLP TREE RF Quarantine 3.401 3.426 3.524 3.191 3.361 3.626 3.836 Normal 4.087 3.976 3.984 3.96 3.932 4.351 4.432 Fraud 4.245 3.83 3.875 3.952 4.204 4.256 4.755 Concerning WAPE, the results be can seen in Table 5and are very similar to the MAE ones. In quarantine period, the best result is given by the GRU model with a WAPE of 0.102. Our new experiments provide a reduction of −0.018 WAPE to the baseline. Concerning normal period, the best result is given by the MLP model with a WAPE of 0.08. Our new experiments provide an increase of +0.2 WAPE to the baseline. Regarding fraud period, the best result is given by the CNN model with a WAPE of 0.056. Our new experiments provide a decrease of −0.011 WAPE to the baseline. Table 5. WAPE results Dataset LSTM CNN TCN GRU MLP TREE RF Quarantine 0.109 0.11 0.113 0.102 0.107 0.115 0.123 Normal 0.083 0.081 0.081 0.081 0.08 0.089 0.09 Fraud 0.063 0.056 0.057 0.058 0.062 0.063 0.07 With respect to models forecasting enhancement(see Tables 6and 7), there are many similarities in MAE and WAPE. The most significant improvements are shown by Deep Learning, especially in TCN, CNN and LSTM models, where the best model is TCN with a mean error decrease of −14.4% in the three periods. Machine Learning models show the slightest improvement. They even show worse results than the baseline. Besides, they offer the worst scores in terms of WAPE. 682 B. Vega-M´arquez et al. Table 6. MAE enhancement (%) Dataset LSTM CNN TCN GRU MLP TREE RF Quarantine −17.451 −10.102 −8.586 −16.246 −10.966 −11.99 −13.739 Normal −10.057 −8.344 −18.809 −9.589 −11.779 +6.459 −9.347 Fraud −6.538 −19.436 −15.961 −8.093 −11.587 +5.739 +5.013 Mean −11.349 −12.627 −14.452 −11.309 −11.444 +0.069 −6.024 Table 7. WAPE enhancement (%) Dataset LSTM CNN TCN GRU MLP TREE RF Quarantine −17.424 −9.836 −8.13 −15 −10.833 −12.878 −13.986 Normal −9.782 −7.954 −19 −10 −12.087 +48.333 +25 Fraud −5.97 −20 −16.176 −3.333 −13.889 −21.25 −23.913 Mean −11.059 −12.596 −14.435 −9.444 −12.27 +4.734 −4.299 4 Conclusions In this paper we propose a comparative framework for price forecasting in the electricity market. This framework is based on the use of different Deep Learning and Machine Learning algorithms to obtain the best possible performance, making use of exogenous variables. To show the improvements added by the use of exogenous variables, we performed the price prediction using only past price information, i.e., we obtained the results from a univariate point of view as a baseline model. The results of our experiments have shown the importance of conducting a prior analysis on which variables can influence the main goal. A total of three datasets and seven different models have been tested. For each of the data sets, the two most correlated variables according to Pearson’s correlation coefficient were chosen and the performance of each of the models was measured with two metrics, MAE and WAPE. The results obtained have shown that the use of exogenous variables for electricity price prediction is a point to be taken into account, as it adds additional information that helps the models in their predictions. As future work, we propose two research directions. On the one hand, to conduct a more exhaustive analysis of the exogenous variables to be used, choosing a larger number of variables, as well as different correlation coefficients in order to be able to make a more exhaustive ranking and search for the best exogenous variables. On the other hand, considering the temporal nature of the data, it would be advisable to study how the past can influence the final prediction, i.e. to use lag variables for each of the time instants to be predicted and see which past time instants have the greatest influence on the results. Comparative Between Univariate and Multivariate Models in EPF 683 References 1. Albawi, S., Mohammed, T.A., Al-Zawi, S.: Understanding of a convolutional neural network. In: 2017 International Conference on Engineering and Technology (ICET), pp. 1–6. IEEE (2017) 2. Atef, S., Eltawil, A.B.: A comparative study using deep learning and support vector regression for electricity price forecasting in smart grids. In: 2019 IEEE 6th International Conference on Industrial Engineering and Applications (ICIEA), pp. 603–607. IEEE (2019) 3. Cruz, A., Mu˜noz, A., Zamora, J.L., Esp´ınola, R.: The effect of wind generation and weekday on Spanish electricity spot price forecasting. Electric Power Syst. Res. 81(10), 1924–1935 (2011) 4. Red El´ectrica de Espa˜na, S.A.U.R.: esios: Sistema de informaci´on del operador del sistema (2020). https://www.esios.ree.es 5. Gardner, M.W., Dorling, S.: Artificial neural networks (the multilayer perceptron)- a review of applications in the atmospheric sciences. Atmos. Environ. 32(14–15), 2627–2636 (1998) 6. Gonz´alez, C., Mira-McWilliams, J., Ju´arez, I.: Important variable assessment and electricity price forecasting based on regression tree models: classification and regression trees, bagging and random forests. IET Generation, Transmission Distribution 9(11), 1120–1128 (2015) 7. He, H., Lu, N., Jiang, Y., Chen, B., Jiao, R.: End-to-end probabilistic forecasting of electricity price via convolutional neural network and label distribution learning. Energy Rep. 6, 1176–1183 (2020) 8. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997) 9. Huang, C.J., Shen, Y., Chen, Y.H., Chen, H.C.: A novel hybrid deep neural network model for short-term electricity price forecasting. Int. J. Energy Res. 45(2), 2511– 2532 (2021) 10. Jahangir, H., et al.: A novel electricity price forecasting approach based on dimension reduction strategy and rough artificial neural networks. IEEE Trans. Industr. Inf. 16(4), 2369–2381 (2019) 11. Jahangir, H., et al.: A novel electricity price forecasting approach based on dimension reduction strategy and rough artificial neural networks. IEEE Trans. Industr. Inf. 16(4), 2369–2381 (2020). https://doi.org/10.1109/TII.2019.2933009 12. Lara-Benıtez, P., Carranza-Garcıa, M., Riquelme, J.C.: An experimental review on deep learning architectures for time series forecasting (2020) 13. Oord, A.v.d., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., Kavukcuoglu, K.: Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 (2016) 14. Pa´ıs, E.: La cnmc multa a endesa y naturgy con 25 millones por alterar los precios de la electricidad (2019) 15. Segal, M.R.: Machine learning benchmarks and random forest regression (2004) 16. Shen, G., Tan, Q., Zhang, H., Zeng, P., Xu, J.: Deep learning with gated recurrent unit networks for financial sequence predictions. Procedia Comput. Sci. 131, 895–903 (2018). https://doi.org/10.1016/j.procs.2018.04.298.https://www. sciencedirect.com/science/article/pii/S1877050918306781, recent Advancement in Information and Communication Technology 17. Torgo, L.: Functional models for regression tree leaves. In: ICML, vol. 97, pp. 385–393. Citeseer (1997)