Probabilistic Forecasting Framework Oriented to Distribution Networks and Microgrids

IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING 1 Probabilistic Forecasting Framework Oriented to Distribution Networks and Microgrids Antonio Parejo , Graduate Student Member, IEEE, Sebastián García , Enrique Personal , Member, IEEE, Juan Ignacio Guerrero , Alejandro Carrasco , Member, IEEE, and Carlos León , Senior Member, IEEE Abstract— In electrical distribution networks an adequate management is key for supporting the deployment of renewable generation sources and microgrids while extracting their maximum potential. Among the existing optimization approaches, stochastic and probabilistic methods are experiencing a growth in their use. However, one of the problems when applying these approaches is the complexity of creating and evaluating the quality of the required stochastic forecasts compared to deterministic forecasts. To mitigate this difficulty, this paper proposes a probabilistic forecasting framework that integrates model creation, their evaluation, and the selection of the best model for predicting. Additionally, two novel methods are proposed for creating scenario sets, and a new metric is defined for evaluating and selecting which model to use. The proposed framework is applied in a case study over a dataset of ten secondary distribution substations from a real distribution network located in Manzanilla (Spain), showing the effect of the selection criteria over the forecasting quality. Note to Practitioners—This article was motivated by the challenge of probabilistic forecasting inclusion in automatic management systems applied to power distribution networks and microgrids. Modern stochastic management optimization methods are fed with probabilistic forecasts, which offer richer information than classic deterministic forecasting. Therefore, the management systems should be able to automatically train a certain number of forecasting models (e.g., machine learning models), evaluate and compare them, and apply the best ones for obtaining the forecasts to feed the management optimization system. Considering the variety of models, techniques, probabilistic forecast types, and evaluation metrics, it can be unclear how to perform this process. For these reasons, this article proposes a probabilistic forecasting framework that integrates methods for the construction of diverse types of predictions (quantiles, intervals, and scenario sets), their evaluation, and the selection of the best model for performing each required prediction for feeding optimization systems. This framework could help to facilitate the implantation of modern stochastic optimization management systems for distribution networks and microgrids, as it simplifies the forecasting process. Manuscript received 22 December 2023; accepted 30 January 2024. This article was recommended for publication by Editor Q. Zhao upon evaluation of the reviewers’ comments. The publication is part of the project TED2021-129702B-I00, funded by MCIN/AEI/10.13039/501100011033 and the European Union “NextGenerationEU”/PRTR. (Corresponding author: Antonio Parejo.) The authors are with the Department of Electronic Technology, Escuela Politécnica Superior, Universidad de Sevilla, 41011 Sevilla, Spain (e-mail: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]). Color versions of one or more figures in this article are available at https://doi.org/10.1109/TASE.2024.3361651. Digital Object Identifier 10.1109/TASE.2024.3361651 Index Terms— Stochastic forecasting, probabilistic forecasting, demand forecasting, microgrids, power distribution networks. I. INTRODUCTION THE recent generalized deployment of renewable generation in the power system over the world is supposing a huge challenge for distribution network management [1]. Specifically, distribution system operators (DSOs) are looking for increasing what they call the “observability” and “controllability” of their networks following the smart grid paradigm [2], obtaining a deeper knowledge about their state. This is required due to the high variability of distributed generation (DG) and the requirements of the upcoming growth of electric vehicles (EVs), which may cause congestion in the network during generation and consumption peaks [3]. In this sense, microgrids are becoming one of the key elements for the power system. A microgrid can be defined as a small electric network that is designed for a reliable and massive integration of distributed generation at the primary and secondary distribution level, especially the renewables (solar, wind, and other low carbon technologies) [4],[5]. For taking full advantage from the control capabilities of distribution networks (and microgrids), it is key to perform an optimized scheduling for their operation. This means, to decide which generation units and loads should be connected and disconnected, when it is convenient to charge a storage system, or when a microgrid should be swapped to zero-energy consumption. It can be used for managing demand response resources [6] and for facilitating the integration of EVs in the grid [7], among other applications. Two main general types of microgrid optimization approaches can be found in the literature. These can be deterministic (as for example in [6]) or stochastic (as in [8]). The latter is sometimes also called “probabilistic” when the associated probabilities are explicitly considered [9]. Depending on the approach to be followed, the requirements of input data, i.e., the variables of interest that must be forecasted or estimated to perform the optimization (e.g., expected load consumption, expected generation, etc.) will vary. Deterministic methods require a deterministic forecasting, while stochastic methods need more enriched information, such as quantiles, intervals, scenarios, etc. Therefore, forecasting systems need to automatically choose which prediction models are better to apply. © 2024 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. 2 IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING The evaluation of each form of probabilistic forecasting can be done using appropriate metrics (e.g., the pinball loss function and the Winkler score [10]). However, in a forecasting system some types of probabilistic forecasts may be obtained from others (e.g., a prediction interval with a certain associated probability can be obtained from a quantile set simply by choosing the upper and lower limits). For this reason, in some cases it could not be totally clear for practitioners of the industry how the evaluation, comparison, and selection of forecasting models could be handled, especially when different types of interrelated probabilistic forecasts are involved. Therefore, to help mitigating these problems, this paper proposes a forecasting framework that defines how to perform the construction of diverse probabilistic forecasts for serving as inputs for optimization systems when several forecasting models are available. Inside the framework, prediction intervals and scenario sets are constructed using quantile sets. The generation of scenario sets is done by two methods proposed in this paper, and their evaluation is performed using a new proposed metric. Furthermore, for performing the evaluation and selection of models, two different approaches are presented: i) choose the model that creates the best-quality quantile set for constructing all the required intervals and scenario sets; ii) evaluate individually the quality of all the available models for creating each type of interval and scenario set that is required, and then use the best-quality model for constructing each of the required intervals and scenario sets. To the best of the authors’ knowledge, the effects of these criteria have not been previously treated in the literature. The original contributions of this article can be summarized as follows: 1) Framework for obtaining probabilistic forecasts that automatically perform the evaluation and selection of forecasting models. It can provide quantiles, intervals, and scenario sets for the forecasted variables. Two possible approaches are presented for performing the model selection. 2) Two methods for creating scenario sets using quantile sets are proposed (called MiAs and ExAs). In this regard, an evaluation metric (called WePin) has been designed to make up for the shortcomings of the current metrics in scenario sets assessment. 3) As a case study, the proposed probabilistic framework has been applied to forecast the demand (obtaining quantiles, intervals, and scenario sets) of ten secondary distribution substations in the town of Manzanilla (Spain). The effects of the two approaches for model selection are compared and discussed for this case, showing how it could affect to the quality of the demand forecasting in a real distribution network. The rest of the paper is organized as follows. Section II makes a review of the state of the art. Section III introduces some methods and metrics from the literature that serve to construct and evaluate probabilistic forecasts, as some of these have been integrated in the framework. Section IV presents the proposed probabilistic forecasting framework. Section V corresponds to the case study. Then, a discussion on the results and next research steps is made in Section VI. Finally, Section VII summarizes the conclusions of the paper. II. STATE OF THE ART As previously said, there are diverse approaches for optimizing the management of microgrids (and, in general, distribution networks). According to [11], some Energy Management System (EMS) optimization approaches are: Model Predictive Control (MPC), Open Loop Feedback Control (OLFC, sometimes referred to as stochastic MPC) and stochastic dynamic programming (SDP). The mentioned stochastic methods can solve the optimization problem considering multiple scenarios with their inherent probabilities of occurrence. Theoretically, it would be possible to generate as many forecasted scenarios as it is desired for a certain variable. However, the inclusion of a high number of scenarios in stochastic optimization methods would drastically increase the computational cost, as it has been pointed out by other authors [12]. Therefore, it is a common solution to perform a limited selection of scenarios. For example, [13] uses 50 scenarios which have the highest probability of occurrence, including wind speed, solar radiation, thermal, and electrical loads as uncertain parameters (i.e., variables to forecast). In [9], the uncertainties are modeled using the Probability Distribution Function (PDF), which is discretized. Then, several scenarios are created using Monte Carlo, and finally a scenario reduction is done. Other optimization approaches consider prediction intervals for uncertain parameters instead of using scenario sets. An example can be seen in [8], where the authors propose a robust optimization the management of microgrids. In [14] the scheduling of distribution systems is done considering that the generations and load demands vary between their forecasted lower and upper bounds (i.e., within the intervals). These optimization methods, that can be applied in distribution networks and microgrids, require the prediction of one or more variables to obtain the optimal management operations, as for example solar generation [15], wind generation [16], etc. In this sense, the literature usually classifies the forecasting methods in two main branches. A deterministic forecasting (or point forecasting) method provides a single point for each interval of time. A probabilistic forecasting method gives the results in the form of quantiles, intervals, density functions, or scenarios [10]. Thus, a deterministic forecasting model provide less information, while the probabilistic models give much more detail about the expected values and their characteristics. According to [17], in some areas of power system the use of probabilistic methods is still an underdeveloped topic, with both academics and practitioners not using the correct evaluation or testing procedures. For the formulation of a probabilistic forecasting problem, four approaches are distinguished in [17], which are historical simulation (or empirical/sample Prediction Intervals, PIs), distribution-based probabilistic forecast, bootstrapped PIs, and Quantile Regression Averaging (QRA). Other methods that are This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. PAREJO et al.: PROBABILISTIC FORECASTING FRAMEWORK ORIENTED TO DISTRIBUTION NETWORKS AND MICROGRIDS 3 commonly found in the literature are Quantile Regression [1], bootstrapped quantiles [18], Bayesian bootstrap quantiles [19], and scenario sets [13]. The scenarios can be created using diverse approaches such as Monte Carlo [9], Quasi-Monte Carlo methods [20], or other approaches [21]. Regarding the forecasting techniques, these are usually classified in statistical, machine learning, etc. Nowadays, some of the most popular ones (both for deterministic and probabilistic forecasting) are machine learning methods, which include for example random forests, neural networks [22], and many others. Forecasting techniques are applied not only for performing energy management in networks, but also for predicting faults and outages during extreme events. For example, [23] present a method for fault prediction under heatwaves, in which the prediction problem is formulated as a binary classification task. In [24], the authors predict storm outages by means of deterministic forecasting techniques, and they indicate that future steps would include the exploration of probabilistic forecasting for this application. Considering the diversity of optimization methods that admit probabilistic information as input, their integration in a real automatic system would require a supply of diverse forecast data. Additionally, the forecasting system should automatically perform the evaluation of the best models that will provide each forecast to the optimization system. For this reason, some authors have made proposals of frameworks and system architectures for performing these tasks. In [25], a forecasting framework is proposed. It can handle the training, comparison, and selection of models, but it only included deterministic methods, not stochastic. The concept of forecast reconciliation is described in [26], which aims to stablish how interrelated forecasts should be used to generate indirect forecasts. However, probabilistic methods are not covered. In [27], reconciliation is applied for obtaining PIs for solar forecast. In [28], a probabilistic framework based on PIs is proposed and tested in the IEEE 104-bus tests system to predict groups of generators. It internally applies PIs and quantile regression; however, it does not integrate the generation of scenario sets. A comparison of models for probabilistic forecast applied to overhead lines dynamic line rating is done in [29]. The authors use diverse metrics for evaluating the ability of the models to provide a good density forecast and the ability to provide a good quantile forecast. Regarding PIs, a single comparison is done using the interval with a probability of 94% (which is evaluated using the mean interval size). Other PIs are not considered in the paper, nor the creation of scenario sets. The different methods included in the mentioned papers are summarized in Table I. As it can be seen, some of these papers include probabilistic methods, but none of them integrates the creation of quantiles, intervals, and scenario sets together. Therefore, it is a clear advantage of the proposed forecasting framework that it includes all these methods, to completely cover the requirements for a stochastic forecasting system essential for the optimization of microgrids and distribution networks. Also, it will be defined how to perform the model selection. TABLE I PROBABILISTIC FORECASTING METHODS IN SOME PAPERS FROM THE LITERATURE The next section will analyze some of the methods that are applied in stochastic forecasting and their evaluation. Some of these will be included in the proposed framework. III. METHODS This section will describe some of the procedures that are usually applied in stochastic forecasting and their evaluation. A. Parametric Construction of Quantile Sets Some authors apply parametric models to obtain the error distribution in the forecasting [30]. In this sense, parametric quantile sets perform the estimation of quantiles considering a certain type of distribution and known parameters. A common method is based on assuming a normal distribution and taking a value for the mean value (µ) and for the standard deviation (σ ). The mean value could correspond to a deterministic forecasting, while the standard deviation can be estimated from the historical data, or directly assumed to be a known value. The assumption of a distribution could be later used to construct “distribution-based” interval forecasts, as in [30]. For example, in [13], the uncertain parameters are assumed to have a continuous probability distribution function with 30% standard deviation. If the 99 quantiles from 0.01 to 0.99 are calculated, the forecasted quantile set for a variable yfor an instant twill be: Qf orect = { ˆyt,q},∀q∈ {0.01,0.02,...,0.99}(1) where: P(ˆyt,q>yt,q)=q(2) B. Non-Parametric Construction of Quantile Sets It is possible to directly construct quantile sets without assuming a known type of distribution, but by means of a method that directly provides a quantile set for the variable to be predicted. One of the methods that can be found in the bibliography is Quantile Regression Forest, which is used in [1]. It is a non-parametric method in which each tree of the forest provides a point, and the quantiles are obtained from the group of points provided by all the trees of the forest. This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. 4 IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING The mathematical expression of the forecasting in this case will be as seen in (1) and (2), although the method for obtaining the values is different. C. Construction of Prediction Intervals As said in [31], a prediction interval is composed of an upper and lower bound. The probability that a variable is between these bounds can be described as (1–α). In the literature, diverse authors have proposed methods for directly constructing PIs without the need of a quantile set. According to [31], some methods for their obtention using neural networks are Bayesian, delta, bootstrap, mean–variance estimation, and upper lower bound estimation. Other authors apply Kalman filter, extreme learning, and bootstrapping. When estimated quantile sets are available, it is possible to directly construct PIs using these. If two quantiles were used for constructing the PI, the probability of such interval would be the difference between the two respective probabilities of the quantiles. Let Utbe the quantile that is chosen to be the upper bound and Ltthe quantile that will be the lower bound. Then, the resulting forecasted interval with a probability (1–α) is: {Ut,Lt, (1−α)}(3) where: (1−α)=P(Ut>yt)−P(Lt>yt)(4) P(Ut<yt<Lt)=1−α(5) For example, the interval defined by the quantiles 0.05 and 0.95 will have an associated probability of 90%. D. Evaluation Metrics for Probabilistic Methods Regarding the evaluation of probabilistic methods, the typical metrics used for deterministic methods are not valid [10]. Therefore, other metrics have been proposed, such as the pinball loss function or the Winkler score [10],[32]. The most common existing metrics for the evaluation of probabilistic forecasts are the pinball loss function (for quantiles) and the Winkler score (for intervals). Their expressions are defined in (1) and (2) [10] respectively: Pinball(ˆyt,q,yt,q)=((1−q)( ˆyt,q −yt), yt<ˆyt,q q(yt− ˆyt,q), yt≥ ˆyt,q (6) Winkler =     δ, Lt≤yt≤Ut δ+2(Lt−yt)/α, yt<Lt δ+2(yt−Ut)/α, yt>Ut, (7) in which ˆyt,qcorresponds to the forecasted value for a specific quantile, ytis the real value to be forecasted, qis the quantile, Ltis the lower bound, Utthe upper bound, δis the difference between the two bounds of the PI (δ=Ut−Lt), and (1−α) is the nominal probability of the prediction interval. The pinball loss function evaluates a forecast considering its associated quantiles. As it is said in [10], the pinball losses can be summed across all targeted quantiles (for example, summing the pinballs for q=0.01, 0.02, 0.03, . . . , 0.99) to obtain the pinball loss of the probabilistic forecast. In the present paper, the average value will be calculated instead: Pinballˆyt,yt=1 99 ·X0.99 q=0.01 Pinball(ˆyt,q,yt,q)(8) In [29], the Pinball loss function receives the name of Quantile Score (QS) when it is used for evaluating the quality of an individual quantile forecast (instead of summing together the values for multiple different quantiles). The Winkler score evaluates an interval considering the upper and lower limit and its associated probability. The main metrics for both types of forecasting are reviewed in [33]. Various metrics for probabilistic forecasting are included, such as pinball loss function, Winkler score, and others. There are other metrics that are oriented to the evaluation of Cumulative Distribution Functions (CDFs) and PDFs instead of evaluating specific quantiles or intervals. Some examples of these metrics are the continuous ranked probability score and the Dawid–Sebastiani score [34]. In [29], the probability density function is evaluated using Continuous Ranked Probability Score (CRPS). Despite there are metrics for evaluating CDFs and PDFs, it has been appreciated that in most of the cases, these distributions are later discretized for performing the scenario generation, as in [9]. As can be seen, while there are diverse methods in the literature for creating quantiles, intervals, and scenarios. However, it has not been found in the literature a defined methodology on how the best forecasting model should be chosen when a certain type of forecast (e.g., a quantile set) is applied to obtain others (e.g., intervals and scenarios). In this sense, the models could be chosen according to their best performance forecasting quantiles, or to the quality of the interval or scenario set that is required, but it is unclear which approach is better. Therefore, a probabilistic framework that englobes different probabilistic models will be proposed. It will define the two different approaches that can be followed in the model selection process. This framework will be applied in a case study to compare these approaches. IV. PROPOSED PROBABILISTIC FRAMEWORK This section will describe the procedure of probabilistic forecasting modelling, the selection of the best models using a new proposed metric, and the generation of scenarios. A. Framework Architecture In this architecture, the input data are used to prepare diverse datasets (with different groups of inputs), which are later used to train forecasting models. These models are compared using their corresponding metric, and then stored in a “ranking of models” (i.e., an ordered list from better to worse according to their quality evaluation metric). When it is required to perform a forecasting, the best possible model will be executed. The reason to save more than a single model for each variable to predict is that, in case the best model could not be executed This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. PAREJO et al.: PROBABILISTIC FORECASTING FRAMEWORK ORIENTED TO DISTRIBUTION NETWORKS AND MICROGRIDS 5 Fig. 1. Proposed procedure for generation and evaluation of quantile sets, intervals, and scenario sets. (because some of the input data were missing), there would be other models that could be used. For performing deterministic forecasting, the models could be evaluated using the Root Mean Square Error (RMSE). However, in the case of probabilistic forecasting, it is more complex. The procedures that have been designed for the inclusion of probabilistic forecasting in the framework are depicted in Figure 1. The steps are: •Step A: Choose the variables to forecast and their related data that could serve as input for the models. •Step B: Train the probabilistic forecasting models of the variables to be predicted. This can be done by stochastic/probabilistic models that directly provide the quantiles, or by applying a known distribution (e.g., normal distribution) with a certain standard deviation over the predictions of a deterministic forecasting. •Step C: Select the group of quantiles that will be used for creating each of the intervals and scenario sets. •Step D: Create prediction intervals. This is done using pairs of quantiles and their probabilities. For constructing each interval, both quantiles will be symmetrical (e.g., quantiles 0.1 and 0.9 for the interval with a probability of 80%). •Step E: Generate scenario sets from the selected quantiles. The proposed procedure will be later explained. •Step F: Evaluate the probabilistic forecasts. The pinball loss function is used for evaluating quantile sets, the Winkler score is used for intervals, and the weighted pinball score (WePin, a metric that is proposed in this paper) is used for scenario sets generated by the proposed method. The methodology for creating scenario sets and evaluating these will be exposed next. B. Quantile Selection and Scenario Generation For generating scenarios, some authors produce all scenarios and their probability, and then select only those most probable scenarios. This approach, while causes a trivial error, reduces the simulation time [13]. In the present paper, an alternative way to perform this process is proposed. Instead of producing all possible scenarios and keeping the most probable ones, a selection of the quantiles to be used will be done, and their corresponding probabilities will be assigned. The method exposed in [13], and other Monte-Carlo-based methods from the literature, despite being able to generate a massive number of different scenarios, do not usually include a way to evaluate which forecasting model produces the scenario sets of a better quality. On the contrary, the proposed method includes an evaluation metric to do so. It provides a methodology to evaluate and choose the probabilistic forecasting model among all those that have been trained and generate the desired number of scenarios and their associated probabilities. For starting the proposed scenario generation, it is necessary to obtain a group of quantiles of the variable to be predicted. Two possible methods are described. The first one, that has been described in [13], obtain the scenarios from a single deterministic forecast by supposing a fixed standard deviation. The second one is to use some probabilistic forecasting technique that provides quantile information. Under the proposed method, the uncertainty values for a scenario Sat the time tfor a certain quantile qwill be equal to This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. 6 IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING the probabilistic forecast during a period of time for a certain quantile q. Therefore, it can be said that: St(q)= ˆyt,q(9) However, the probability of the scenario is not equal to that of the quantile. This decision is made considering that the sum of the probabilities of all considered scenarios for an uncertainty must be equal to 1(to reach the 100% of probability). This will depend on the number of scenarios, and which are the quantiles that are chosen. For distributing the probability to the scenarios, the proposed method is as follows. Being Qf orect ={ˆ yt,q},∀q∈{0.01, 0.02, . . . , 0.99} the array of 99 forecasted quantiles (from quantile 0.01 to 0.99) of a variable at a certain instant t. Being Qindex ={0, 1, . . . , 98} the array that contains the position indexes of the elements of the array Qf orect . Being Cindex ={c0, c1, . . . , cK−1}the ordered array of K quantile indexes that will compose the scenario set (i.e., chosen quantiles), in which the position k=0is the lowest quantile of them and k=K-1 is the highest quantile. Note that in this array, a value qk=0means “the quantile 0.01 is used as one of the scenarios of the set,” a value qk=1means “the quantile 0.02 is used as one of the scenarios of the set,” etc. The reason for this behavior is that the elements qkexpress the indexes inside the array of 99 quantiles. Being St={st,0, st,1, . . . , st,K−1}={Q f orect |i∈Cindex }the ordered array of Kscenarios corresponding to the Kquantile values chosen, in which the position k=0is the scenario with the lowest quantile of them and k=K-1 is the scenario with the highest associated quantile. Being Gt={g0, g1, . . . , gK−1}={P(st,0), P(st,1), . . . , P(st,K−1)} the array of probabilities of each scenario belonging to the scenario set St, given that PK−1 k=0Pst,k=1. If it is considered that the probability between two scenarios is equally divided between them, then the probabilities of the array Gshould be distributed between the scenarios according to the next expression: gk=                qk+qk+1−qk 2,if k =0 qk+1−qk 2+qk−qk−1 2=qk+1−qk−1 2, if 0 <k<K−1 1−qk+qk−qk−1 2,if k =K−1 (10) Therefore, (4) establishes how the probabilities should be distributed to the scenarios once the set of quantiles has been chosen. However, it is necessary to choose these quantiles first. Two novel procedures have been defined in this paper: •MiAs (middle assignation): choose the rest of the scenarios equally distributed among the available quantiles. •ExAs (extreme assignation): forces that the two extremes of the quantile array (the lowest one and the highest one) are chosen to create scenarios and the rest of them are equally distributed between these two extremes. The procedure that is proposed for the automatic selection of quantiles for creating scenarios, and the assignation of probabilities for such scenarios is expressed in Algorithm 1. To exe- Algorithm 1 Create a set of scenarios applying the chosen method. The algorithm gives the percentile indexes that correspond to a set of scenarios and the probability that should be assigned to each of these scenarios. The total number of scenarios is equal to the input Nscen . The two available methods are “MiAs” and “ExAs”. This is selected according to the input methodscen . The output Cindex contains the indexes of the quantile array that are chosen to serve as scenarios (therefore, it has a length of Nscen elements). The output Gcontains the probabilities of each scenario. Input Nscen ,methodscen Output Cindex ,G Ensure: Nscen is an integer number Ensure: Nscen ≥1 Ensure: Nscen ≤99 Ensure: (methodscen ="MiAs")or (methodscen ="ExAs") Declare Cindex array of Nscen integers Declare Garray of Nscen floats if methodscen is “MiAs” then n←1 position ←0 while n<2·Nscen do indexn←100·n 2·Nscen indexn←round(indexn) indexn←indexn−1▷Position 0 is percentile 1% in the array of quantiles. Cindex position←indexn n←n+2 position ←position +1 end while n←0 while n<Nscen do ▷The probabilities are considered equal for all the scenarios in the MiAs set. probabn←1 Nscen G[n]←probabn n←n+1 end while else if methodscen is “ExAs” then n←0 if Nscen =1then Cindex [n]←49 ▷Save the percentile 50% in array. G[n]←1.0▷Assign a probability of 1 (i.e., 100%) to the scenario. else Cindex [n]←0▷Save the lowest percentile (1%) in array. n←1 while n<Nscen −1do indexn←100·n Nscen −1 indexn←round(indexn) indexn←indexn−1 Cindex [n]←indexn n←n+1 end while Cindex [n]←98 ▷Save the highest percentile (99%) in array. n←0 while n<Nscen do if n=0then ▷Store probability for lower extreme scenario. probabn←Cindex [n]+1 100 +Cindex [n+1]−Cindex [n] 2·100 else if n=Nscen −1then ▷Store probability for upper extreme scenario. probabn←100−1−Cindex [n] 100 +Cindex [n]−Cindex [n−1] 2·100 else probabn←Cindex [n]−Cindex [n−1] 2·100 +Cindex [n+1]−Cindex [n] 2·100 end if G[n]←probabn n←n+1 end while end if end if Note 1: This algorithm considers that the forecasting for each point is composed by a sorted array of 99 float numbers, where the number in position 0 corresponds to the prediction for percentile 1% and the position 98 corresponds to the prediction for percentile 99%. Otherwise, the algorithm should be adapted appropriately. Note 2: For the calculation of scenarios under the method “ExAs”, the lower extreme corresponds with the percentile 1% (whose index is 0) and the upper extreme corresponds with the percentile 99% (whose index is 98). Under the method “MiAs”, the extremes are automatically chosen depending on the number of scenarios. cute the algorithm, it is only required to select the desired number of scenarios and choose if the extreme quantiles should be included in the pool (ExAs method) or not (MiAs method). For simplifying the proposed algorithm, it has been considered that the vector Qhas 99 elements, the value for This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. PAREJO et al.: PROBABILISTIC FORECASTING FRAMEWORK ORIENTED TO DISTRIBUTION NETWORKS AND MICROGRIDS 7 quantile 0.01 (i.e., percentile 1%) is stored in position 0, and the value for quantile 0.99 (i.e., percentile 99%) is stored in position 98. However, the same idea could be adapted to other cases adapting the values and indexes in the given algorithm. The proposed algorithm for the creation of scenario sets is as follows. It considers that the number of available quantiles for the variable whose scenarios will be obtained is equal to 99 (from quantile 1 to quantile 99). The algorithm will create a number Nscen of scenarios composed by the array Cindex (the indexes of the quantiles that corresponds with the scenarios) and the array of probabilities G(the probabilities for the respective scenarios). The symbol “▷” indicates a comment. C. Weighted Pinball (WePin) Evaluation Metric The weighted pinball (WePin) is a metric proposed in this paper for performing the evaluation of the scenario sets created by the aforesaid method. This metric is based on the pinball loss function, but it assigns weights to each scenario according to its occurrence probability. Consider a probabilistic forecast of a time series variable yt that provides a group of Kscenarios from st,0to st,K−1, each of them corresponding to a certain quantile q(that goes from q0to qK−1). Moreover, each scenario has a probability of occurrence gfrom g0to gK−1. The WePin score for a certain day d(i.e., WePindailyd )that goes from t=0to t=T-1 is defined as: WePindailyd=XK−1 k=0"gk·XT−1 t=0"Pinballst,k,yt,qk T##, (11) where Tis the number of steps considered within a day (in the case of hourly data, Tis equal to 24). qkwill be the quantile value associated to the scenario st,k. Therefore: qk=Qindex ·(Cindex (k)) +1 100 ,(12) Similarly, the mean daily WePin score for a group of D days will be equal to: WePinD_days =XD d=1 WePindailyd D,(13) With these proposed metrics, the generated scenario sets can be evaluated. D. Execution of the Framework Next, it will be defined how the forecasting system decide which of the trained model instances to use for performing the required prediction. Let Mi(Xi,t)be a trained model instance that receives the inputs Xi,tand return a forecasting in the shape a quantile set. Let {Mi}:i∈{1,. . . ,n} be a group of ntrained model instances that are available in the forecasting system. Let IF( ) be a function that receives a set of quantiles and a decimal number between 0 and 1 (the probability) as inputs and return an interval with its associated probability. Let AF( ) be a function that receives a set of quantiles and an integer number (the number of scenarios) as inputs and return a scenario set with the associated probabilities of the scenarios. The forecasting system needs to choose which of the model instances to use among those that are available for each required prediction (i.e., to obtain quantiles, and to feed the functions IFand AFfor creating intervals and scenario sets). This will be decided according to the evaluation metric that the model instances obtain during the validation process. For the evaluation, Pinballvalidat ( ) is a function that returns the average value of the Pinball of the quantiles during the period of time used for the validation of models (of course, this will be calculated by comparing the forecasting with the real data in that period). Similarly, Winklervalidat ( ) returns the average value of the Winkler during the validation period for a certain forecasted interval with an associated probability (1–α), and WePinvalidat ( ) returns the average value of the WePin during the validation period for a scenario set. Two approaches are proposed for choosing the model instances: i) General approach. Choose the model that creates the best-quality quantile set for constructing all the required intervals and scenario sets (therefore, the same model instance will be used for all the predictions). The model instances that will be chosen are, for creating quantiles: Mquantilesgen =arg min Mi {Pinballvalidat (Mi(Xi))}n i=1(14) for creating an interval with probability (1−α): Mintervalgen =Mquantilesgen (15) for creating a scenario set with Nscenarios: Mscenariosetgen =Mquantilesgen (16) ii) Specific approach. Evaluate individually the quality of all the available models for creating each type of interval and scenario set that is required, and then use the best-quality model for constructing each of the required intervals and scenario sets. The model instances that will be chosen are, for creating quantiles: Mquantilesspe =arg min Mi {Pinballvalidat (Mi(Xi))}n i=1(17) for creating an interval with probability (1−α): Mintervalspe =arg min Mi {Winklervalidat (IF(Mi(Xi),1−α))}n i=1 (18) for creating a scenario set with Nscenarios: Mintervalspe =arg min Mi {WePinvalidat (AF(Mi(Xi),N))}n i=1 (19) Therefore, the forecasting system can operate with any of the two approaches. The model instance that is chosen will be used for performing the required forecasting. This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. 8 IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING E. Integration of the Forecasting With Other Applications The framework will provide forecasting of one or several variables of interest by giving the configured quantiles (usually, 0.01, 0.02, . . . 0.99), the configured interval probabilities, and the configured scenario sets. The forecasting outputs, that will be then sent to feed other applications (e.g., optimization management systems), corresponding to the expected values for a predicted variable during an instant (or time period) t, are as follows: The forecasted quantile set is: Qf orect = { ˆyt,q},∀q∈ {0.01,0.02,...,0.99}(20) The forecasted interval with a probability (1−α) is: {Ut,Lt, (1−α)}(21) The scenario set of scenarios Swith their probabilities G is: {St,Gt} = st,0,st,1,...,st,K−1,{g0,g1,...,gK−1}(22) The quantile set can be used to represent the expected behavior of variables, or to perform other actions based on that information. The later use of the intervals in an optimization system can be done as in [14], where the uncertainty variables are modeled using their lower/upper bounds (in that paper, PR i,tis the lower bound and PL i,tis the upper bound for load demands at a node i during the instant t). The use of scenarios can be done as shown in [9], where Nsscenarios sand their associated probabilities πsare applied to represent forecasted values, and [11], where σscenarios wσand their probabilities πσare used in the stochastic optimization for microgrid management. V. CASE STUDY This section will expose a case study of application of the proposed framework to forecast the consumption in a distribution network. Specifically, the objective will be the day-ahead hourly prediction of the consumption of each of the secondary distribution substations in the town of Manzanilla (Spain). It will be possible to obtain different types of forecasts, deterministic, probabilistic distribution, intervals, and scenario sets. A. Dataset Description The available datasets correspond to the power demand data of 10 secondary substations from the year 2017 to the year 2020. Their location and their average demand can be seen in Figure 2. These data include the hourly power demand of each of the secondary substations (expressed in W). Additionally, hourly weather data (temperature in ◦C, humidity in %, and rain in mm), and calendar-related information (the day of the week, the month, if the day is a holiday or not, etc.) has been included too. These data fields will be combined to create several datasets for training machine learning forecasting models, as it will be described next. Fig. 2. Secondary substations of the town of Manzanilla. a) Location; b) Average demand in kW. B. Models and Input Data Each of the substations will be modeled individually. Therefore, the model instances will have 24 outputs that correspond to those hourly demands for a single substation for the next day. Regarding the inputs, it can include calendar information, weather information, and in some of them the demand of previous days is included (as it can be helpful to forecast the future demand), other include the average hourly demand of the previous week (or weeks), and others can include some different processed information about the previous demand. From the combination of diverse subgroups of these fields, a total of 48 datasets (with different subgroups of input fields) were obtained for the period under study. The reason for doing so is to obtain diverse model instances with different input requirements. C. Modeling Techniques The two probabilistic modeling techniques that will be applied for obtaining quantile sets are: •Random Forest Regressor (RFR) and normal distribution (whose standard deviation is proportional to the forecasted values) for obtaining a probabilistic forecasting. This is a parametric technique (see Section III-A). The standard deviation, which is an hyperparameter, will have three possible values: ◦10% of standard deviation, labeled “RFR_prob SIGMAFIX10”. ◦20%, labeled “RFR_probSIGMAFIX20”. ◦30%, labeled “RFR_probSIGMAFIX30”. •Quantile Regression Forest. This is a non-parametric technique (see Section III-B). It will be labeled “RFR_prob”. In this study, the hyperparameter value for the number of trees will be 101 for both techniques. Therefore, four technique This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. PAREJO et al.: PROBABILISTIC FORECASTING FRAMEWORK ORIENTED TO DISTRIBUTION NETWORKS AND MICROGRIDS 9 Fig. 3. Train, validation, and test splits in the case study. variants (considering the three different values of standard deviation) are included. Once obtained the quantiles by mean of these techniques, the required prediction intervals could be obtained (as seen in Section III-C). For obtaining the required scenario sets, the quantiles will be processed by following the proposed methods MiAs and ExAs (as seen in Section IV-C). Additionally, the proposed forecasting system is also able to provide deterministic forecasting applying Multi-Layer Perceptron Regressor (MLPR), and RFR. However, the objective of the present case study is to compare the ways of performing the selection of probabilistic models (i.e., the two proposed approaches for the framework). Therefore, the deterministic forecasting techniques will not be considered (as these would simply be chosen according to their RMSE). D. Train/Validation/Test Splits The application of the proposed forecasting system is done by training the model instances using a part of the input dataset, then using another part to perform the validation (the evaluation metrics that each model instance obtains during the validation period are used to order the instances from better to worse) and then using the rest of the data for testing (which corresponds to the period in which the forecasting system is operating to provide the required predictions using the available models). In this case study, this process has been done 16 times, performing a time-series split with train/validation/test. In total, the test period has been 16 months. Figure 3indicates how the splits were done. The models are trained (using data of train period) and evaluated (with data of validation period) to create model rankings (models ordered according to their evaluation metric during the validation for each type of probabilistic forecasting). Then, these will be used to perform the forecasting tasks for one month (during the test period). For each day, the best possible model according to the corresponding ranking will be executed. After that time, a new cycle starts, and all the models are retrained, and evaluated (to obtain the new ranking of models). E. Description of the Experiments The objective of the experiments is applying the probabilistic forecasting framework in a case study, and checking which of the two proposed approaches for choosing models performs better. In this case study, 16 train/validation/test cycles will be done in total. After each training process there will be 192 trained model instances (48 datasets of inputs and 4 technique variants) for obtaining probabilistic forecasts of each secondary substation. The decision of which of the available model instances is better is done according to the evaluation metrics obtained in the validation phase, but this can be evaluated by following the general approach or by the specific approach. The model that is chosen as better (for each type of forecast), will be applied during the test period. In this case study, the global performance of the applied framework under the two possible approaches will be compared, as each of them has its own advantages and disadvantages: •General approach. The procedure is simpler. •Specific approach. This procedure requires that each available model is used to construct each type of interval and scenario to evaluate their performance. In this sense, as the framework is able to provide different types of probabilistic forecasting (quantile sets, intervals, and sets of scenarios), it will be analyzed what is the effect of choosing the best models only by the pinball loss function (which is a simpler method) or using the specific metric for each type of model (Pinball, Winkler score, or WePin, which is a more complex method). This will show the consequence of taking into consideration the specific type of uncertainty that is going to be created when performing the model selection (i.e., the comparison of models in the validation period to choose the best one). Given than the value of the evaluation metric during the test period applying the general approach is Egen and under the specific approach is Espe, the comparison between them will be done by calculating the Improvement (expressed as a percentage): Improvement(%)=Egen −Espe Egen ·100,(23) This way of comparing changes in error metrics (in which a lower metric value express a lower error) is similarly followed in [24], where it is called “percent improvement”. If the value of Improvement is a positive value, it means that the specific approach achieves better quality forecasts than the general approach. The type of probabilistic uncertainty representation for which the comparison has been made (by calculating the Improvement) are: •Intervals: 6 types of intervals (whose probabilities are 98%, 94%, 90%, 80%, 70%, and 60%) •Sets of scenarios created using the MiAs method: 48 different scenario sets. •Sets of scenarios created using the ExAs method: 48 different scenario sets. Therefore, considering that there are 10 secondary distribution substations, there will be 60 intervals, 480 sets of MiAs scenarios, and 480 sets of ExAs scenarios to be forecasted. This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.