Online forecasting using neighbor-based incremental learning for electricity markets

S.I.: INCREMENTAL LEARNING Online forecasting using neighbor-based incremental learning for electricity markets L. Melgar-Garcı ´a 1 •D. Gutie ´rrez-Avile ´s 2 •C. Rubio-Escudero 2 •A. Troncoso 3 Received: 3 October 2023 / Accepted: 2 December 2024 The Author(s) 2025 Abstract Electricity market forecasting is very useful for the different actors involved in the energy sector to plan both the supply chain and market operation. Nowadays, energy demand data are data coming from smart meters and have to be processed in real-time for more efficient demand management. In addition, electricity prices data can present changes over time such as new patterns and new trends. Therefore, real-time forecasting algorithms for both demand and prices have to adapt and adjust to online data in order to provide timely and accurate responses. This work presents a new algorithm for electricity demand and prices forecasting in real-time. The proposed algorithm generates a prediction model based on the k-nearest neighbors algorithm, which is incrementally updated in an online scenario considering both changes to existing patterns and adding new detected patterns to the model. Both time-frequency and error threshold based model updates have been evaluated. Results using energy demand from 2007 to 2016 and prices data for different time periods from the Spanish electricity market are reported and compared with other benchmark algorithms. Keywords Real-time forecasting Incremental learning Streaming time series Electricity 1 Introduction Nowadays, the topic of time series forecasting is receiving increasing attention, especially because of its interdisciplinary nature. Almost all scientific disciplines consist of data sampled over time, which makes forecasting a task of utmost importance and complexity. Big data streaming is becoming one of most widely used trends in big data in recent years. The most important feature for big data streaming is the velocity referred to large continuous flows of data, called streams. Algorithms dealing with big data streaming aim to yield results in realtime. For this reason, streams need to be processed and modeled in a special way considering specific requirements [1]. In addition to obtaining fast results when working in streaming mode, it is important to develop a model that must be always ready to give responses. However, streaming flows usually variate and suffer transformations which could lead to a non-accurate response if the model does not adapt to them [2]. Thus, prediction models for streaming data must be updated as the data arrives in realtime to best match its new behavior. This update must be in real-time, so the re-training of the model should be discarded in favor of incremental learning. Participants in electricity markets (both demand and prices) are particularly interested in forecasting, as obtaining forecasts is critical for many areas in order to increase profits or reduce costs. In addition, climate change is one of the most concerning topics of recent decades. Climate change has led to the growth of renewable energies &L. Melgar-Garcı ´a [email protected] D. Gutie ´rrez-Avile ´s [email protected] C. Rubio-Escudero [email protected] A. Troncoso [email protected] 1 Department of Artificial Intelligence, Universidad Polite ´cnica de Madrid, 28660 Madrid, Spain 2 Department of Computer Science, University of Seville, Avda. Reina Mercedes s/n, 41012 Seville, Spain 3 Data Science and Big Data Lab, Pablo de Olavide University, 41013 Seville, Spain 123 Neural Computing and Applications https://doi.org/10.1007/s00521-024-10876-x(0123456789().,-volV)(0123456789().,-volV) in many countries in recent years. Due to the volatility and intermittency of the renewable energies, the power grid currently presents enormous uncertainty and electricity markets present changes and fluctuations. These changes can lead to shifts in existing patterns in the data, the emergence of new patterns or even anomalous behavior. Therefore, forecasting electricity prices in an online scenario becomes essential. In this context, the current progress of Internet of Things (IoT) devices is leading to the possibility of monitoring energy consumption. Furthermore, IoT devices provide extensive amounts of high-dimensional data in streaming [3]. This type of data opens up a huge field of study in the big data streaming paradigm with the aim of obtaining real-time solutions that lead to energetic efficiency. In this work, we propose an algorithm to predict electricity demand and prices time series in real-time. This algorithm, named StreamWNN, uses the K-nearest neighbors to compute the final prediction and the neighbors are updated over time through incremental learning. In particular, the StreamWNN algorithm is made up of two phases: a batch phase in which a historical model based on nearest neighbors is created and an online phase to forecast and update the model. Therefore, the online phase keeps the model always adjusted to the current data also adding in the model new patterns called novelties and detecting possible anomalies. Results using energy consumption data in Spain from 2007 to 2016 and electricity prices data for three different time periods are evaluated to show the accuracy of the predictions, the response time of the algorithm and the improvements obtained when the model is updated. The aim of this research paper is to show how the proposed incremental learning for the nearest neighbor method improves the online predictions. The rest of the paper is structured as follows. Section 2 presents a review of forecasting algorithms for energy time series focusing on real-time and data received in streaming. In Sect. 3, the methodology of the StreamWNN algorithm is described, including how the updating of the prediction model is performed. Section 4defines the electricity demand and prices datasets used and presents the discussion of the results. The paper ends with some final conclusions and ideas for future approaches in Sect. 5. 2 Related work Time series forecasting has been extensively studied to solve numerous problems in different fields, with a wide range of applications. For example, predicting telecommunications network traffic as a time series is useful for assigning resources according to demand, optimizing routing, detecting anomalies or designing, monitoring and managing the network [4]. In [5], this problem was solved using a combination of CNN-LSTM neural networks after proving that classical methods such as ARIMA and SVR were not able to correctly predict these complex time series patterns. In [6], this time series prediction problem was solved using the hybrid CNN-LSTM model together with a feature selection module and a hyperparameter search optimization module. Most of the published prediction models for big data time series work in batch mode. There is still a lot of research to do in relation to streaming environments. The challenges of high-dimensional massive data mining in real-time are related to storage, processing and obtaining useful knowledge. Understanding and analyzing data in streaming is necessary to assist in the decision-making process [7]. Thanks to the increasing amount of massive data from electronic devices, there are lots of applications for this type of data. In terms of streaming time series forecasting, authors in [8] presented several models to predict the evolution of the COVID-19 pandemic in realtime. In [9] a forecasting model for streaming taxi demand was presented. Other types of algorithms are starting to have more influence in streaming environments. A triclustering algorithm [10] was developed for the real-time processing in [11,12]. Three-dimensional patterns from environmental sensor and medical streaming data were obtained. In the real-time environment, a review of forecasting algorithms for streaming data from year 2000 to 2015 was presented in [13]. In [14] a new algorithm for streaming energy demand data forecasting was proposed. It used three different algorithms adapted for streaming data: k-means, nearest neighbors and Naive Bayes. In [15] a randomized version of neural network with a incremental learning using the electric load datasets from Australian energy market was proposed to improve both efficiency and accuracy. In [16] a forecasting online self-adaptive model based on random forest was applied to electricity market price. Authors demonstrated that this online model, that took into consideration the fluctuations and volatility of price, was more accurate than the benchmark algorithms in the literature. In [17] a XGBoost was developed for realtime price forecast in Singapore. Similarly, electricity demand forecasting have been studied mostly in the batch or traditional mode. Regarding the traditional approach, authors in [18] presented a big data approach for electricity consumption in smart cities. Predictions for electricity demand using nearest neighbors from Apache Spark framework for big data were made in [19]. Regarding electricity price forecasting, a review of current electricity price forecasting methods was presented in [20]. In [21] a multivariate logistic regression model was used to forecast day-ahead extremely low and high Neural Computing and Applications 123 Australian electricity prices, taking into account influencing variables. Authors focused on understanding the dynamics of extreme electricity prices. A forecasting algorithm based on genetic optimization for Nordic electricity spot price data was developed in [22]. A LSTM model was deployed to forecast the average monthly spot prices in Spain in [23]. Concerning streaming modeling, this type of data can variate and change its behavioral patterns while time passes. It is important to have a streaming model that adapts to the new streams of data. Many of the approaches in the literature for model updating in streaming environment are based on external algorithms as the Kalman filter. Authors in [24] presented a coupled methodology of the K-nearest neighbor algorithm with a Kalman filter for real-time flood forecasting. The state transition matrix of the Kalman filter was recalculated using the forecasting method to improve the performance of the model and obtain accurate results. In [25] concepts of ensemble Kalman filter were used to update a rainfall runoff model for forecasting large floods in real-time. Another ensemble Kalman filter was also used in [26]. The ensemble was used as a data assimilation algorithm to update water temperature forecasting considering sensor instances. Recently, deep neural networks have been used to predict streaming data by taking into account the changes that the data may undergo over time. A wavelet-neural network with an error-updating scheme was proposed in [27] for meteorological predictions in streaming. The need of systematic error-updating for stream flows was proven. Authors in [28] proposed an incremental update method based on support vector machine (SVM) and gate recurrent unit (GRU) considering concept drift for forecasting in real-time. Training models were updated based on the error between batch test results and real values. In [29] a combination of a real-time autoregression and a deep Long Short-Term Memory (LSTM) recurrent network to predict streaming data from industrial processes was proposed. The deep LSTM found temporal relationships in data while the autoregression model addressed overlap and transfer between different recurring concepts drifts thus improving the accuracy. The high volatility and noise characteristics of electricity price pose great challenges in predicting it. In [30] real-time electricity price forecasting was accurately performed by applying transfer learning and GRU. However, there is no reference to concept drift in the article. The authors emphasized that transfer learning models on hybrid datasets are robust to changes in the input data and therefore more generalizable. In [31] and [32] two new methods were presented for the same objective: real-time forecasting of electricity price. The former used a CNNGRU parallel statistical model, while the latter employed a CNN-based autoencoder. The last model obtained the most accurate results. The developed models made tight predictions taking into account the high noise, volatility and nonlinearity of the data. 3 Methodology The StreamWNN streaming algorithm [33] is based on the K-nearest neighbors [34] which main idea is that nearest data share similar properties or characteristics. This is crucial for the problem to be solved, i.e., time series forecasting with model update while receiving real-time data. Online model updates and accurate predictions are performed based on similar properties or characteristics of the past data and the newly received data. This incremental learning approach of the machine learning algorithm is key to obtaining more accurate results on data with similarities over time, as electricity markets data. In electricity markets, both energy consumption and market prices are time series and their prediction can be formalized with the theoretical problem described below. The goal of this time series forecasting problem is to predict the next values considering past ones. The time series used is divided into Ninstances where the i-th instance consists of xirepresenting its wpast features and yi representing its hnext classes, i.e., next hvalues to predict: Xt¼fðx1 ;y1Þ;...;ðxN ;yNÞg xi2Rwyi2Rhð1Þ The dataset is divided into three chronologically ordered sets: setneighbors, setpatterns and setstreaming. The StreamWNN algorithm uses the offline-online learning approach to model data streams [35]. The first two data sets are used in the batch or offline phase of the algorithm and the last set in the streaming or online phase. Big data streams requirements have to be fulfilled only in the online phase. These requirements are: only a limited set of past data can be stored; the model has to adapt to concept drift quickly and has to be always ready to make predictions; the model has to work in distributed computational environments so that its computation is fast [36]. In this work, an innovative incremental learning approach during the online phase is presented. In addition, novel and unexpected patterns in the incoming streaming data are detected. Novelties are, as well, added into the model. In this way, the algorithm is always up to date. The proposed algorithm is built on Apache Spark 2.3.4 and uses HDFS file system on Hadoop 2.7.7 and the Kappa Architecture on Apache Kafka 2.11 as streaming platform, i.e., a single pipeline is specifically designed for the job. The methodology of this algorithm is defined in five sections. Section 3.1 describes the offline phase. Neural Computing and Applications 123 Section 3.2 presents the online phase and how real-time forecasting is performed. Section 3.3 focuses on the incremental learning approach of the StreamWNN algorithm. Then, Sect. 3.4 details how novel patterns and unusual behavior data are detected in real-time. Finally, Sect. 3.5 presents the model’s approach to combine incremental learning and the inclusion of novelties in the model. 3.1 Offline learning model The creation of the offline learning model is the first task of the algorithm. It is a very important part as it is the base model for the online phase where streaming data start to arrive. This first phase is based on batch processing but using distributed programming, which allows to get a good model computed in a short time. The offline phase uses the setneighbors and the setpatterns representing approximately 70% and 30% of the chronologically ordered data used for the offline stage. The batch model associates each feature instance of the setpatterns with its Kclosest instances of the setneighbors. The offline model is represented as: M¼\xi ;\yðneighbor1ðxiÞÞ;...;yðneighborKðxiÞÞ [[ ð2Þ where xi2Rware the features of the i-th instance of the setpatterns and yðneighborjðxiÞÞ 2 Rhare the classes of the instance of the setneighbors selected as the j-th closest neighbor of xi. The proximity between the attributes of two instances, one from the set of patterns setpatterns and the other from the set of neighbors setneighbors, is calculated using a userspecified distance metric such as Euclidean, Manhattan or Chebyshev distances. The user can also define his/her own distance metric. It can be represented as dðxi ;xjÞwhere xi refers to the features of the i-th instance of the set of patterns and xjto the features of the j-th instance of the set of neighbors. Due to the nature of the data used in the experiment of this scientific paper and the analyses performed on it in [37], the distance used in this experiment is the Euclidean distance. Once the offline model is generated, predictions of the next hvalues are made and its performance is tested by calculating an error metric. In the offline phase, predictions for each value lin the prediction horizon of length hare computed by: byl¼1 PK j¼1ajX K j¼1 ajyðneighborjðxiÞÞl1lhð3Þ where ajrepresents the distance of dðxi ;xjÞdefined as follows: aj¼1 dðxi ;xjÞ2ð4Þ Closest data will have a greater ajdistance. Finally, the error of the predictions made in the offline phase is computed. For this work, Mean Absolute Percentage Error (MAPE) and Mean Absolute Error (MAE) metrics are used. Thus, in the offline stage, the final errors MAPEoffline and MAEoffline are the mean of all MAPEiand MAEi, respectively, where MAPEiand MAEiare the errors of the predictions for each i-th instance of the setpatterns and they are defined as: MAPEi¼1 hX h l¼1 ylbyl yl 100 1 lhð5Þ MAEi¼1 hX h l¼1 jylbylj1lhð6Þ where ylcorresponds to the lclass of the i-th instance and bylis the offline prediction defined by Equation (3). 3.2 Real-time forecasting Once the offline model has been computed from Eq. (2) the online phase of the StreamWNN algorithm starts. Data are received in streaming and collected in instances of wfeatures, these data can be called xstreaming. For each xstreaming, the Euclidean distances between it and all xiinstances of the model Mare calculated. Note that the number of distances calculated is the number of instances of the set of patterns setpatterns. The one with the minimum distance dmin is selected as the nearest xi, called xmin from now on. Once xstreaming is associated with its nearest xmin, predictions are computed. In this case, online predictions consider not only the K-nearest neighbors of xmin but also xmin itself as a neighbor for each value lto predict in the prediction horizon of length h. That is: byl¼1 aX K j¼1 ajyðneighborjðxminÞÞl ! þaminyðxminÞl ! ð7Þ where aand amin are defined as: Neural Computing and Applications 123 a¼X K j¼1 aj ! þamin ð8Þ amin ¼1 dðxmin ;xstreamingÞ2ð9Þ Then, the MAPEstreaming and MAEstreaming error metrics for each i-th instance of the setstreaming can be calculated using Equations (5) and (6) when the real class of stream data xstreaming is received. Thus, in the online stage, the final errors MAPEonline and MAEonline are the mean of all MAPEstreaming and MAEstreaming, respectively. Therefore, real-time forecast is performed considering the historical data stored in the model Mobtained in the offline phase. Figure 1presents a summary of the main steps of the model when there is no online update. However, the StreamWNN algorithm includes three more different possibilities of dynamically updating the model during the online phase. These updates are explained in Sects. 3.3,3.4 and 3.5. 3.3 Incremental learning The goal of online incremental learning is to keep the model up to date. This is a very important task in data streaming algorithms, as new data patterns may appear and need to be included in the model. Without this incremental update, the model may age and not be suitable for real-time forecasting. The incremental learning is performed by updating the neighbors of xmin, i.e., neighborjðxminÞwith j¼1;...;K. For this reason, it is said that the model is internally updated, since the dimensions of the model Mare maintained but the components are updated considering the new patterns in the streaming data. The update of neighbors uses a buffer Bof possible updates. Let dmin be the distance between xstreaming and its nearest pattern xmin, and let dKbe the distance between xmin and its farthest neighbor in the model neighborKðxminÞ. That is, dmin ¼dðxmin ;xstreamingÞð10Þ dK¼dðxmin ;neighborKðxminÞÞ ð11Þ Therefore, once the real-time forecast is performed, if dmin is less than dKit means that the actual xstreaming is a more accurate neighbor of the xmin than its current neighbors in the model M. If it occurs, xstreaming is added to the possible update buffer with its corresponding xmin as follows: B¼fðxmin ;xstreaming ;dminÞg ð12Þ The buffer is filled as many times as necessary by adding instances meeting the condition dmin\dKfor each instance xstreaming in the setstreaming. The buffer of possible updates is checked at a specific time and therefore, the model is updated at that specific moment. This so-called specific time can depend on a Fig. 1 Summary of the online phase of the model without update Neural Computing and Applications 123 temporal value or on an error value. On the one hand, the temporal update can be performed: every day, every month, every three months, etc. On the other hand, the other update can be performed when the error of the actual forecast is higher than a defined threshold. When the so-called specific moment occurs, the K nearest instances of xmin in the current model Mare selected along with all the xstreaming associated with xmin in the buffer. Then, all neighbors both from the current model and from the buffer are sorted by distance and the K smallest ones are kept. All neighbors selected are the updated neighbors of the xmin in the model M. It is possible that all Kneighbors of a xmin are updated in the model with data streaming instances. An example of an update of the model when considering three neighbors (K¼3) is illustrated in the following equation: M¼\xmin ;\yðxstreaming1Þ;yðneighbor1ðxminÞÞ; yðxstreaming2Þ[[ ð13Þ It can be seen that the nearest and furthest neighbors are updated with two streaming instances of the setstreaming and the neighbor one of the offline model is still considered a good neighbor, more specifically, it is the second closest neighbor. Figure 2represents graphically the procedure of the StreamWNN when the incremental update depends on a temporal value (tin the Figure). 3.4 Detection and learning of unknown patterns A data streaming model requires the ability to dynamically learn novel patterns and differentiate them from unexpected patterns. In this research paper, both terms, i.e., novel and unexpected patterns, are considered unknown patterns since the model could not learn them from the offline or static phase as they were not included in the offline data. Each of these types of unknown patterns needs to be treated independently taking into account their nature. On the one hand, a novel pattern or novelty represents a newly emerging concept in the incoming data. Some of the future incoming data are expected to have a behavior similar to that of the emerging novelty [38]. Considering this, novelties are added to the online model as new instances that update the model externally, i.e., the dimensions of the model increase. A novel instance xstreaminglis added to the base model M(see Eq. 2)as follows: M¼\xstreamingl ;\yðneighbor1ðxstreaminglÞÞ;...; yðneighborKðxstreaminglÞÞ [[ ð14Þ where xstreamingl2Rware the features of the l-th instance of the streaming data (setstreaming) that is identified as a novelty and yðneighborjðxstreaminglÞÞ 2 Rhare the classes of the instance of the entire offline historical data (setpatterns and setneighbors) selected as the j-th closest neighbor of xstreamingl. On the other hand, unexpected patterns or outliers in streams represent an usual behavior which is not supposed to be repeated in the future incoming data. Therefore, the online model does not update when an unexpected pattern is detected. Nevertheless, it triggers an alarm to warn the user to get a deeper insight into this data. The identification of the l-th streaming instance xstreamingl as normal, novel or unexpected follows an unsupervised learning approach, since the time series are not labeled. Consequently, the evolution of the error metric over time is the selected approach to determine whether an unknown pattern is correctly identified or not. Specifically, the error committed between the l-th forecast class ystreamingland the l-th actual class ystreaminglis monitored accordingly: xst ¼ Unexpected if error(c yst ;ystÞ[upthr Novel if lowthr\errorðc yst ;ystÞ\upthr Not an unknown pattern otherwise 8 > < > :ð15Þ where upthr and lowthr are the user-defined upper and lower error thresholds for classifying instances as normal, novel or unexpected. Prior to the execution of the model, the user must determine which error metric to consider (see Eqs. 6 and 5) and how to define the upper and lower thresholds. Typically, these thresholds are defined as a combination of the mean and standard deviation of the error of the offline model. This conception for determining the type of pattern is evidenced in several studies such as [39] where the behavior of unknown patterns using K-nearest neighborsbased techniques were investigated. Authors concluded with the idea that outliers or anomalies occur far away from their nearest neighbors. Figure 3shows graphically the StreamWNN procedure when unusual patterns and novelties are detected in the streaming data and how the novelties are added to the base model. 3.5 Incremental learning and novelties update The same execution of the StreamWNN can perform at the same time both the incremental learning approach and the novelties approach. As defined in Sect. 3.3, the first approach updates neighbors in the model. The second approach, defined in Sect. 3.4, identifies unusual patterns in real-time and updates the model with novelties. The first Neural Computing and Applications 123 one is an internal update and the second one is an external update. Figure 4presents an overview of the entire StreamWNN procedure. In this case, both learning approaches are included so that the model can perform them at the same time. 4 Results The proposed algorithm is run on two types of electricityrelated data, i.e., electricity demand and electricity prices, in Sects. 4.1 and 4.2, respectively. In particular, Sects. 4.1.1 and 4.2.1 presents the datasets and parameters used in each experiment. Sections 4.1.2 and 4.2.2 discuss the StreamWNN results obtained. Sections 4.1.3 and 4.2.3 compare the StreamWNN results with those obtained by benchmark forecasting algorithms. Finally, Sect. 4.3 Fig. 2 Summary of the online phase of the model with incremental update Neural Computing and Applications 123 analyzes the scalability and computational complexity of the proposed algorithm. 4.1 Electricity demand 4.1.1 Dataset and experimental setting description The time series used in this experiment consists of the electrical energy consumption in megawatt (MW) in Spain. The time series has 497,832 samples measured every 10 min. The historical data contain all samples from January 1st 2007 to June 21st 2016, after a pre-processing step (e.g., adjustment of time shift days data) [33]. The whole dataset is divided in 3 sets: 70% for the offline part (70% for the setneighbors and 30% for the setpatterns, respectively) and 30% for the online data (setstreaming). Thus, the offline set consists of data from January 2007 to mid August 2013 Fig. 3 Summary of the online phase of the model with unknown patterns detection and learning Neural Computing and Applications 123 and the streaming set from mid August 2013 to mid June 2016. Data are available online at https://www.omie.es. This research paper reports the results obtained for the following parameters: 4 h of prediction horizon (h¼24), one day of past-window features (w¼144) and four neighbors for each pattern (k¼4). An extensive study of the behavior of different parameters for the same dataset can be found in [37]. The selected parameters are the ones that obtained the most accurate results. Fig. 4 Overview of the entire process of StreamWNN Neural Computing and Applications 123 4.3 Scalability and complexity analysis A very important aspect of algorithms that work in streaming is to achieve real-time results during the online phase. For this, it is very important that the algorithm is scalable so that the execution times are not affected exponentially when the set of past data increases. Figure 12 presents the execution time spent for the prediction of the demand electricity dataset for four different forecast horizons with online daily update. These prediction horizons are: h¼24 (with w¼144), h¼24 (with w¼288), h¼ 72 (w¼576) and h¼144 (w¼864). The conclusions presented are the same as those obtained for the price datasets. It can be noted that the number of iterations for each prediction horizon is different as wand hvalues are different. The execution time increases linearly versus iterations for all prediction horizons, but the shorter the prediction horizon, the higher the scalability of the proposed prediction algorithm, as a line with a smaller slope can be seen in the Figure. Thus, it can be concluded that the algorithm is scalable and suitable to deal with big data. Another important consideration given the nature of the StreamWNN is its computational complexity. In the offline phase, where there are no streaming constraints, it is oðknwÞ, being kthe number of neighbors to consider, n the length of the setneighbors and wthe fixed length of the past values used to predict, as defined in Sect. 3. In the online phase, the real-time requirements are met and the computational complexity is o(1) since the entire model is already created [37]. 5 Conclusions In this paper, a new forecasting algorithm has been proposed to predict electricity demand and electricity prices time series in real-time. As previous step to the prediction, an initial forecasting model based on the K-nearest neighbors algorithm has been obtained using historical data. This model is composed of a set of patterns along with its kcorresponding nearest neighbors. Thus, the final prediction has been computed using the K-nearest neighbors of the nearest pattern to the streaming data. In this way, predictions can be obtained in real-time because it is not necessary to compute the K-nearest neighbors each time a prediction is performed as these Kneighbors are already computed. The algorithm has been successfully applied, obtaining accurate predictions over time by incrementally updating the model. This update consists of updating the neighbors in real-time with the new data Fig. 11 Examples of novelties detected in energy price normal dataset Neural Computing and Applications 123 received in streaming. In addition, the algorithm has been able to find novel patterns that it has incorporated into the model by updating it. Different unusual patterns in the realtime predictions have also been detected. Different parameters for the update’s options have been tested and the runtimes for calculating the real-time prediction show that the algorithm is efficient as well as scalable with respect to the number of iterations. In future work the algorithm will be developed to be suitable to work with streaming multivariate time series and to solve both regression and classification problems in real-time. Acknowledgements The authors would like to thank the Spanish Ministry of Science, Innovation and Universities for the support under Projects PID2020-117954RB-C2, TED2021-131311B-C22 and PID2023-146037OB-C22. Funding Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. Data availability The data that support the findings of this study are available in https://www.esios.ree.es and https://www.omie.es. Declarations Conflict of interest The authors declare that they have no conflict of interest. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons. org/licenses/by/4.0/. References 1. Sahal R, Breslin JG, Ali MI (2020) Big data and stream processing platforms for Industry 4.0 requirements mapping for a predictive maintenance use case. J Manuf Syst 54:138–151 2. Bifet A, Hammer B, Schleif FM (2019) Recent trends in streaming data analysis, concept drift and analysis of dynamic data sets. In: 27th European symposium on artificial neural networks (ESANN), pp 421–430 3. Shivaraman N, Saki S, Liu Z, Ramanathan S, Easwaran A, Steinhorst S (2020) Real-time energy monitoring in IoT-enabled mobile devices . In: Proceedings of the 23th design, automation test in europe conference exhibition (DATE), pp 991–994 4. Alizadeh M, Beheshti MTH, Ramezani A, Bolouki S (2023) An optimized hybrid methodology for short-term traffic forecasting in telecommunication networks. Trans Emerg Telecommun Technol 34(12):4860 5. Alizadeh M, Beheshti MTH, Ramezani A, Saadatinezhad H (2020) Network traffic forecasting based on fixed telecommunication data using deep learning. In: 2020 6th Iranian conference on signal processing and intelligent systems (ICSPIS), pp 1–7 6. Alizadeh M, Mousavi SE, Beheshti MTH, Ostadi A (2021) Combination of feature selection and hybrid classifier as to network intrusion detection system adopting fa, gwo, and bat optimizers. In: 2021 7th International conference on signal processing and intelligent systems (ICSPIS), pp 1–7 7. Alothali E, Alashwal H, Harous S (2019) Data stream mining techniques: a review. Telecommun Comput Electron Control 17(2):728–737 Table 5 Forecast error results for the streaming phase of the electricity price dataset Algorithm MAE (€/MWh) MAPE (%) StreamWNN online daily ?novelties 3.57 8.35 MLP 3.79 8.52 Passive-aggressive learning 27.5 13.73 AMRules 5.09 12.93 Hoeffding Tree 3.75 9.11 Hoedding Adaptive Tree 3.31 7.19 (a) Normal period StreamWNN online daily ?novelties 2.44 8.87 MLP 2.72 9.93 Passive-aggressive learning 6.11 19.94 AMRules 3.03 11.94 Hoeffding tree 3.44 12.25 Hoedding adaptive tree 3.66 12.93 (b) Quarantine period StreamWNN online daily ?novelties 3.68 5.47 MLP 3.73 5.69 Passive-aggressive learning 8.45 12.66 AMRules 5.42 8.09 Hoeffding tree 3.75 5.68 Hoedding adaptive tree 3.40 5.14 (c) Fraud period The best results are presented in bold Fig. 12 Execution time versus number of iterations for daily neighbor updating in the online phase Neural Computing and Applications 123 8. Roosa K, Lee Y, Luo R, Kirpich A, Rothenberg R, Hyman JM, Yan P, Chowell G (2020) Real-time forecasts of the COVID-19 epidemic in China from February 5th to February 24th, 2020. Infect Dis Model 5:256–263 9. Moreira-Matias L, Gama J, Ferreira M, Mendes-Moreira J, Damas L (2013) Predicting taxi-passenger demand using streaming data. IEEE Trans Intell Transp Syst 14:1393–1402 10. Melgar-Garcı ´a L, Godinho MT, Espada R, Gutie ´rrez-Avile ´sD, Brito IS, Martı ´nez-A ´lvarez F, Troncoso A, Rubio-Escudero C (2021) Discovering spatio-temporal patterns in precision agriculture based on triclustering. In: Proceedings of the 15th international conference on soft computing models in industrial and environmental applications (SOCO). Springer, Cham, pp 226–236 11. Melgar-Garcı ´a L, Gutie ´rrez-Avile ´s D, Rubio-Escudero C, Troncoso A (2021) Discovering three-dimensional patterns in realtime from data streams: an online triclustering approach. Inf Sci 558:174–193 12. Melgar-Garcı ´a L, Gutie ´rrez-Avile ´s D, Rubio-Escudero C, Troncoso A (2020) High-content screening images streaming analysis using the STriGen methodology. In: Proceedings of the 35th annual association for computing machinery symposium on applied computing (SAC), pp 537–539 13. Yaseen ZM, El-shafie A, Jaafar O, Afan HA, Sayl KN (2015) Artificial intelligence based models for stream-flow forecasting: 2000–2015. J Hydrol 530:829–844 14. Jime ´nez-Herrera P, Melgar-Garcı ´a L, Asencio-Corte ´s G, Troncoso A (2020) A new forecasting algorithm based on neighbors for streaming electricity time series. In: Proceedings of the 15th international conference on hybrid artificial intelligent systems (HAIS). Springer, Cham, pp 522–533 15. Qiu X, Suganthan PN, Amaratunga GAJ (2018) Ensemble incremental learning random vector functional link network for short-term electric load forecasting. Knowl-Based Syst 145:182–196 16. Wang P, Xu K, Ding Z, Du Y, Liu W, Sun B, Zhu Z, Tang H (2022) An online electricity market price forecasting method via random forest. IEEE Trans Ind Appl 58(6):7013–7021 17. Sai W, Pan Z, Liu S, Jiao Z, Zhong Z, Miao B, Chan SH (2023) Event-driven forecasting of wholesale electricity price and frequency regulation price using machine learning algorithms. Appl Energy 352:121989 18. Perez-Chaco ´n R, Talavera-Llames RL, Martı ´nez-A ´lvarez F, Troncoso A (2016) Finding electric energy consumption patterns in big time series data. In: Proceedings of the 13th international conference on distributed computing and artificial intelligence (DCAI), pp 231–238 19. Talavera-Llames RL, Pe ´rez-Chaco ´n R, Martı ´nez-Ballesteros M, Troncoso A, Martı ´nez-A ´lvarez F (2016) A nearest neighboursbased algorithm for big time series data forecasting. In: Proceedings of the 11th international conference on hybrid artificial intelligent systems (HAIS), pp 174–185 20. Weron R (2014) Electricity price forecasting: a review of the state-of-the-art with a look into the future. Int J Forecast 30(4):1030–1081 21. Liu L, Bai F, Su C, Ma C, Yan R, Li H, Sun Q, Wennersten R (2022) Forecasting the occurrence of extreme electricity prices using a multivariate logistic regression model. Energy 247:123417 22. Mehrdoust F, Noorani I, Brahim Belhaouari S (2023) Forecasting nordic electricity spot price using deep learning networks. Neural Comput Appl 19169–19185 23. Magalha ˜es BG, Bento PMR, Pombo JAN, Calado MRA, Mariano SJPS (2023) Spot price forecasting for best trading strategy decision support in the Iberian electricity market. Expert Syst Appl 224:120059 24. Liu K, Li Z, Yao C, Chen J, Zhang K, Saifullah M (2016) Coupling the k-nearest neighbor procedure with the Kalman filter for real-time updating of the hydraulic model in flood forecasting. Int J Sedim Res 31(2):149–158 25. Komma J, Blo ¨schl G, Reszler C (2008) Soil moisture updating by ensemble Kalman filtering in real-time flood forecasting. J Hydrol 357(3):228–242 26. Thomas RQ, Figueiredo RJ, Daneshmand V, Bookout BJ, Puckett LK, Carey CC (2020) A near-term iterative forecasting system successfully predicts reservoir hydrodynamics and partitions uncertainty in real time. Water Resour Res 56(11):20 27. Nanda T, Sahoo B, Chatterjee C (2019) Enhancing real-time streamflow forecasts with wavelet-neural network based errorupdating schemes and ECMWF meteorological predictions in variable infiltration capacity model. J Hydrol 575:890–910 28. Yan L, Feng J, Wu Y, Hang T (2020) Data-driven fast real-time flood forecasting model for processing concept drift. cloud computing. In: Smart grid and innovative frontiers in telecommunications. Springer, Cham, pp 363–374 29. Sun L, Ji Y, Zhu M, Gu F, Dai F, Li K (2021) A new predictive method supporting streaming data with hybrid recurring concept drifts in process industry. Comput Ind Eng 161:107625 30. Yang H, Schell KR (2021) Real-time electricity price forecasting of wind farms with deep neural network transfer learning and hybrid datasets. Appl Energy 299:117242 31. Yang H, Schell KR (2022) Ghtnet: tri-branch deep learning network for real-time electricity price forecasting. Energy 238:122052 32. Yang H, Schell KR (2022) Qcae: a quadruple branch CNN autoencoder for real-time electricity price forecasting. Int J Electr Power Energy Syst 141:108092 33. Melgar-Garcı ´a L, Gutie ´rrez-Avile ´s D, Rubio-Escudero C, Troncoso A (2021) Nearest neighbors-based forecasting for electricity demand time series in streaming. In: Advances in artificial intelligence. Springer, Cham, pp 185–195 34. Jeffers J, Reinders J, Sodani A (2016) Chapter 24-Machine learning. In: Intel Xeon Phi processor high performance programming, 2nd edn, pp 527–548 35. Larranaga P, Atienza D, Rozo JD, Ogbechie A, Puerto-Santana C, Bielza C (2018) Industrial applications of machine learning. CRC Press, Boca Raton 36. Benczu ´r AA, Kocsis L, Pa ´lovics R (2019) Online machine learning algorithms over data streams. In: Encyclopedia of big data technologies. Springer, pp 1199–1207 37. Melgar-Garcı ´a L, Gutie ´rrez-Avile ´s D, Rubio-Escudero C, Troncoso A (2023) Identifying novelties and anomalies for incremental learning in streaming time series forecasting. Eng Appl Artif Intell 123:106326 38. Faria ER, Gonc¸alves IJCR, Carvalho ACPLF, Gama J (2016) Novelty detection in data streams. Artif Intell Rev 45:235–269 39. Chandola V, Banerjee A, Kumar V (2009) Anomaly detection: a survey. ACM Comput Surv 41(3) 40. Trull O, Garcı ´a-Dı ´az JC, Troncoso A (2019) Application of discrete-interval moving seasonalities to Spanish electricity demand forecasting during Easter. Energies 12(6):1083 41. Trull O, Garcı ´a-Dı ´az JC, Troncoso A (2021) One-day-ahead electricity demand forecasting in holidays using discrete-interval moving seasonalities. Energy 231:120966 42. Vega-Ma ´rquez B, Rubio-Escudero C, Nepomuceno-Chamorro IA, Arcos-Vargas A (2021) Use of deep learning architectures for day-ahead electricity price forecasting over different time periods in the spanish electricity market. Appl Sci 11(13) 43. Python: River. https://riverml.xyz/latest/. Accessed 01 Sept 2024 (2024) Neural Computing and Applications 123