Anticipating Future Behavior of an Industrial Press Using LSTM Networks

applied sciences A icle An icipa ing Fu u e Beha io o an Indus ial P ess Using LSTM Ne wo ks Balduíno Césa Ma eus 1,2,* , Ma eus Mendes 3,4,* , José To es Fa inha 3,5 and An ónio Ma ques Ca doso 2   Ci a ion: Ma eus, B.C.; Mendes, M.; Fa inha, J.T.; Ca doso, A.M. An icipa ing Fu u e Beha io o an Indus ial P ess Using LSTM Ne wo ks. Appl. Sci. 2021,11, 6101. h ps://doi.o g/10.3390/app11136101 Academic Edi o s: Ma lene Amo im, Yu al Cohen and João Reis Recei ed: 30 Ap il 2021 Accep ed: 25 June 2021 Published: 30 June 2021 Publishe ’s No e: MDPI s ays neu al wi h ega d o ju isdic ional claims in published maps and ins i u ional a il- ia ions. Copy igh : © 2021 by he au ho s. Licensee MDPI, Basel, Swi ze land. This a icle is an open access a icle dis ibu ed unde he e ms and condi ions o he C ea i e Commons A ibu ion (CC BY) license (h ps:// c ea i ecommons.o g/licenses/by/ 4.0/). 1EIGeS—Resea ch Cen e in Indus ial Enginee ing, Managemen and Sus ainabili y, Lusó ona Uni e si y, Campo G ande, 376, 1749-024 Lisboa, Po ugal 2 CISE—Elec omecha onic Sys ems Resea ch Cen e, Uni e si y o Bei a In e io , Calçada Fon e do Lamei o, P–62001-001 Co ilhã, Po ugal; [email p o ec ed] 3Poly echnic o Coimb a, ISEC, 3045-093 Coimb a, Po ugal; [email p o ec ed] 4Ins i u e o Sys ems and Robo ics, Uni e si y o Coimb a, 3004-531 Coimb a, Po ugal 5Cen e o Mechanical Enginee ing, Ma e ials and P ocesses—CEMMPRE, 3030-788 Coimb a, Po ugal *Co espondence: [email p o ec ed] (B.C.M.); [email p o ec ed] (M.M.) Abs ac : P edic i e main enance is e y impo an in indus ial plan s o suppo decisions aiming o maximize main enance in es men s and equipmen ’s a ailabili y. This pape p esen s p edic i e models based on long sho - e m memo y neu al ne wo ks, applied o a da ase o senso eadings. The aim is o o ecas u u e equipmen s a uses based on da a om an indus ial pape p ess. The da ase s con ain da a om a h ee-yea pe iod. Da a a e p e-p ocessed and he neu al ne wo ks a e op imized o minimize p edic ion e o s. The esul s show ha i is possible o p edic u u e beha io up o one mon h in ad ance wi h easonable con idence. Based on hese esul s, i is possible o an icipa e and op imize main enance decisions, as well as con inue esea ch o imp o e he eliabili y o he model. Keywo ds: ime se ies p edic ion; LSTM p edic ion; deep lea ning p edic ion; p edic i e main enance 1. In oduc ion Mode n p ocesso s, compu e s and high speed ne wo ks make i possible o acqui e, ans e and s o e la ge quan i ies o da a in eal ime. Acquisi ion and combina ion o da a om di e en senso s makes i possible o gain an insigh ul iew o he s a e o ac o ies, indus ial plan s and o he acili ies. La ge da ase s can be cons uc ed, s o ed and p ocessed using in o ma ion echnologies such as Big Da a, cloud compu ing, cu ing- edge compu ing, and a i icial in elligence ools. The In e ne o Things (IoT) is a ecen concep , which p o ides many bene i s o di e en a eas, such as main enance and p oduc- ion managemen , because i acili a es he au oma ion o asks such as moni o ing and main enance. This esul s in he popula iza ion o in elligen sys ems, which a e highly dependen on Big Da a [ 1 ] and a e an impo an a ea o s udy, since hey o e he ools and me hods o acqui e and p ocess la ge olumes o da a such as his o ical p oduc ion p ocesses, including many p oduc ion and ope a ing pa ame e s. Mode n ime-se ies and o he da a analysis echniques ha e been used wi h success o di e en asks, such as eeway a ic analysis [ 2 ] and addi i e manu ac u ing [ 3 ]. Di e en app oaches ha e also been p oposed in he ield o p edic i e main enance [ 4 , 5 ]. Sa is ac o y esul s we e ob ained using Big Da a eco ds as suppo o PCA models, which esul ed in a wa ning ala m se e al days be o e a po en ial ailu e happened [6]. Li e cycle op imiza ion has been an impo an conce n o decades. A physical asse wi h p ope main enance will ha e a longe use ul li e wi h a g ea e e u n on in es men o he o ganiza ion [7]. P edic i e main enance equi es good quali y da a. The in o ma ion ha is ex ac ed om he online o o line da a mus be eliable, and so he esul s mus be good enough o jus i y he in es men in da a collec ion and analysis. The p ocess s a s om he Appl. Sci. 2021,11, 6101. h ps://doi.o g/10.3390/app11136101 h ps://www.mdpi.com/jou nal/applsci Appl. Sci. 2021,11, 6101 2 o 16 co ec calib a ion o he eading senso s and equipmen [ 8 ]. The da a a e hen s o ed and p ocessed using di e en models, such as P incipal Componen Analysis (PCA) and Neu al Ne wo ks [ 9 ]. Main enance planning in ol es he use o se e al algo i hms, he mos common being ime se ies [10]. Main enance o equipmen in he indus y becomes a sensi i e and impo an poin ha a ec s he equipmen ’s ope a ing ime and e iciency [ 5 ]. This makes main enance one o he s a egic poin s o he de elopmen and g ow h o compe i i eness is-à- is compe i o s. Chen and Tseng s udied he o al expec ed cos o main aining a lo a ion sys em, including he cos o los p oduc ion, he cos o epai s, and he cos o s andby machines [11]. Daniyan e al. p opose he in eg a ion o A i icial In elligence (AI) sys ems, which will b ing many bene i s in diagnosing condi ion p oblems o indus ial machines [ 12 ]. They highligh he iabili y o AI ha combines he use o A i icial Neu al Ne wo ks (ANNs) wi h a dynamic ime se ies model, o aul diagnos ics, o op imize he equipmen in e en ion ime. Hsu e al. demons a ed ha neu al ne wo ks can be a g ea echnology in he suppo and decision making o la ge and small companies [ 13 ]. The e is a end o use hose ools in p edic i e main enance sys ems wi h he aim o making he p edic ion sys ems mo e in elligen [14]. Acco ding o Jimenez e al., he e is a g ea e o in he de elopmen o p edic i e models o applica ion in p edic i e main enance [ 15 ]. Ay az and Alpay apply Long Sho -Te m Memo y (LSTM) neu al ne wo k app oaches o p edic eal p oduc ion da a, ob aining sa is ac o y esul s, supe io o con en ional models [ 16 ]. In hei s udy o imp o e main enance planning o minimize unexpec ed s ops, hey apply a new me hod ha consis s o he combined use o decomposi ion in empi ical mode o ensemble and long- e m memo y. Thei esul s showed a pe o mance supe io o o he s a e o he a models. LSTM ne wo ks use se e al po s wi h di e en unc ions o con ol neu ons and o s o e in o ma ion. The LSTM cell can e ain impo an in o ma ion o a longe pe iod in which i is used. This p ope y o in o ma ion main enance allows he LSTM o exhibi a good pe o mance in he classi ica ion, p ocessing, o o ecas ing o a complex dynamic sequences [17]. The p esen wo k uses di e en LSTM models o p edic u u e ends o six a iables, on a da ase con aining h ee yea s o da a samples g abbed in an indus ial p ess, which aims o ope a e con inuously wi h minimum down ime. Di e en da a p e-p ocessing echniques, ne wo k a chi ec u es and hype pa ame e s we e es ed in o de o de e mine he models ha bes i he da a and p o ide he lowes p edic ion e o s. Sec ion 2con ains a summa y o ela ed wo k. Sec ion 3desc ibes he heo y o he LSTM ne wo ks. Sec ion 4desc ibes he me hods used o he p esen wo k. Sec ion 5 de- sc ibes he esul s and alida ion o he p edic i e model. Sec ion 7d aws some conclusions and sugges ions o u u e wo k. 2. Rela ed Wo k 2.1. P edic i e Main enance In sma indus ies, p edic i e main enance is one o he mos used echniques o imp o e condi ion moni o ing, as i allows one o e alua e he condi ions o speci ic equipmen in o de o p edic p oblems be o e ailu e [ 18 ]. Fo good pe o mance o p edic i e models, i is impo an ha he senso da a collec ed a e o good quali y. Deep neu al models ha e been used wi h success o imp o e p edic ion o condi ion moni o ing o indus ial equipmen . Wang e al. [ 19 ] use a model o long sho - e m ecu en neu al ne wo ks (LSTM- RNN) wi h he objec i e o p edic i e main enance based on pas da a. The main objec i e o p edic i e main enance is o make an accu a e es ima e o a sys em’s Remaining Use ul Li e (RUL). T adi ional sys ems a e only able o wa n he use when i is oo la e and he Appl. Sci. 2021,11, 6101 3 o 16 ailu e occu s, causing an unp edic able o line pe iod du ing which he sys em canno ope a e p ope ly wi h a consequen was e o ime and esou ces [20]. In o de o assess he condi ion o a sys em, he p edic i e main enance app oach employs senso s o di e en kinds. Some examples a e empe a u e, ib a ion, eloci y o noise senso s, which a e a ached o he main componen s whose ailu e would comp omise he en i e ope a ion o he sys em. In his sense, p edic i e main enance analyzes he his o y o a sys em in e ms o he measu emen s collec ed by he senso s ha a e dis ibu ed among he componen s, wi h he objec i e o ex ac ing a “ ailu e pa e n” ha can be exploi ed o plan an op imal main enance s a egy and hus educing o line pe iods [ 21 ]. In a case ela ed o he s eel indus y, Re . [ 22 ] used neu al ne wo ks o classi ica ion o main enance ac i i ies, so ha in e en ions a e planned acco ding o he ac ual s a us o he machine and no in ad ance. Using mul iple neu al ne wo ks o iden i y s a us and RUL a a highe esolu ion can be e y di icul , as he sys em can p edic ailu e classi ica ions and may no be able o ecognize neighbo ing s a es. One limi a ion a ises om he need o main enance eco ds o label da ase s and he need o la ge amoun s o da a o adequa e quali y wi h main enance e en s, such as componen ailu es. When sys ems s a o be e y complex o he numbe o senso measu emen s o manage is e y la ge, i can be di icul o es ima e a ailu e. Fo his eason, in ecen yea s, machine lea ning echniques a e used mo e and mo e o p edic wo king condi ions o a componen . Ma hew e al. [ 23 ] p opose se e al app oaches o machine lea ning such as suppo ec o machines (SVMs), decision ees (DTs), Random Fo es s (RFs), and o he s ha show which echnique has he bes pe o mance in RUL o ecas o u bo an engines. A majo challenge in ope a ions managemen is ela ed o p edic ing machine speed, which can be used o dynamically adjus p oduc ion p ocesses based on di e en sys em condi ions, op imize p oduc ion pe o mance and minimize ene gy consump ion [ 24 ]. Essien and Gianne i [ 25 ] use a deep con olu ional LSTM encode –decode a chi ec u e model on eal da a, ob ained om a me al packaging ac o y. They show ha i is possible o pe o m combina ions o LSTM wi h o he ne wo ks o signi ican ly imp o e he esul s. 2.2. P edic ion wi h LSTM Models LSTM neu al ne wo ks achie ed he bes pe o mance in a numbe o compu a ional sequence labeling asks, including speech ecogni ion and machine ansla ion [ 26 ]. The e a e a a ie y o enginee ing p oblems ha can be sol ed using p edic i e neu al models. Besh and Za zou a used neu al ne wo k models o p edic p oblems o suspended oad b idge s uc u es based on global na iga ion sa elli e sys em obse a ions [ 27 ]. Sak e al. demons a ed ha he p oposed LSTM a chi ec u es exhibi be e pe o mances compa ed o deep neu al ne wo ks (DNNs) in a la ge ocabula y speech ecogni ion ask wi h a la ge numbe o ou pu s a es [ 28 ]. Chen e al. adop ed LSTMs o p edic ing he ailu e o hea y uck ai comp esso s [ 29 ]. They concluded ha he use o LSTMs leads o mo e consis ency in p edic ions o e ime compa ed o models ha igno e his o y, such as andom o es models. Gosh e al. [ 30 ] p esen ed an ex ension ha hey called Con ex ual LSTM (CLSTM). This model was also used o he o ecas ing o pollu an s. The e is also he p oposal o a gene ic long sho - e m memo y (GLSTM), which has been used in he s udy o wind ene gy o ecas ing [ 31 ]. Guo e al. p esen ed a combina ion me hod based on eal- ime p edic ion e o s in which he suppo ec o eg ession (SVR) and LSTM ou pu s a e combined in he inal esul s o he model’s p edic ion, hus ob aining esul s o g ea e p ecision [32]. Ren e al. used a combina ion o a Con olu ion Neu al Ne wo ks (CNNs) and LSTM in o de o ex ac mo e in-dep h in o ma ion om da a o p edic he use ul li e o ion ba e ies [ 33 ]. Niu e al. used an LSTM and de eloped an e ec i e speed p edic ion model o sol e p edic ion p oblems o e ime [ 34 ]. Feng e al. epo ha he LSTM algo i hm is supe io and, acco ding o hem, i pe o ms be e han con en ional neu al ne wo k models [35]. Appl. Sci. 2021,11, 6101 4 o 16 The a chi ec u e o an LSTM ne wo k includes he numbe o hidden laye s and he numbe o delay uni s, which is he numbe o p e ious da a poin s ha a e conside ed o aining and es ing. Cu en ly, he e is no gene al ule o selec ing he numbe o delays and hidden laye s [ 36 ]. A deep LSTM can be buil by s acking mul iple LSTM laye s, which gene ally wo ks be e han a single laye . Deep LSTM ne wo ks ha e been applied o sol e many eal-wo ld sequence modelling p oblems [ 37 ]. The LSTM can also be used o planning s udies [38], namely o planning he analysis o oad a ic speed. To p oduce a p edic ion model wi h good accu acy, i is necessa y o op imize neu al models’ hype pa ame e s. While simple models can o en p oduce good esul s wi h de aul hype pa eme e s, he op imiza ion p ocess can g ea ly imp o e he esul s [ 39 – 41 ]. The selec ion o hype pa ame e s o en makes he di e ence be ween unde pe o mance and s a e-o - he-a pe o mance. Op imiza ion is o en pe o med using machine lea ning algo i hms, such as g id sea ch, g ey wol op imiza ion o pa icle swa m op imiza ion. In he p esen p edic ion model, howe e , he hype pa ame e s we e op imized manually, ollowing a ial and e o guided p ocess, one a iable a a ime. This me hod was ollowed because i was he mos con enien conside ing he limi ed compu ing powe a ailable. 2.3. LSTM wi h Encode and Decode Expe imen s we e pe o med wi h a p edic i e model based on he LSTM wi h encode and decode a chi ec u e. The model consis s o wo LSTMs, in which he i s LSTM has he unc ion o p ocessing an inpu sequence and gene a ing an encoded s a e. The encoded s a e comp esses he in o ma ion in he inpu s eam. The second LSTM, called a decode , uses he encoded s a e o p oduce an ou pu sequence. Those inpu and ou pu sequences can be o di e en leng hs. This echnique has al eady been used o sol e p oblems such as he p edic ion o ehicle ajec o ies based on deep lea ning [ 42 ]. This a chi ec u e [ 43 ] has shown g ea pe o mance o asks o ansla ing om sequence o sequence. LSTM encode –decode models ha e also been p oposed o lea ning asks such as au oma ic ansla ion [ 43 , 44 ]. The e is he applica ion o his model o sol e many p ac ical p oblems, such as he s udy o he equipmen condi ion, applica ions in language ansla ions, among o he s [45–47]. 3. Theo e ical Backg ound The p esen wo k uses LSTM ne wo ks, conside ing he e e ed di e en s udies showing hei use ulness o ime se ies p edic ions [ 48 , 49 ]. The LSTM is a deep lea ning ecu en neu al ne wo k a chi ec u e ha is a a ia ion o adi ional ecu en neu al ne wo ks (RNNs). I was in oduced by Hoch ei e and Schmidhube in 1997. The mos popula e sion is a modi ica ion e ined by many wo ks in he li e a u e [ 50 , 51 ], which is called anilla LSTM (he eina e e e ed o as LSTM). The LSTM is excellen a handling ime se ies da a only wi h i s ne wo k pa ame e s. Fo example, weigh s and pola iza ion a e adjus ed o op imized [ 52 ]. The p ima y modi ica ion o he LSTM when compa ed o he RNN a chi ec u e is he s uc u e o he hidden laye [ 53 ]. The LSTM model is a powe ul ype o ecu en neu al ne wo k (RNN), capable o lea ning long- e m depen- dencies [ 54 ]. They became popula due o hei powe o ep esen a ion and e ec i eness in cap u ing long- e m dependencies [55]. Many ne wo ks showed ins abili y when dealing wi h exploding o anishing g adien p oblems du ing lea ning. Those p oblems happen when he g adien o he e o is oo la ge o oo small. I i is oo la ge, i o e lows and he e o s canno p opaga e p ope ly h ough di e en laye s du ing lea ning. I i is oo small, i anishes and he ne wo k does no lea n. Di e en me hods we e p oposed o sol e hose p oblems, known as a kind o o “doo con ol” ha is used in RNN models. Fo example, Ga ed Recu en uni (GRU) algo i hms [ 56 , 57 ], as he LSTMs [ 58 , 59 ], a e o a la ge ex en immune o he g adien p oblems and lea n well. The LSTM ne wo k s uc u e is based on h ee po s whose unc ion is o egula e he low. Those po s a e called he en ance doo , he o ge ga e, and he exi doo . The Appl. Sci. 2021,11, 6101 5 o 16 main po o en y is o egula e he en y o new memo y da a; he o ge ga e has he unc ion o egula ing he s o age ime in he ne wo k memo y and he ou pu po in ends o egula e how much he alue e ained in memo y in luences he ac i a ion o he ou pu block [60]. Kong e al. demons a e some ele an conclusions such as (1) LSTM has a good p edic i e capaci y; (2) hei use can signi ican ly imp o e he p o i o se ice p o ide s, so he e is an oppo uni y when i comes o explo ing he o ecas in eal ime [ 61 ]. LSTM ne wo ks a e he de ac o gold s anda d o deep lea ning algo i hms o analyzing ime se ies da a [55]. Figu e 1shows he in e nal a chi ec u e o an LSTM uni cell. Acco ding o [ 62 , 63 ], he in e nal calcula ion o mulae o he LSTM uni a e de ined as ollows: i =σ(x Ui+h −1Wi+bi)(1) =σ(x U +h −1W +b )(2) o =σ(x Uo+h −1Wo+bo)(3) a = an(x UC+h −1WC+bC)(4) whe e Ui , U , Uo and UC a e he weigh ma ices o mapping he cu en inpu laye on h ee po s and he s a e o he cu en inpu cell. Figu e 1. De ailed layou o a long sho - e m memo y uni [63]. Wi , W , Wo and WC a e he weigh ma ices o mapping he p e ious ou pu laye on h ee po s and he cu en s a e o he inpu cell. b , bi , bo , and bc a e pola iza ion ec o s o calcula ing he s a e o he doo and he inpu cell. σ is he ga e ac i a ion unc ion, which is no mally a sigmoid unc ion. an is he hype bolic angen unc ion which is he ac i a ion unc ion o he cu en s a e o he inpu cell. Then, he cu en s a e o he ou pu cell and he ou pu laye can be calcula ed using he ollowing equa ions. C =σ( ×C −1+i ×a )(5) h = anh(C )×o (6) To assess he quali y o he p edic ion model, one o he mos popula me ics is he Roo Mean Squa e E o (RMSE), which is gi en by Equa ion (7): RMSE =s1 n n ∑ =1 (Y −ˆ Y)2(7) whe e Y ep esen s he desi ed ( eal) alue and ˆ Y is he p edic ed (ob ained om he model) alue. The di e ence be ween Y and ˆ Y is he e o be ween he alue expec ed Appl. Sci. 2021,11, 6101 6 o 16 o ob ain and he alue ac ually ob ained om he ne wo k. n ep esen s he numbe o samples used in he es se . The RMSE, howe e , is an absolu e e o . The e o e, he e a e also he Mean Absolu e Pe cen age E o (MAPE) and he Mean Absolu e E o (MAE). Those e o s a e gi en by he ollowing o mulae: MAE =1 n n ∑ =1 |Y −ˆ Y |(8) MAPE =1 n n ∑ =1 |Y −ˆ Y | |Y |(9) whe e Y ep esen s he eal alue, ˆ Y he p edic ed alue and n ep esen s he o al numbe o samples. 4. Da a P epa a ion Da a a e key o de eloping e icien modeling and planning. Howe e , o be aluable, da a need o be p ocessed and s uc u ed be o e being analyzed. 4.1. The P oblem The main goal o he p esen wo k is o p edic po en ial ailu es in an indus ial d ying p ess be o e hey happen. Da a come om six senso s ins alled in he p ess. Those senso s moni o he ope a ion o he p ess, wi h a sampling pe iod o one minu e. The moni o ed a iables a e: (1) elec ic cu en in ensi y; (2) oil le el a he hyd aulic uni ; (3) VAT p essu e; (4) o a ion speed; (5) empe a u e in he hyd aulic uni ; and (6) o que. The da ase con ains six ime se ies, one o each senso , wi h he alues s o ed in he da abase om 2016 o Augus 2020. Figu e 2shows a plo o he six ime se ies, be o e any p ocessing is applied. These da a p esen some uppe and lowe ex emes, which may be disc epan da a. Those disc epan samples may be due o eading e o s o pe iods when he equipmen was o o in ano he a ypical s a e. Figu e 2. Plo o he o iginal da ase alues. The a iables a e elec ic cu en in ensi y, hyd aulic uni oil le el, VAT p essu e, mo o eloci y, empe a u e a he hyd aulic uni , and o que. Some o he samples, such as hose when he equipmen was o bu he senso s we e s ill eading, can comp omise he aining o he machine lea ning models o be de eloped. Table 1shows some s a is ical pa ame e s such as mean, s anda d de ia ion (s d), minimum, hi d quan iles, and maximum alue. Appl. Sci. 2021,11, 6101 7 o 16 Table 1. S a is ical pa ame e s o he da ase a iables, be o e p ocessing: C. in ensi y, hyd aulic uni oil le el, o que, VAT p essu e, eloci y, and empe a u e. C. In ensi y Hyd aulic To que VAT Veloci y Tempe a u e mean 30.26 75.90 15.28 18.25 4.59 38.22 s d 1.36 4.54 0.69 2.67 0.98 1.62 min 26.34 62.93 13.59 9.67 1.27 33.19 Q1—25% 29.30 72.86 14.90 17.13 3.92 37.17 Q2—50% 30.46 75.53 15.43 18.72 4.57 38.33 Q3—75% 31.28 79.52 15.78 19.97 5.28 39.35 max 34.26 88.97 17.09 26.17 7.87 43.10 4.2. Cleaning Disc epan Da a In o de o acili a e he aining p ocess, disc epan samples we e iden i ied and emo ed using he quan iles me hod. Samples which a e beyond he Q1− 3 ×s d o Q3+ 3 ×s d a e eplaced by he mean alue. The ex eme alues we e eplaced wi h he a e age. Figu e 3shows he same a iables a e disc epan da a samples we e emo ed. As he igu e shows, he lines a e now smoo he and easie o ead. Figu e 4shows ha he samples a e e enly dis ibu ed a e he wi hd awal o disc epan da a. Figu e 3. Plo o he da ase alues a e cleaning disc epan da a. The a iables a e cu en in ensi y, hyd aulic uni oil le el, VAT p essu e, eloci y, empe a u e, and o que. Figu e 4. Dis ibu ion o da a poin s o all he senso s, wi h lowly and highly disc epan da a cleaned. The p edic i e models o be used a e obus and ole an o noise. Howe e , he cleane da a a e expec ed o show be e esul s. As an example, a p o isional expe imen o ain Appl. Sci. 2021,11, 6101 8 o 16 a neu al ne wo k LSTM model wi h a his o ical window o 70 samples and 40 LSTM uni cells showed highe and unde e mined e o s. The model was no able o lea n o p edic some a iables, as shown in Table 2. Wi h clean da a, he e we e be e and de e minable esul s, as shown in Table 3. The ables show he MAPE and MAE o all inpu a iables, as de e mined in he es se . They also show he RMSE, as calcula ed in he ain and es se s, globally o all a iables. Table 2. P edic ion esul s wi hou cleaning disc epan da a in he da abase, wi h a window o 70 samples and 40 LSTM uni s. Window 70 Days C. In ensi y Hyd aulic To que VAT Veloci y Tempe a u e MAPE in 8.46 in 98.19 in 11.59 MAE 3.52 6.57 24.73 10.53 14.88 4.21 T ain Tes RMSE 79.52 79.64 Table 3. Fo ecas esul s wi h ea men s in he da abase wi h 40 LSTM uni s. Window 70 Days C. In ensi y Hyd aulic To que VAT Veloci y Tempe a u e MAPE 2.52 3.02 2.44 13.10 in 2.48 MAE 0.76 2.28 0.37 1.32 0.57 0.94 T ain Tes RMSE 1.71 1.97 5. Expe imen s and Resul s Expe imen s we e pe o med wi h he aim o alida ing he model ha has he bes pe o mance in p edic ing da a om he indus ial p ess. The es s a e di ided in o wo subsec ions, i s wi h esampling o da a o one sample pe day and hen wi h esampling o a sample each 12 h. 5.1. LSTM Models and Da ase Pa i ion A e p ocessing he da a, expe imen s we e pe o med wi h an LSTM model. The model included an encode and decode , wi h one hidden LSTM laye in he middle and a dense laye a he ou pu . The model was used o ain and p edic , wi h six a iables ha ep esen da a coming om he pape p ess senso s. The goal was o o ecas he alue o hose a iables wi h he highes possible le el o con idence so ha i b ings added bene i s in p edic i e main enance. Figu e 5desc ibes he a chi ec u e o one o he ne wo k models used. The models we e implemen ed in Py hon using he Tenso Flow lib a y and Ke as. The expe imen s we e pe o med aiming o ob ain a p edic ion o all a iables one mon h in ad ance, om a window o a numbe o pas samples. The LSTM models ecei ed, as an inpu , a sequence consis ing o he composi ion o a numbe o samples o each a iable. The numbe o samples depended on he window size and he esampling a e used. The ou pu sequence is composed o he alues p edic ed o each o he a iables. Appl. Sci. 2021,11, 6101 9 o 16 Figu e 5. Model summa y o one o he LSTM ne wo ks used. The model ecei es a window o n samples o each a iable and p edic s he alue o hose a iables as p edic ed 30 days ahead. To ain and es he models, he da ase was di ided in o ain and es subse s. Valida ion was pe o med using he es se , bu hose samples we e no inco po a ed in o he aining se . The aining se con ained 85% o he samples and he es se he emaining 15% o samples. These alues a e adequa e o con e gence du ing lea ning. As an example, Figu e 6shows a lea ning cu e o a model wi h 70 uni s in he middle laye and a window o 30 lag samples. The igu e shows ha lea ning con e ges and akes ewe han 10 epochs. The emainde expe imen s we e pe o med using 100 epochs. Figu e 6. Example o lea ning cu e, showing he loss measu ed du ing aining o an LSTM model. 5.2. Expe imen s o De e mine His o ical Window Size and Numbe o LSTM Uni s Using One Sample pe Day The i s expe imen s pe o med aimed o de e mine he bes window size o use. The smalle he window, he smalle and as e he model ha can be used. Howe e , i he window is oo small, i may be insu icien o make accu a e p edic ions. The o iginal da ase had 1,445,760 da a poin s, which is e y la ge and would equi e a lo o memo y and ime o ain and es . The expe imen s we e pe o med a e down- sampling he da a, so ha he e is only one sample pe day. Tha sample is he a e age o 1004 o iginal samples. The downsampled da ase is, he e o e, less han he one housand o he o iginal da ase . The esul s a e measu ed in he es se . The igu e abo e shows he MAPE and MAE measu ed o each a iable. I also shows he global RMSE measu ed globally o he ain and es se s. As Figu e 7shows, models wi h windows o 40 and 50 samples allow be e lea ning and p oduce smalle p edic ion e o s. Appl. Sci. 2021,11, 6101 16 o 16 42. Pa k, S.H.; Kim, B.; Kang, C.M.; Chung, C.C.; Choi, J.W. Sequence- o-Sequence P edic ion o Vehicle T ajec o y ia LSTM Encode -Decode A chi ec u e. In P oceedings o he 2018 IEEE In elligen Vehicles Symposium (IV), Changshu, China, 26–30 June 2018; pp. 1672–1678. [C ossRe ] 43. Cho, K.; an Me ienboe , B.; Gülçeh e, Ç.; Bouga es, F.; Schwenk, H.; Bengio, Y. Lea ning Ph ase Rep esen a ions using RNN Encode -Decode o S a is ical Machine T ansla ion. a Xi 2014, a Xi :1406.1078. 44. Su ske e , I.; Vinyals, O.; Le, Q.V. Sequence o Sequence Lea ning wi h Neu al Ne wo ks. a Xi 2014, a Xi :1409.3215. 45. Wang, T.; Chen, P.; Ama al, K.; Qiang, J. An Expe imen al S udy o LSTM Encode -Decode Model o Tex Simpli ica ion. a Xi 2016, a Xi :1609.03663. 46. Bengio, S.; Vinyals, O.; Jai ly, N.; Shazee , N. Scheduled Sampling o Sequence P edic ion wi h Recu en Neu al Ne wo ks. a Xi 2015, a Xi :1506.03099. 47. Du, S.; Li, T.; Yang, Y.; Ho ng, S.J. Mul i a ia e ime se ies o ecas ing ia a en ion-based encode –decode amewo k. Neu ocompu ing 2020,388, 269–279. [C ossRe ] 48. Ge s, F.A.; Eck, D.; Schmidhube , J. Applying LSTM o Time Se ies P edic able Th ough Time-Window App oaches. In Neu al Ne s WIRN Vie i-01; Taglia e i, R., Ma ina o, M., Eds.; Pe spec i es in Neu al Compu ing; Sp inge : Be lin/Heidelbe g, Ge many, 2002; pp. 193–200._20. [C ossRe ] 49. Meng, Q.; Wang, H.; He, M.; Gu, J.; Qi, J.; Yang, L. Displacemen p edic ion o wa e -induced landslides using a ecu en deep lea ning model. Eu . J. En i on. Ci . Eng. 2020. [C ossRe ] 50. Sa ikaya, R.; Hin on, G.E.; Deo as, A. Applica ion o Deep Belie Ne wo ks o Na u al Language Unde s anding. IEEE/ACM T ans. Audio Speech Lang. P ocess. 2014,22, 778–784. [C ossRe ] 51. Sunde meye , M.; Schlü e , R.; Ney, H. LSTM neu al ne wo ks o language modeling. In P oceedings o he Thi een h Annual Con e ence o he In e na ional Speech Communica ion Associa ion, Po land, OR, USA, 9–13 Sep embe 2012. 52. Hu, Y.; Sun, X.; Nie, X.; Li, Y.; Liu, L. An Enhanced LSTM o T end Following o Time Se ies. IEEE Access 2019 ,7, 34020–34030. [C ossRe ] 53. Ge s, F.A.; Schmidhube , J.; Cummins, F. Lea ning o Fo ge : Con inual P edic ion wi h LSTM. In P oceedings o he 1999 Nin h In e na ional Con e ence on A i icial Neu al Ne wo ks ICANN 99 (Con . Publ. No. 470), Edinbu gh, UK, 7–10 Sep embe 1999; pp. 850–855. [C ossRe ] 54. G e , K.; S i as a a, R.K.; Kou ník, J.; S euneb ink, B.R.; Schmidhube , J. LSTM: A Sea ch Space Odyssey. IEEE T ans. Neu al Ne w. Lea n. Sys . 2017,28, 2222–2232. [C ossRe ] [PubMed] 55. Hoch ei e , S.; Schmidhube , J. Long Sho -Te m Memo y. Neu al Compu . 1997,9, 1735–1780. [C ossRe ] [PubMed] 56. Li, C.; Jiang, P.; Zhou, A. Rigo ous solu ion o slope s abili y unde seismic ac ion. Compu . Geo ech. 2019 ,109, 99–107. [C ossRe ] 57. Li, C.; Hu, B.C.; Hu, D.; Xu, X.F.; Zong, X.C.; Li, J.P.; Wu, M.C. S e eoselec i e ing-opening o s y ene oxide a ele a ed concen a ion by Phaseolus ulga is epoxide hyd olase, P EH2, in he o ganic/aqueous biphasic sys em. Ca al. Commun. 2019 , 123, 1–5. [C ossRe ] 58. Qin, Y.; Li, K.; Liang, Z.; Lee, B.; Zhang, F.; Gu, Y.; Zhang, L.; Wu, F.; Rod iguez, D. Hyb id o ecas ing model based on long sho e m memo y ne wo k and deep lea ning neu al ne wo k o wind signal. Appl. Ene gy 2019,236, 262–272. [C ossRe ] 59. Zhang, Z.; Ye, L.; Qin, H.; Liu, Y.; Wang, C.; Yu, X.; Yin, X.; Li, J. Wind speed p edic ion me hod using Sha ed Weigh Long Sho -Te m Memo y Ne wo k and Gaussian P ocess Reg ession. Appl. Ene gy 2019,247, 270–284. [C ossRe ] 60. Zhao, R.; Yin, Y.; Shi, Y.; Xue, Z. In elligen in usion de ec ion based on ede a ed lea ning aided long sho - e m memo y. Phys. Commun.2020,42, 101157. [C ossRe ] 61. Kong, X.; Kong, D.; Bai, L.; Xiao, J. Online p icing o demand esponse based on long sho - e m memo y and ein o cemen lea ning. Appl. Ene gy 2020,271, 114945. [C ossRe ] 62. Li, Y.; Lu, Y. LSTM-BA: DDoS De ec ion App oach Combining LSTM and Bayes. In P oceedings o he 2019 Se en h In e na ional Con e ence on Ad anced Cloud and Big Da a (CBD), Suzhou, China, 21–22 Sep embe 2019; pp. 180–185. [C ossRe ] 63. Alamee , Z.; Fa halla, A.; Li, K.; Ye, H.; Jianhua, Z. Mul is ep-ahead o ecas ing o coal p ices using a hyb id deep lea ning model. Resou . Policy 2020,65, 101588. [C ossRe ]