applied
sciences
A icle
An icipa ing Fu u e Beha io o an Indus ial P ess Using
LSTM Ne wo ks
Balduíno Césa Ma eus 1,2,* , Ma eus Mendes 3,4,* , José To es Fa inha 3,5 and An ónio Ma ques Ca doso 2


Ci a ion: Ma eus, B.C.; Mendes, M.;
Fa inha, J.T.; Ca doso, A.M.
An icipa ing Fu u e Beha io o an
Indus ial P ess Using LSTM
Ne wo ks. Appl. Sci. 2021,11, 6101.
h ps://doi.o g/10.3390/app11136101
Academic Edi o s: Ma lene Amo im,
Yu al Cohen and João Reis
Recei ed: 30 Ap il 2021
Accep ed: 25 June 2021
Published: 30 June 2021
Publishe ’s No e: MDPI s ays neu al
wi h ega d o ju isdic ional claims in
published maps and ins i u ional a il-
ia ions.
Copy igh : © 2021 by he au ho s.
Licensee MDPI, Basel, Swi ze land.
This a icle is an open access a icle
dis ibu ed unde he e ms and
condi ions o he C ea i e Commons
A ibu ion (CC BY) license (h ps://
c ea i ecommons.o g/licenses/by/
4.0/).
1EIGeS—Resea ch Cen e in Indus ial Enginee ing, Managemen and Sus ainabili y, Lusó ona Uni e si y,
Campo G ande, 376, 1749-024 Lisboa, Po ugal
2
CISE—Elec omecha onic Sys ems Resea ch Cen e, Uni e si y o Bei a In e io , Calçada Fon e do Lamei o,
P–62001-001 Co ilhã, Po ugal; [email p o ec ed]
3Poly echnic o Coimb a, ISEC, 3045-093 Coimb a, Po ugal; [email p o ec ed]
4Ins i u e o Sys ems and Robo ics, Uni e si y o Coimb a, 3004-531 Coimb a, Po ugal
5Cen e o Mechanical Enginee ing, Ma e ials and P ocesses—CEMMPRE, 3030-788 Coimb a, Po ugal
*Co espondence: [email p o ec ed] (B.C.M.); [email p o ec ed] (M.M.)
Abs ac :
P edic i e main enance is e y impo an in indus ial plan s o suppo decisions aiming
o maximize main enance in es men s and equipmen ’s a ailabili y. This pape p esen s p edic i e
models based on long sho - e m memo y neu al ne wo ks, applied o a da ase o senso eadings.
The aim is o o ecas u u e equipmen s a uses based on da a om an indus ial pape p ess. The
da ase s con ain da a om a h ee-yea pe iod. Da a a e p e-p ocessed and he neu al ne wo ks a e
op imized o minimize p edic ion e o s. The esul s show ha i is possible o p edic u u e beha io
up o one mon h in ad ance wi h easonable con idence. Based on hese esul s, i is possible o
an icipa e and op imize main enance decisions, as well as con inue esea ch o imp o e he eliabili y
o he model.
Keywo ds:
ime se ies p edic ion; LSTM p edic ion; deep lea ning p edic ion; p edic i e
main enance
1. In oduc ion
Mode n p ocesso s, compu e s and high speed ne wo ks make i possible o acqui e,
ans e and s o e la ge quan i ies o da a in eal ime. Acquisi ion and combina ion o
da a om di e en senso s makes i possible o gain an insigh ul iew o he s a e o
ac o ies, indus ial plan s and o he acili ies. La ge da ase s can be cons uc ed, s o ed
and p ocessed using in o ma ion echnologies such as Big Da a, cloud compu ing, cu ing-
edge compu ing, and a i icial in elligence ools. The In e ne o Things (IoT) is a ecen
concep , which p o ides many bene i s o di e en a eas, such as main enance and p oduc-
ion managemen , because i acili a es he au oma ion o asks such as moni o ing and
main enance. This esul s in he popula iza ion o in elligen sys ems, which a e highly
dependen on Big Da a [
1
] and a e an impo an a ea o s udy, since hey o e he ools
and me hods o acqui e and p ocess la ge olumes o da a such as his o ical p oduc ion
p ocesses, including many p oduc ion and ope a ing pa ame e s.
Mode n ime-se ies and o he da a analysis echniques ha e been used wi h success
o di e en asks, such as eeway a ic analysis [
2
] and addi i e manu ac u ing [
3
].
Di e en app oaches ha e also been p oposed in he ield o p edic i e main enance [
4
,
5
].
Sa is ac o y esul s we e ob ained using Big Da a eco ds as suppo o PCA models,
which esul ed in a wa ning ala m se e al days be o e a po en ial ailu e happened [6].
Li e cycle op imiza ion has been an impo an conce n o decades. A physical asse
wi h p ope main enance will ha e a longe use ul li e wi h a g ea e e u n on in es men
o he o ganiza ion [7].
P edic i e main enance equi es good quali y da a. The in o ma ion ha is ex ac ed
om he online o o line da a mus be eliable, and so he esul s mus be good enough
o jus i y he in es men in da a collec ion and analysis. The p ocess s a s om he
Appl. Sci. 2021,11, 6101. h ps://doi.o g/10.3390/app11136101 h ps://www.mdpi.com/jou nal/applsci
Appl. Sci. 2021,11, 6101 2 o 16
co ec calib a ion o he eading senso s and equipmen [
8
]. The da a a e hen s o ed and
p ocessed using di e en models, such as P incipal Componen Analysis (PCA) and Neu al
Ne wo ks [
9
]. Main enance planning in ol es he use o se e al algo i hms, he mos
common being ime se ies [10].
Main enance o equipmen in he indus y becomes a sensi i e and impo an poin
ha a ec s he equipmen ’s ope a ing ime and e iciency [
5
]. This makes main enance
one o he s a egic poin s o he de elopmen and g ow h o compe i i eness is-à- is
compe i o s. Chen and Tseng s udied he o al expec ed cos o main aining a lo a ion
sys em, including he cos o los p oduc ion, he cos o epai s, and he cos o s andby
machines [11].
Daniyan e al. p opose he in eg a ion o A i icial In elligence (AI) sys ems, which
will b ing many bene i s in diagnosing condi ion p oblems o indus ial machines [
12
].
They highligh he iabili y o AI ha combines he use o A i icial Neu al Ne wo ks
(ANNs) wi h a dynamic ime se ies model, o aul diagnos ics, o op imize he equipmen
in e en ion ime.
Hsu e al. demons a ed ha neu al ne wo ks can be a g ea echnology in he suppo
and decision making o la ge and small companies [
13
]. The e is a end o use hose ools
in p edic i e main enance sys ems wi h he aim o making he p edic ion sys ems mo e
in elligen [14].
Acco ding o Jimenez e al., he e is a g ea e o in he de elopmen o p edic i e
models o applica ion in p edic i e main enance [
15
]. Ay az and Alpay apply Long
Sho -Te m Memo y (LSTM) neu al ne wo k app oaches o p edic eal p oduc ion da a,
ob aining sa is ac o y esul s, supe io o con en ional models [
16
]. In hei s udy o
imp o e main enance planning o minimize unexpec ed s ops, hey apply a new me hod
ha consis s o he combined use o decomposi ion in empi ical mode o ensemble and
long- e m memo y. Thei esul s showed a pe o mance supe io o o he s a e o he
a models.
LSTM ne wo ks use se e al po s wi h di e en unc ions o con ol neu ons and o
s o e in o ma ion. The LSTM cell can e ain impo an in o ma ion o a longe pe iod in
which i is used. This p ope y o in o ma ion main enance allows he LSTM o exhibi a
good pe o mance in he classi ica ion, p ocessing, o o ecas ing o a complex dynamic
sequences [17].
The p esen wo k uses di e en LSTM models o p edic u u e ends o six a iables,
on a da ase con aining h ee yea s o da a samples g abbed in an indus ial p ess, which
aims o ope a e con inuously wi h minimum down ime. Di e en da a p e-p ocessing
echniques, ne wo k a chi ec u es and hype pa ame e s we e es ed in o de o de e mine
he models ha bes i he da a and p o ide he lowes p edic ion e o s.
Sec ion 2con ains a summa y o ela ed wo k. Sec ion 3desc ibes he heo y o he
LSTM ne wo ks. Sec ion 4desc ibes he me hods used o he p esen wo k.
Sec ion 5
de-
sc ibes he esul s and alida ion o he p edic i e model. Sec ion 7d aws some conclusions
and sugges ions o u u e wo k.
2. Rela ed Wo k
2.1. P edic i e Main enance
In sma indus ies, p edic i e main enance is one o he mos used echniques o
imp o e condi ion moni o ing, as i allows one o e alua e he condi ions o speci ic
equipmen in o de o p edic p oblems be o e ailu e [
18
]. Fo good pe o mance o
p edic i e models, i is impo an ha he senso da a collec ed a e o good quali y. Deep
neu al models ha e been used wi h success o imp o e p edic ion o condi ion moni o ing
o indus ial equipmen .
Wang e al. [
19
] use a model o long sho - e m ecu en neu al ne wo ks (LSTM-
RNN) wi h he objec i e o p edic i e main enance based on pas da a. The main objec i e
o p edic i e main enance is o make an accu a e es ima e o a sys em’s Remaining Use ul
Li e (RUL). T adi ional sys ems a e only able o wa n he use when i is oo la e and he
Appl. Sci. 2021,11, 6101 3 o 16
ailu e occu s, causing an unp edic able o line pe iod du ing which he sys em canno
ope a e p ope ly wi h a consequen was e o ime and esou ces [20].
In o de o assess he condi ion o a sys em, he p edic i e main enance app oach
employs senso s o di e en kinds. Some examples a e empe a u e, ib a ion, eloci y o
noise senso s, which a e a ached o he main componen s whose ailu e would comp omise
he en i e ope a ion o he sys em. In his sense, p edic i e main enance analyzes he his o y
o a sys em in e ms o he measu emen s collec ed by he senso s ha a e dis ibu ed
among he componen s, wi h he objec i e o ex ac ing a “ ailu e pa e n” ha can be
exploi ed o plan an op imal main enance s a egy and hus educing o line pe iods [
21
].
In a case ela ed o he s eel indus y, Re . [
22
] used neu al ne wo ks o classi ica ion o
main enance ac i i ies, so ha in e en ions a e planned acco ding o he ac ual s a us
o he machine and no in ad ance. Using mul iple neu al ne wo ks o iden i y s a us
and RUL a a highe esolu ion can be e y di icul , as he sys em can p edic ailu e
classi ica ions and may no be able o ecognize neighbo ing s a es. One limi a ion a ises
om he need o main enance eco ds o label da ase s and he need o la ge amoun s o
da a o adequa e quali y wi h main enance e en s, such as componen ailu es.
When sys ems s a o be e y complex o he numbe o senso measu emen s o
manage is e y la ge, i can be di icul o es ima e a ailu e. Fo his eason, in ecen yea s,
machine lea ning echniques a e used mo e and mo e o p edic wo king condi ions o a
componen . Ma hew e al. [
23
] p opose se e al app oaches o machine lea ning such as
suppo ec o machines (SVMs), decision ees (DTs), Random Fo es s (RFs), and o he s
ha show which echnique has he bes pe o mance in RUL o ecas o u bo an engines.
A majo challenge in ope a ions managemen is ela ed o p edic ing machine speed,
which can be used o dynamically adjus p oduc ion p ocesses based on di e en sys em
condi ions, op imize p oduc ion pe o mance and minimize ene gy consump ion [
24
].
Essien and Gianne i [
25
] use a deep con olu ional LSTM encode –decode a chi ec u e
model on eal da a, ob ained om a me al packaging ac o y. They show ha i is possible
o pe o m combina ions o LSTM wi h o he ne wo ks o signi ican ly imp o e he esul s.
2.2. P edic ion wi h LSTM Models
LSTM neu al ne wo ks achie ed he bes pe o mance in a numbe o compu a ional
sequence labeling asks, including speech ecogni ion and machine ansla ion [
26
]. The e
a e a a ie y o enginee ing p oblems ha can be sol ed using p edic i e neu al models.
Besh and Za zou a used neu al ne wo k models o p edic p oblems o suspended oad
b idge s uc u es based on global na iga ion sa elli e sys em obse a ions [
27
]. Sak e al.
demons a ed ha he p oposed LSTM a chi ec u es exhibi be e pe o mances compa ed
o deep neu al ne wo ks (DNNs) in a la ge ocabula y speech ecogni ion ask wi h a
la ge numbe o ou pu s a es [
28
]. Chen e al. adop ed LSTMs o p edic ing he ailu e
o hea y uck ai comp esso s [
29
]. They concluded ha he use o LSTMs leads o mo e
consis ency in p edic ions o e ime compa ed o models ha igno e his o y, such as
andom o es models.
Gosh e al. [
30
] p esen ed an ex ension ha hey called Con ex ual LSTM (CLSTM).
This model was also used o he o ecas ing o pollu an s. The e is also he p oposal o a
gene ic long sho - e m memo y (GLSTM), which has been used in he s udy o wind ene gy
o ecas ing [
31
]. Guo e al. p esen ed a combina ion me hod based on eal- ime p edic ion
e o s in which he suppo ec o eg ession (SVR) and LSTM ou pu s a e combined in
he inal esul s o he model’s p edic ion, hus ob aining esul s o g ea e p ecision [32].
Ren e al. used a combina ion o a Con olu ion Neu al Ne wo ks (CNNs) and LSTM
in o de o ex ac mo e in-dep h in o ma ion om da a o p edic he use ul li e o ion
ba e ies [
33
]. Niu e al. used an LSTM and de eloped an e ec i e speed p edic ion model
o sol e p edic ion p oblems o e ime [
34
]. Feng e al. epo ha he LSTM algo i hm
is supe io and, acco ding o hem, i pe o ms be e han con en ional neu al ne wo k
models [35].
Appl. Sci. 2021,11, 6101 4 o 16
The a chi ec u e o an LSTM ne wo k includes he numbe o hidden laye s and he
numbe o delay uni s, which is he numbe o p e ious da a poin s ha a e conside ed o
aining and es ing. Cu en ly, he e is no gene al ule o selec ing he numbe o delays
and hidden laye s [
36
]. A deep LSTM can be buil by s acking mul iple LSTM laye s, which
gene ally wo ks be e han a single laye . Deep LSTM ne wo ks ha e been applied o
sol e many eal-wo ld sequence modelling p oblems [
37
]. The LSTM can also be used o
planning s udies [38], namely o planning he analysis o oad a ic speed.
To p oduce a p edic ion model wi h good accu acy, i is necessa y o op imize neu al
models’ hype pa ame e s. While simple models can o en p oduce good esul s wi h
de aul hype pa eme e s, he op imiza ion p ocess can g ea ly imp o e he esul s [
39
–
41
].
The selec ion o hype pa ame e s o en makes he di e ence be ween unde pe o mance
and s a e-o - he-a pe o mance. Op imiza ion is o en pe o med using machine lea ning
algo i hms, such as g id sea ch, g ey wol op imiza ion o pa icle swa m op imiza ion. In
he p esen p edic ion model, howe e , he hype pa ame e s we e op imized manually,
ollowing a ial and e o guided p ocess, one a iable a a ime. This me hod was ollowed
because i was he mos con enien conside ing he limi ed compu ing powe a ailable.
2.3. LSTM wi h Encode and Decode
Expe imen s we e pe o med wi h a p edic i e model based on he LSTM wi h encode
and decode a chi ec u e. The model consis s o wo LSTMs, in which he i s LSTM has
he unc ion o p ocessing an inpu sequence and gene a ing an encoded s a e. The encoded
s a e comp esses he in o ma ion in he inpu s eam. The second LSTM, called a decode ,
uses he encoded s a e o p oduce an ou pu sequence. Those inpu and ou pu sequences
can be o di e en leng hs.
This echnique has al eady been used o sol e p oblems such as he p edic ion o
ehicle ajec o ies based on deep lea ning [
42
]. This a chi ec u e [
43
] has shown g ea
pe o mance o asks o ansla ing om sequence o sequence. LSTM encode –decode
models ha e also been p oposed o lea ning asks such as au oma ic ansla ion [
43
,
44
].
The e is he applica ion o his model o sol e many p ac ical p oblems, such as he s udy
o he equipmen condi ion, applica ions in language ansla ions, among o he s [45–47].
3. Theo e ical Backg ound
The p esen wo k uses LSTM ne wo ks, conside ing he e e ed di e en s udies
showing hei use ulness o ime se ies p edic ions [
48
,
49
]. The LSTM is a deep lea ning
ecu en neu al ne wo k a chi ec u e ha is a a ia ion o adi ional ecu en neu al
ne wo ks (RNNs). I was in oduced by Hoch ei e and Schmidhube in 1997. The mos
popula e sion is a modi ica ion e ined by many wo ks in he li e a u e [
50
,
51
], which is
called anilla LSTM (he eina e e e ed o as LSTM). The LSTM is excellen a handling
ime se ies da a only wi h i s ne wo k pa ame e s. Fo example, weigh s and pola iza ion
a e adjus ed o op imized [
52
]. The p ima y modi ica ion o he LSTM when compa ed
o he RNN a chi ec u e is he s uc u e o he hidden laye [
53
]. The LSTM model is a
powe ul ype o ecu en neu al ne wo k (RNN), capable o lea ning long- e m depen-
dencies [
54
]. They became popula due o hei powe o ep esen a ion and e ec i eness
in cap u ing long- e m dependencies [55].
Many ne wo ks showed ins abili y when dealing wi h exploding o anishing g adien
p oblems du ing lea ning. Those p oblems happen when he g adien o he e o is oo
la ge o oo small. I i is oo la ge, i o e lows and he e o s canno p opaga e p ope ly
h ough di e en laye s du ing lea ning. I i is oo small, i anishes and he ne wo k
does no lea n. Di e en me hods we e p oposed o sol e hose p oblems, known as a
kind o o “doo con ol” ha is used in RNN models. Fo example, Ga ed Recu en
uni (GRU) algo i hms [
56
,
57
], as he LSTMs [
58
,
59
], a e o a la ge ex en immune o he
g adien p oblems and lea n well.
The LSTM ne wo k s uc u e is based on h ee po s whose unc ion is o egula e
he low. Those po s a e called he en ance doo , he o ge ga e, and he exi doo . The
Appl. Sci. 2021,11, 6101 5 o 16
main po o en y is o egula e he en y o new memo y da a; he o ge ga e has he
unc ion o egula ing he s o age ime in he ne wo k memo y and he ou pu po in ends
o egula e how much he alue e ained in memo y in luences he ac i a ion o he ou pu
block [60].
Kong e al. demons a e some ele an conclusions such as (1) LSTM has a good
p edic i e capaci y; (2) hei use can signi ican ly imp o e he p o i o se ice p o ide s,
so he e is an oppo uni y when i comes o explo ing he o ecas in eal ime [
61
]. LSTM
ne wo ks a e he de ac o gold s anda d o deep lea ning algo i hms o analyzing ime
se ies da a [55].
Figu e 1shows he in e nal a chi ec u e o an LSTM uni cell. Acco ding o [
62
,
63
],
he in e nal calcula ion o mulae o he LSTM uni a e de ined as ollows:
i =σ(x Ui+h −1Wi+bi)(1)
=σ(x U +h −1W +b )(2)
o =σ(x Uo+h −1Wo+bo)(3)
a = an(x UC+h −1WC+bC)(4)
whe e
Ui
,
U
,
Uo
and
UC
a e he weigh ma ices o mapping he cu en inpu laye on
h ee po s and he s a e o he cu en inpu cell.
Figu e 1. De ailed layou o a long sho - e m memo y uni [63].
Wi
,
W
,
Wo
and
WC
a e he weigh ma ices o mapping he p e ious ou pu laye on
h ee po s and he cu en s a e o he inpu cell.
b
,
bi
,
bo
, and
bc
a e pola iza ion ec o s
o calcula ing he s a e o he doo and he inpu cell.
σ
is he ga e ac i a ion unc ion,
which is no mally a sigmoid unc ion.
an
is he hype bolic angen unc ion which is he
ac i a ion unc ion o he cu en s a e o he inpu cell.
Then, he cu en s a e o he ou pu cell and he ou pu laye can be calcula ed using
he ollowing equa ions.
C =σ( ×C −1+i ×a )(5)
h = anh(C )×o (6)
To assess he quali y o he p edic ion model, one o he mos popula me ics is he
Roo Mean Squa e E o (RMSE), which is gi en by Equa ion (7):
RMSE =s1
n
n
∑
=1
(Y −ˆ
Y)2(7)
whe e
Y
ep esen s he desi ed ( eal) alue and
ˆ
Y
is he p edic ed (ob ained om he
model) alue. The di e ence be ween
Y
and
ˆ
Y
is he e o be ween he alue expec ed

Appl. Sci. 2021,11, 6101 6 o 16
o ob ain and he alue ac ually ob ained om he ne wo k.
n
ep esen s he numbe o
samples used in he es se .
The RMSE, howe e , is an absolu e e o . The e o e, he e a e also he Mean Absolu e
Pe cen age E o (MAPE) and he Mean Absolu e E o (MAE). Those e o s a e gi en by
he ollowing o mulae:
MAE =1
n
n
∑
=1
|Y −ˆ
Y |(8)
MAPE =1
n
n
∑
=1
|Y −ˆ
Y |
|Y |(9)
whe e
Y
ep esen s he eal alue,
ˆ
Y
he p edic ed alue and
n
ep esen s he o al numbe
o samples.
4. Da a P epa a ion
Da a a e key o de eloping e icien modeling and planning. Howe e , o be aluable,
da a need o be p ocessed and s uc u ed be o e being analyzed.
4.1. The P oblem
The main goal o he p esen wo k is o p edic po en ial ailu es in an indus ial
d ying p ess be o e hey happen. Da a come om six senso s ins alled in he p ess. Those
senso s moni o he ope a ion o he p ess, wi h a sampling pe iod o one minu e. The
moni o ed a iables a e: (1) elec ic cu en in ensi y; (2) oil le el a he hyd aulic uni ;
(3) VAT p essu e; (4) o a ion speed; (5) empe a u e in he hyd aulic uni ; and (6) o que.
The da ase con ains six ime se ies, one o each senso , wi h he alues s o ed in he
da abase om 2016 o Augus 2020.
Figu e 2shows a plo o he six ime se ies, be o e any p ocessing is applied. These
da a p esen some uppe and lowe ex emes, which may be disc epan da a. Those
disc epan samples may be due o eading e o s o pe iods when he equipmen was o
o in ano he a ypical s a e.
Figu e 2.
Plo o he o iginal da ase alues. The a iables a e elec ic cu en in ensi y, hyd aulic
uni oil le el, VAT p essu e, mo o eloci y, empe a u e a he hyd aulic uni , and o que.
Some o he samples, such as hose when he equipmen was o bu he senso s
we e s ill eading, can comp omise he aining o he machine lea ning models o be
de eloped. Table 1shows some s a is ical pa ame e s such as mean, s anda d de ia ion
(s d), minimum, hi d quan iles, and maximum alue.
Appl. Sci. 2021,11, 6101 7 o 16
Table 1.
S a is ical pa ame e s o he da ase a iables, be o e p ocessing: C. in ensi y, hyd aulic uni
oil le el, o que, VAT p essu e, eloci y, and empe a u e.
C. In ensi y Hyd aulic To que VAT Veloci y Tempe a u e
mean 30.26 75.90 15.28 18.25 4.59 38.22
s d 1.36 4.54 0.69 2.67 0.98 1.62
min 26.34 62.93 13.59 9.67 1.27 33.19
Q1—25% 29.30 72.86 14.90 17.13 3.92 37.17
Q2—50% 30.46 75.53 15.43 18.72 4.57 38.33
Q3—75% 31.28 79.52 15.78 19.97 5.28 39.35
max 34.26 88.97 17.09 26.17 7.87 43.10
4.2. Cleaning Disc epan Da a
In o de o acili a e he aining p ocess, disc epan samples we e iden i ied and
emo ed using he quan iles me hod. Samples which a e beyond he
Q1−
3
×s d
o
Q3+
3
×s d
a e eplaced by he mean alue. The ex eme alues we e eplaced wi h he
a e age. Figu e 3shows he same a iables a e disc epan da a samples we e emo ed.
As he igu e shows, he lines a e now smoo he and easie o ead. Figu e 4shows
ha he samples a e e enly dis ibu ed a e he wi hd awal o disc epan da a.
Figu e 3.
Plo o he da ase alues a e cleaning disc epan da a. The a iables a e cu en in ensi y,
hyd aulic uni oil le el, VAT p essu e, eloci y, empe a u e, and o que.
Figu e 4.
Dis ibu ion o da a poin s o all he senso s, wi h lowly and highly disc epan da a cleaned.
The p edic i e models o be used a e obus and ole an o noise. Howe e , he cleane
da a a e expec ed o show be e esul s. As an example, a p o isional expe imen o ain
Appl. Sci. 2021,11, 6101 8 o 16
a neu al ne wo k LSTM model wi h a his o ical window o 70 samples and 40 LSTM uni
cells showed highe and unde e mined e o s. The model was no able o lea n o p edic
some a iables, as shown in Table 2. Wi h clean da a, he e we e be e and de e minable
esul s, as shown in Table 3. The ables show he MAPE and MAE o all inpu a iables,
as de e mined in he es se . They also show he RMSE, as calcula ed in he ain and es
se s, globally o all a iables.
Table 2.
P edic ion esul s wi hou cleaning disc epan da a in he da abase, wi h a window o
70 samples and 40 LSTM uni s.
Window 70 Days
C. In ensi y Hyd aulic To que VAT Veloci y Tempe a u e
MAPE in 8.46 in 98.19 in 11.59
MAE 3.52 6.57 24.73 10.53 14.88 4.21
T ain Tes
RMSE 79.52 79.64
Table 3. Fo ecas esul s wi h ea men s in he da abase wi h 40 LSTM uni s.
Window 70 Days
C. In ensi y Hyd aulic To que VAT Veloci y Tempe a u e
MAPE 2.52 3.02 2.44 13.10 in 2.48
MAE 0.76 2.28 0.37 1.32 0.57 0.94
T ain Tes
RMSE 1.71 1.97
5. Expe imen s and Resul s
Expe imen s we e pe o med wi h he aim o alida ing he model ha has he bes
pe o mance in p edic ing da a om he indus ial p ess. The es s a e di ided in o
wo subsec ions, i s wi h esampling o da a o one sample pe day and hen wi h
esampling o a sample each 12 h.
5.1. LSTM Models and Da ase Pa i ion
A e p ocessing he da a, expe imen s we e pe o med wi h an LSTM model. The
model included an encode and decode , wi h one hidden LSTM laye in he middle and a
dense laye a he ou pu . The model was used o ain and p edic , wi h six a iables ha
ep esen da a coming om he pape p ess senso s. The goal was o o ecas he alue o
hose a iables wi h he highes possible le el o con idence so ha i b ings added bene i s
in p edic i e main enance.
Figu e 5desc ibes he a chi ec u e o one o he ne wo k models used. The models
we e implemen ed in Py hon using he Tenso Flow lib a y and Ke as.
The expe imen s we e pe o med aiming o ob ain a p edic ion o all a iables one
mon h in ad ance, om a window o a numbe o pas samples.
The LSTM models ecei ed, as an inpu , a sequence consis ing o he composi ion o a
numbe o samples o each a iable. The numbe o samples depended on he window size
and he esampling a e used. The ou pu sequence is composed o he alues p edic ed
o each o he a iables.
Appl. Sci. 2021,11, 6101 9 o 16
Figu e 5.
Model summa y o one o he LSTM ne wo ks used. The model ecei es a window o n
samples o each a iable and p edic s he alue o hose a iables as p edic ed 30 days ahead.
To ain and es he models, he da ase was di ided in o ain and es subse s.
Valida ion was pe o med using he es se , bu hose samples we e no inco po a ed
in o he aining se . The aining se con ained 85% o he samples and he es se he
emaining 15% o samples. These alues a e adequa e o con e gence du ing lea ning.
As an example, Figu e 6shows a lea ning cu e o a model wi h 70 uni s in he middle
laye and a window o 30 lag samples. The igu e shows ha lea ning con e ges and akes
ewe han 10 epochs. The emainde expe imen s we e pe o med using 100 epochs.
Figu e 6.
Example o lea ning cu e, showing he loss measu ed du ing aining o an LSTM model.
5.2. Expe imen s o De e mine His o ical Window Size and Numbe o LSTM Uni s Using One
Sample pe Day
The i s expe imen s pe o med aimed o de e mine he bes window size o use. The
smalle he window, he smalle and as e he model ha can be used. Howe e , i he
window is oo small, i may be insu icien o make accu a e p edic ions.
The o iginal da ase had 1,445,760 da a poin s, which is e y la ge and would equi e
a lo o memo y and ime o ain and es . The expe imen s we e pe o med a e down-
sampling he da a, so ha he e is only one sample pe day. Tha sample is he a e age o
1004 o iginal samples. The downsampled da ase is, he e o e, less han he one housand
o he o iginal da ase .
The esul s a e measu ed in he es se . The igu e abo e shows he MAPE and MAE
measu ed o each a iable. I also shows he global RMSE measu ed globally o he ain
and es se s.
As Figu e 7shows, models wi h windows o 40 and 50 samples allow be e lea ning
and p oduce smalle p edic ion e o s.
Appl. Sci. 2021,11, 6101 16 o 16
42.
Pa k, S.H.; Kim, B.; Kang, C.M.; Chung, C.C.; Choi, J.W. Sequence- o-Sequence P edic ion o Vehicle T ajec o y ia LSTM
Encode -Decode A chi ec u e. In P oceedings o he 2018 IEEE In elligen Vehicles Symposium (IV), Changshu, China, 26–30
June 2018; pp. 1672–1678. [C ossRe ]
43.
Cho, K.; an Me ienboe , B.; Gülçeh e, Ç.; Bouga es, F.; Schwenk, H.; Bengio, Y. Lea ning Ph ase Rep esen a ions using RNN
Encode -Decode o S a is ical Machine T ansla ion. a Xi 2014, a Xi :1406.1078.
44. Su ske e , I.; Vinyals, O.; Le, Q.V. Sequence o Sequence Lea ning wi h Neu al Ne wo ks. a Xi 2014, a Xi :1409.3215.
45.
Wang, T.; Chen, P.; Ama al, K.; Qiang, J. An Expe imen al S udy o LSTM Encode -Decode Model o Tex Simpli ica ion. a Xi
2016, a Xi :1609.03663.
46.
Bengio, S.; Vinyals, O.; Jai ly, N.; Shazee , N. Scheduled Sampling o Sequence P edic ion wi h Recu en Neu al Ne wo ks.
a Xi 2015, a Xi :1506.03099.
47.
Du, S.; Li, T.; Yang, Y.; Ho ng, S.J. Mul i a ia e ime se ies o ecas ing ia a en ion-based encode –decode amewo k.
Neu ocompu ing 2020,388, 269–279. [C ossRe ]
48.
Ge s, F.A.; Eck, D.; Schmidhube , J. Applying LSTM o Time Se ies P edic able Th ough Time-Window App oaches. In Neu al Ne s
WIRN Vie i-01; Taglia e i, R., Ma ina o, M., Eds.; Pe spec i es in Neu al Compu ing; Sp inge : Be lin/Heidelbe g, Ge many,
2002; pp. 193–200._20. [C ossRe ]
49.
Meng, Q.; Wang, H.; He, M.; Gu, J.; Qi, J.; Yang, L. Displacemen p edic ion o wa e -induced landslides using a ecu en deep
lea ning model. Eu . J. En i on. Ci . Eng. 2020. [C ossRe ]
50.
Sa ikaya, R.; Hin on, G.E.; Deo as, A. Applica ion o Deep Belie Ne wo ks o Na u al Language Unde s anding. IEEE/ACM
T ans. Audio Speech Lang. P ocess. 2014,22, 778–784. [C ossRe ]
51.
Sunde meye , M.; Schlü e , R.; Ney, H. LSTM neu al ne wo ks o language modeling. In P oceedings o he Thi een h Annual
Con e ence o he In e na ional Speech Communica ion Associa ion, Po land, OR, USA, 9–13 Sep embe 2012.
52.
Hu, Y.; Sun, X.; Nie, X.; Li, Y.; Liu, L. An Enhanced LSTM o T end Following o Time Se ies. IEEE Access
2019
,7, 34020–34030.
[C ossRe ]
53.
Ge s, F.A.; Schmidhube , J.; Cummins, F. Lea ning o Fo ge : Con inual P edic ion wi h LSTM. In P oceedings o he 1999 Nin h
In e na ional Con e ence on A i icial Neu al Ne wo ks ICANN 99 (Con . Publ. No. 470), Edinbu gh, UK, 7–10 Sep embe 1999;
pp. 850–855. [C ossRe ]
54.
G e , K.; S i as a a, R.K.; Kou ník, J.; S euneb ink, B.R.; Schmidhube , J. LSTM: A Sea ch Space Odyssey. IEEE T ans. Neu al
Ne w. Lea n. Sys . 2017,28, 2222–2232. [C ossRe ] [PubMed]
55. Hoch ei e , S.; Schmidhube , J. Long Sho -Te m Memo y. Neu al Compu . 1997,9, 1735–1780. [C ossRe ] [PubMed]
56.
Li, C.; Jiang, P.; Zhou, A. Rigo ous solu ion o slope s abili y unde seismic ac ion. Compu . Geo ech.
2019
,109, 99–107. [C ossRe ]
57.
Li, C.; Hu, B.C.; Hu, D.; Xu, X.F.; Zong, X.C.; Li, J.P.; Wu, M.C. S e eoselec i e ing-opening o s y ene oxide a ele a ed
concen a ion by Phaseolus ulga is epoxide hyd olase, P EH2, in he o ganic/aqueous biphasic sys em. Ca al. Commun.
2019
,
123, 1–5. [C ossRe ]
58.
Qin, Y.; Li, K.; Liang, Z.; Lee, B.; Zhang, F.; Gu, Y.; Zhang, L.; Wu, F.; Rod iguez, D. Hyb id o ecas ing model based on long sho
e m memo y ne wo k and deep lea ning neu al ne wo k o wind signal. Appl. Ene gy 2019,236, 262–272. [C ossRe ]
59.
Zhang, Z.; Ye, L.; Qin, H.; Liu, Y.; Wang, C.; Yu, X.; Yin, X.; Li, J. Wind speed p edic ion me hod using Sha ed Weigh Long
Sho -Te m Memo y Ne wo k and Gaussian P ocess Reg ession. Appl. Ene gy 2019,247, 270–284. [C ossRe ]
60.
Zhao, R.; Yin, Y.; Shi, Y.; Xue, Z. In elligen in usion de ec ion based on ede a ed lea ning aided long sho - e m memo y. Phys.
Commun.2020,42, 101157. [C ossRe ]
61.
Kong, X.; Kong, D.; Bai, L.; Xiao, J. Online p icing o demand esponse based on long sho - e m memo y and ein o cemen
lea ning. Appl. Ene gy 2020,271, 114945. [C ossRe ]
62.
Li, Y.; Lu, Y. LSTM-BA: DDoS De ec ion App oach Combining LSTM and Bayes. In P oceedings o he 2019 Se en h In e na ional
Con e ence on Ad anced Cloud and Big Da a (CBD), Suzhou, China, 21–22 Sep embe 2019; pp. 180–185. [C ossRe ]
63.
Alamee , Z.; Fa halla, A.; Li, K.; Ye, H.; Jianhua, Z. Mul is ep-ahead o ecas ing o coal p ices using a hyb id deep lea ning model.
Resou . Policy 2020,65, 101588. [C ossRe ]