Academic Edi o s: Pio A. Kowalski
and Ra al Sche e
Recei ed: 29 No embe 2024
Re ised: 27 Janua y 2025
Accep ed: 6 Feb ua y 2025
Published: 7 Feb ua y 2025
Ci a ion: Solís-Ma ín, D.;
Galán-Páez, J.; Bo ego-Díaz, J. A
Model o Lea ning-Cu e Es ima ion
in E icien Neu al A chi ec u e
Sea ch and I s Applica ion in
P edic i e Heal h Main enance.
Ma hema ics 2025,13, 555. h ps://
doi.o g/10.3390/ma h13040555
Copy igh : © 2025 by he au ho s.
Licensee MDPI, Basel, Swi ze land.
This a icle is an open access a icle
dis ibu ed unde he e ms and
condi ions o he C ea i e Commons
A ibu ion (CC BY) license
(h ps://c ea i ecommons.o g/
licenses/by/4.0/).
A icle
A Model o Lea ning-Cu e Es ima ion in E icien Neu al
A chi ec u e Sea ch and I s Applica ion in P edic i e
Heal h Main enance
Da id Solís-Ma ín †, Juan Galán-Páez †and Joaquín Bo ego-Díaz *
Depa men o Compu e Science and A i icial In elligence, Uni e sidad de Se illa, 41012 Se illa, Spain;
[email p o ec ed] (D.S.-M.); [email p o ec ed] (J.G.-P.)
*Co espondence: jbo [email p o ec ed]
†These au ho s con ibu ed equally o his wo k.
Abs ac : A pe sis en challenge in machine lea ning is he compu a ional ine iciency o
neu al a chi ec u e sea ch (NAS), pa icula ly in esou ce-cons ained domains like p e-
dic i e main enance. This wo k in oduces a no el lea ning-cu e es ima ion amewo k
ha educes NAS compu a ional cos s by o e 50% while main aining model pe o mance,
add essing a c i ical bo leneck in au oma ed machine lea ning design. By de eloping a
da a-d i en es ima o ained on 62 di e en p edic i e main enance da ase s, we demon-
s a e a gene alized app oach o ea ly-s opping ials du ing neu al ne wo k op imiza ion.
Ou me hodology no only educes compu a ional esou ces bu also p o ides a ans-
e able echnique o e icien neu al ne wo k a chi ec u e explo a ion ac oss complex
indus ial moni o ing asks. The p oposed app oach achie es a ema kable balance be-
ween compu a ional e iciency and model pe o mance, wi h only a 2% pe o mance
deg ada ion, showcasing a signi ican ad ancemen in au oma ed neu al a chi ec u e
op imiza ion s a egies.
Keywo ds: lea ning cu es; neu al a chi ec u e sea ch; p edic i e main enance; Bayesian
op imiza ion
MSC: 68U99
1. In oduc ion
In ecen yea s, deep lea ning (DL) has signi ican ly impac ed a ious indus ial sec-
o s, achie ing ema kable success in ields such as image p ocessing and na u al language
(o audio) p ocessing. Mo eo e , in ela i ely less-explo ed indus ial domains like p og-
nos ics and heal h managemen (PHM), DL has demons a ed p omising esul s. PHM
e e s o he p ocess o moni o ing, analyzing, and p edic ing he heal h and pe o mance o
equipmen and machine y. PHM sys ems aim o de ec po en ial ailu es be o e hey occu
by assessing he condi ion o asse s h ough a ious senso s and da a analysis echniques.
This helps op imize main enance schedules, educe down ime, and ex end equipmen
li espan. The ul ima e goal o PHM is o enhance he eliabili y and e iciency o indus ial
ope a ions by p edic ing ailu es, imp o ing sa e y, and minimizing cos s associa ed wi h
unplanned main enance and epai s. Addi ionally, syn hesizing explana ions o machine y
expe s [
1
] and aiding decision-making p ocesses o domain expe s [
2
] a e equally c i ical.
The economic signi icance o p edic i e heal h moni o ing (PHM) has been inc eas-
ingly ecognized ac oss indus ial sec o s. Recen s udies highligh subs an ial inancial
Ma hema ics 2025,13, 555 h ps://doi.o g/10.3390/ma h13040555
Ma hema ics 2025,13, 555 2 o 36
bene i s, showing ha o ganiza ions implemen ing PHM echnologies expe ience signi i-
can educ ions in main enance cos s and ope a ional down ime [3].
1.1. P ognos ics and Heal h Managemen
P edic i e heal h moni o ing (PHM) sys ems ep esen an ad anced in eg a ion o
sensing echnologies, da a analy ics, and machine lea ning. The diag am in Figu e 1
illus a es he wo k low o a PHM sys em, speci ically o p edic i e main enance in an
indus ial se ing.
Figu e 1. P edic i e main enance ecosys em.
The p ocess begins wi h indus ial machine y as he p ima y sou ce o ope a ional da a,
equipped wi h senso s ha con inuously moni o pe o mance and heal h me ics. Raw
senso da a—such as empe a u e, p essu e, and ib a ion le els—a e logged and eco ded.
A p ocess lowcha ou lines he machine y’s ole wi hin indus ial ope a ions, p o iding
insigh s in o in e ac ions and po en ial ailu e poin s. All collec ed da a, including senso
eadings and p ocess low in o ma ion, a e s o ed in a cen alized eposi o y o ensu e
e icien agg ega ion and a ailabili y o analysis.
The h ee uppe -le el componen s ep esen indus ial suppo sys ems. The ERP
(en e p ise esou ce planning) sys em manages co e business p ocesses, op imizing in-
en o y, o de s, and main enance using s o ed da a. Quali y enginee ing applies quali y
managemen p inciples, analyzing da a o ensu e machine y mee s s anda ds and imple-
men ing imp o emen s. The manu ac u ing execu ion sys em (MES) moni o s and con ols
p oduc ion in eal ime, adjus ing p ocesses o enhance e iciency and minimize down ime.
The AI co e is he ML models which a e applied o he s o ed da a o p edic u u e
machine y ailu es be o e hey occu . These models analyze pa e ns and anomalies in
he da a o iden i y po en ial issues ha migh lead o equipmen ailu e. The esul s
om he ML models a e isualized in a use - iendly dashboa d, p o iding eal- ime
insigh s in o he heal h and pe o mance o he machine y, highligh ing a eas ha need
a en ion and p edic ing u u e main enance needs. I includes a ious me ics and KPIs
(key pe o mance indica o s) ha help in decision making.
Da a-d i en me hods ha e gained popula i y o hei abili y o le e age as amoun s
o da a om mode n senso s and sys ems (e.g., he In e ne o Things). These me hods
p ocess his o ical da a o iden i y pa e ns and de ec deg ada ion ends. Va ious ap-
p oaches ha e been explo ed, including neu al ne wo ks [
4
], deep lea ning [
5
], XAI [
1
],
and ensemble me hods based on decision ees [
6
,
7
]. Suppo ec o machines (SVMs)
Ma hema ics 2025,13, 555 3 o 36
ha e also been applied o RUL in s udies such as [
8
,
9
]. To add ess unce ain y, Bayesian
ne wo ks [
10
] and uzzy logic-based sys ems [
11
] ha e been p oposed, enhancing he
obus ness o RUL p edic ions.
1.2. Neu al A chi ec u e Sea ch
The design o neu al a chi ec u es is a c i ical ac o in ex ac ing ask- ele an ea u es,
signi ican ly impac ing he pe o mance o he esul ing model. O e he cou se o neu al
ne wo k esea ch, a ious high-le el a chi ec u es ha e been in oduced, including eed o -
wa d ne wo ks (FFNs), con olu ional neu al ne wo ks (CNNs), ecu en neu al ne wo ks
(RNNs), and T ans o me s, among o he s. Ne e heless, iden i ying he op imal hype -
pa ame e con igu a ion—such as he numbe and ypes o laye s, ac i a ion unc ions,
numbe o neu ons, lea ning a e, and o he ela ed pa ame e s— emains a challenging
ask ha equi es subs an ial expe ise and i e a i e ine- uning.
Neu al a chi ec u e sea ch (NAS) is an eme ging app oach in he au oma ed design o
neu al ne wo ks, aiming o sys ema ically iden i y he op imal ne wo k a chi ec u e o a
speci ic ask. This pa adigm explo es a ious ne wo k s uc u es o de e mine which one
deli e s he bes pe o mance, op imizing bo h accu acy and esou ce e iciency.
Th ee key componen s de ine NAS. Fi s ly, a i s co e is he a chi ec u e sea ch space,
which encompasses all possible ne wo k con igu a ions ha can be e alua ed o a gi en
p oblem. Secondly, he e alua ion p ocess plays a c i ical ole in de e mining which a chi-
ec u e bes mee s he equi emen s, such as p edic ion accu acy o compu a ional e iciency.
Las ly, o ind he op imal a chi ec u e, NAS employs op imiza ion me hods ha guide he
sea ch, using echniques such as e olu iona y algo i hms o ein o cemen lea ning. In his
way, NAS enables he au oma ion and subs an ial imp o emen o neu al ne wo k design,
educing he need o manual in e en ion in selec ing pa ame e s and s uc u es.
In machine lea ning (ML) p ac ices, NAS is ypically s uc u ed wi h an ou e loop ha
sea ches o op imal ne wo k hype pa ame e s and an inne loop ha op imizes he model
pa ame e s (i.e., he ne wo k weigh s) using hose hype pa ame e s. This wo k ocuses
on implemen ing ea ly-s opping mechanisms wi hin he inne loop o imp o e e iciency
in NAS.
NAS ypically equi es aining and e alua ing a la ge numbe o candida e mod-
els—o en in he hund eds— h oughou he sea ch p ocess. This makes NAS inhe en ly
esou ce-in ensi e and ime-consuming. Fo ins ance, in [
12
], Zoph e al. in oduce NAS-
Ne , which equi es 2000 GPU days o pe o m a sea ch using ein o cemen lea ning, while
AmoebaNe -A, p esen ed by Real e al. in [
13
], demands 3150 GPU days using e olu iona y
algo i hms. Consequen ly, de eloping s a egies o educe he compu a ional esou ces
and ime equi ed o NAS is o pa amoun impo ance, enabling b oade accessibili y and
g ea e e iciency in deploying NAS-based me hodologies.
NAS aces se e al compu a ional and me hodological challenges. One key challenge
is combina o ial complexi y, as he a chi ec u e sea ch space g ows exponen ially wi h he
dep h and wid h o he ne wo k. This makes inding he op imal a chi ec u e inc easingly
di icul as he design space expands. Ano he signi ican hu dle is compu a ional cos ,
as e alua ing mul iple a chi ec u es equi es subs an ial esou ces in bo h ime and ha d-
wa e. Addi ionally, ans e abili y poses a challenge, as op imal a chi ec u es may a y
ac oss di e en domains, meaning ha a ne wo k a chi ec u e e ec i e o one ask may no
pe o m as well in ano he . Finally, in e p e abili y emains a majo issue, as unde s anding
why one a chi ec u e ou pe o ms ano he is o en no s aigh o wa d, making i di icul
o gain insigh s in o he ac o s d i ing pe o mance di e ences.
Ma hema ics 2025,13, 555 4 o 36
NAS o PHM
While he gene al p inciples o NAS a e uni e sal, hei applica ion in speci ic domains
like p ognos ics and heal h managemen (PHM) equi es impo an adap a ions.
In e p e abili y cons ain s, eliabili y equi emen s, handling o limi ed o noisy da a,
and he need o explainabili y all come in o play. The ansi ion om gene ic NAS o
NAS specialized o PHM in ol es add essing hese unique domain-speci ic cons ain s.
In PHM, ensu ing ha he sys em can explain i s p edic ions and ope a e eliably unde
unce ain condi ions is pa icula ly impo an . Addi ionally, he limi ed a ailabili y o
clean da a in many PHM scena ios adds complexi y o he design p ocess, necessi a ing
me hods ha pe o m well e en wi h noisy o incomple e in o ma ion [14].
While NAS p o ides a amewo k o au oma ing model design, accu a ely es ima ing
he po en ial pe o mance o a model wi h minimal compu a ional esou ces emains a
signi ican challenge. This issue is especially c i ical in domains like PHM, whe e bo h
compu a ional e iciency and p edic i e accu acy a e c ucial. Ou app oach aims o de elop
a lea ning-cu e es ima ion echnique ha p o ides ea ly, eliable insigh s in o he po en-
ial pe o mance o a model, he eby minimizing he compu a ional o e head ypically
associa ed wi h adi ional NAS me hods [15].
1.3. On Es ima ing Lea ning Cu es om Ini ial Da a
In he s udy o neu al ne wo ks (and ML in gene al), analyzing he ma hema ical
p ope ies o lea ning cu es is o signi ican impo an . These cu es, which ack he
e olu ion o e o o loss du ing aining, p o ide aluable insigh s in o model con e gence
and o e all beha io .
Key p ope ies o lea ning cu es ha o e c i ical insigh s in o he beha io o DL
models and guide mo e e ec i e aining s a egies include [16] he ollowing:
•
Con e gence a e: The speed a which e o dec eases du ing aining i e a ions,
e lec ing he op imiza ion e iciency o he p ocess.
•
Smoo hness: The deg ee o egula i y o s abili y in lea ning cu es. Smoo he cu es
o en indica e mo e s able and p edic able aining p ocesses.
•
Con exi y: Whe he lea ning cu es exhibi con ex beha io , which simpli ies analysis
and op imiza ion.
No ably, hese ac o s acili a e he s udy o he ini ial cu a u e o lea ning cu es,
which may con ain p edic i e in o ma ion abou a model’s inal pe o mance [
17
]. A s eepe
ini ial cu a u e, ep esen ing a apid dec ease in e o , could indica e be e inal pe o -
mance. In es iga ing he ela ionship be ween ini ial lea ning cu e cu a u e and inal
pe o mance ac oss a ious neu al ne wo k a chi ec u es and da ase s is he e o e a p omis-
ing a ea o esea ch.
This subsec ion does no aim o comp ehensi ely add ess he ounda ional p ob-
lem [
16
,
17
]. Ins ead, i ocuses on le e aging ini ial beha io (ea ly s opping) o p edic
u u e pe o mance and iden i y he bes model. Typical me hods o i ing lea ning cu es
in ol e c oss- alida ion o di ec pa ame e uning o ex apola e pe o mance o unseen
da ase sizes. Pa ame ic app oaches, such as modeling loss unc ions wi h exponen ial o
powe -law beha io s, a e pa icula ly use ul in ML se ings whe e da a complexi y a ies
widely [
16
]. These app oaches o m he basis o he p ac ical es ima ion s a egies explo ed
he e. We b ie ly discuss o he app oaches, along wi h hei ad an ages and disad an ages,
which jus i y u he explo a ion o al e na i e solu ions.
1.3.1. Ex apola ion o Lea ning Cu es
In he ML li e a u e, he e m “lea ning cu e” is unde s ood in wo di e en ways.
The i s , mo e gene al, meaning e e s o he lea ning cu e as a unc ion o he size o he
Ma hema ics 2025,13, 555 5 o 36
aining se . These ypes o lea ning cu es ha e been s udied o ex apola e pe o mance
om smalle o la ge da ase s and a e no he ocus o his pape . The second, mo e
commonly used, meaning e e s o he lea ning cu e as a unc ion o he numbe o
aining i e a ions (neu al ne wo k epochs in his wo k).
In his pape , we conside he lea ning cu e
y( ) = L(x
,
θ )
, whe e
L
ep esen s he
loss unc ion o he model applied o he da ase xwi h pa ame e s θ a epoch .
An in ui i e app oach o p edic ing lea ning cu es is based on he me hod p oposed
by Domhan e al. [
18
], which conside s a amily o pa ame ic unc ions o ex apola e he
lea ning cu e om i s ini ial obse a ions.
In his amewo k, he cu es a e modeled as a weigh ed linea combina ion o
k
basis
unc ions
{ϕi(
,
θi)}1≤i≤n
, each dependen on ime
and pa ame e ec o s
θi
. The cen al
assump ion is ha he cu e being es ima ed can be modeled as
ˆ
y( |Θ,w) =
k
∑
i=1
wiϕi( ,θi),
whe e
Θ
ep esen s he se o all pa ame e s
θi
, and
w
deno es he weigh ec o associa ed
wi h he pa ame e s used in he basis unc ions. The p edic ion p ocess also assumes
obse a ional noise a ound he unknown ue alue ( ), modeled as
y( )∼ N(ˆ
( |Θ,w),σ2),
whe e a p io is de ined o he pa ame e s. Using a g adien - ee Ma ko chain Mon e
Ca lo (MCMC) me hod, samples om he pos e io dis ibu ion a e ob ained o p edic
u u e alues o he lea ning cu e.
The app oach is lexible due o he inclusion o a bi a y pa ame ic unc ions, allow-
ing i o adap o a ious neu al ne wo k a chi ec u es and hype pa ame e con igu a ions.
Howe e , a key limi a ion is ha he model does no le e age p e iously e alua ed hy-
pe pa ame e con igu a ions, equi ing obse a ion o a signi ican po ion o he lea ning
cu e be o e i s p edic ions become eliable.
A c ucial aspec o his me hodology is es ima ing he cu a u e o he lea ning cu e
o p edic i s u u e beha io , such as de ec ing con exi y. Using he cen al limi heo em,
he dis ibu ion o sample means app oaches no mali y o su icien ly la ge sample sizes.
The e o e, o es ima e he cu a u e,
S
independen expe imen s a e pe o med, gene a ing
p edic ions o la ge enough epochs
m
o e eal eliable ends. This p o ides a solid
ounda ion o e alua ing he p og ession o he lea ning cu e. Fo ins ance, based on
Donham’s me hod, i ˆ
y(m)is he p edic ed lea ning cu e
ˆ
y(m) = Ehy(m)|{y( )}N
=1i≈1
S
S
∑
s=1
ˆ
s(m|Θs,ws),
hen he cu a u e o ˆ
yis
d2ˆ
ym
dm2≃1
S
S
∑
s=1
∂2ˆ
∂m2(m|θs,ws) = 1
S
S
∑
s=1
d2
dm2 k
∑
j=1
wsjϕj(m,θsj)!=1
S
S
∑
s=1
k
∑
j=1
wsj
d2ϕj(m,θsj)
dm2.
which can p o ide aluable insigh s in o he e olu ion o he lea ning cu e. Thus, he cu -
a u e depends on he alues o
θsj
,
wsj
. I he cu a u e is posi i e, he cu e is con ex,
which ypically indica es ha he model is imp o ing i s pe o mance a an inc easing a e.
Con e sely, a nega i e cu a u e (conca e) sugges s ha he imp o emen a e is slowing
down, o en signaling ha he model pe o mance is s abilizing.
Ma hema ics 2025,13, 555 6 o 36
1.3.2. The Powe -Law Hypo hesis
Be o e in oducing a new me hod o NAS and i s applica ion in he con ex o PHM, i
is c ucial o e alua e whe he a pa ame ic app oach is app op ia e. In pa icula , by using
powe -law unc ions.
The answe is no clea -cu . In some cases, pa ame e iza ions o powe -law unc ions
can be e ec i e, while in o he s, hey may no be applicable. The ques ion o whe he
lea ning cu es in PHM can be eliably modeled wi h powe laws o equi e al e na i e
unc ional o ms is undamen al gi en he complexi y o PHM en i onmen s. Le us b ie ly
examine his issue.
Taking he cu e as in ou case (as a unc ion o epochs), se e al models exis ha use
a pa ame ic powe law o model he lea ning cu e. Fo ins ance, Kad a e al. [
19
] model
lea ning cu es using a neu al ne wo k ensemble, wi h he ou pu being condi ioned o
ollow a powe law, and [
20
] Tissue e al. model he lea ning cu e using a powe law bu
as a unc ion o epochs and lea ning a e annealing.
S a ing om he hypo hesis ha he lea ning cu e ollows a powe law (o a pa a-
me ic sum o such unc ions),
y(m)≃a+cm−(1+α),m≥m0
he p oblem can be na owed o es ima ing he pa ame e s
(a
,
c
,
α)
. These can be e ec i ely
in e ed h ough me hods like he Domhan app oach, as p e iously desc ibed, o ia
specialized echniques o powe -law de ec ion (c . [
21
]). In [
21
], Clause e al. in oduce
a s a is ical me hod o es ima e powe -law pa ame e s and conduc goodness-o - i es s
using obus echniques, including he Kolmogo o –Smi no es , o compa e powe -law
i s wi h al e na i e dis ibu ions.
The u ili y o powe -law models in PHM has been ecognized o hei gene aliza ion
capabili ies ( o ins ance, in [
22
] Hes ness e al. show he powe -law ela ion be ween DL
models’ gene aliza ion e o and ac o s such as he amoun o aining da a, model size,
and compu e esou ces), hough al e na i e unc ional o ms a e o en needed o cap u e
in ica e lea ning dynamics [
23
]. PHM scena ios equen ly in ol e di e se equipmen and
ope a ional condi ions, leading o lea ning cu es ha de ia e om powe -law beha io .
Fo example, exponen ial unc ions e ec i ely model apid ini ial e o educ ion in sys ems
wi h s able signal- o-noise a ios [
18
]. Complex PHM sys ems may also exhibi phase
ansi ions, equi ing hyb id o piecewise models o e pu ely pa ame ic app oaches.
Selec ing unc ional o ms should depend on empi ical alida ion, model e alua ion,
and domain expe ise.
1.3.3. Modeling Lea ning Cu es by S ochas ic P ocesses
An al e na i e app oach o modeling lea ning cu es is o ha ness hei s ochas ic
na u e. Speci ically, conside a s ochas ic p ocess
{Y( )} ≥0
, whe e
Y( )
ep esen s he
loss a ime
(see [
24
]). While s ochas ic p ocesses a e widely used in machine lea ning
(e.g., [
25
]), he s ochas ici y he e a ises om pa ame e upda es a each epoch. In he exp es-
sion
Y( ) = L(x|θ )
, he pa ame e ec o
θ
is de ined ecu si ely as
θ =θ (L(x|θ −1))
,
leading o he ela ionship
Y( ) = L(x|θ (L(x|θ −1)))
Modeling lea ning cu es as s ochas ic p ocesses o e s se e al ad an ages. Fi s , i
cap u es he inhe en andomness in he lea ning p ocess, which is in luenced by ac o s
such as weigh ini ializa ion, he o de o aining da a, and noise in he da a. Addi ionally,
i allows o he quan i ica ion o unce ain y in p edic ing he inal pe o mance o he
Ma hema ics 2025,13, 555 7 o 36
model, a c ucial aspec o in o med decision making. S ochas ic modeling also p o ides
access o powe ul analy ical ools om s a is ics and p obabili y heo y, which can be used
o s udy he p ope ies o lea ning cu es and p edic u u e pe o mance.
I is use ul o hypo hesize ha
Y( )
ollows a classic s ochas ic di usion p ocess,
exp essed as a di e en ial s ochas ic equa ion o he o m
dY( ) = µ(Y( ), )d +σ(Y( ), )dW( )(1)
whe e
µ(Y( )
,
)
is he d i e m, ep esen ing he de e minis ic end o he lea ning cu e;
σ(Y( )
,
)
is he di usion coe icien , cap u ing s ochas ic a iabili y; and las ly
W( )
is a
s anda d B ownian mo ion. This kind o s ochas ic di e en ial equa ions has been ex en-
si ely s udied. Fo example, B oga -Mo e e al. [
26
] p esen a me hod o es ima ing bo h
he d i and di usion coe icien s o con inuous, mul idimensional, nonlinea s ochas ic
di e en ial equa ions (SDEs) ha a e in luenced by con ol inpu s.
Ou lea ning-cu e es ima o
ˆ
N
can p edic inal pe o mance based on
N
ini ial
epochs in s ochas ic e ms:
ˆ
YN(m) = EhY(m)|{Y( )}N
=1i(2)
The in e es in s ochas ic modeling lies in he a ailable ools o es ima ing he lea ning
cu e. Using he classical esul o I ô [
27
], i we model he cu e wi h Equa ion (1) and
(
,
Y )
is wice di e en iable in
Y
and di e en iable in
, he change in
(
,
Y )
is gi en by
d ( ,Y ) = ∂
∂ +µ∂
∂x+1
2σ2∂2
∂x2d +σ∂
∂xdW .
This solu ion is use ul o s ochas ic modeling o lea ning cu es. Fo example, us-
ing
X = (
,
Y )
o
(
,
Y ) = ∂
∂ (Y )
, we would ob ain a s ochas ic exp ession o he
cu a u e o Y .
In eg a ing bo h sides o he lemma o I ô, we ob ain
(T,YT) = (0,Y0) + ZT
0
∂
∂ ( ,Y )d +ZT
0
∂
∂x( ,Y )dY +1
2ZT
0
∂2
∂x2( ,Y )σ2( ,Y )d .
Subs i u ing dY wi h Equa ion (1) in he second in eg al e m, we ob ain
(T,YT) = (0,Y0) + ZT
0∂
∂ +µ∂
∂x+1
2σ2∂2
∂x2d +ZT
0σ∂
∂xdW .
This in eg al o m o he lemma is use ul o e alua ing
(
,
X )
a he inal ime
T
,
based on i s ini ial alue and he in eg al con ibu ions o e ime. Fo
(
,
Y) = Y( )
, as in
ou case, i p o ides a de ailed es ima ion.
Despi e i s bene i s, s ochas ic modeling o lea ning cu es comes wi h some limi-
a ions. One key challenge is complexi y: some s ochas ic models can be in ica e and
di icul o i o he a ailable da a. Fu he mo e, hese models ely on assump ions abou
he lea ning p ocess ha may no always be alid, which can a ec hei accu acy in ce ain
con ex s. Las ly, s ochas ic models equi e a su icien amoun o da a o accu a ely es ima e
hei pa ame e s, posing a p oblem in si ua ions whe e da a a e limi ed.
1.4. Bayesian Op imiza ion o Neu al A chi ec u e Sea ch
In he op imiza ion p ocess desc ibed abo e, he inpu o he objec i e unc ion
consis s o he model a chi ec u e and aining hype pa ame e s, while he ou pu is he
Ma hema ics 2025,13, 555 8 o 36
pe o mance o he model ained wi h he gi en se ings. Bayesian op imiza ion (BO) is
pa icula ly e ec i e in scena ios whe e e alua ing is compu a ionally expensi e.
The BO amewo k p o ides a p obabilis ic app oach o op imizing expensi e- o-
e alua e objec i e unc ions, especially when he unc ion lacks a known analy ical o m o
is compu a ionally demanding. BO uses a su oga e model, ypically a Gaussian p ocess,
o app oxima e he objec i e unc ion based on p io e alua ions. An acquisi ion unc ion
hen guides he selec ion o he nex e alua ion poin , balancing explo a ion o unce ain
egions wi h exploi a ion o p omising a eas. This i e a i e app oach makes BO highly
e icien , equi ing signi ican ly ewe e alua ions compa ed o b u e- o ce o g id-sea ch
me hods. BO is widely applied in hype pa ame e op imiza ion, NAS, and expe imen al
design. Fo ins ance, in [
28
], Kandasamy e al. p esen NASBOT, a BO amewo k o NAS
using a no el dis ance me ic, in he space o neu al ne wo k a chi ec u es.
A ypical applica ion o BO in ML in ol es model selec ion asks [
29
], whe e he
gene aliza ion pe o mance o a s a is ical model canno be de e mined analy ically and
mus ins ead be e alua ed empi ically. Fo example, BO can be used o selec he pa ame e
λand he ke nel bandwid h h o SVMs.
Me hods based on BO p o ide a sys ema ic and au oma ed al e na i e o adi ional
p ac ices commonly employed in PHM. Typically, human expe s inspec lea ning cu es
du ing aining o iden i y and e mina e uns wi h poo hype pa ame e se ings, he eby
accele a ing manual hype pa ame e op imiza ion. In PHM, lea ning cu es play a cen al
ole in e alua ing he pe o mance o algo i hms wi h espec o esou ces such as he
numbe o aining examples o i e a ions [
17
]. These cu es ha e a a ie y o applica ions,
including guiding da a acquisi ion s a egies, implemen ing ea ly-s opping c i e ia o
p e en o e i ing, and acili a ing model selec ion p ocesses [
18
]. BO enhances hese asks
by au oma ing he explo a ion and op imiza ion o hype pa ame e s, educing he need o
manual in e en ion while main aining e iciency.
1.5. Aim o he Pape
This wo k ocuses on le e aging lea ning cu es o elimina e unp omising models
ea ly in he NAS p ocess using BO. Building on p io esea ch [
30
], he p esen wo k aims
o op imize NAS speci ically o PHM applica ions by in oducing a no el pe o mance
es ima ion amewo k. This amewo k is designed o s eamline he NAS p ocess by
signi ican ly educing compu a ional cos s while main aining high-quali y model selec ion.
The co e o he p oposed app oach lies in an in elligen es ima o capable o p edic ing
he long- e m pe o mance o a model by analyzing only a ew ini ial aining and alida ion
epochs. To achie e his, he es ima o is buil using an ex ensi e da ase comp ising
62 p edic i e main enance p oblems, ensu ing i s obus ness and gene aliza ion. This
es ima o is hen seamlessly in eg a ed in o he BO loop, enabling he p ocess o e icien ly
p une unp omising candida es and alloca e esou ces o models wi h highe po en ial,
he eby enhancing he o e all e iciency o NAS in PHM asks.
In ou p oposed model, p io assump ions abou he unc ional o m o he cu es a e
no equi ed. While such assump ions could simpli y he p oblem, hey a e no necessa y
wi hin ou amewo k. As a esul , ou me hod gene alizes beyond pu ely pa ame ic
app oaches, making i be e sui ed o en i onmen s cha ac e ized by he e ogenei y, such
as in he con ex o PHM.
1.6. S uc u e o he Pape
This wo k is s uc u ed as ollows. Sec ion 2p esen s some ela ed wo ks, whe e
key de elopmen s and me hodologies ela ed o he p oposed app oach a e discussed.
Sec ion 3p o ides he ounda ions o Bayesian op imiza ion, o e ing an in oduc ion o he
Ma hema ics 2025,13, 555 9 o 36
echnique, whe eas Sec ion Bayesian Op imiza ion Based on a Gaussian P ocess elabo a es
on Bayesian op imiza ion, explaining i s in eg a ion wi h Gaussian p ocesses.
In Sec ion 4, he da ase s used in his s udy a e desc ibed in de ail, ocusing on hei
cha ac e is ics and applicabili y o p edic i e main enance asks. Sec ion 5ou lines he
main app oach and me hodology de eloped in his wo k.
Sec ion 6de ails he a chi ec u es o he pe o mance es ima o s used in he s udy and
he aining p ocedu e. Finally, he pe o mance o he es ima o s is assessed and analyzed
unde di e en condi ions.
In Sec ion 7, he ea ly-s opping me hodology applied du ing BO-based NAS is dis-
cussed. This sec ion de ines he me ics used o e alua e he pe o mance o he ea ly-
s opping p ocess, p esen s he esul s ob ained, and compa es he app oach wi h di e -
en baselines.
Sec ion 8is de o ed o conduc ing a heo e ical analysis o wo undamen al aspec s
o he me hod. Fi s ly, he impac o addi i e and au o eg essi e noise on lea ning cu es
and ea ly-s opping e iciency is in es iga ed. Secondly, he con e gence o obus ea ly-
s opping me hods unde noise is analyzed.
Finally, Sec ion 9con ains a discussion o he esul s and insigh s ob ained om he
expe imen s and heo e ical analysis, and Sec ion 10 concludes he wo k, summa izing he
indings and sugges ing po en ial di ec ions o u u e esea ch.
2. Rela ed Wo k
Building upon he ideas in oduced in he Domhan e al. pape [
18
] men ioned abo e,
Bake e al. p opose a me hod o enabling ea ly s opping du ing he aining o neu al
ne wo ks by le e aging lea ning cu es [
31
]. This me hod employs an es ima o o p edic
he inal pe o mance o a model du ing i s aining. I he es ima ed pe o mance is lowe
han he bes pe o mance obse ed so a , he aining p ocess o ha model is e mina ed
p ema u ely. This ea ly-s opping app oach is in eg a ed in o he Hype band op imiza ion
algo i hm [
32
], which is designed o e icien ly alloca e compu a ional esou ces du ing
hype pa ame e op imiza ion. Fu he mo e, Klein e al. [
17
] explo e he use o Bayesian
ne wo ks o modeling lea ning cu es wi hin he Hype band amewo k, enhancing i s
p edic i e capabili ies du ing hype pa ame e sea ch.
I is impo an o highligh ha he Hype band algo i hm ope a es as a esou ce alloca-
ion s a egy and does no u ilize pe o mance da a om p e ious i e a ions when selec ing
new con igu a ions o e alua e. Ins ead, Hype band accele a es andom sea ch by apply-
ing an adap i e ea ly-s opping mechanism ha dynamically alloca es esou ces—such as
aining i e a ions, da a samples, o ea u es—based on in e media e pe o mance me ics.
Wi hin he con ex o Bayesian op imiza ion (BO), a ious me hods ha e been de el-
oped o imp o e e iciency by inco po a ing ea ly-s opping mechanisms. These me hods
aim o hal e alua ions o unp omising hype pa ame e con igu a ions wi hin he inne
loop o he op imiza ion p ocess. A no able example is BOHB [
33
], which combines he
s eng hs o Bayesian op imiza ion and Hype band. BOHB le e ages in o ma ion om
p e iously sampled con igu a ions o guide he sea ch p ocess while e aining he ea ly-
s opping s a egy o Hype band o disca d unp omising candida es, he eby imp o ing
bo h compu a ional e iciency and solu ion quali y.
Dai e al. [
34
] p oposed a Bayesian model o es ima e he con e gence o a model being
ained du ing a Bayesian op imiza ion (BO) i e a ion. This es ima ion allowed o he
ea ly e mina ion o aining, he eby educing he numbe o unnecessa y aining epochs.
Howe e , a limi a ion o his me hod is ha models wi h limi ed po en ial o ou pe o m
he cu en bes con igu a ion could s ill unde go a subs an ial numbe o aining epochs
be o e being hal ed.
Ma hema ics 2025,13, 555 16 o 36
0 20 40 60 80
epochs
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
loss
Valida ion cu es
Figu e 7. Example o lea ning cu es used in his wo k. On he op, he aining pe o mance cu es
a e depic ed, and on he bo om, he alida ion pe o mance cu es, bo h o he same model se ings.
As can be seen, he alida ion cu es a e pa icula ly noisy du ing he i s epochs.
5. O e all Desc ip ion o he P oposal
Ou p oposal is based on wo main componen s: a lea ning-cu e es ima o (Sec ion 6)
and a BO p ocess o NAS ha uses he a o emen ioned es ima o o ea ly s opping
(Sec ion 7). Le us b ie ly desc ibe hese elemen s.
Lea ning-cu e es ima o :
The goal is o de elop a model capable o p edic ing he inal
pe o mance o a DL model based solely on he i s epochs o i s aining and
alida ion cu es. Two e sions o he es ima o a e p oposed: An es ima o based
only on he lea ning cu es, and an es ima o condi ioned on he hype pa ame e s o
he neu al ne wo k a chi ec u e.
In eg a ion in o BO o NAS:
The lea ning-cu e es ima o is used o accele a e he BO
p ocess in he sea ch o op imal neu al ne wo k a chi ec u es o PHM asks. When
he es ima o p edic s ha he pe o mance o a model will be signi ican ly wo se
han he bes model ound so a , he aining o ha model is s opped, hus sa ing
compu a ional esou ces.
The model is ained and e alua ed using a da ase o 61,000 lea ning cu es gene a ed
om 62 di e en da ase s in he con ex o PHM, as desc ibed in Sec ion 4.
6. Pe o mance Es ima o s
The p oposed me hodology cen e s on a model designed o es ima e he inal alida-
ion pe o mance
y(T)
o a neu al ne wo k. This es ima ion allows o he applica ion o
an ea ly-s opping s a egy du ing NAS, imp o ing e iciency and educing compu a ional
cos s by hal ing aining ea ly based on p edic ed pe o mance.
To cons uc he es ima o , he ini ial s ep in ol es selec ing a speci ic machine lea ning
model. The p ocess uses he i s
N
obse ed poin s o aining and alida ion pe o mance
ex ac ed om he lea ning cu es. These poin s p o ide a basis o analyzing and p edic -
ing ends by wo king on pai s o cu es: yi=⟨y ain
i,y al
i⟩.
The aining se o build such a model is composed o a se o o me aining and alida ion
pe o mances and hei co esponding a ge s. Using he no a ion
y↾[0,N]=⟨y ain
↾[0,N]
,
y al
↾[0,N]⟩
:
SLC =n(y1
↾[0,N],y1(T)),(y2
↾[0,N],y2(T)), . . . , (yM
↾[0,N],yM(T))o
Ma hema ics 2025,13, 555 17 o 36
whe e Mis he numbe o samples in he da ase .
No e ha he numbe o aining epochs o each model could be di e en (see
Figu e 7
). The a ia ion is due o he models being ained using an ea ly-s opping c i e ion.
This is qui e di e en om o he wo ks, whe e he numbe o epochs is always he same.
In a second modeling s age, i is s udied how he in o ma ion o he hype pa ame e s
impac s he lea ning-cu e es ima o . Thus, he hype pa ame e s
θi
will be included as
pa o he es ima o .
Since he da ase con ains a ious ypes o a chi ec u es and only a gene al lea ning-
cu e es ima o is ained, he hype pa ame e ec o is pai ed wi h a one-ho ec o ha
de e mines whe he each hype pa ame e is se o ha ne wo k. Thus, he aining se
collec ing such in o ma ion o his kind o es ima o is o he o m
SLC+H=n((y1,θ1),y1
T),((y2,θ2),y2
T), . . . , ((y)M,θM),yM
T)o
A i s app oxima ion o app oach he es ima ion could be o s a om
ˆ
y(x|Θ,ω)≃
M
∑
j=1
ωjy ain
j(x|θj) (+
M
∑
j=1
ωjy al
j(x|θj))
Thus, he app oach p oposed by Domhan could be applicable. Howe e , as p e iously
men ioned, pa ame e es ima ion using MCMC can be compu a ionally expensi e, and he
mos app op ia e pa ame ic amily (especially in he con ex o PHM) is unclea . An al-
e na i e, which we aim o demons a e in his pape , is o use a neu al ne wo k-based
model, ained on he a ailable da a, o compu e
ˆ
y(T)
. The a chi ec u e o he ne wo k is
ou lined below.
6.1. Pe o mance Es ima o A chi ec u es
The selec ed a chi ec u e o he inal pe o mance es ima o is he LSTM, chosen o
i s excellen pe o mance in p ocessing and modeling ime-se ies da a, e ec i ely cap u ing
bo h pas and u u e dependencies. The a chi ec u e begins wi h LSTM cells consis ing o
128 uni s. Following hese, wo ully connec ed laye s wi h 128 and 64 neu ons, espec i ely,
a e added, each u ilizing ReLU as he ac i a ion unc ion. To imp o e aining s abili y and
con e gence, ba ch no maliza ion is applied immedia ely a e he LSTM cells.
The inpu o he model co esponds o aining and alida ion pe o mance cu es,
wi h a a iable shape anging om 2 o 9 obse ed epochs. The comple e a chi ec u e is
illus a ed in Figu e 8.
To analyze he impac o ne wo k hype pa ame e s on he lea ning p ocess, a condi-
ioned ne wo k was designed. The hype pa ame e s de ine he a chi ec u e o he ne wo k
esponsible o gene a ing he lea ning cu e. Gi en ha he da ase includes di e en
ypes o ne wo ks (FFN, CNN, RNN, and T ans o me ), ce ain hype pa ame e s a e no
applicable ac oss all a chi ec u es (see Table 1).
To add ess his, a one-ho ec o mask was in oduced o indica e he ele ance o each
o he 28 hype pa ame e s o a speci ic ne wo k ype. Bo h he hype pa ame e ec o
and he mask, each wi h dimensions 28
×
2, a e p ocessed h ough wo ully connec ed
laye s con aining 32 and 16 neu ons, espec i ely. These laye s u ilize he Swish ac i a ion
unc ion, de ined as (x) = x·sigmoid(x)(see Figu e 8B).
The ou pu om he inal ully connec ed laye is conca ena ed wi h he ou pu o he
Bi-LSTM cells. This combined ep esen a ion is hen passed o he wo inal dense laye s o
he ne wo k. De ailed summa ies o he a chi ec u es selec ed a e p o ided in Table A1.
Ma hema ics 2025,13, 555 18 o 36
Conca ena e
Dense laye
Ba ch No maliza ion
D opou
Ou pu
Bi-LSTM
128 Bi-LSTM Uni s
x128
ReLU
x64
ReLU
([2-9], 2)
Inpu Shape
128 Bi-LSTM Uni s
x128
ReLU
([2-9], 2)
Inpu Shape
(28, 2)
Inpu Shape
x64
ReLU
x32
Swish
x16
Swish
A)
B)
Figu e 8. Pe o mance es ima o a chi ec u es. (A) Es ima o a chi ec u e using only obse ed
aining and alida ion pe o mance cu es. (B) Es ima o a chi ec u e using pe o mance cu es
and ne wo k hype pa ame e s.
Table 1. Hype pa ame e s used o condi ion he pe o mance es ima o . Some hype pa ame e s a e
no applicable, depending on he ype o ne wo k.
Hype pa ame e Desc ip ion MLP CNN RNN T ans o me
MLP Ac i a ion Ac i a ion unc ion
Ba ch No maliza ion Use ba ch no maliza ion
Bidi ec ional Bidi ec ional RNN
Block Size Numbe o con olu ional blocks
RNN Cell Type Recu en cell ype
Con Ac i a ion Con olu ion ac i a ion unc ion
Dense Ac i a ion Dense block ac i a ion unc ion
Dila ion Ra e Con olu ion ac i a ion a e
D opou D opou egula iza ion a e
F1, F2, F3 MS-CNN con olu ion size
FC1, FC2 Dense laye size
Fil e s Con olu ion il e s
Ke nel Size Con olu ion ke nel size
L1, L2 L1 and L2 egula iza ion a es
Dense Dim Dense block pa ame e s
Model Dim To al model pa ame e s
MS Blocks Numbe o MS-CNN blocks
Con Blocks Numbe o con olu ional blocks
Laye s Numbe o laye s
Heads Numbe o T ans o me heads
Ou pu Dim Ou pu dimension
RNN Uni s Numbe o RNN uni s
Segmen Size Pa ch segmen size
6.2. T aining P ocedu e and Es ima o Pe o mance
The pe o mance es ima o s we e designed o p edic he inal alida ion loss o a
ne wo k, wi h he choice o loss unc ion depending on he na u e o he ask. Fo eg ession
asks, he mean squa ed e o (MSE) was used, while classi ica ion asks elied on c oss-
en opy loss.
The hype pa ame e s o he pe o mance es ima o ne wo k o be op imized include
he numbe o LSTM laye s, he size o he LSTM cells, he bidi ec ionali y, and he lea ning
a e. To ensu e obus aining and e alua ion, he da ase o pe o mance cu es was
spli a he da ase le el (known as g oup-based spli ing) o mi iga e o e i ing. A o al
o 30% o he da ase s we e andomly assigned o he es se , while he emaining 70%
we e used wi hin a h ee- old c oss- alida ion ( ha is, hey we e u he di ided, alloca ing
66% o he aining olds and 33% o he alida ion old), o enhance he esul s’ eliabili y.
Ma hema ics 2025,13, 555 19 o 36
Addi ionally, all expe imen s we e epea ed six imes wi h di e en andom seeds o
alida e he esul s ac oss di e se es condi ions and ne wo k ini ializa ions. Table 2shows
he ange o hype pa ame e s s udied.
Table 2. Hype pa ame e ange s udied o he pe o mance es ima o .
Hype pa ame e Values
Lea ning Ra e {0.01, 0.001, 0.0001}
Numbe o LSTM Laye s {1, 2, 3, 4}
Numbe o LSTM Cells {16, 32, 64, 128}
Bidi ec ional LSTM {Yes, No}
The aining p ocess inco po a ed se e al op imiza ion echniques o ensu e e i-
ciency and s abili y. Ea ly s opping was applied o e mina e aining when alida ion
pe o mance ailed o imp o e o eigh consecu i e epochs, educing unnecessa y com-
pu a ion and minimizing o e i ing. A lea ning a e decay s a egy, inspi ed by he wo k
o You e al. [
88
], was also employed. I alida ion pe o mance showed no imp o emen
o i e consecu i e epochs, he lea ning a e was educed by a ac o o 0.1. The Adam
op imize [89] was used o ain he es ima o s e ec i ely.
The pe o mance o he es ima o s, as illus a ed in Figu e 9, is in luenced by he
numbe o obse ed aining epochs. As an icipa ed, he es ima ion e o dec eases as
he numbe o obse ed epochs inc eases, demons a ing he ad an age o inco po a ing
mo e aining da a. In e es ingly, he inclusion o ne wo k hype pa ame e s enhances
he accu acy o he es ima o s, p ima ily when he numbe o obse ed epochs is small.
Addi ionally, he lea ning a e appea s o ha e a lesse impac on he non-condi ional
app oach, whe eas lowe lea ning a es a e necessa y o he condi ional app oach o
achie e op imal pe o mance.
23456789
Obse ed epochs
0.05
0.06
0.07
0.08
0.09
0.10
0.11
Es ima o Pe o mance (MAE)
Lea ning a e
0.0001
0.001
0.01
Es ima o condi ioned
False
T ue
Figu e 9. E iciency o he inal pe o mance es ima o s, g ouped by lea ning a e and whe he he
model was condi ioned o no .
Figu e 10 illus a es he ela ionship be ween he alida ion MAE and es MAE as
measu ed by he pe o mance es ima o . The igu e highligh s ha a lowe numbe o
obse ed epochs is associa ed wi h a educed likelihood o o e i ing, whe e he model
main ains simila pe o mance on bo h alida ion and es se s. This end is pa icula ly e -
iden in he ke nel densi y es ima ion (KDE) plo , which shows a mo e consis en alignmen
be ween alida ion and es e o s as he numbe o obse ed epochs dec eases.
Ma hema ics 2025,13, 555 20 o 36
0.0 0.2 0.4 0.6
MAE (Valida ion)
0.0
0.1
0.2
0.3
0.4
0.5
MAE (Tes )
0.0 0.2 0.4
MAE (Tes )
Figu e 10. E ec o he pe o mance es ima o on he es se in ela ion o alida ion pe o mance.
6.3. Robus ness Analysis
In his sec ion, we analyze he sensi i i y o he es ima o pe o mance o a ious
hype pa ame e s using he Sobol sensi i i y index. The Sobol index quan i ies he con ibu-
ion o each inpu pa ame e o he ou pu a iance, p o iding insigh in o how obus he
model is o changes in hype pa ame e s. The Sobol index o a gi en hype pa ame e
Xi
is
compu ed as ollows:
Si=Va [E[Y|Xi]]
Va [Y]
whe e
Y
is he ou pu o he model (e.g., es RMSE) and
Xi
is he hype pa ame e unde
s udy. The nume a o measu es he a iance o he ou pu due o a ia ions in
Xi
, while
he denomina o ep esen s he o al a iance in he ou pu . The Sobol index alues o
each hype pa ame e a e summa ized in Table 3.
Table 3. Sobol sensi i i y index o each hype pa ame e s udied o he pe o mance es ima o .
Hype pa ame e Sobol Index
Lea ning a e 0.0587
Ne wo k dep h −0.0303
Bidi ec ional 0.0109
Recu en uni s (wid h) 0.0306
Fo he lea ning a e, he Sobol index was ound o be 0.0587. This ela i ely mode a e
alue sugges s ha he lea ning a e has a meaning ul impac on model pe o mance. Thus,
ine- uning he lea ning a e is likely o ha e he mos no iceable e ec on imp o ing model
pe o mance. In con as , he ne wo k dep h had a Sobol index o
−
0.0303, indica ing
a small nega i e in luence on he pe o mance o he model. The nega i e Sobol index
sugges s ha a shallowe a chi ec u e could be mo e e ec i e o his p oblem, and adding
ex a laye s migh no con ibu e posi i ely o pe o mance.
The bidi ec ional se up yielded a Sobol index o 0.0109, which indica es a e y small
posi i e e ec on pe o mance. This sugges s ha p ocessing in o ma ion in bo h o wa d
and backwa d di ec ions has a mino bene i . Finally, o he numbe o ecu en uni s,
he Sobol index was 0.0306, showing a small posi i e in luence on model pe o mance.
While his index indica es a modes e ec , i s ill sugges s ha he wid h o he ecu en
Ma hema ics 2025,13, 555 21 o 36
laye s plays a ole in cap u ing mo e complex pa e ns in he da a. Howe e , he con i-
bu ion is small compa ed o he lea ning a e, and adjus men s o he ecu en uni s may
lead o ma ginal imp o emen s a he han d as ic changes in pe o mance.
7. Ea ly S opping Du ing NAS wi h BO
A e aining he inal pe o mance es ima o , i s e ec i eness in educing he numbe
o epochs equi ed du ing BO o NAS was e alua ed. Since unning he BO p ocess ac oss
62 da ase s o a ious a chi ec u es is compu a ionally expensi e, and hese expe imen s
had al eady been conduc ed o gene a e he lea ning cu es, we simula ed he BO p ocess
using he esul s om hose expe imen s. Speci ically, we ollowed he same sequence o
es se expe imen s o each da ase as hei espec i e aining uns.
The de ails o his simula ion p ocedu e a e ou lined in Algo i hm 2. The condi ion
es ima ed_loss <
2
·bes _loss
was in oduced as a heu is ic a he han a pa ame e de i ed
om uning. This h eshold was se a he beginning o ou s udy based on he hypo hesis ha
i he p edic ed loss o a model is wice as high as he bes loss obse ed so a , i is unlikely ha
he model will ou pe o m he cu en bes . The alue o 2 was chosen o p o ide a balance
be ween il e ing ou poo ly pe o ming models and allowing su icien explo a ion. While
his speci ic h eshold was no ex ensi ely uned, i e lec s an in ui i e assump ion abou he
ela ionship be ween ea ly loss es ima es and inal pe o mance. Fu u e wo k could explo e
al e na i e h esholds o be e unde s and hei impac on model selec ion.
7.1. Me ics
Two key me ics we e compu ed o e alua e he impac o ea ly s opping du ing he
BO p ocess. The i s me ic quan i ies he o al numbe o aining epochs skipped du ing
he BO p ocess. Ea ly s opping is applied o educe he compu a ional cos by a oiding
unnecessa y aining i e a ions. Le
E o al
deno e he o al numbe o epochs a model would
un wi hou ea ly s opping, and
Eobse ed
ep esen he numbe o epochs obse ed be o e
ea ly s opping is applied. The numbe o skipped epochs, Eskipped, is de ined as
Eskipped =E o al −Eobse ed
This me ic p o ides a measu e o he compu a ional ime sa ed by applying
ea ly s opping.
The second me ic measu es he pe o mance d op pe cen age,
Dpe o mance
. This
me ic measu es he ela i e educ ion in pe o mance caused by ea ly s opping, compa ed
o he bes possible pe o mance achie ed i all epochs we e ully obse ed.
Le
P ull
deno e he pe o mance (e.g., es loss) o he bes model when ained o all
epochs, and
Pea ly
he pe o mance o he model selec ed by ea ly s opping. The pe o -
mance d op pe cen age is compu ed as
Dpe o mance =P ull −Pea ly
P ull
This me ic quan i ies he cos o applying ea ly s opping in e ms o pe o mance deg ada ion.
I is no ewo hy ha he e exis s a ade-o be ween hese wo me ics. As he numbe
o aining epochs skipped inc eases, he p obabili y o disca ding he bes models selec ed
du ing he BO p ocess wi hou ea ly s opping inc eases. The e o e, i is impo an o ind
an op imal balance, whe e ea ly s opping e ec i ely educes compu a ional cos wi hou
signi ican ly comp omising model pe o mance.
Ma hema ics 2025,13, 555 22 o 36
Algo i hm 2: Ea ly s opping wi hin BO simula ion
Inpu : E(·): Final pe o mance es ima o
BO: Bayesian Op imiza ion expe imen eco ds
N: Numbe o obse ed epochs
Ou pu : Bes Loss
Ini ialize a iables: bes _loss ←∞, bes _hype pa ame e s ←NaN;
o 1 o 100 do
h←nex hype pa ame e se aken (in same o de ) om a o me ly execu ed
BO p ocess;
lc ← ain and alida ion pe o mance cu es o he N i s epochs;
es ima ed_loss ←E(lc, h);
i es ima ed_loss <bes _loss * 2 hen
/* The expe imen is un un il he end */
inal_loss ←ac ual loss ob ained in BO p ocess
end
else
/* The expe imen is p uned */
inal_loss ←es ima ed_loss
end
i inal_loss <bes _loss hen
bes _loss ← inal_loss;
end
end
7.2. Resul s
The esul s o he simula ion a e p esen ed in Figu e 11. Bo h pe o mance es ima o s
sa e app oxima ely 42% o 65% o compu a ion ime (skipped epochs). Howe e , an in e se
end is obse ed: as ewe epochs a e obse ed, mo e epochs a e disca ded when a non-
p omising a chi ec u e is de ec ed ea ly, which sligh ly educes he o e all pe o mance.
Numbe o Obse ed T aining Epochs
2 3 4 5 6 7 8 9
Model
LSTM ARIMA Random
Figu e 11. BO ea ly-s opping simula ion esul s. Le : Pe cen age o imes he bes solu ion (g ound
u h) was ound. Righ : Pe cen age o aining epochs skipped. Bo h plo s compa e hese me ics
agains he mean pe cen age o loss educ ion and he numbe o obse ed aining epochs.
Ma hema ics 2025,13, 555 23 o 36
Addi ionally, we measu e he equency wi h which he es ima o iden i ies he bes
solu ion, o g ound u h. This likelihood inc eases as he numbe o obse ed epochs
g ows, sugges ing ha ha ing mo e da a leads o mo e accu a e es ima ions.
The e ec o obse ing ewe epochs on he mean alida ion loss is also analyzed.
On a e age, he inc ease in loss is minimal—app oxima ely 2% in he wo s -case scena io
when only wo epochs a e obse ed (no e ha he lea ning cu es a e no malized be ween
0 and 1). This is e iden om he s a ma ke s in Figu e 12. These esul s sugges ha
he nega i e impac o ea ly s opping in BO, in e ms o pe o mance d op (
Dpe o mance
),
is negligible, e en when he numbe o obse ed epochs is limi ed. This inding aligns
wi h he wo k o Egele e al. [
36
], who demons a ed excellen pe o mance e en when
obse ing only a single epoch.
23456789
Numbe o Obse ed T aining Epochs
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
Mean Loss Pe cen age D op
Figu e 12. The igu e illus a es he dis ibu ion o he mean pe cen age loss educ ion ac oss
a ying numbe s o obse ed aining epochs. The x-axis ep esen s he numbe o obse ed aining
epochs, while he y-axis shows he mean pe cen age educ ion in loss. S a s indica e he bes models,
selec ed based on he alida ion loss o he pe o mance es ima o , whe eas ed do s ep esen he
pe o mance in he BO p ocess o models wi h he highes alida ion loss. The hype pa ame e s o
hese a chi ec u es a e de ailed in Table A1.
An impo an ques ion, pa icula ly ega ding s abili y and eliabili y, is whe he he
p edic i e accu acy o he pe o mance es ima o ansla es e ec i ely o he BO p ocess.
The s a ma ke s in Figu es 12 and 13 illus a e he co ela ion be ween he alida ion loss
o he pe o mance es ima o and he mean inc ease in alida ion loss achie ed du ing
he BO p ocess. The esul s indica e ha a lowe alida ion loss o he pe o mance
es ima o co esponds o a lowe inc ease in he mean alida ion loss du ing he BO
p ocess, highligh ing he ole o he es ima o in guiding e icien model selec ion.
The obse ed mode a e posi i e co ela ions (bo h Pea son and Spea man) indica e a
ela ionship be ween he alida ion loss o he pe o mance es ima o and he BO e iciency
o he p ocess in iden i ying well-pe o ming models, e en when a signi ican pe cen age
o epochs a e disca ded. No ably, he highe Spea man co ela ion compa ed o he Pea son
co ela ion sugges s ha while he ela ionship may no be s ic ly linea , i ollows a
consis en mono onic end. This insigh highligh s he impo ance o accu a e pe o mance
es ima ion in enhancing he e ec i eness o he BO p ocess.
The inclusion o hype pa ame e s in he pe o mance es ima o esul s in jus ma ginal
imp o emen s. While i enhances he p edic ion accu acy o he cu e es ima o , he bene i s
o inco po a ing ne wo k ea u es become less p onounced i one looks a he BO p ocess
esul s. Condi ioning aids in disca ding poo -pe o ming models, wi h only a minimal
dec ease in he mean loss pe cen age d op (see Table 4), pa icula ly when using only wo
Ma hema ics 2025,13, 555 24 o 36
obse ed epochs. This inding aligns wi h he g ea e a iabili y in pe o mance obse ed
wi h ewe epochs, as illus a ed in Figu e 9. Once bo h app oaches con e ge, he amoun o
disca ded aining da a becomes compa able, leading o simila a e age BO pe o mance.
0.02 0.04 0.06 0.08 0.10 0.12
Pe o mance Es ima o Valida ion Loss
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
Mean Valida ion Loss D op in BO
T end
Figu e 13. Co ela ion be ween he alida ion loss o he pe o mance model and he mean alida ion
loss d op du ing he BO p ocess. The da a poin s ep esen indi idual expe imen s, and he ed line
shows he eg ession end.
7.3. Pe o mance Es ima o A chi ec u e Impac in he BO P ocess
Addi ionally, he po en ial impac o he hype pa ame e s o he es ima o was ana-
lyzed. The hype pa ame e s s udied included lea ning a e, ne wo k dep h, bidi ec ionali y,
and ecu en uni s (wid h). Figu e 14 illus a es he e ec o di e en anges o alues
o each hype pa ame e . Mos hype pa ame e s appea o ha e no signi ican impac on
he BO p ocess, wi h he excep ion o he lea ning a e. Fo he non-condi ioned es ima o ,
highe lea ning a es a e p e e ed, whe eas o he condi ioned es ima o , his ela ionship
is e e sed.
Table 4. Compa ison o mean loss pe cen age d op and pe cen age o skipped epochs o a ying
numbe s o obse ed epochs, wi h and wi hou ne wo k ea u e condi ioning. The able highligh s
he impac o condi ioning on bo h me ics, ac oss di e en le els o obse ed epochs ( om 2 o 9).
Dpe o mance Eskipped
Ne wo k Fea u es
Condi ioned No Yes No Yes
Obse ed Epochs
2 0.043709 0.045064 0.646917 0.746301
3 0.033079 0.038057 0.625684 0.626886
4 0.044258 0.048155 0.604485 0.602086
5 0.034555 0.038707 0.573473 0.574904
6 0.032851 0.036993 0.540953 0.549439
7 0.038565 0.042210 0.514593 0.523448
8 0.030337 0.039286 0.483932 0.496455
9 0.032317 0.037472 0.458637 0.468747
7.4. Compa ison wi h Baseline Models
The p oposed app oach was e alua ed agains h ee baseline me hods o assess
i s pe o mance.
The i s baseline is a andom app oach, whe e a pe cen age o aining epochs is dis-
ca ded a andom. In Figu es 11 and 15, he esul s o he andom baseline a e ep esen ed
wi h black plus symbols.
Ma hema ics 2025,13, 555 25 o 36
23456789
Numbe o Obse ed T aining Epochs
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
Mean Loss Pe cen age D op
Non-Condi ioned Es ima o
23456789
Numbe o Obse ed T aining Epochs
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
Mean Loss Pe cen age D op
Condi ioned Es ima o
Lea ning a e
0.0001
0.001
0.01
23456789
Numbe o Obse ed T aining Epochs
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
Mean Loss Pe cen age D op
23456789
Numbe o Obse ed T aining Epochs
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
Mean Loss Pe cen age D op
Ne wo k dep h
1
2
3
4
23456789
Numbe o Obse ed T aining Epochs
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
Mean Loss Pe cen age D op
23456789
Numbe o Obse ed T aining Epochs
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
Mean Loss Pe cen age D op
Bidi ec ional
False
T ue
23456789
Numbe o Obse ed T aining Epochs
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
Mean Loss Pe cen age D op
23456789
Numbe o Obse ed T aining Epochs
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
Mean Loss Pe cen age D op
Recu en uni s (wid h)
16
32
64
128
Figu e 14. Boxplo s illus a ing he impac o hype pa ame e con igu a ions on he mean loss
pe cen age d op ac oss di e en numbe s o obse ed aining epochs. Each ow co esponds o a
speci ic hype pa ame e : lea ning a e, ne wo k dep h, bidi ec ionali y, and ecu en uni s (wid h).
The le column shows esul s o non-condi ioned es ima o s, while he igh column shows esul s
o condi ioned es ima o s.
The second baseline le e ages he “las seen alue” as an app oxima ion o he
inal pe o mance. This app oach is o en compe i i e wi h mo e sophis ica ed lea ning
cu e ex apola ion me hods [
90
]. Despi e i s simplici y, i has been e ec i ely u ilized
in amewo ks such as Hype band [
32
]. To ensu e consis ency in compa isons, he same
ea ly-s opping ule applied o ou app oach is also applied o his baseline. The esul s o
he las -seen baseline a e depic ed in Figu es 11 and 15 using squa e ma ke s.
Ma hema ics 2025,13, 555 32 o 36
Fu u e wo k will ocus on de eloping a mo e sophis ica ed s a e-space model o lea n-
ing cu es ha cap u es he in ica e dynamics o neu al ne wo k aining. Fo ins ance,
his could in ol e modeling wi h an ex ended s ochas ic di e en ial equa ion o he o m
dy( ) = A( )y( ) + a ch(θ) + da a(X) + op (α,β)d +Σ( )dW( )(21)
which inco po a es a chi ec u e-dependen pa ame e s h ough
a ch(θ)
, which cap u es
laye complexi y, connec ion ypes, and s uc u al cha ac e is ics. Da ase dynamics a e
modeled by
da a(X)
, in eg a ing dimensionali y, en opy, and sepa abili y measu es. Op-
imiza ion p ocesses a e ep esen ed by
op (α
,
β)
, explici ly accoun ing o lea ning a e,
momen um, and adap i e op imiza ion s a egies. The noise p ocess
Σ( )dW( )
is modeled
as a co ela ed, ime-dependen s ochas ic p ocess, allowing o mo e nuanced cap u e o
empo al dependencies and noise cha ac e is ics in lea ning dynamics.
Finally, i should be no ed ha he cu en me hod does no ully exploi he po en ial
knowledge and adap a ion capabili ies o he pe o mance cu es gene a ed du ing appli-
ca ion o a new da ase . Inco po a ing ew-sho lea ning o le e age hese pe o mance
cu es in ongoing expe imen s could be ano he a ea o imp o emen .
10. Conclusions
This s udy p esen s a me hodology o enhance he e iciency o neu al a chi ec u e
sea ch (NAS) in he PHM domain h ough he in eg a ion o a pe o mance es ima o .
By le e aging he ini ial aining and alida ion pe o mance cu es, he p oposed es ima o
p edic s he inal alida ion pe o mance o a model, o e ing a subs an ial educ ion in
compu a ional cos . The expe imen s demons a e ha he app oach achie es o e 50%
ime sa ings wi h an a e age pe o mance d op o jus 2%, highligh ing i s minimal impac
on model pe o mance.
The p oposed amewo k e ec i ely add esses he ade-o be ween compu a ional
e iciency and inal pe o mance. I s obus design and adap abili y make i applicable
o a wide ange o domains beyond PHM. While he cu en wo k p o ides signi ican
insigh s, u u e esea ch should explo e u he op imiza ion o he es ima o , inco po a ion
o addi ional me a-a ibu es, and applica ion o eal-wo ld scena ios o ully ealize i s
po en ial. O e all, his me hodology con ibu es o ad ancing esou ce-e icien NAS while
main aining high-quali y model selec ion.
Au ho Con ibu ions: Concep ualiza ion, D.S.-M. and J.G.-P., me hodology D.S.-M. and J.G.-P.;
so wa e, D.S.-M.; alida ion, D.S.-M. and J.G.-P.; o mal analysis, J.B.-D.; in es iga ion, D.S.-M.,
J.G.-P. and J.B.-D.; esou ces, D.S.-M.; da a cu a ion, D.S.-M. and J.G.-P.; w i ing—o iginal d a
p epa a ion, D.S.-M., J.G.-P. and J.B.-D.; w i ing— e iew and edi ing,
D.S.-M.
, J.G.-P. and J.B.-D.; isu-
aliza ion, D.S.-M. and J.G.-P.; supe ision, J.G.-P. and J.B.-D.; p ojec adminis a ion, D.S.-M.; unding
acquisi ion, J.B.-D. All au ho s ha e ead and ag eed o he published e sion o he manusc ip .
Funding: G an PID2023-147198NB-I00 unded by MICIU/AEI/10.13039/501100011033 (Agencia
Es a al de In es igación) and by FEDER, UE, and by he Minis y o Science and Educa ion o
Spain h ough he na ional p og am “Ayudas pa a con a os pa a la o mación de in es igado es en
emp esas (DIN2019-010887/AEI/10.13039/50110001103)”, o S a e P og amme o Science Resea ch
and Inno a ions 2017–2020.
Da a A ailabili y S a emen : To ensu e he ep oducibili y and anspa ency o ou wo k, we ha e
made he sou ce code and da ase publicly a ailable. These esou ces can be accessed a he ollowing
link: h ps://gi hub.com/dasolma. The da ase s used o gene a e he cu e da ase a e all publicly
a ailable and we e accessed using he phmd Py hon package [
42
], which can be ins alled ia he
pip
package manage . Addi ionally, he gene a ed cu e da ase has been included as pa o he phmd
ool o acili a e u u e esea ch.
Ma hema ics 2025,13, 555 33 o 36
Con lic s o In e es : The au ho s decla e no con lic s o in e es .
Appendix A. Bes Pe o mance Es ima o A chi ec u es
Table A1 summa izes he a chi ec u e hype pa ame e s ha achie ed he bes ali-
da ion losses du ing c oss- alida ion. The pe o mance o hese models du ing he BO
p ocess is shown in Figu e 12, indica ed wi h s a ma ke s.
Table A1. A chi ec u e hype pa ame e s wi h bes mean alida ion loss du ing c oss- alida ion o
he pe o mance es ima o .
Obse ed Epochs Condi ioned LSTM Blocks LSTM Cells Bidi ec ional Lea ning Ra e
2 No 2 64 Yes 0.010
3 No 1 16 Yes 0.010
4 No 2 32 Yes 0.010
5 No 3 128 Yes 0.001
6 No 3 64 No 0.001
7 No 2 128 No 0.010
8 No 4 16 No 0.010
9 No 1 16 Yes 0.010
Re e ences
1.
Solís-Ma ín, D.; Galán-Páez, J.; Bo ego-Díaz, J. On he Soundness o XAI in P ognos ics and Heal h Managemen (PHM).
In o ma ion 2023,14, 256. [C ossRe ]
2. Bo ego-Díaz, J.; Galán-Páez, J. Explainable A i icial In elligence in Da a Science. Minds Mach. 2022,32, 485–531. [C ossRe ]
3.
Bousdekis, A.; Apos olou, D.; Men zas, G. P edic i e Main enance in he 4 h Indus ial Re olu ion: Bene i s, Business Oppo uni-
ies, and Manage ial Implica ions. IEEE Eng. Manag. Re . 2020,48, 57–62. [C ossRe ]
4.
Tian, Z.; Wong, L.; Sa aei, N. A neu al ne wo k app oach o emaining use ul li e p edic ion u ilizing bo h ailu e and suspension
his o ies. Mech. Sys . Signal P ocess. 2010,24, 1542–1555. [C ossRe ]
5.
Deu sch, J.; He, D. Using deep lea ning-based app oach o p edic emaining use ul li e o o a ing componen s. IEEE T ans. Sys .
Man, Cybe n. Sys . 2017,48, 11–20. [C ossRe ]
6.
Ma hew, V.; Toby, T.; Singh, V.; Rao, B.M.; Kuma , M.G. P edic ion o Remaining Use ul Li e ime (RUL) o u bo an engine using
machine lea ning. In P oceedings o he 2017 IEEE In e na ional Con e ence on Ci cui s and Sys ems (ICCS), San F ancisco, CA,
USA, 25–27 Oc obe 2017; IEEE: Pisca away, NJ, USA, 2017; pp. 306–311.
7.
Remaining Use ul Li e (RUL) P edic ion o Rolling Elemen Bea ing Using Random Fo es and G adien Boos ing Technique;
Vol. 13: Design, Reliabili y, Sa e y, and Risk. In P oceedings o he ASME In e na ional Mechanical Enginee ing Cong ess and
Exposi ion, Pi sbu gh, PA, USA, 9–15 No embe 2018.
8.
Nie o, P.G.; Ga cía-Gonzalo, E.; Lashe as, F.S.; de Cos Juez, F.J. Hyb id PSO–SVM-based me hod o o ecas ing o he emaining
use ul li e o ai c a engines and e alua ion o i s eliabili y. Reliab. Eng. Sys . Sa . 2015,138, 219–231. [C ossRe ]
9.
O dóñez, C.; Lashe as, F.S.; Roca-Pa dinas, J.; de Cos Juez, F.J. A hyb id ARIMA–SVM model o he s udy o he emaining
use ul li e o ai c a engines. J. Comp. Appl. Ma h. 2019,346, 184–191. [C ossRe ]
10.
Ng, S.S.; Xing, Y.; Tsui, K.L. A nai e Bayes model o obus emaining use ul li e p edic ion o li hium-ion ba e y. Appl. Ene gy
2014,118, 114–123. [C ossRe ]
11.
Ba aldi, P.; Di Maio, F.; Al-Dahidi, S.; Zio, E.; Mangili, F. P edic ion o indus ial equipmen emaining use ul li e by uzzy
simila i y and belie unc ion heo y. Expe Sys . Appl. 2017,83, 226–241. [C ossRe ]
12.
Zoph, B.; Vasude an, V.; Shlens, J.; Le, Q.V. Lea ning ans e able a chi ec u es o scalable image ecogni ion. In P oceedings o
he he IEEE Con e ence on Compu e Vision and Pa e n Recogni ion, Sal Lake Ci y, UT, USA, 18–23 June 2018; pp. 8697–8710.
13.
Real, E.; Agga wal, A.; Huang, Y.; Le, Q.V. Regula ized e olu ion o image classi ie a chi ec u e sea ch. In P oceedings o he
AAAI Con e ence on A i icial In elligence, Honolulu, HI, USA, 27 Janua y–1 Feb ua y 2019; Volume 33, pp. 4780–4789.
14.
Biggio, L.; Kas anis, I. P ognos ics and Heal h Managemen o Indus ial Asse s: Cu en P og ess and Road Ahead. F on . A i .
In ell. 2020,3, 578613. [C ossRe ] [PubMed]
15.
Fink, O.; Wang, Q.; S ensén, M.; De sin, P.; Lee, W.J.; Duco e, M. Po en ial, challenges and u u e di ec ions o deep lea ning in
p ognos ics and heal h managemen applica ions. Eng. Appl. A i . In ell. 2020,92, 103678. [C ossRe ]
16. Hu e , M. Lea ning Cu e Theo y. a Xi 2021, a Xi :2102.04074.
17.
Klein, A.; Falkne , S.; Sp ingenbe g, J.T.; Hu e , F. Lea ning Cu e P edic ion wi h Bayesian Neu al Ne wo ks. In P oceedings o
he In e na ional Con e ence on Lea ning Rep esen a ions, Toulon, F ance, 24–26 Ap il 2017.
Ma hema ics 2025,13, 555 34 o 36
18.
Domhan, T.; Sp ingenbe g, J.T.; Hu e , F. Speeding up au oma ic hype pa ame e op imiza ion o deep neu al ne wo ks by
ex apola ion o lea ning cu es. In P oceedings o he 24 h In e na ional Con e ence on A i icial In elligence, Tokyo, Japan,
3–7 July 2023; AAAI P ess: Washing on, DC, USA, 2023; IJCAI’15; pp. 3460–3468.
19.
Kad a, A.; Janowski, M.; Wis uba, M.; G abocka, J. Scaling Laws o Hype pa ame e Op imiza ion. In Ad ances in Neu al
In o ma ion P ocessing Sys ems; Oh, A., Naumann, T., Globe son, A., Saenko, K., Ha d , M., Le ine, S., Eds.; Cu an Associa es,
Inc.: San F ancisco, CA, USA, 2023; Volume 36, pp. 47527–47553.
20. Tissue, H.; Wang, V.; Wang, L. Scaling Law wi h Lea ning Ra e Annealing. a Xi 2024, a Xi :2408.11029.
21.
Clause , A.; Shalizi, C.R.; Newman, M.E.J. Powe -Law Dis ibu ions in Empi ical Da a. SIAM Re . 2009,51, 661–703. [C ossRe ]
22.
Hes ness, J.; Na ang, S.; A dalani, N.; Diamos, G.; Sengup a, S.; Ande son, T.; Pa wa y, M.M.A.; Yang, Y.; Smelyanskiy, M. Deep
lea ning scaling is p edic able, empi ically. a Xi 2017, a Xi :1712.00409.
23.
Smi h, L.N. A disciplined app oach o neu al ne wo k hype -pa ame e s: Pa 1–lea ning a e, ba ch size, momen um, and
weigh decay. a Xi 2018, a Xi :1803.09820.
24.
Mand , S.; Ho man, M.D.; Blei, D.M. S ochas ic g adien descen as app oxima e Bayesian in e ence. J. Mach. Lea n. Res. 2017,
18, 4873–4907.
25.
Chaudha i, P.; Cho omanska, A.; Soa o, S.; LeCun, Y.; Baldassi, C.; Bo gs, C.; Chayes, J.; Sagun, L.; Zecchina, R. En opy-SGD:
Biasing g adien descen in o wide alleys. J. S a . Mech. Theo y Exp. 2019,2019, 124018. [C ossRe ]
26.
B oga -Mo e, L.; Bonalli, R.; Rudi, A. Lea ning Con olled S ochas ic Di e en ial Equa ions. a Xi 2024, a Xi :s a .ML/2411.01982.
27. I ô, K. On S ochas ic P ocesses. Ann. Ma h. 1944,45, 202–220.
28.
Kandasamy, K.; Neiswange , W.; Schneide , J.; Poczos, B.; Xing, E.P. Neu al a chi ec u e sea ch wi h bayesian op imisa ion and
op imal anspo . Ad . Neu al In . P ocess. Sys . 2018,31, 2020–2029.
29.
Snoek, J.; La ochelle, H.; Adams, R.P. P ac ical bayesian op imiza ion o machine lea ning algo i hms. Ad . Neu al In . P ocess.
Sys . 2012,25, 2951–2959.
30.
Solís-Ma ín, D.; Galán-Páez, J.; Bo ego-Díaz, J. Bayesian Model Selec ion P uning in P edic i e Main enance. In P oceedings
o he Hyb id A i icial In elligen Sys ems: 19 h In e na ional Con e ence, HAIS 2024, Salamanca, Spain, 9–11 Oc obe 2024;
P oceedings, Pa I; Sp inge : Be lin/Heidelbe g, Ge many, 2024; pp. 263–274. [C ossRe ]
31.
Bake , B.; Gup a, O.; Raska , R.; Naik, N. Accele a ing Neu al A chi ec u e Sea ch using Pe o mance P edic ion. In P oceedings
o In e na ional Con e ence on Lea ning Rep esen a ions (ICLR 2018)-Wo kshop T ack, Camb idge, MA, USA, 2018. A ailable
online: h ps://open e iew.ne /pd ?id=BJypUGZ0Z (accessed on 1 Decembe 2024).
32.
Li, L.; Jamieson, K.G.; DeSal o, G.; Ros amizadeh, A.; Talwalka , A. Hype band: Bandi -based con igu a ion e alua ion o
hype pa ame e op imiza ion. In P oceedings o he ICLR (Pos e ), Toulon, F ance, 24–26 Ap il 2017; p. 53.
33.
Falkne , S.; Klein, A.; Hu e , F. BOHB: Robus and e icien hype pa ame e op imiza ion a scale. In P oceedings o he
In e na ional Con e ence on Machine Lea ning, PMLR, S ockholm, Sweden, 10–15 July 2018; pp. 1437–1446.
34.
Dai, Z.; Yu, H.; Low, B.K.H.; Jaille , P. Bayesian op imiza ion mee s Bayesian op imal s opping. In P oceedings o he In e na ional
Con e ence on Machine Lea ning, PMLR, Long Beach, CA, USA, 9–15 June 2019; pp. 1496–1506.
35.
Ad iaensen, S.; Rako oa ison, H.; Mülle , S.; Hu e , F. E icien bayesian lea ning cu e ex apola ion using p io -da a i ed
ne wo ks. Ad . Neu al In . P ocess. Sys . 2024,36, 19858–19886.
36.
Egele, R.; Moh , F.; Vie ing, T.; Balap akash, P. The un easonable e ec i eness o ea ly disca ding a e one epoch in neu al
ne wo k hype pa ame e op imiza ion. Neu ocompu ing 2024,597, 127964. [C ossRe ]
37.
Rako oa ison, H.; Ad iaensen, S.; Mallik, N.; Ga ibo , S.; Be gman, E.; Hu e , F. In-Con ex F eeze-Thaw Bayesian Op imiza ion
o Hype pa ame e Op imiza ion. a Xi 2024, a Xi :2404.16795.
38.
Ruhkop , T.; Mohan, A.; Deng, D.; To nede, A.; Hu e , F.; Lindaue , M. Masi : Me a-lea ned algo i hm selec ion using implici
ideli y in o ma ion. T ans. Mach. Lea n. Res. 2023, 2835–8856.
39.
Wu, K.; Kim, K.; Ga ne , R.; Ga dne , J.R. The Beha io and Con e gence o Local Bayesian Op imiza ion. a Xi 2024,
a Xi :2305.15572.
40.
Jones, D.R.; Schonlau, M.; Welch, W.J. E icien Global Op imiza ion o Expensi e Black-Box Func ions. J. Glob. Op im. 1998,
13, 455–492. [C ossRe ]
41. Bull, A.D. Con e gence Ra es o E icien Global Op imiza ion Algo i hms. J. Mach. Lea n. Res. 2011,12, 2879–2904.
42.
Solís-Ma ín, D.; Galán-Páez, J.; Bo ego-Díaz, J. PHMD: An easy da a access ool o p ognosis and heal h managemen da ase s.
So wa eX 2025,29, 102039. [C ossRe ]
43. Bole, B.; Kulka ni, C.; Daigle, M. Randomized ba e y usage da a se . NASA AMES P ogn. Da a Repos. 2014,70 .
44.
Saha, B.; Goebel, K. Ba e y Da a Se . NASA AMES P ognos ics Da a Reposi o y. 2007. A ailable online: h p:// i.a c.nasa.go /
p ojec /p ognos ic-da a- eposi o y (accessed on 1 Decembe 2024).
45.
Capaci o Elec ical S ess-2–Ca alog—ca alog.da a.go . 2024. A ailable online: h ps://ca alog.da a.go /da ase /capaci o -
elec ical-s ess-2 (accessed on 8 Ap il 2024).
Ma hema ics 2025,13, 555 35 o 36
46.
Celaya, J.R.; Saxena, A.; Saha, S.; Goebel, K.F. P ognos ics o powe MOSFETs unde he mal s ess accele a ed aging using
da a-d i en and model-based me hodologies. In P oceedings o he Annual Con e ence o he PHM Socie y, Shenzhen, China,
24–25 May 2011; Volume 3, pp. 1–10.
47.
F icke, K.; Nascimen o, R.; Co be a, M.; Kulka ni, C.; Viana, F. P ognosis o Li-ion Ba e ies Unde La ge Load Va ia ions Using
Hyb id Physics-In o med Neu al Ne wo ks. In P oceedings o he Annual Con e ence o he PHM Socie y, Sal Lake Ci y, UT,
USA, 28 Oc obe –2 No embe 2023; Volume 15, pp. 1–12.
48.
Saxena, A.; Goebel, K.; Simon, D.; Eklund, N. Damage p opaga ion modeling o ai c a engine un- o- ailu e simula ion. In
P oceedings o he 2008 In e na ional Con e ence on P ognos ics and Heal h Managemen , Den e , CO, USA, 6–9 Oc obe 2008;
IEEE: Pisca away, NJ, USA, 2008; pp. 1–9.
49.
P ognos ics Cen e o Excellence Da a Se Reposi o y—NASA—nasa.go . 2024. A ailable online: h ps://www.nasa.go /
in elligen -sys ems-di ision/disco e y-and-sys ems-heal h/pcoe/pcoe-da a-se - eposi o y/ (accessed on 8 Ap il 2024).
50.
Kulka ni, C.S.; Gua ne os Luna, A. Desc ip ion o Simula ed Small Sa elli e Ope a ion Da a Se s; Technical Repo ; NASA: Washing on,
DC, USA, 2018.
51.
Kulka ni, C.; Hogge, E.; Goebel, K. HIRF Ba e y Da a Se . NASA Ames P ognos ics Da a Reposi o y. A ailable online:
h ps://www.nasa.go /in elligen -sys ems-di ision/disco e y-and-sys ems-heal h/pcoe/pcoe-da a-se - eposi o y/ (accessed
on 8 Ap il 2024).
52.
A ias Chao, M.; Kulka ni, C.; Goebel, K.; Fink, O. Ai c a engine un- o- ailu e da ase unde eal ligh condi ions o p ognos ics
and diagnos ics. Da a 2021,6, 5. [C ossRe ]
53. Seijun Chung, S. Da a Challenge a PHM Asia Paci ic 2021. Kaggle. 2021. A ailable online: h ps://kaggle.com/compe i ions/
phmap21-classi ica ion- ask (accessed on 2 Oc obe 2024).
54.
Danilo Gio dano, M.S. PHME Da a Challenge. 2024. [Eu opean Con e ence o he P ognos ics and Heal h Managemen Socie y].
A ailable online: h ps://gi hub.com/PHME-Da achallenge/Da a-Challenge-2024 (accessed on 3 Oc obe 2024).
55.
2010 PHM Socie y Con e ence Da a Challenge–PHM Socie y—phmsocie y.o g. 2024. A ailable online: h p://www.phmsocie y.
o g/compe i ion/phm/10 (accessed on 8 Ap il 2024).
56.
2023, P.A.P. Da a Challenge a PHM Asia Paci ic 2023. 2023. A ailable online: h ps://phmap.jp/p og am-da a/ (accessed on
3 Sep embe 2024).
57.
Annual Con e ence o he P ognos ics and Heal h Managemen Socie y 2018—PHM Socie y—phmsocie y.o g. 2024. A ailable
online: h p://www.phmsocie y.o g/compe i ion/phm/18 (accessed on 8 Ap il 2024).
58.
˙
Ince, K.; Si keci, E.; Genç, Y. Remaining Use ul Li e P edic ion o Expe imen al Fil a ion Sys em: A Da a Challenge. PHM Soc.
Eu . Con . 2020,5, 1–10. [C ossRe ]
59.
Faul Da a Se s—Socie y Fo Machine y Failu e P e en ion Technology—m p .o g. 2024. A ailable online: h ps://www.m p .
o g/ aul -da a-se s/ (accessed on 8 Ap il 2024).
60.
He, J.; Guan, X.; Peng, T.; Liu, Y.; Saxena, A.; Celaya, J.; Goebel, K. A mul i- ea u e in eg a ion me hod o a igue c ack de ec ion
and c ack leng h es ima ion in i e ed lap join s using Lamb wa es. Sma Ma e . S uc . 2013,22, 105007. [C ossRe ]
61.
Bosello, M.; Falcome , C.; Rossi, C.; Pau, G. To Cha ge o o Sell? EV Pack Use ul Li e Es ima ion ia LSTMs, CNNs, and
Au oencode s. Ene gies 2023,16, 2837. [C ossRe ]
62.
Cao, P.; Zhang, S.; Tang, J. P ep ocessing- ee gea aul diagnosis using small da ase s wi h deep con olu ional neu al ne wo k-
based ans e lea ning. IEEE Access 2018,6, 26241–26253. [C ossRe ]
63.
Lu, C.; Wang, Y.; Ragulskis, M.; Cheng, Y. Faul diagnosis o o a ing machine y: A me hod based on image p ocessing. PLoS
ONE 2016,11, e0164111. [C ossRe ]
64.
Kollmeye , P.; Ca los, V.; Naguib, M.; Skells, M. LG 18650HG2 Li-Ion Ba e y Da a and Example Deep Neu al Ne wo k xEV SOC
Es ima o Sc ip . 2020. Mendeley Da a. A ailable online: h ps://da a.mendeley.com/da ase s/cp3473x7x /3 (accessed on
3 Oc obe 2024).
65. Gea Da ase s. 2024. A ailable online: h ps://www.kau-sdol.com/kaug (accessed on 8 Ap il 2024).
66.
Wang, B.; Lei, Y.; Li, N.; Li, N. A hyb id p ognos ics app oach o es ima ing emaining use ul li e o olling elemen bea ings.
IEEE T ans. Reliab. 2018,69, 401–412. [C ossRe ]
67.
Bea ing Da a Cen e | Case School o Enginee ing|Case Wes e n Rese e Uni e si y—enginee ing.case.edu. 2024. A ailable
online: h ps://enginee ing.case.edu/bea ingda acen e (accessed on 8 Ap il 2024).
68.
So o-Ocampo, C.R.; Ma o o, J.; Cano-Mo eno, J.D.; Me a, J.M. Bea ing Da abase—E alua ion o Isola ed Cases—zenodo.o g.
2023. A ailable online: h ps://zenodo.o g/ eco ds/8241764 (accessed on 8 Ap il 2024).
69.
Nehasil, O.; Dobiášo á, L.; Mazanec, V.; Ši ok
`
y, J. Ve sa ile AHU aul de ec ion–Design, ield alida ion and p ac ical applica ion.
Ene gy Build. 2021,237, 110781. [C ossRe ]
70.
Lessmeie , C.; Kimo ho, J.K.; Zimme , D.; Sex o, W. Condi ion moni o ing o bea ing damage in elec omechanical d i e sys ems
by using mo o cu en signals o elec ic mo o s: A benchma k da a se o da a-d i en classi ica ion. In P oceedings o he PHM
Socie y Eu opean Con e ence, Bilbao, Spain, 5–8 July 2016; Volume 3, pp. 1–10.
Ma hema ics 2025,13, 555 36 o 36
71.
Co addu, A.; One o, L.; Ghio, A.; Sa io, S.; Angui a, D.; Figa i, M. Machine lea ning app oaches o imp o ing condi ion-based
main enance o na al p opulsion plan s. P oc. Ins . Mech. Eng. Pa M J. Eng. Ma i . En i on. 2016,230, 136–153. [C ossRe ]
72.
Cipollini, F.; One o, L.; Co addu, A.; Mu phy, A.J.; Angui a, D. Condi ion-based main enance o na al p opulsion sys ems wi h
supe ised da a analysis. Ocean Eng. 2018,149, 268–278. [C ossRe ]
73.
Yang, X.; Yan, R.; Gao, R.X. Induc ion mo o aul diagnosis using mul iple class ea u e selec ion. In P oceedings o he 2015
IEEE In e na ional Ins umen a ion and Measu emen Technology Con e ence (I2MTC) P oceedings, Pisa, I aly, 11–14 May 2015;
IEEE: Pisca away, NJ, USA, 2015; pp. 256–260.
74.
Bi kl, C. Diagnosis and P ognosis o Deg ada ion in Li hium-Ion Ba e ies. Ph.D. Thesis, Uni e si y o Ox o d, Ox o d, UK, 2017.
75.
Li, K.; Ping, X.; Wang, H.; Chen, P.; Cao, Y. Sequen ial uzzy diagnosis me hod o mo o olle bea ing in a iable ope a ing
condi ions based on ib a ion analysis. Senso s 2013,13, 8013–8041. [C ossRe ]
76.
Ba bie i, G.; Sanchez-Londoño, D.; Ca aneo, L.; Fumagalli, L.; Rome o, D. A case s udy o p oblem-based lea ning educa ion in
aul diagnosis assessmen . IFAC-Pape sOnLine 2020,53, 107–112. [C ossRe ]
77.
Se e son, K.A.; A ia, P.M.; Jin, N.; Pe kins, N.; Jiang, B.; Yang, Z.; Chen, M.H.; Aykol, M.; He ing, P.K.; F aggedakis, D.; e al.
Da a-d i en p edic ion o ba e y cycle li e be o e capaci y deg ada ion. Na . Ene gy 2019,4, 383–391. [C ossRe ]
78. So o-Ocampo, C.R.; Me a, J.M.; Cano-Mo eno, J.D.; Ga cia-Be na do, J.L. Low-cos , high- equency, da a acquisi ion sys em o
condi ion moni o ing o o a ing machine y h ough ib a ion analysis-case s udy. Senso s 2020,20, 3493. [C ossRe ]
79.
Ko alenko, I.; Saez, M.; Ba on, K.; Tilbu y, D. SMART: A sys em-le el manu ac u ing and au oma ion esea ch es bed. Sma
Sus ain. Manu . Sys . 2017,1, 20170006. [C ossRe ]
80.
Lei, M.; Jackson, L. IEEE 2014 Da a Challenge Da a. 2014. A ailable online: h ps:// eposi o y.lbo o.ac.uk/a icles/da ase /
IEEE_2014_Da a_Challenge_Da a/3518141/1? ile=5604290 (accessed on 1 Oc obe 2024).
81.
Von Bi gelen, A.; Bu a i, D.; Mage , J.; Niggemann, O. Sel -o ganizing maps o anomaly localiza ion and p edic i e main enance
in cybe -physical p oduc ion sys ems. P ocedia Ci p 2018,72, 480–485. [C ossRe ]
82.
UCI Machine Lea ning Reposi o y—a chi e.ics.uci.edu. 2024. A ailable online: h ps://a chi e.ics.uci.edu/da ase /421/aps+
ailu e+a +scania+ ucks (accessed on 8 Ap il 2024).
83.
Da ban, Z.Z.; Webb, G.I.; Pan, S.; Agga wal, C.C.; Salehi, M. Deep lea ning o ime se ies anomaly de ec ion: A su ey. a Xi
2022, a Xi :2211.05244.
84.
Helwig, N.; Pignanelli, E.; Schü ze, A. Condi ion moni o ing o a complex hyd aulic sys em using mul i a ia e s a is ics. In
P oceedings o he 2015 IEEE In e na ional Ins umen a ion and Measu emen Technology Con e ence (I2MTC) P oceedings,
Pisa, I aly, 11–14 May 2015; IEEE: Pisca away, NJ, USA, 2015; pp. 210–215.
85.
Ba e y Da a|Cen e o Ad anced Li e Cycle Enginee ing—calce.umd.edu. 2024. A ailable online: h ps://calce.umd.edu/
ba e y-da a (accessed on 8 Ap il 2024).
86.
Lee, J.; Qiu, H.; Yu, G.; Lin, J.; Rexno d, I.M.S. Uni e si y o Cincinna i. Bea ing Da a Se , NASA Ames P ognos ics Da a Reposi o y;
NASA Ames Resea ch Cen e : Mo e Field, CA, USA, 2007.
87.
Noguei a, F. Bayesian Op imiza ion: Open Sou ce Cons ained Global Op imiza ion Tool o Py hon. 2014. A ailable online:
h ps://gi hub.com/ m n/BayesianOp imiza ion (accessed on 3 Oc obe 2024).
88.
You, K.; Long, M.; Wang, J.; Jo dan, M.I. How does lea ning a e decay help mode n neu al ne wo ks? a Xi 2019,
a Xi :1908.01878.
89. Kingma, D.P.; Ba, J. Adam: A me hod o s ochas ic op imiza ion. a Xi 2014, a Xi :1412.6980.
90.
Ga giani, M.; Klein, A.; Falkne , S.; Hu e , F. P obabilis ic ollou s o lea ning cu e ex apola ion ac oss hype pa ame e
se ings. a Xi 2019, a Xi :1910.04522.
91.
Wang, J.; Wang, H.; Pe a, C.G.; Chiang, N.Y. On he con e gence o noisy Bayesian Op imiza ion wi h Expec ed Imp o emen .
a Xi 2025, a Xi :2501.09262.
92.
Ai alio is, P.; Geo goulias, K.; Ch yssolou is, G. The use o Digi al Twin o p edic i e main enance in manu ac u ing. In . J.
Compu . In eg . Manu . 2019,32, 1067–1080. [C ossRe ]
Disclaime /Publishe ’s No e: The s a emen s, opinions and da a con ained in all publica ions a e solely hose o he indi idual
au ho (s) and con ibu o (s) and no o MDPI and/o he edi o (s). MDPI and/o he edi o (s) disclaim esponsibili y o any inju y o
people o p ope y esul ing om any ideas, me hods, ins uc ions o p oduc s e e ed o in he con en .
