Mehdiye , Nija ; Majla ow, Maxim; Fe ke, Pe e
A icle — Published Ve sion
Quan i ying and explaining machine lea ning unce ain y
in p edic i e p ocess moni o ing: an ope a ions esea ch
pe spec i e
Annals o Ope a ions Resea ch
P o ided in Coope a ion wi h:
Sp inge Na u e
Sugges ed Ci a ion: Mehdiye , Nija ; Majla ow, Maxim; Fe ke, Pe e (2024) : Quan i ying and
explaining machine lea ning unce ain y in p edic i e p ocess moni o ing: an ope a ions esea ch
pe spec i e, Annals o Ope a ions Resea ch, ISSN 1572-9338, Sp inge US, New Yo k, NY, Vol. 347,
Iss. 2, pp. 991-1030,
h ps://doi.o g/10.1007/s10479-024-05943-4
This Ve sion is a ailable a :
h ps://hdl.handle.ne /10419/323301
S anda d-Nu zungsbedingungen:
Die Dokumen e au EconS o dü en zu eigenen wissenscha lichen
Zwecken und zum P i a geb auch gespeiche und kopie we den.
Sie dü en die Dokumen e nich ü ö en liche ode komme zielle
Zwecke e iel äl igen, ö en lich auss ellen, ö en lich zugänglich
machen, e eiben ode ande wei ig nu zen.
So e n die Ve asse die Dokumen e un e Open-Con en -Lizenzen
(insbesonde e CC-Lizenzen) zu Ve ügung ges ell haben soll en,
gel en abweichend on diesen Nu zungsbedingungen die in de do
genann en Lizenz gewäh en Nu zungs ech e.
Te ms o use:
Documen s in EconS o may be sa ed and copied o you pe sonal
and schola ly pu poses.
You a e no o copy documen s o public o comme cial pu poses, o
exhibi he documen s publicly, o make hem publicly a ailable on he
in e ne , o o dis ibu e o o he wise use he documen s in public.
I he documen s ha e been made a ailable unde an Open Con en
Licence (especially C ea i e Commons Licences), you may exe cise
u he usage igh s as speci ied in he indica ed licence.
h p://c ea i ecommons.o g/licenses/by/4.0/
Annals o Ope a ions Resea ch (2025) 347:991–1030
h ps://doi.o g/10.1007/s10479-024-05943-4
ORIGINAL RESEARCH
Quan i ying and explaining machine lea ning unce ain y in
p edic i e p ocess moni o ing: an ope a ions esea ch
pe spec i e
Nija Mehdiye 1,2 ·Maxim Majla ow1,2 ·Pe e Fe ke1,2
Recei ed: 3 Ap il 2023 / Accep ed: 11 Ma ch 2024 / Published online: 3 Ap il 2024
© The Au ho (s) 2024
Abs ac
In he apidly e ol ing landscape o manu ac u ing, he abili y o make accu a e p edic ions
is c ucial o op imizing p ocesses. This s udy in oduces a no el amewo k ha combines
p edic i e unce ain y wi h explana o y mechanisms o enhance decision-making in com-
plex sys ems. The app oach le e ages Quan ile Reg ession Fo es s o eliable p edic i e
p ocess moni o ing and inco po a es Shapley Addi i e Explana ions (SHAP) o iden i y he
d i e s o p edic i e unce ain y. This dual- ace ed s a egy se es as a aluable ool o
domain expe s engaged in p ocess planning ac i i ies. Suppo ed by a eal-wo ld case s udy
in ol ingamedium-sized Ge manmanu ac u ing i m, hea icle alida es he model’s e ec-
i eness h ough igo ous e alua ions, including sensi i i y analyses and es s o s a is ical
signi icance. By seamlessly in eg a ing unce ain y quan i ica ion wi h explainable a i icial
in elligence, his esea ch makes a no el con ibu ion o he e ol ing discou se on in elligen
decision-making in complex sys ems.
Keywo ds Explainable a i icial in elligence (XAI) ·Unce ain y quan i ica ion (UQ) ·
P edic i e p ocess moni o ing ·In o ma ion sys ems (IS)
1 In oduc ion
In oday’s highly compe i i e and complex business en i onmen , o ganiza ions a e unde
cons an p essu e o op imize hei pe o mance and decision-making p ocesses. Acco ding
o He be Simon, enhancing o ganiza ional pe o mance elies on e ec i ely channeling
BNija Mehdiye
nija .mehdiye[email p o ec ed]
Maxim Majla ow
maxim.majla o[email p o ec ed]
Pe e Fe ke
pe e [email p o ec ed]
1Ge man Resea ch Cen e o A i icial In elligence (DFKI), Campus D 3.2, 66123 Saa b ücken,
Saa land, Ge many
2Saa land Uni e si y, Campus D 3.2, Saa b ücken 66123, Saa land, Ge many
123
992 Annals o Ope a ions Resea ch (2025) 347:991–1030
ini e human a en ion owa ds c i ical da a o decision-making, necessi a ing he in eg a-
ion o in o ma ion sys ems (IS), a i icial in elligence (AI) and ope a ions esea ch (OR)
insigh s (Simon, 1997). Recen OR esea ch p o ides e idence in suppo o his p oposi-
ion, as he discipline has wi nessed a ans o ma ion due o he abundan a ailabili y o
ich and oluminous da a om a ious sou ces coupled wi h ad ances in machine lea ning
(ML) (F azze o e al., 2019). As o la e, heigh ened academic a en ion has been de o ed o
p esc ip i e analy ics, a discipline ha sugges s combining he esul s o p edic i e analy ics
wi h op imiza ion echniques in a p obabilis ic amewo k o gene a e esponsi e, au oma ed,
es ic ed, ime-sensi i e, and ideal decisions (Lepenio i e al., 2020).
The inc easing p ominence o da a-d i en solu ions in ope a ional decision-making is e i-
den , pa icula ly in he domain o manu ac u ing in elligence (Mehdiye and Fe ke, 2021).
One o he pi o al applica ions o his end is he use o p edic i e analy ics o p oduc ion
planning and scheduling. Howe e , o ha ness i s ull po en ial o suppo p oduc ion plan-
ne s, ce ain limi a ions mus be add essed. One o he signi ican gaps is ha he majo i y o
s udies p ima ily concen a e on he p edic ion o non- echnical pa ame e s, such as demand
and supply luc ua ions. These pa ame e s o en se e as cons ain s o a e in eg a ed in o
he objec i e unc ion o selec ed op imiza ion amewo ks. On he o he hand, echnical
aspec s inhe en o he p oduc ion p ocess—like ope a ional yield, p oduc ion lead ime,
quali y conce ns, and po en ial sys em ailu es— emain unde explo ed (Chaa i e al., 2014).
This disc epancy can be a ibu ed o he sca ci y o pe inen da a om in o ma ion sys ems
a he shop loo le el. As a esul , many cu en es ima ions ega ding echnical pa ame e s
a e g ounded mo e in in ui ion o assump ions a he han conc e e da a, leading o esul s
ha o en all sho o op imal.
Ano he conside able gap in he ield pe ains o he ou pu p oduced du ing he p edic-
i e analy ics s age. Upon close examina ion o s udies ha in eg a e da a-d i en pa ame e
es ima ion be o e op imiza ion, i becomes appa en ha hey o e whelmingly p oduce poin
o ecas s (Mi en sis and Lens, 2022). This app oach, howe e , leads o he applica ion o
de e minis ic op imiza ion me hods, which may no ully cap u e he complexi ies and unce -
ain ies o eal-wo ld scena ios. Despi e he exis ence o nume ous op imiza ion me hods ha
conside unce ain y, as p oposed in Mula e al. (2006), hei in eg a ion wi h he p eceding
p edic i e analy ics s age has ye o be es ablished. Consequen ly, a ising demand exis s o
he p oduc ion o ML ou pu s ha can p ecisely and comp ehensi ely cap u e and quan i y
he p edic i e unce ain y wi hin he speci ic ope a ional esea ch con ex being examined.
Las ly, e en i unce ain y associa ed wi h he op imiza ion pa ame e o in e es can be
es ima ed, a mo e ac ionable and ad an ageous app oach would in ol e explaining i s unde -
lying sou ce. This can be accomplished h ough an explainable a i icial in elligence (XAI)
app oach ha iden i ies inpu pa e ns ha lead o unce ain p edic ions. By pinpoin ing he
speci ic inpu ea u es con ibu ing o p edic i e unce ain y, p ac i ione s can gain insigh s
in o egions whe e aining da a is spa se o whe e speci ic ea u es exhibi anomalous beha -
io (An o án e al., 2020). These insigh s would ale necessa y adjus men s o he model’s
decision-making p ocess o ou comes p io o i s subsequen ope a ionaliza ion.
To add ess he h ee iden i ied gaps, we p opose a mul i-s age ML app oach ha inco -
po a es unce ain y awa eness and explainabili y. We demons a e he e ec i eness o his
app oach h ough i s applica ion o a eal-wo ld p oduc ion planning scena io. The con i-
bu ion o his s udy is mul i ace ed. To add ess he i s gap, we use a supe ised lea ning
app oach o p obabilis ically es ima e a p oduc ion- ela ed pa ame e , speci ically he p o-
cessing ime o p oduc ion e en s. To achie e his objec i e, we employ p ocess e en da a
sou ced om manu ac u ing execu ion sys ems (MES). These sys ems a e p ocess-awa e
in o ma ion sys ems (PAIS) ha acili a e he coo dina ion o unde lying ope a ional p o-
123
Annals o Ope a ions Resea ch (2025) 347:991–1030 993
cesses and cap u e he digi al oo p in s o p ocess e en s du ing execu ion. The esul ing
e en log consis s o sequen ially eco ded e en s associa ed wi h a pa icula case, along
wi h a ious a ibu es such as imes amps, esou ces (human o machine) esponsible o
p ocess execu ion, and o he case-speci ic de ails. To be mo e p ecise, he p oblem a hand
is o mula ed as a p edic i e p ocess moni o ing p oblem. This necessi a es he u iliza ion
o speci ic p e-p ocessing, encoding, and ea u e enginee ing echniques o accoun o he
inhe en business and ope a ional p ocess da a equi emen s.
To ackle he second gap, we u ilize Quan ile Reg ession Fo es s (QRF), an ML me hod
de eloped speci ically o es ima e condi ional quan iles o high-dimensional p edic o a i-
ables (Meinshausen, 2006). QRF is an ex ension o he adi ional andom o es s echnique,
o e ing a non-pa ame ic and p ecise app oach o es ima ing p edic ion in e als. These
p edic ion in e als p o ide aluable in o ma ion on he unce ain y o model ou comes,
allowing o a be e unde s anding o he p edic i e powe o he model and i s limi a ions.
To b idge he hi d gap, we o e an explana ion o he main d i e s o unce ain y by
examining he impac o ea u e alues on p edic ion in e als. To accomplish his, we u ilize
local and global pos -hoc explana ions using SHapley Addi i e Explana ions (SHAP) (Lund-
be g and Lee, 2017). Ou app oach di e s om he s a e-o - he-a use o his echnique in
ha we use he p edic ion in e al wid h as he ou pu , which p o ides a di ec explana ion
o ea u e a ibu ions o unce ain y. Fu he mo e, we e ine ou explana ions on a g anula
le el, such as o di e en unce ain y p o iles o indi idual p oduc ion ac i i ies, esul ing
in a mo e nuanced unde s anding o he unde lying d i e s o unce ain y.
The emainde o his pape iss uc u edas ollows:Sec .2ou lines he eal-wo ldscena io
ha se es as he backd op o ou p oposed me hod. Sec ion3de ails he co e me hodology,
while Sec .4desc ibes he expe imen al design and e alua ion me ics. Sec ion5p o ides an
exhaus i e analysis o ou me hod’s pe o mance. This is ollowed by Sec .6, which del es
in o he p ac ical and scien i ic implica ions o hese esul s. Sec ion7 e iews pe inen
li e a u e, and Sec .8o e s he inal ema ks.
2 Mo i a ing usage scena io
In his sec ion, we ou line he p oduc ion p ocesses ele an o he collec ed da a and cla i y
how he ecommended me hodology is applied in p ac ice. This se s he s age o unde -
s anding he con ex . Impo an ly, he sugges ed app oach o p ocess p edic ion, which
inco po a es bo h unce ain y and explainabili y, is adap able ac oss a ious planning sce-
na ios. This case s udy is pa o a join esea ch p ojec wi h a medium-sized Ge man
manu ac u e specializing in cus om and s anda dized essel componen s. The p oduc ion
p ocess in ol es mul iple s ages and u ilizes ma e ials such as s ainless s eel, aluminum, and
ca bon s eel, equi ing specialized equipmen and expe ise.
A he s a o he manu ac u ing p ocess, cus ome o de s a e sou ced om he pa -
ne ’s p oduc ca alog. Once an o de is ecei ed, he manu ac u ing i m assesses i s p io i y
and de e mines he equi ed sequence o p oduc ion ac i i ies, which may be ei he p ese
o sligh ly adjus ed based on he cus ome ’s speci ica ions. These speci ica ions encompass
a a ie y o a ibu es, such as a icle g oup iden i ie , ma e ial g oup iden i ie , weigh ,
bend adius, base diame e , shee wid h, quan i y, and welding speci ica ions. These ac-
o s signi ican ly impac he ime needed o each p oduc ion ac i i y. Despi e possessing a
sequence o ac i i ies o each cus ome o de , p ocess expe s p esen ly depend on in ui ion
123
994 Annals o Ope a ions Resea ch (2025) 347:991–1030
Table 1 P oduc ion p ocess e en log
Case S a End Diame e Wo ke
N Ac i i y Time Time Base ... ID ...
162384 Plasma 2019-04-18 2019-04-18 1800 .. 409
Welding 06:26:47 09:51:25
162384 G inding 2019-04-18 2019-04-18 1800 .. 108
Weld. Seam 12:11:30 19:07:14
162384 Dishing 2019-04-23 2019-04-23 1800 .. 150
P ess (2) 10:50:31 18:34:11
162384 Bead 2019-04-24 2019-04-24 1800 .. 726
Small 10:20:13 19:57:45
162384 X-Ray 2019-04-25 2019-04-25 1800 .. 703
Examina ion 10:26:23 10:26:32
162384 Edge 2019-04-26 2019-04-26 1800 .. 742
Debu ing 09:08:38 17:50:27
.. .. .. .. .. .. .. ..
177566 3D Mic o- 2021-06-21 2021-06-21 3680 .. 139 ..
s ep Ci cle 07:04:38 10:26:37 .. ..
.. .. .. .. .. .. .. ..
o expe ience-based es ima ions o asce ain hei du a ion. This inabili y o quan i y his
i al ime-speci ic p oduc ion pa ame e esul s in subop imal planning ou comes.
To add ess his issue, he pa ne has implemen ed an MES solu ion o cap u e he p ocess
execu ion de ails o p oduc ion ac i i ies o each cus ome o de . In ou use case, he p ocess
da a adhe es o a pa icula s uc u e. Each cus ome o de is ep esen ed by a case wi h a
unique case iden i ie , and a p ocess case comp ises he causal and empo al sequence o
se e al e en s ela ed o he p oduc ion o he co esponding cus ome o de . A p ocess
e en encompasses he ac i i y desc ibing he p oduc ion s ep execu ed, he s a and end
imes amps o execu ion, case a ibu es such as cus ome o de speci ica ions de ailed abo e,
and e en -speci ic a ibu es like machine o human esou ces esponsible. The examined use
case in ol es 30 dis inc ac i i ies, including o ming o ma e ial on dishing p esses, manual
welding, plasma welding, su ace g inding, manual sanding, debu ing o edges, e c. All
p ocess execu ion da a o his o ical cus ome o de s a e expo ed and s o ed in an e en log.
Table 1p esen s an exce p om he e en log o illus a i e pu poses.
Using his o ical e en da a, expe s can now accu a ely calcula e he du a ion o each
p oduc ion ac i i y, also known as e en p ocessing ime. This is done by measu ing he ime
di e ence be ween he s a and end imes amps o each ac i i y. In a highly compe i i e
ma ke , p ecise es ima ion o hese p ocessing imes is c ucial o e ec i e planning. Wi h
his in o ma ion, expe s can o ecas he o al p oduc ion ime needed o comple e an en i e
o de . To be e unde s and how e en p ocessing ime ela es o o he ime me ics com-
monly discussed in he ield, please e e o Fig.1. In his con ex , e en p ocessing ime is a
componen o he o e all cycle ime, including wai ing o idle pe iods.
Upon iden i ying he a ge pa ame e o in e es , p obabilis ic machine lea ning solu-
ions should be employed o gene a e da a-d i en es ima ions and co esponding unce ain y
in o ma ion. This is supplemen ed wi h ele an explana ion mechanisms, allowing use s o
123
Annals o Ope a ions Resea ch (2025) 347:991–1030 995
Fig. 1 Rela ionship o e en p ocessing ime wi h o he ime concep s as desc ibed in Dumas e al. (2018)
comp ehend he model unce ain y ela ed o indi idual ac i i y du a ions. The unce ain y-
awa e ou pu s a e hen u ilized as inpu o decision augmen a ion scena ios o as inpu o
he adop ed op imiza ion app oach o gene a e p oduc ion plans.
3 Me hodology
The s udy in oduces a me hod ha in eg a es IS and AI o add ess challenges in OR. Mo e
speci ically, he p ima y objec i e is o de elop an ML-based solu ion using p ocess e en
da a om ele an in o ma ion sys ems, namely MES (see Fig.2). Impo an conside a ions
in his ega d include he need o quan i y unce ain y and ensu e explainabili y as pa o he
OR amewo k.This sec ionou lines heme hodology,whichincludes de ining and p epa ing
p ocess e en da a o supe ised lea ning, employing QRF o unce ain y quan i ica ion,
cons uc ingunce ain yp o iles,andusing heSHAPme hod o explainingp edic i emodel
unce ain y.
3.1 P ocess da a p epa a ion
This sec ion desc ibes he p ocedu e o con e ing a p ocess e en log da a om MES in o a
abula da ase and o mula ing he du a ion p edic ion o each ac i i y in he unning aces
as a supe ised lea ning ask. To accomplish his, i is c ucial o iden i y he inpu a iables
om he examined unning aces and align hem wi h he espec i e a ge alues. Fo cla i y,
we ini ially in oduce no a ions and o mal de ini ions o elemen s like e en s, e en logs,
aces, pa ial aces, and e en du a ion, d awing om es ablished li e a u e (Pola o e al.,
2014; an de Aals , 2016;Teinemaae al.,2019).
De ini ion 1 (E en )Ane en is a uple e=a,c, s a , comple e,
1,...,
n,whe e
•a∈Ais he co esponding p ocess ac i i y;
•c∈Cis he case id;
• s a ∈Ts a is he s a imes amp o he e en (de ined as seconds since 1/1/1970
which is a Unix epoch ime ep esen a ion);
• comple e ∈Tcomple e is he comple ion imes amp o he e en ;
• 1,...,
n ep esen s he lis o e en speci ic a ibu es, whe e ∀1≤i≤n: i∈Vi,Vi
deno ing he domain o he i h a ibu e.
123
996 Annals o Ope a ions Resea ch (2025) 347:991–1030
Fig. 2 O e iew o he p oposed unce ain y explainabili y app oach
Consequen ly, E=A×C×Ts a ×Tcomple e ×V1×···×Vnis de ined as he uni e se o
e en s. Mo eo e , we de ine he ollowing p ojec unc ions gi en he e en e∈E:
•pa:E→A,pa(e)=a,
•pc:E→C,pc(e)=c,
•p s a :E→Ts a ,p s a (e)= s a ,
•p comple e :E→Tcomple e,p comple e (e)= comple e,
•p i:E→Ei,p i(e)= i,∀1≤i≤n
De ini ion 2 (T aces and E en Log)A ace σ∈E∗is a ini e sequence o e en s σc=
e1,e2,...,e|σc|, o which each ei∈σoccu s no mo e han once and ∀ei,ej∈σ,pc(ei)=
pc(ej)∧pTS(ei)pTSej,i 1≤i<j<|σc|.Thee en log ECis de ined as a se o
comple ed aces, EC={σc|c∈C}.
De ini ion 3 (Pa ial T aces) Two op ions a e gi en o ob aining pa ial aces, depending
on he p edic i e p ocess moni o ing use case. De ined o e σ he ollowing hdi(σc)and
li(σc)gene a e he p e ixes and su ixes espec i ely as ollows:
•selec ion ope a o (.): σc(i)=σi,∀1≤i≤n;
•hdi(σc)=e1,e2,...,emin(i,n) o i∈[1,|σc|]⊂N
• li(σc)=ew,ew+1,...,enwhe e w=max(n−i+1,1);
123
Annals o Ope a ions Resea ch (2025) 347:991–1030 997
•|σ|=n(i.e. he ca dinali y o leng h o he ace).
We deno e he se o pa ial aces gene a ed by he li(σc) unc ion as γ. These pa ial
aces o m he basis o cons uc ing a abula da ase ha p edic s he du a ion o he
emaining e en s in a unning ace. To elucida e, ou p oposed app oach in his s udy is
pe o med be o e ini ia ing a case. None heless, he model and da a s uc u e can be adap ed
o accommoda e upda es ollowing each e en wi hin he unning aces, i needed. Hence,
he applica ion o he li(σc) unc ion p o es pe inen o shaping he aining da a s uc u e.
In he li e a u e o p edic i e p ocess moni o ing, a ious p ocess pe o mance indica o s
(PPIs) o en se e as a ge s o in e es . These a ge s can ange om cos and quali y me ics
o ime- ela ed indica o s. The p oac i e analysis o such ime-based me ics is ins umen al
in enhancing bo h he ope a ional and s a egic capabili ies o o ganiza ions. In his s udy,
he ocus is on he p ocessing ime o an e en , which is de ined as he du a ion o he
co esponding ac i i y. This du a ion is compu ed as he di e ence be ween he comple ion
and s a imes amps o he e en :
De ini ion 4 (E en P ocessing Time/Labeling) Gi en a non-emp y ace σ=∈E∗,a
labeling unc ion esp :E→Y, also e e ed o as anno a ion unc ion, maps an e en
e∈σ o he co esponding alue o i s esponse a iable esp(e)∈Y.Wede ine hee en
p ocessing ime as ou esponse a iable, calcula ed as ollows:
esp(e)=p comple e (e)−p s a (e), (1)
wi h he domain o he de ined esponse a iable being Y⊂R+.
De ini ion 5 (Fea u e Ex ac ion) The ea u e ex ac ion unc ion in his s udy is de ined as
a unc ion ea :E∗→X∗which ex ac s he ea u e alues om a gi en non-emp y ace
σ=∈E∗, wi h X∈Rdim deno ing he domain o he ea u es and dim being he inpu
dimension. Fo a gi en ace σc=e1,e2,...,e|σc|, he ea u e ex ac ion unc ion ea
gene a es a se o ea u es xi,1, ..., xi,dim o each e en ei. n addi ion o case-speci ic and
e en -speci ic ea u e alues, he ea u e ex ac ion unc ion enables he e ie al o in a-
case-speci ic ea u es, such as n-g ams.
Fo ase o p edic o a iablesx=x1, ..., xpand a esponse a iable y,wede ine
D={(xi,yi)}N
i=1as he da ase o associa ed (x,y) alues, wi h N deno ing o al amoun o
obse a ions. The da ase is spli in o h ee pa s: D=D ain ∪D al ∪D es ,ino de ouse
D ain o aining, D al o hype pa ame e op imiza ion and as a calib a ion se o de i e
unce ain y p o iles (see Sec .5.2), and D es o e alua ion, wi h N ain,N al,N es being
he espec i e amoun o ins ances in each subse .
3.2 In e al p edic ion wi h quan ile eg ession o es s
Random Fo es s (RF) is a obus machine lea ning me hod ha cap u es complex non-linea
ela ionships in da a (B eiman, 2001). The p ima y objec i e o RF is o p oduce accu a e
p edic ions by o ming an ensemble o decision ees du ing he aining phase. The model
hen ou pu s ei he he mode o he classes o classi ica ion asks o he mean p edic ion o
eg ession asks based on hese indi idual ees. Howe e , he s anda d RF app oach does
no del e in o he comple e condi ional dis ibu ion o he esponse a iable.
Fo a comp ehensi e unde s anding o such dis ibu ions, QRF was in oduced as an
ex ension o RF (Meinshausen, 2006). This app oach aims o es ima e condi ional quan iles,
123
998 Annals o Ope a ions Resea ch (2025) 347:991–1030
o e ing deepe insigh s in o bo h he dis ibu ion o he da a and he unce ain y associa ed
wi h p edic ions. The capabili y o p oduce p edic ion in e als ende s QRF aluable ac oss
mul iple OR applica ions. In hese con ex s, accu a ely po aying p edic i e unce ain ies is
i al o in o med decision-making and s a egic planning.
In acco dance wi h (Meinshausen, 2006), le θbe a andom pa ame e ec o guiding
ee g ow h wi hin he RF, T(θ) he associa ed ee, B he space o a da a poin Xwi h
dimensionali y pand R⊆Bbe a ec angula subspace o any lea o e e y ee wi hin
he RF. Any x∈Bcan be alloca ed o exac ly one lea o any ee o he RF such ha x∈R,
hus deno ing he speci ic lea o he co esponding ee ia (x,θ). Fi s , he weigh unc ion
wi(x,θ)is de ined o each obse a ion iand ee T(θ),gi enby
wi(x,θ)=1{Xi∈R(x,θ)}
#j:Xj∈R(x,θ),(2)
whe e 1{Xi∈R(x,θ)}is an indica o unc ion ha equals o 1 i he obse a ion i alls in he
lea node co esponding o (x,θ),and#j:Xj∈R(x,θ)is he numbe o obse a ions
ha all in ha same lea node.
Nex , he weigh s a e a e aged o e mul iple ees o ob ain he inal weigh unc ion
wi(x):
wi(x)=k−1
k
=1
wi(x,θ
),(3)
whe e kis he numbe o ees and θ ep esen s he - h ee.
Finally, he es ima ed condi ional quan ile unc ion
F(y|X=x)is calcula ed as a
weigh ed sum o he indica o unc ion 1{Yi≤y} o each obse a ion i,whe eYiis he esponse
a iable:
F(y|X=x)=
n
i=1
wi(x)1{Yi≤y}.(4)
Using his es ima ed condi ional quan ile unc ion, he quan ile unc ion Qα(x) o a gi en
quan ile le el αcan be ob ained as
Qα(x)=in {y:
F(y|X=x)≥α}.(5)
Based on he quan ile unc ion Qα(x), p edic ion in e als o speci ic le els can be de i ed
using
I1−2α(x)=Qα(x), Q1−α(x),(6)
Fo he p edic ion o he condi ional mean, as yielded by egula RF, QRF allow he
u iliza ion o
F(y|X=x)=
n
i=1
wi(x)Yi.(7)
In he ield o OR, p edic ion in e als se e as a obus ool o quan i ying unce ain y
in di e se decision-making scena ios. Unlike s anda d p edic ion models, which p o ide
only a single-poin es ima e, p edic ion in e als o e a ange o possible ou comes. This
ange enhances he eliabili y o decisions by conside ing he inhe en a iabili y in he
unde lying model. Fo ins ance, p edic ion in e als may se e mul iple pu poses in OR:
hey can e alua e he p obabili y o exceeding speci ic h esholds, unc ion as pa ame e s in
123
Annals o Ope a ions Resea ch (2025) 347:991–1030 1005
and He e a, 2008). The (CD) is calcula ed using he o mula:
CD =qk(k+1)
6N(19)
In his con ex , CDis he benchma k o disce ning whe he he pe o mance dispa i ies
be ween any wo me hodologies a e s a is ically signi ican , based on hei a e age ankings.
U ilizing he F iedman–Nemenyi es sui e o e s mul iple bene i s o model e alua-
ion. Fi s ly, i s non-pa ame ic cha ac e is ic ensu es ha he assessmen emains obus
e en when he da a dis ibu ion is non-no mal. Secondly, he me hod’s inhe en capabili y
o simul aneously compa e mul iple machine lea ning algo i hms p o ides a comp ehensi e
e alua ion landscape. Finally, inco po a ing a pos -hoc es educes he likelihood o com-
mi ing Type I e o s, a c ucial aspec when conduc ing mul iple compa isons. Collec i ely,
hese ea u es make he F iedman–Nemenyi es ing amewo k a s a is ically sound and com-
pu a ionally e icien ool o iden i ying he mos e ec i e machine-lea ning app oach o a
speci ic ask.
4.4 So wa e ools
Fo he asks o da ap ocessingand ea u e enginee ing, he“ idy e se” collec iono lib a ies
was employed, wi h a pa icula emphasis on he “dply ” lib a y. The QRF model was imple-
men ed in R, p ima ily using he “ idymodels” and “ ange ” lib a ies. These lib a ies also
acili a ed he compa a i e assessmen o o he models, such as XGBoos . Addi ionally, he
lib a ies employed included “glmne ” o linea eg ession, “ pa ” o decision ees, and
“dba s” o Bayesian Addi i e Reg ession T ees (BART). In he ealm o model e alua ion,
“PMCMR” and “PMCMRplus” we e used o conduc ing signi icance es s. The calcula-
ion o SHAP alues was execu ed h ough he “ke nelshap” lib a y, in collabo a ion wi h
“doPa allel” o pa allel compu ing. Fo isualiza ion pu poses, “ggplo 2,” “ggs a splo ,”
and “ggbeeswa m” we e he p ima y lib a ies employed. This a ay o specialized lib a ies
enabled a comp ehensi e app oach o da a p epa a ion, model implemen a ion, e alua ion,
and isualiza ion, aligning well wi h he p ojec ’s analy ical and p edic i e objec i es.
5 Resul s
This sec ion analyzes he esul s o he p oposed app oach. Fi s , he esul s o he model
e alua ion a e p esen ed, en ailing he cons uc ion o a baseline o compa a i e model
analysis, he analysis o he sensi i i y o hype pa ame e s o he QRF model as well as he
e alua ion o soundness wi h ega ds o UQ. Nex , he cons uc ion o unce ain y p o iles is
p esen ed, ollowed by a desc ip ion o key indings. Las ly, an analysis o model unce ain y
u ilizing SHAP is conduc ed, comp ising examina ions on a local and global le el as well as
on he basis o unce ain y p o iles.
5.1 Model e alua ion
This subsec ion p esen s a de ailed examina ion o se e al ace s pe aining o model pe o -
mance. I commences by in oducing a baseline model. This baseline ac s as a compa a i e
measu e and is de i ed om he manu ac u e ’s cu en me hodology o es ima ing p o-
cessing imes, as in oduced in Sec .2. Following es ablishing he baseline, he subsec ion
123
1006 Annals o Ope a ions Resea ch (2025) 347:991–1030
ansi ions in o a compa a i e analysis ocused on poin p edic ions. This e alua ion le e -
ages he ou comes om he hype pa ame e op imiza ion phase o assess he e icacy o he
op-pe o ming models. Subsequen ly, he sec ion explo es hype pa ame e sensi i i y o he
QRF model. This pa aims o asce ain he ex en o which exhaus i e hype pa ame e un-
ing is equisi e o achie ing op imal pe o mance. Finally, he subsec ion w aps up wi h an
analysis a ge ing model eliabili y in e ms o unce ain y. This is conduc ed by jux aposing
he pe o mances o QRF and BART, speci ically in hei abili y o quan i y unce ain y.
5.1.1 Compa a i e analysis o poin p edic ions
As ou lined in Sec .2, he p ocess planne s employ domain expe ise and his o ical da a
o es ima e p ocessing imes o indi idual p oduc ion s eps. Despi e inco po a ing a ious
ac o s such as e en -speci ic de ails and his o ical cases, hese es ima es ha e been ound o
be un eliable in mul iple ins ances. Documen ed in he manu ac u e ’s planning da a, hese
es ima es se e as a baseline o ou poin p edic ion e alua ion. Fo compa a i e analysis,
we ocus on models op imized h ough hype pa ame e uning, as discussed in Sec .4.2.The
se ings o each op imized model a e summa ized in Table 4.
Table 5o e s a side-by-side compa ison o model pe o mance o poin p edic ion,
p esen ing alida ion and es da ase s. A e age esul s and s anda d de ia ions a e p esen ed
o he alida ion da a. The baseline pe o mance, ep esen ed by MAE alues o 55.6 and
51.9 and RMSE alues o 117.8 and 99.5 o he alida ion and es da ase s espec i ely,
Table 4 Hype pa ame e op imiza ion esul s o BART, DT, LR, QRF and XGBoos models
Model Pa ame e Value Model Pa ame e Value
BART p io _ou come_ ange 5 QRF min_n 20
p io _ e minal_node_expo 1 m y 70
ees 1000 ees 100
XGBoos lea n_ a e 0.0163
DT cos _complexi y 2.51e−05 loss_ educ ion 0.164
min_n 32 min_n 6
ee_dep h 15 m y 99
sample_size 0.923
ees 1832
LR penal y 1.17 ee_dep h 12
Table 5 Compa a i e analysis o model pe o mance (poin p edic ion)
Da ase and me ic Baseline BART DT LR QRF XGBoos
Valida ion da a
MAE 55.6±6.81a38.5±4.58 44.5±6.06 43 ±5.38 37.4±4.91 36.5±4.73
RMSE 117.8±11.176.1±10 95 ±13.380.1±12.481.4±11.678.3±11.1
Tes da a
MAE 51.9 36.3 41.5 43.6 35.51 33.4
RMSE 99.5 64.7 80.5 75.9 66.88 64
aS anda d De ia ion ac oss C oss-Valida ion Folds
123
Annals o Ope a ions Resea ch (2025) 347:991–1030 1007
Fig. 3 F iedman–Nemenyi es esul s o ML model compa ison
se es as ou e e ence poin . Ac oss all e alua ion me ics, e e y ML model es ed su passed
his baseline. No ewo hy among hese a e he QRF and XGboos models, which s and ou
as he mos e ec i e. XGBoos achie es he lowes MAE and RMSE alues ac oss bo h
da ase s, eco ding MAE sco es o 35.5 and 33.4, and RMSE sco es o 78.3 and 64. These
igu es a e closely ollowed by he QRF model, which pos s MAE and RMSE alues o 37.4
and 35.51, and 81.4 and 66.88, espec i ely.
Figu e 3p esen s he esul s o he F iedman–Nemenyi es o p edic i e models BART,
DT, LR, QRF, and XGBoos , u ilizing a 10- old o e lapping sliding window c oss- alida ion.
The F iedman es p oduces a chi-squa ed alue o 38 and a ma kedly small p- alue o
1.1206e−7, signi ying signi ican pe o mance dispa i ies among he models unde consid-
e a ion. To deepen ou unde s anding, we applied he Nemenyi es o pai wise compa isons.
Figu e3illus a es hese compa isons, whe e he p esence o connec ing lines be ween mod-
els deno es s a is ically signi ican di e ences in pe o mance. The es con i ms ha he
pe o mance ad an ages o QRF and XGBoos o e inhe en ly in e p e able models a e s a-
is ically signi ican , no me ely coinciden al.
No ably, he absence o a line be ween XGBoos and QRF implies hei pe o mance
di e ence is s a is ically negligible. Al hough XGBoos eco ds a sligh ly lowe MAE han
QRF, his disc epancy is no s a is ically meaning ul. Consequen ly, bo h models a e consid-
e ed s a is ically equi alen o his speci ic analysis. In p ac ical e ms, he choice be ween
QRF and XGBoos should hinge on o he ac o s, such as QRF’s abili y o quan i y p edic ion
unce ain y o XGBoos ’s scalabili y and di e se pe o mance ad an ages. In ou con ex ,
QRF’s inhe en capabili y o quan i y model unce ain y makes i he p e e able op ion o e
XGBoos .
5.1.2 Sensi i i y analysis o hype pa ame e op imiza ion o QRF
The sensi i i y o he QRF model o a ious hype pa ame e se ings is examined in Fig.4.
This igu e p esen s mean MAE alues o each hype pa ame e con igu a ion o he QRF
model ac oss all alida ion olds, as ou lined in Sec .4.2. Ve ical ba s in he igu e indica e
he co esponding s anda d de ia ions. The model demons a es a obus pe o mance wi h
123
1008 Annals o Ope a ions Resea ch (2025) 347:991–1030
Fig. 4 Sensi i i y analysis o hype pa ame e se ing o QRF based on 10- old o e lapping sliding window
c oss alida ion
Fig. 5 Sensi i i y analysis o hype pa ame e se ing o QRF based on 10- old o e lapping sliding window
c oss alida ion o each o he me ics min_n,m y, ees
a ela i ely na ow gap o app oxima ely 2.5min be ween he bes and wo s ou comes.
Addi ionally, he s anda d de ia ion luc ua es wi hin a modes ange o 4.9 o 5.5min ac oss
he alida ion olds. O e all, he QRF model shows commendable s abili y wi h espec o
hype pa ame e a ia ions.
Figu e 5displays a boxplo e alua ing he pe o mance o he QRF model based on MAE
alues ac oss di e en hype pa ame e se ings. The hype pa ame e s unde conside a ion
include he numbe o a iables andomly sampled a each spli (m y), he o al numbe
o ees in he o es ( ees), and he minimum numbe o obse a ions in e minal nodes
(min_n). The MAE alues p edominan ly ange be ween 37.4 and 40.0, highligh ing he
model’s gene al obus ness o hype pa ame e a ia ion.This is pa icula ly no ewo hy gi en
he ex ensi e ange o se ings e alua ed, encompassing min_n alues om 2 o 32, m y om
40 o 90, and ees om 50 o 100 (see Sec .4.2).
I is c ucial o acknowledge ha op imizing indi idual hype pa ame e s in isola ion may
yield subop imal esul s, gi en hei complex in e dependencies. Fo ins ance, he bes -
pe o ming QRF model in ou s udy had a min_n o 20, a m y o 70, and ees o 100
(see Table 4). This complexi y o en ende s simplis ic, uni a ia e op imiza ion app oaches
ine ec i e. While he s abili y o MAE ac oss a ious se ings is ad an ageous, i should no
o e shadow he in ica e ela ionships among hype pa ame e s and hei collec i e impac
on he model’s pe o mance.
123
Annals o Ope a ions Resea ch (2025) 347:991–1030 1009
5.1.3 E alua ion o unce ain y soundness
To e alua e he quali y o unce ain y es ima es, we conduc ed a quali a i e compa ison
be ween QRF and BART. The models we e con igu ed o p oduce p edic ion in e als, as
desc ibed in Sec .3.2 o QRF and analogously o BART. Following he app oach in He e
al. (2017), Ehsan e al. (2019), he models we e compa ed using PICP, MPIW, and MRPIW
me ics. The compa a i e esul s a e summa ized in Table 6. While QRF and BART show
simila pe o mance in e ms o PICP and MPIW, hey di e subs an ially in hei MRPIW
alues. Speci ically, QRF’s MRPIW on he alida ion da a is 3.01, signi ican ly lowe han
BART’s 11.86. This pa e n is consis en on he es da a, whe e QRF sco es an MRPIW o
1.74 compa ed o BART’s 4.83. Fu he , he s anda d de ia ion o QRF’s MRPIW (0.352) is
much lowe han ha o BART (4.29). This sugges s ha QRF o e s mo e s able pe o mance
and be e adap s i s p edic ion in e als ac oss alida ion olds.
Theno able educ ion in MRPIW o QRF unde sco esi se ec i enessingene a ingmo e
eliable p edic ion in e als. This is suppo ed by lowe s anda d de ia ions, which es i y
o QRF’s obus ness in p o iding eliable unce ain y es ima es. Al hough bo h models o e
simila co e age(PICP) and compa ableMPIW esul s(pNemenyi =0.53),QRF ou pe o ms
BART signi ican ly in MRPIW (pNemenyi =0.0016) on he alida ion da a, as isualized in
Fig.6.
Rega ding he e alua ion o indi idual p edic ions, wo key igu es, namely Figs.7and
8, o e salien obse a ions. These igu es no only p o ide poin p edic ions and associa ed
esiduals bu also p esen he co e age ensu ed by each model’s p edic ion in e als on he
es da a. I is no ewo hy ha he p edic ion in e als p o ided by he QRF model adap
acco ding o he magni ude o he co esponding poin p edic ions, a ea u e conspicuously
absen in he BART model. In mo e quan i iable e ms, he minimal wid h o he p edic ion
Table 6 Compa a i e analysis o model unce ain y o BART and QRF
Da ase and me ic BART QRF Da ase and Me ic BART QRF
Valida ion da a Tes da a
PICP 0.949 ±0.01a0.949 ±0.006 PICP 0.919 0.912
MPIW 250.93 ±44.5 243.44 ±28.3 MPIW 179.2 171.8
MRPIW 11.87 ±4.29 3.01 ±0.352 MRPIW 4.83 1.74
aS anda d De ia ion ac oss C oss-Valida ion Folds
Fig. 6 F iedman–Nemenyi es esul s o BART and QRF o each o he unce ain y me ics MPIW and
MRPIW
123
1010 Annals o Ope a ions Resea ch (2025) 347:991–1030
Fig. 7 Visualiza ion o model p edic ions, ac ual alues, p edic ion in e als and co e age o BART and QRF
o es da a. The poin s ep esen he ela ionship be ween he model p edic ions on he x-axis and he ac ual
alues on he y-axis. Co esponding p edic ion in e als a e depic ed as blue e ical lines, spanning om he
alue o he uppe bounda y o he alue o he lowe bounda y. The colo o he poin s indica e i he ac ual
alue was cap u ed by he p edic ion in e al (g een) o no (o ange o ac ual alues below lowe bounda y,
ed o ac ual alues abo e uppe bounda y). A loga i hmic scale was used o allow a clea e depic ion o
alues below he lowe bounda y
Fig. 8 Visualiza ion o esiduals, p edic ion in e als and co e age o BART and QRF o es da a. The
poin s ep esen he ela ionship be ween he model p edic ions on he x-axis and esidual alues on he y-
axis. Co esponding p edic ion in e als a e depic ed as blue e ical lines, spanning om he alue o he
uppe bounda y o he alue o he lowe bounda y educed by he model p edic ion. The colo o he poin s
indica e i he ac ual alue was cap u ed by he p edic ion in e al (g een) o no (o ange o ac ual alues
below lowe bounda y, ed o ac ual alues abo e uppe bounda y)
123
Annals o Ope a ions Resea ch (2025) 347:991–1030 1011
in e al o BART s ands a 80.3min wi h a s anda d de ia ion o 18.2min. This s a is ical
obse a ion c i ically hampe s BART’s u ili y in he con ex o UQ, pa icula ly o lowe
a ge alues. In s a k con as , he QRF model mani es s a much mo e adap i e beha io ,
wi h a minimal p edic ion in e al wid h o 0.22min and a no ably wide s anda d de ia ion
o 194.6min. Such adap abili y becomes especially pe inen o model p edic ions alling
below he 100-minu e ma k. Fu he , Fig.7elucida es ha BART se s a minimal alue o
he lowe bounda y o i s p edic ion in e als, s emming om i s in insic es ic ion o non-
nega i e alues.To summa ize, heQRF model exhibi sa mo enuancedcapabili yin ailo ing
i s p edic ion in e als based on speci ic poin p edic ions, he eby making i subs an ially
mo e i ing o applica ions ha necessi a e eliable unce ain y es ima es.
To u he alida e he ele ance and soundness o he UQ esul s deli e ed by he QRF
model, an in e iew was conduc ed wi h a p ocess expe om he manu ac u ing pa ne . The
endo semen om he p ocess expe p o ides aluable quali a i e alida ion o he QRF
model’s app oach o es ima ing model unce ain ies. This is signi ican because he expe
has a deep unde s anding o he complexi ies and a iabili ies in he manu ac u ing p ocess,
o e ing a eal-wo ld pe spec i e ha complemen s s a is ical e alua ions.
The use o a dedica ed dashboa d in ou e alua ion o model e alua ion was a key ac o in
making he complex QRF model accessible o he expe . The dashboa d no only showcased
he model’s p edic ions and ela ed unce ain ies bu also allowed he expe o in e ac wi h
da a on bo h ace and e en le els. This in e ac i e componen p o ides a dynamic way o
es he model’s capabili ies and limi a ions, adding ano he laye o i s alida ion p ocess.
Mo eo e , he dashboa d is a p o o ype o how he model could be in eg a ed in o exis ing
managemen sys ems, illus a ing i s ope a ional easibili y. The expe ’s a i ma ion speaks
o he model’s p ac ical u ili y. By examining he dashboa d, he expe was able o ela e he
model’s UQ ou pu s o e e yday ope a ional decisions. The bes - and wo s -case scena ios
gene a ed by he model, as mani es ed in he p edic ion in e als, can se e as ac ionable
guidance o p oduc ion planning, helping o mi iga e isks and op imize esou ce alloca ion.
Addi ionally, he expe ’s assessmen o he model as “in ui i ely comp ehensible” indi-
ca es ha heQRFmodelcouldbein eg a edin oexis ingwo k lowswi hminimaldis up ion.
I s “adap i e p edic ion in e als” we e also deemed “su icien ly sound and sa is ac o y”,
unde lining he model’s abili y o adap o he unique cha ac e is ics o indi idual p oduc ion
s eps, hus enhancing i s eal-wo ld applicabili y. In summa y, he expe ’s alida ion is mo e
han jus an endo semen ; i p o ides a mul i- ace ed e alua ion ha unde sco es he QRF
model’s obus ness, p ac ical u ili y, and adap abili y. This aligns well wi h he model’s s a is-
ical alida ion, he eby ein o cingi s iabili y asa eliable ool o unce ain y quan i ica ion
in complex manu ac u ing p ocesses.
5.2 Unce ain y p o ile cons uc ion and e alua ion
In esponse o he ope a ional needs exp essed by p ocess expe s, we ha e ex ended ou
model o include a sys em o unce ain y p o iles—speci ically ca ego ized as low, medium,
andhigh.Theaimis oo e anuancedlens h oughwhichmodelp edic ionscanbee alua ed,
p o iding p ac i ione s wi h an e ec i e ool o isk mi iga ion and decision-making. While
p edic ion in e als a e use ul as aw unce ain y measu es, hey may no be immedia ely
in e p e able in a p ac ical se ing.
Anini ial a emp wasmade o ca ego izeunce ain y iape cen ile-based p o iling, ocus-
ing on he wid hs o p edic ion in e als. Howe e , his app oach p o ed o be subop imal;
he wid h o he p edic ion in e al was obse ed o co ela e s ongly wi h he ac ual ou pu
123
1012 Annals o Ope a ions Resea ch (2025) 347:991–1030
Fig. 9 Visualiza ion o unce ain y p o ile h esholds o alida ion da a. Fi s , da a se was so ed in ascending
o de o Wid h alues and a nume ical index was in oduced o depic he o de . Each poin in he plo depic s
i s index on he x-axis and he co esponding Wid h alue on he y-axis. The g een and ed e ical line
di ide poin s espec i ely a he 25 h and 75 h pe cen ile. The g een and ed ho izon al lines depic he
co esponding Wid h alues, which espec i ely sepa a e he “low” om he “medium” ( Wid h = 1.685) and
he “medium” om he “high” p o ile ( Wid h = 2.738). The poin s a e colo ed o ep esen hei a ilia ion
wi h he co esponding p o ile: g een o “low”, o ange o “medium” and ed o “high”
alues. Consequen ly, ac i i ies wi h longe du a ions exhibi ed in la ed p edic ion in e als,
and he e e se was ue o ac i i ies wi h sho e du a ions. To add ess his issue, we u ned
o an al e na i e me ic: ela i e wid h in e als, as delinea ed in Sec .4.3. This no malized
app oach accoun s o he inhe en a iabili y in ac i i y du a ions, he eby p o iding a mo e
accu a e and eliable ep esen a ion o associa ed unce ain ies. By le e aging ela i e wid h
in e als o ca ego ize unce ain ies, we o e p ocess expe s an enhanced unde s anding o
he con idence le els o each p edic ion. This, in u n, allows o in o med decision-making
conce ning po en ial adjus men s in ope a ional p ocesses (see Fig.9).
The cons uc ion o unce ain y p o iles le e ages he aining da ase o calib a ion,
which in ol es sco ing he da ase using he i ed QRF model and calcula ing he Wid h o
each model p edic ion. Ins ances we e hen so ed in ascending o de o Wid h alues, and
he 25 h and 75 h pe cen ile h esholds we e used o de ine he unce ain y p o iles. Values
below he 25 h pe cen ile h eshold we e classi ied as “low” p o ile, wi h Wid h alues below
1.685, while alues abo e he 75 h pe cen ile h eshold we e classi ied as “high” p o ile, wi h
Wid h alues abo e 2.738. The emaining alues we e assigned o he “medium” p o ile.
Figu e9 isualizes he esul s, wi h he e ical lines indica ing he 25 h and 75 h pe cen ile
h esholds and he ho izon al lines ep esen ing he Wid h alues co esponding o hese
h esholds.
The e alua ion me ics o each unce ain y p o ile om aining da a a e p esen ed in
Table 7. The PICP alue is no ably high, egis e ing a 0.997 ac oss all p o iles. This can
be a ibu ed o he model’s high le el o amilia i y wi h he aining da a, ensu ing almos
maximump edic ionin e alco e age. Whenexamining heMPIW me ic, he“high” p o ile
shows he wides p edic ion in e als,wi h a alueo 260.8.This is ollowed by he “medium”
p o ile a 203.7, and he “low” p o ile a 174.4. This esul aligns wi h he expec a ion ha
highe unce ain y p o iles will na u ally ha e la ge p edic ion in e als. As o MRPIW, he
123
Annals o Ope a ions Resea ch (2025) 347:991–1030 1013
Table 7 E alua ion me ics o he QRF model o unce ain y p o iles
Da ase PICP MPIW MRPIW MAE Baseline Numbe o e en s
T aining Da a 0.997 210.6 2.69 21.3 55.6 45,188
P o ile: “low” 0.997 174.4 1.42 23.4 64.1 11,297 (25%)
P o ile: “medium” 0.997 203.7 2.15 20.5 55.6 22,594 (50%)
P o ile: “high” 0.997 260.8 5.06 20.8 50.4 11,297 (25%)
Tes Da a 0.912 171.8 1.74 35.5 51.9 3,389
P o ile: “low” 0.893 131.9 1.30 31.8 54.8 1,948 (58%)
P o ile: “medium” 0.938 220.2 2.10 39.8 76.0 1,201 (35%)
P o ile: “high” 0.944 269.8 3.80 41.1 77.3 240 (7%)
end indica es he “high” p o ile leading wi h a alue o 5.06. I is ollowed by he “medium”
p o ile a 2.15 and he “low” p o ile a 1.42. This dis ibu ion is consis en wi h he ini ial
cons uc ion me hodology o he unce ain y p o iles, which elies on MRPIW alues.
To e alua e he model’s unce ain y on he es da ase , we ollow a me hodology anal-
ogous o he one applied o he aining da ase . Fi s , he es da ase is sco ed o calcula e
he Wid h, PICP, MPIW, and MRPIW me ics. Subsequen ly, using he unce ain y p o ile
h esholds de ined du ing he calib a ion s ep, indi idual es da a p edic ions a e ca ego-
ized in o co esponding unce ain y p o iles. Table 7p esen s hese me ics o each p o ile,
enabling a compa a i e assessmen wi h he aining da a.
In he con ex o PICP, he “low” p o ile exhibi s educed co e age wi h a alue o 0.893,
whe eas he “medium” and “high” p o iles egis e highe co e age alues o 0.938 and 0.944
espec i ely. This phenomenon unde sco es ha he QRF model o e s imp o ed co e age o
p edic ionswi hele a edunce ain yle els,albei a heexpenseo la ge p edic ionin e als.
Consequen ly, his highligh s a ade-o be ween p edic ion co e age and unce ain y. Fo
MPIW, he “low” p o ile has been op imized wi h a alue o 131.9, while he “medium”
and “high” p o iles mani es ex ended p edic ion in e al anges, egis e ing alues o 220.2
and 269.8, espec i ely. I is c ucial o no e ha hese a iances a e pa ly a ibu able o he
imbalanced s uc u e o he es da ase (see Sec .4.1). The MRPIW ends o he es da a
aligncloselywi h hoseobse ed o he alida ionda a. Addi ionally, he es da ase ’s p o ile
alloca ion dis ibu ion indica es ha 58% o e en s a e ca ego ized unde he “low” p o ile,
35% unde he “medium,” and 7% unde he “high” p o ile. This dis ibu ion sugges s ha
he es da ase , delibe a ely ex ac ed o ep esen a ch onological sample om he comple e
da ase , is imbalanced.
Rega ding he Mean Absolu e E o (MAE), he es da a e eals a nuanced pa e n: he
“low”p o ile eco ds hesmalles MAEo 31.8, ollowedby he“medium”and“high”p o iles
wi h MAEs o 39.8 and 41.1, espec i ely. This sugges s ha he MAE inc eases wi h he
model’s unce ain y le els. Fu he mo e, he QRF model demons a es supe io pe o mance
in gene a ing poin es ima es ac oss all unce ain y p o iles when compa ed o he baseline
p edic ions. Rema kably, he Mean Absolu e E o (MAE) o ins ances ca ego ized unde
he “high” unce ain y p o ile ou pe o ms e en he baseline esul s o ins ances wi hin he
“low” p o ile. This ou come accen ua es he obus ness o he QRF model, no jus in e ms
o unce ain y quan i ica ion, bu also in he accu acy o i s poin es ima es.
In scena ios in ol ing u gen o de s and ime-sensi i e deadlines, p ocess expe s empha-
sized he impo ance o p oac i ely iden i ying c i ical s ages in p oduc ion. They also called
o enhanced me hods o es ima ing bes -case and wo s -case scena ios. As ou lined in
123
1014 Annals o Ope a ions Resea ch (2025) 347:991–1030
Sec .5.1.3, in e iews wi h hese expe s con i med ha he exis ing esul s me hei equi e-
men s. The inco po a ion o unce ain y p o iles in o he p oduc ion planning p ocess o e s
wo subs an ial ad an ages. Fi s , i s eamlines he communica ion o model unce ain y
among s akeholde s. By ca ego izing unce ain ies in o dis inc p o iles-low, medium, and
high- hese p o iles p o ide a use - iendly mechanism o quickly assess he le el o isk o
eliabili y associa ed wi h each p edic ion. This ca ego iza ion allows o a mo e a ge ed
discussion and enables decision-make s o quickly iden i y a eas equi ing u he sc u iny
o al e na i e planning. Second, he unce ain y p o iles con ibu e o he op imiza ion o
he p oduc ion schedule. They no only p o ide poin es ima es bu also p edic ion in e als
o each p oduc ion s ep. This addi ional laye o in o ma ion acili a es mo e obus plan-
ning by conside ing no jus he mos likely ou comes bu also possible a ia ions. Planne s
can he e o e sequence p oduc ion s eps mo e e ec i ely, aking in o accoun bo h he es i-
ma ed p ocessing imes and hei associa ed unce ain ies. Fo example, asks alling unde
he “high” unce ain y p o ile may be scheduled wi h added bu e imes o could igge
addi ional e i ica ion s eps o manage isk.
5.3 SHAP analysis o model unce ain y
Thissubsec iondel esin o he analysiso SHAP alues oexamine hein luenceo indi idual
ea u es on he esul ing p edic ion in e als. We pe o m his analysis on wo dis inc le els:
a local le el, concen a ing on a single da a ins ance, and a global le el, which assesses he
model’s gene al pe o mance. Ini ially, we sc u inize SHAP alues a he local le el, a ge ing
he p edic ion in e al o a designa ed ins ance. To deepen ou unde s anding, we ex end his
local analysis o encompass SHAP alues o lowe and uppe p edic ion bounda ies and
poin es ima ion. This mul i ace ed examina ion helps us unde s and he a ia ion in ea u e
con ibu ions unde di e en unce ain y le els.
T ansi ioning o he global le el, we di ec ou ocus o he model’s o e all beha io .
This s age o analysis in ol es e alua ing bo h he SHAP ea u e impo ance ankings and
he co esponding SHAP summa y plo s o he p edic ion in e als. Subsequen ly, we d aw
compa isons be ween SHAP summa y plo s ac oss di e en unce ain y p o iles, ca ego ized
as “low,” “medium,” and “high.” This compa a i e s udy enables us o iden i y dispa i ies in
ea u e con ibu ions unde luc ua ing unce ain y condi ions. To comple e he global anal-
ysis, we p esen he SHAP Dependence plo s o selec ed a iables wi hin each unce ain y
p o ile. This inal s ep allows us o de ec eme ging ends o pa e ns in how ea u es in e ac
ac oss di e se unce ain y scena ios.
5.3.1 Local SHAP analysis
Figu e 10 elucida es he in luence o a ious ac o s on he p edic ion in e al wid h o
a pa icula es ins ance. This ins ance, classi ied unde he “low” unce ain y p o ile, is
ela ed o he “Dishing P ess” ac i i y ca ied ou on he “Dishing P ess 5” machine. In he
plo , a iables a e anked along he y-axis based on hei absolu e impac on he p edic ion
in e al wid h, wi h he mos in luen ial a iables appea ing a he op. Fo his speci ic
ins ance, he numbe o i ems p oduced (“Quan i y = 4”) eme ges as he mos signi ican
ac o in widening he p edic ion in e al, ex ending i by 82.1min. Con e sely, he his o ical
mean p ocessing ime o an i em in he ele an a icle g oup (“Mean P ocessing Time =
25.27”) is he p incipal ac o in con ac ing he p edic ion in e al, educing i s wid h by
33.3min. Gi en he many a iables, he plo ocuses on he op en ac o s based on hei
123
Annals o Ope a ions Resea ch (2025) 347:991–1030 1021
Fig. 18 SHAP summa y plo o he p o ile “high”. The same app oach as in Fig.15 was used, bu he da a
was es ic ed o ins ances pe aining o he “high” unce ain y g oup
Fig. 19 SHAP dependence plo o he a iable “Diame e Base”, wi h he seconda y a iable “Weigh ” o he
“low”, “medium” and “high” unce ain y p o ile. Each poin ep esen s he ela ionship be ween he “Diame e
Base” alue o an ins ance, as seen on he x-axis, he co esponding SHAP alue, as seen on he y-axis, and
he co esponding ela i e “Weigh ” alue, p o ided ia colo coding. A black smoo hing cu e, calcula ed
ia a gene al addi i e model, p o ides a isual aid o each plo
a posi i e co ela ion be ween “Diame e Base” and “Weigh ” in e ms o hei impac on
SHAP alues. Pa icula ly no able is a end in he “low” p o ile, whe e a egion o nega i e
SHAP alues appea s o “Diame e Base” alues be ween 1500 and 2000. This speci ic
beha io is a enua ed in he “medium” p o ile and comple ely absen in he “high” p o ile.
The use o global SHAP analysis on da a subse s dis inguished by a ying unce ain y
p o ileso e samul i- ace ed iewin o hemodel’sbeha io .Thisapp oachenhancesbo h he
model’s obus ness and i s applicabili y ac oss di e se ope a ional condi ions. By ocusing
on hese dis inc subse s, p ocess expe s can pinpoin a iables ha ha e a p onounced
in luence in speci ic con ex s o unce ain y. This ine-g ained unde s anding enables a ge ed
model e inemen , illumina ingpa hs o pe o manceimp o emen . Fu he mo e, examining
hese da a subse s can e eal inconsis encies in he model’s sensi i i y o ce ain ea u es
ac oss di e en unce ain y p o iles. Such insigh s a e aluable o unde s anding he model’s
123
1022 Annals o Ope a ions Resea ch (2025) 347:991–1030
limi a ions and i s esilience o di e se inpu condi ions, he eby aiding in he calib a ion
o i s p edic i e capabili ies. The end esul o his comp ehensi e app oach is a machine
lea ning model be e equipped o eal-wo ld applica ions. I allows o he de elopmen o
cus omized s a egies o mi iga e isks and unce ain ies in a ious scena ios, hus enhancing
he model’s u ili y and us wo hiness in decision-making p ocesses.
6 Discussion
6.1 Rele ance o ope a ions esea ch
The ele ance o ou me hodology o OR is mul i ace ed, wi h each componen —be i
unce ain y es ima ion o explana ion—ha ing dis inc implica ions. The p oposed app oach
aligns wi h he “p edic - hen-op imize” model commonly ound in OR. In his app oach, ML
is used o p edic essen ial pa ame e s o an op imiza ion model be o e o simul aneously
as he op imiza ion models a e sol ed (Miši´c and Pe akis, 2020). The co e o ou wo k lies
in he “p edic ” phase, u ilizing he QRF echnique o p oduce o ecas s ha come wi h
quan i ied unce ain y. These o ecas s hen may se e as inpu s o op imiza ion models.
Unlike adi ional OR models ha o en ely on de e minis ic o o e ly simplis ic s ochas ic
pa ame e s, ou app oach cap u es he in ica e, possibly non-linea , na u e o unce ain y.
This e ined unde s anding o unce ain y is hen in eg a ed in o a ious OR models, he eby
imp o ing he obus ness and eliabili y o he op imiza ion solu ions. Essen ially, ou wo k
lays a sophis ica ed ounda ion o he subsequen op imiza ion phase.
Mo e speci ically, ocusing on p edic i e p ocess moni o ing, ou p ima y conce n is com-
p ehending dynamic sys em beha io . Wi hin he OR amewo k, ou p oblem is o mula ed
o imp o e sys em esponses o luc ua ing inpu s. Ou app oach e ines decision-making by
na owing he ange o op ions, hus enabling he in eg a ion o da a analy ics in o ope a-
ional op imiza ion. The use case, cen e ed on quan i ying unce ain y in p ocess p edic ions,
ex ends adi ional OR p oblems o e alua e sys em pe o mance unde di e se condi ions.
Mo eo e , ou model p o ides mul i-le el insigh s in o p edic i e unce ain y, aluable o
bo h ac ical and ope a ional planning. These insigh s a e pa icula ly use ul o esou ce
alloca ion and p oduc ion scheduling in ou pa ne manu ac u ing i m. Fo example, he
use o SHAP analysis o iden i y he mos in luen ial ea u es con ibu ing o o e all unce -
ain y pa allels esou ce alloca ion challenges in OR. This unde s anding enables a ge ed
in e en ions and esou ce ealloca ions o minimize unce ain y in eal-wo ld p ocesses.
The o e a ching objec i e o ou esea ch, akin o many OR ini ia i es, is o o e obus
decision suppo . By quan i ying unce ain y, we p o ide decision-make s wi h a comp ehen-
si e unde s anding o po en ial ou comes, including bes -case, wo s -case, and mos -likely
scena ios. This aligns wi h OR’s emphasis on isk mi iga ion, whe e decisions accoun o
he a iabili y and unce ain y o ou comes. Gi en he speci ic con ex , i is clea ha ou
app oach is deeply oo ed in OR me hodologies. OR’s ex ensi e oolki has been pi o al in
shaping ou esea ch, making i igo ous and applicable o eal-wo ld challenges. This mu u-
ally bene icial ela ionship ensu es ha ou con ibu ions a e bo h g ounded in es ablished
me hods and inno a i e in he ield o p edic i e p ocess moni o ing.
123
Annals o Ope a ions Resea ch (2025) 347:991–1030 1023
6.2 Implica ions o domain expe s
The u iliza ion o a mul i-s age machine lea ning app oach ha in eg a es unce ain y awa e-
ness and explainabili y holds signi ican implica ions o decision-make s ac oss di e se
business ope a ions. Th ough he use o machine lea ning models ha accoun o unce -
ain y, expe s in a gi en ield can gain a deepe unde s anding o po en ial ou comes and hei
associa ed a iabili y. This enhanced comp ehension can acili a e mo e e icen decision-
making and ul ima ely lead o imp o ed isk managemen . The heigh ened awa eness o
unce ain y leads o imp o ed ope a ional p ocesses in o ganiza ions, p omo ing a cul u e o
decision-making based on da a, which ul ima ely esul s in inc eased e iciency and e ec i e-
ness. The in eg a ion o ML explana ion componen s in he decision-making p ocess o e s
he undamen al bene i o being able o disce n he undamen al ac o s ha con ibu e o
p edic i e unce ain y. This ac o p o ides decision-make s wi h he abili y o concen a e
hei endea o s on mi iga ing he pe inen o igins o unp edic abili y, he eby enhancing he
esilience and us wo hiness o he decision-making mechanism.
Mo eo e , he knowledge acqui ed om ou unce ain y-awa e explainable app oach
can be u ilized o enhance esou ce alloca ion, acili a ing o ganiza ions o gi e p io i y o
esou ces in domains wi h signi ican ola ili y and alle ia e ela ed isks. This solu ion can
also p o ide bene i s o domain expe s in he a eas o s a egic planning and o ganiza ional
adap abili y. By comp ehending he magni ude o unce ain ies, indi iduals can o mula e
mo e eliable and adap able s a egic plans ha co espond wi h he objec i es o he o ga-
niza ion and ensu e sus ained p ospe i y. In addi ion, he capaci y o measu e and cla i y
unce ain ies p o ides p o essionals wi h he necessa y asse s o adap o e ol ing ci cum-
s ances and add ess possible in e up ions, augmen ing he compe i i eness o he en e p ise
in a dynamic comme cial se ing.
6.3 Theo e ical/scien i ic implica ions
The use o ou p oposed app oach ha in eg a es unce ain y awa eness and explainabili y
has no ewo hy heo e ical and scien i ic implica ions o he domains o p esc ip i e ana-
ly ics, OR, and AI. This s udy con ibu es o he ad ancemen o scien i ic knowledge by
add essing gaps in he exis ing li e a u e. Speci ically, i emphasizes he signi icance o in e-
g a ing echnical p oduc ion pa ame e s, p oducing machine lea ning ou pu s ha accoun
o unce ain ies, and elucida ing he o igins o such unce ain ies. The inco po a ion o hese
componen s wi hin he decision-making amewo k has he po en ial o enhance he e icacy
o he model, p oduce mo e esilien op imiza ion esul s, and os e a deepe comp ehension
o in icacies inhe en in p ac ical scena ios.
F om a me hodological pe spec i e, he p oposed app oach expands he use o ML me h-
ods, such as QRF and SHAP, o gene a e p edic ion in e als and a ibu e unce ain y o
pa icula inpu ea u es. The p og ess made in his ield no only acili a es a mo e ho -
ough comp ehension o he inhe en unce ain ies in p oblems ela ed o OR bu also lays
he g oundwo k o he c ea ion o no el me hodologies and echniques ha u he augmen
he combina ion o unce ain y and explana ion in models used o op imiza ion. Conse-
quen ly, o hcoming s udies may u ilize hese me hodological ad ancemen s o de elop
no el app oaches ha ackle a di e se ange o in ica e comme cial challenges.
Finally, by illus a ing i s applicabili y o eal p oduc ion planning scena ios, he p o-
posed me hod makes a signi ican addi ion o he scien i ic communi y. This use case se es
as a p oo -o -concep , demons a ing how he mul i-s age ML s a egy is e ec i e a man-
123
1024 Annals o Ope a ions Resea ch (2025) 347:991–1030
aging unce ain y and deli e ing use ul insigh s. The success ul applica ion o he sugges ed
s a egy in a p ac ical se ing may inspi e addi ional in es iga ion and s udy in ela ed a eas,
os e ing in e disciplina y coope a ion and encou aging he c ea ion o new heo ies, me hod-
ologies, and applica ions ha ad ance scien i ic unde s anding gene ally.
6.4 Th ea s o alidi y
While he p oposed me hodology shows p omising esul s in he ield o p edic i e p ocess
moni o ing, i ’s c ucial o acknowledge po en ial h ea s o he s udy’s alidi y. Recognizing
hese limi a ions no only p o ides a mo e comple e unde s anding o he s udy’s cons ain s
bu also encou ages u he esea ch aimed a add essing hese issues.
The alidi y o he indings can be signi ican ly in luenced by he quali y and ep esen a-
i eness o he da a u ilized in his s udy. The case s udy’s indings migh no be gene alizable
i he da a used don’ accu a ely e lec he eal-wo ld scena io o i hey ha e biases, incon-
sis encies, o e o s. Mo eo e , i is impe a i e o ha e an adequa ely la ge sample size o
mi iga e he impac o andom a ia ions o anomalies on he esul s. The alidi y o he s udy
may be impac ed by he assump ions made du ing he de elopmen o he ML models. I is
impo an o no e ha he assump ions ega ding he unde lying dis ibu ion o he da a and
he in e ac ions be ween a iables may no be applicable in all scena ios. Thus, he e icacy
o he sugges ed me hodology may exhibi a iabili y con ingen upon hese a o emen ioned
ac o s.
While SHAP p o ides insigh s in o he sou ces o unce ain y, he scope o in e p e -
ing and cla i ying hese explana ions may be limi ed. The unde s andabili y o con ibu ing
ac o s can be hinde ed by complex in e ac ions among a iables o high-dimensional da a,
posing challenges o domain expe s. Fu u e esea ch should ocus on c ea ing mo e acces-
sible and unde s andable explana ions, he eby imp o ing communica ion wi h s akeholde s.
By add essing hese po en ial limi a ions, subsequen s udies can e ine he p oposed me hod-
ology and deepen he o e all unde s anding o UQ and XAI wi hin he con ex o p edic i e
p ocess moni o ing.
7 Rela ed wo k
The p incipal objec i e o OR is o ha monize me hods o e ec i e manage ial decision-
making. In eg al o his aim is in eg a ing in o ma ion sys ems and decision-suppo ools,
as a icula ed by Simon (1997). The ela ionship be ween OR and AI is mu ually bene icial;
AI app oaches o en equi e he solu ion o op imiza ion p oblems—a co e componen o
OR me hodologies. Con e sely, AI echniques ind applica ion in OR o p edic ing c ucial
pa ame e s and o mula ing heu is ics o complex op imiza ion asks (Benne and Pa ado-
He nández, 2006). In he specialized a ea o condi ional-s ochas ic op imiza ion, he wo k by
Be simas and Kallus (2020) illus a es he p omise o using p edic i e analy ics o es ima e
condi ionally expec ed cos s o a ious inpu s. This add esses he challenging ask o min-
imizing unce ain cos s in he p esence o incomple e in o ma ion. The s udy se s he s age
o inno a i e applica ions in p esc ip i e analy ics by showing how p edic i e echniques
can sol e complex op imiza ion issues. Fu he explo ing his syne gy, he pape by Bengio
e al. (2021) del es in o he combina i e po en ial o ML and combina o ial op imiza ion.
The au ho s ad oca e o a no el app oach ha iews op imiza ion p oblems as da a poin s.
123
Annals o Ope a ions Resea ch (2025) 347:991–1030 1025
This enables he iden i ica ion o p oblem dis ibu ions and enhances decision-making capa-
bili ies, mo ing beyond he limi a ions o adi ional heu is ics.
T adi ionally, he OR ield has been cons ained by limi ed da a a ailabili y and com-
pu a ional esou ces, necessi a ing eliance on models de i ed om mic oeconomic heo y,
game heo y, op imiza ion, and s ochas ic models (Miši´c and Pe akis, 2020). Howe e , he
mode n landscape has e ol ed signi ican ly due o ad ancemen s in compu a ional powe
and algo i hms, as well as inc eased da a a ailabili y. This e olu ion has ele a ed AI o a
cen al ole in OR. Speci ically, AI has been ins umen al in enhancing ou unde s anding o
unde lying p ocesses in a eas such as scheduling, he eby enabling mo e e icien ope a ions
(Isaksson e al., 2018). Da a-d i en analy ics ha e demons a ed e ec i eness ac oss a wide
spec um o OR challenges. These ange om capaci y planning (Youn e al., 2022) and p o-
duc ion planning (Usuga Cada id e al., 2020) o dis ibu ion planning (Kuma e al., 2020)
and in en o y managemen ( an Jaa s eld and Schelle -Wol , 2015). Fu he applica ions
include anspo a ion (Chung e al., 2017), sales and ope a ions planning (Thomé e al.,
2012), as well as dynamic p icing and e enue managemen (Xue e al., 2016). The applica-
ion o da a-d i en decision-making wi hin hese sec o s yields subs an ial bene i s. Among
hese a e an enhanced e u n on in es men , op imized asse u iliza ion, and an inc ease in
ma ke alue (Mehdiye and Fe ke, 2021).
In his s udy, we ocus on a speci ic ML p oblem, namely p edic i e p ocess moni o -
ing, which is a echnique wi hin he b oade ield o p ocess mining ha includes p ocess
disco e y, con o mance checking, and p ocess enhancemen (Van De Aals e al., 2012).
P edic i e p ocess moni o ing le e ages his o ical execu ion da a o p o ide use s wi h p e-
dic ions abou a a ge o in e es o a gi en p ocess execu ion (Maggi e al., 2014). P ocess
mining encompasses a se o echniques aimed a ex ac ing aluable insigh s om da a
gene a ed by p ocess-awa e in o ma ion sys ems du ing p ocess execu ion. I se es as an
in e media y be ween p ocess science (including OR) and da a science (encompassing ields
such as p edic i e and p esc ip i e analy ics), o e ing me hods o da a-d i en p ocess anal-
ysis ( an de Aals , 2022). As illus a ed in Fig.20 and p esen ed in Rehse e al. (2019),
he e a e h ee cen al p edic ion asks based on he a ge o in e es and i s cha ac e is ics:
p ocess ou come p edic ion (Teinemaa e al., 2019), nex e en p edic ion (Tax e al., 2017;
E e mann e al., 2017), and emaining ime p edic ion (Ve enich e al., 2019;Teinemaae
al., 2018).
Nume ous e iew a icles ha e been published on he subjec o p edic i e p ocess mon-
i o ing. Fo example, Di F ancescoma ino e al. (2018) classi ied 51 p ocess p edic ion
me hods based on hei p edic ion a ge s using a alue-d i en amewo k. These me h-
ods exhibi ed di e en p edic ion a chi ec u es and we e ca ego ized in o a ious ca ego ies,
Fig. 20 O e iew o p edic i e p ocess analy ics (Rehse e al., 2019)
123
1026 Annals o Ope a ions Resea ch (2025) 347:991–1030
including ca ego ical ou come, cos s, in e -case me ics, isk, sequence o alues, and ime.
Teinemaa e al. (2019) conduc ed a sys ema ic e iew and p oposed a axonomy o ou come-
o ien ed p edic i e p ocess moni o ing. The au ho s iden i ied and compa ed 14 ele an
pape s based on se e al c i e ia, including classi ica ion algo i hm, il e ing, p e ix ex ac ion,
sequence encoding, and ace bucke ing. Addi ionally, an expe imen al e alua ion cap u ing
he impac o di e en quali a i e c i e ia was conduc ed using he au ho s’ own implemen-
a ion. Ve enich e al. (2019) conduc ed a su ey on me hods o p edic ing emaining ime
in business p ocesses, examining and compa ing 25 ele an pape s published be ween 2008
and 2017 based on c i e ia such as applica ion domain, inpu da a, p edic ion algo i hm,
and p ocess awa eness. A quan i a i e compa ison was pe o med ia a benchma k o 16
emaining ime p edic ion me hods on a ious publicly a ailable da ase s.
While black-box ML algo i hms excel in p edic i e accu acy o p ocess moni o ing, hei
inhe en opaci y o en lea es use s elian on less e ec i e bu anspa en models (Neu e al.,
2022;A ie ae al.,2020).In hislandscape,XAIhasa isenasa i ala eao esea ch,aimeda
b idging he gap be ween AI pe o mance and human in e p e abili y. By doing so, XAI seeks
o imp o e use us and acili a e e ec i e collabo a ion be ween in elligen sys ems and
human ope a o s (Mehdiye and Fe ke, 2021; Guido i e al., 2018). Se e al comp ehensi e
e iews ha e con ibu ed o he unde s anding o a ious aspec s o explana o y echniques
wi hin AI and ML (Emme -S eib e al., 2020).
One c i ical ocus has been he explo a ion o local e sus global explana ion me hods.
These me hods di e in hei app oaches and implica ions, o e ing cus omized s a egies
o explana ion based on he con ex o hei applica ion (Adadi and Be ada, 2018). Ex end-
ing his, esea ch has gone in o unde s anding he in e ac ion be ween speci ic explana ion
echniques and he models hey seek o make anspa en . The ca ego iza ion o hese ech-
niques as ei he model-cen ic o model-agnos ic has been ins umen al in guiding hei
applica ion (Angelo e al., 2021). Fu he mo e, an inclusi e app oach has been adop ed o
in ol e di e se s akeholde s in he design and deploymen s ages o explana ion echniques.
This inclusi i y allows o a mo e nuanced implemen a ion ha mee s a ied equi emen s
and pe spec i es (A ie a e al., 2020). Simul aneously, he e has been a conce ed e o o
elucida e he o e a ching goals o explana o y mechanisms. These s udies e eal he mo i-
a ions and in ended ou comes d i ing hei de elopmen and deploymen (Mehdiye and
Fe ke, 2021). The mul i-disciplina y na u e o his esea ch has enabled he inco po a ion o
insigh s om cogni i e and social sciences. This app oach en iches ou unde s anding o he
human ac o s ha a ec he comp ehension and accep ance o machine-gene a ed explana-
ions (Mille , 2019). Addi ionally, he con ex ual elemen s ha e been conside ed in assessing
explana o y echniques, unde lining hei impo ance in a eal-wo ld, decision-making en i-
onmen . E alua ion c i e ia and benchma ks ha e also been es ablished o a igo ous and
objec i e assessmen o explana o y me hods’ e icacy and u ili y (Vilone and Longo, 2021;
an de Waa e al., 2021).
In he con ex o p edic i e p ocess moni o ing, he ocus has equen ly been on employ-
ing XAI echniques, p ima ily h ough pos -hoc explana ion mechanisms (Ha l e al., 2020;
S e ens e al., 2022; Velmu ugan e al., 2021; Mehdiye and Fe ke, 2021,2020). Ano he
eme ging end in ML esea ch and p edic i e p ocess moni o ing is UQ, which aims o cap-
u e and e ec i ely communica e he inhe en unce ain ies in model p edic ions. UQ o e s
an addi ional laye o anspa ency, augmen ing he comp ehensibili y o decisions de i ed
om machine in elligence (Bha e al., 2021). The u ili y o UQ ex ends o aiding s ake-
holde s in asce aining when o us model p edic ions, hus enhancing he unc ionali y o
au oma ed decision sys ems. By quan i ying and inco po a ing unce ain y sys ema ically,
UQ c ea es mo e obus and eliable decision-making amewo ks, pa icula ly when in o -
123
Annals o Ope a ions Resea ch (2025) 347:991–1030 1027
ma ionis ambiguous (Ghanem e al., 2017). The bene i s o employing UQme hodologies a e
mul idisciplina y, p o ing use ul in sec o s anging om enginee ing and inance o en i on-
men al managemen (Smi h, 2013). Se e al ecen e o s ha e explo ed unce ain y wi hin
p edic i e p ocess moni o ing (Wey jens and De Wee d , 2022; Shoush and Dumas, 2022).
Despi e he indi idual ad ancemen s in UQ and XAI, he in e sec ion o hese wo ields
emains la gely unexplo ed in he academic li e a u e. Limi ed s udies ha e conside ed he
bidi ec ional in eg a ion o UQ and XAI, ocusing on cla i ying he o igins o unce ain ies
and in es iga ing he unce ain ies embedded wi hin explana ions hemsel es (Slack e al.,
2021; An o án e al., 2020; Moosbaue e al., 2021). This a icle endea o s o add ess his
esea ch gap. I aims o make he unce ain ies associa ed wi h ML models mo e accessible o
domain expe s, pa icula ly wi hin he con ex o p edic i e p ocess moni o ing. To he bes
o ou knowledge, his s udy is unique in i s app oach o me ge UQ and XAI me hodologies
speci ically o p edic i ep ocessmoni o ing p oblems.I is apionee inge o in hisnascen
ield, a ge ing he o mula ion o mo e anspa en , obus , and eliable decision suppo
8 Conclusion
Insumma y, hiss udyp esen sacomp ehensi eapp oach oadd essing hecomplexp oblem
o p edic ing he ime o comple ion o a ious manu ac u ing p ocesses, while also quan i-
ying and explaining he associa ed unce ain ies. By le e aging ad anced machine lea ning
echniques such as QRF and SHAP analysis, we ha e been able o gene a e explainable,
unce ain y-awa e p edic ions ha a e c ucial o eal-wo ld applica ions. These p edic ions
se e as a sophis ica ed p epa a o y laye o subsequen op imiza ion s eps, aligning closely
wi h he “p edic - hen-op imize” pa adigm p e alen in OR. Ou compa a i e analysis p o-
ides a obus amewo k o e alua ing he e icacy o ou model agains bo h indus y
p ac ices and a ange o al e na i e p edic i e models. The inclusion o s a is ical es s and
eal-wo ld eedback om p ocess owne s adds u he c edibili y o ou indings. By o e ing
bo h heo e ical and empi ical alida ion, we belie e his wo k makes a signi ican con-
ibu ion o he ields o ML and OR, pa icula ly in he con ex o manu ac u ing. The
me hodology de eloped he e is no only applicable o he speci ic use case p esen ed bu
also holds p omise o b oade applica ions, he eby opening a enues o u u e esea ch and
p ac ical implemen a ions.
Funding Open Access unding enabled and o ganized by P ojek DEAL. This esea ch was unded in pa by
he Ge man Fede al Minis y o Educa ion and Resea ch unde g an numbe 01IS21006B (p ojec ExP o).
Decla a ions
Con lic o in e es The au ho s decla e ha hey ha e no Con lic o in e es .
E hical app o al This a icle does no con ain any s udies wi h human pa icipan s o animals pe o med by
any o he au ho s.
Open Access This a icle is licensed unde a C ea i e Commons A ibu ion 4.0 In e na ional License, which
pe mi s use, sha ing, adap a ion, dis ibu ion and ep oduc ion in any medium o o ma , as long as you gi e
app op ia e c edi o he o iginal au ho (s) and he sou ce, p o ide a link o he C ea i e Commons licence,
and indica e i changes we e made. The images o o he hi d pa y ma e ial in his a icle a e included in he
a icle’s C ea i e Commons licence, unless indica ed o he wise in a c edi line o he ma e ial. I ma e ial is
no included in he a icle’s C ea i e Commons licence and you in ended use is no pe mi ed by s a u o y
egula ion o exceeds he pe mi ed use, you will need o ob ain pe mission di ec ly om he copy igh holde .
To iew a copy o his licence, isi h p://c ea i ecommons.o g/licenses/by/4.0/.
123
1028 Annals o Ope a ions Resea ch (2025) 347:991–1030
Re e ences
Adadi, A., & Be ada, M. (2018). Peeking inside he black-box: A su ey on explainable a i icial in elligence
(XAI). IEEE access, 6, 52138–52160.
Angelo , P. P., Soa es, E. A., Jiang, R., A nold, N. I., & A kinson, P. M. (2021). Explainable a i icial in elli-
gence: An analy ical e iew. Wiley In e disciplina y Re iews: Da a Mining and Knowledge Disco e y,
11(5), 1424.
An o án, J., Bha , U., Adel, T., Welle , A., & He nández-Loba o, J. M. (2020). Ge ing a clue: A me hod o
explaining unce ain y es ima es. a Xi p ep in a Xi :2006.06848.
A ie a, A. B., Díaz-Rod íguez, N., Del Se , J., Benne o , A., Tabik, S., Ba bado, A., Ga cía, S., Gil-López, S.,
Molina, D., Benjamins, R., e al. (2020). Explainable a i icial in elligence (xai): Concep s, axonomies,
oppo uni ies and challenges owa d esponsible ai. In o ma ion Fusion, 58, 82–115.
Bengio, Y., Lodi, A., & P ou os , A. (2021). Machine lea ning o combina o ial op imiza ion: A me hod-
ological ou d’ho izon. Eu opean Jou nal o Ope a ional Resea ch, 290(2), 405–421.
Benne , K. P., & Pa ado-He nández, E. (2006). The in e play o op imiza ion and machine lea ning esea ch.
The Jou nal o Machine Lea ning Resea ch, 7, 1265–1281.
Be simas, D., & Kallus, N. (2020). F om p edic i e o p esc ip i e analy ics. Managemen Science, 66(3),
1025–1044.
Bha , U., An o án, J., Zhang, Y., Liao, Q. V., Sa ige i, P., Foglia o, R., Melançon, G., K ishnan, R., S anley,
J., & Tickoo, O. (2021). Unce ain y as a o m o anspa ency: Measu ing, communica ing, and using
unce ain y. In P oceedings o he 2021 AAAI/ACM Con e ence on AI, E hics, and Socie y (pp. 401–413).
B eiman, L. (2001). Random o es s. Machine Lea ning, 45(1), 5–32.
Chaa i, T., Chaabane, S., Aissani, N., & T en esaux, D. (2014). Scheduling unde unce ain y: Su ey and
esea ch di ec ions. In: 2014 In e na ional Con e ence on Ad anced Logis ics and T anspo (ICALT)
(pp. 229–234). IEEE.
Chai,T.,&D axle , R.R.(2014).Roo meansqua ee o (RMSE)o meanabsolu ee o (MAE)?—A gumen s
agains a oiding RMSE in he li e a u e. Geoscien i ic Model De elopmen , 7(3), 1247–1250.
Chung, S. H., Ma, H. L., & Chan, H. K. (2017). Cascading delay isk o ai line wo k o ce deploymen s wi h
c ew pai ing and schedule op imiza ion. Risk Analysis, 37(8), 1443–1458.
DiF ancescoma ino,C., Ghidini,C.,Maggi,F.M., &Milani,F. (2018).P edic i ep ocessmoni o ingme hods:
Which one sui s me bes ? In Business P ocess Managemen : 16 h In e na ional Con e ence, BPM 2018,
Sydney, NSW, Aus alia, Sep embe 9–14, 2018, P oceedings 16 (pp. 462–479). Sp inge .
Dumas, M., La Rosa, M., Mendling, J., Reije s, H. A., Dumas, M., La Rosa, M., Mendling, J., & Reije s, H. A.
(2018). In oduc ion o business p ocess managemen . Fundamen als o Business P ocess Managemen ,
1, 1–33.
Ehsan, B. M. A., Begum, F., Ilham, S. J., & Khan, R. S. (2019). Ad anced wind speed p edic ion using
con ec i ewea he a iables h oughmachinelea ningapplica ion.Applied Compu ing and Geosciences,
1, 100002.
Emme -S eib, F., Yli-Ha ja, O., & Dehme , M. (2020). Explainable a i icial in elligence and machine
lea ning: A eali y oo ed pe spec i e. Wiley In e disciplina y Re iews: Da a Mining and Knowledge
Disco e y, 10(6), 1368.
E e mann, J., Rehse, J.-R., & Fe ke, P. (2017). P edic ing p ocess beha iou using deep lea ning. Decision
Suppo Sys ems, 100, 129–140.
F azze o,D.,Nielsen, T.D., Pede sen, T.B., & Šikšnys,L.(2019).P esc ip i eanaly ics:A su eyo eme ging
ends and echnologies. The VLDB Jou nal, 28, 575–595.
F iedman, M. (1937). The use o anks o a oid he assump ion o no mali y implici in he analysis o a iance.
Jou nal o he Ame ican S a is ical Associa ion, 32(200), 675–701.
Ga cia, S., & He e a, F. (2008). An ex ension on “s a is ical compa isons o classi ie s o e mul iple da a
se s” o all pai wise compa isons. Jou nal o Machine Lea ning Resea ch,9(12), 1.
Ghanem, R., Higdon, D., Owhadi, H., e al. (2017). Handbook o Unce ain y Quan i ica ion (Vol. 6). Sp inge .
Guido i, R., Mon eale, A., Ruggie i, S., Tu ini, F., Gianno i, F., & Ped eschi, D. (2018). A su ey o me hods
o explaining black box models. ACM Compu ing Su eys (CSUR), 51(5), 1–42.
Ha l, M., Weinzie l, S., S ie le, M., & Ma zne , M. (2020). Explainable p edic i e business p ocess moni o ing
using ga ed g aph neu al ne wo ks. Jou nal o Decision Sys ems, 29(sup1), 312–327.
He,J., Wanik,D. W., Ha man, B. M.,Anagnos ou, E.N., As i ha,M., &F ediani, M.E. (2017). Nonpa ame ic
ee-based p edic i e modeling o s o m ou ages on an elec ic dis ibu ion ne wo k. Risk Analysis, 37(3),
441–458.
Isaksson, A. J., Ha junkoski, I., & Sand, G. (2018). The impac o digi aliza ion on he u u e o con ol and
ope a ions. Compu e s & Chemical Enginee ing, 114, 122–129.
123
Annals o Ope a ions Resea ch (2025) 347:991–1030 1029
Jø gensen, M., Teigen, K. H., & Moløkken, K. (2004). Be e su e han sa e? o e -con idence in judgemen
based so wa e de elopmen e o p edic ion in e als. Jou nal o Sys ems and So wa e, 70(1), 79–93.
Klas,M., T endowicz, A.,Ishigai,Y.,& Nakao, H.(2011). Handling es ima ionunce ain y wi h boo s apping:
Empi ical e alua ion in he con ex o hyb id p edic ion me hods. In 2011 In e na ional Symposium on
Empi ical So wa e Enginee ing and Measu emen (pp. 245–254).
Kuma , R., Ganapa hy, L., Gokhale, R., & Tiwa i, M. K. (2020). Quan i a i e app oaches o he in eg a ion
o p oduc ion and dis ibu ion planning in he supply chain: A sys ema ic li e a u e e iew. In e na ional
Jou nal o P oduc ion Resea ch, 58(11), 3527–3553.
Lepenio i, K., Bousdekis, A., Apos olou, D., & Men zas, G. (2020). P esc ip i e analy ics: Li e a u e e iew
and esea ch challenges. In e na ional Jou nal o In o ma ion Managemen , 50, 57–70.
Lipo e sky, S., & Conklin, M. (2001). Analysis o eg ession in game heo y app oach. Applied S ochas ic
Models in Business and Indus y, 17(4), 319–330.
Lundbe g, S. M., & Lee, S.-I. (2017). A uni ied app oach o in e p e ing model p edic ions. In P oceedings o
he 31s in e na ional con e ence on neu al in o ma ion p ocessing sys ems. NIPS’17 (pp. 4768–4777).
Maggi, F. M., Di F ancescoma ino, C., Dumas, M., & Ghidini, C. (2014) P edic i e moni o ing o business
p ocesses. In Ad anced In o ma ion Sys ems Enginee ing: 26 h In e na ional Con e ence, CAiSE 2014,
Thessaloniki, G eece, June 16-20, 2014. P oceedings 26 (pp. 457–472). Sp inge .
Mehdiye , N., & Fe ke, P. (2020). P esc ip i e p ocess analy ics wi h deep lea ning and explainable a i icial
in elligence. In 28 h Eu opean Con e ence on In o ma ion Sys ems (ECIS). An Online AIS Con e ence.
Mehdiye , N., & Fe ke, P. (2021). Explainable a i icial in elligence o p ocess mining: A gene al o e iew
and applica ion o a no el local explana ion app oach o p edic i e p ocess moni o ing. In In e p e able
A i icial In elligence: A Pe spec i e o G anula Compu ing (pp. 1–28).
Mehdiye , N., & Fe ke, P. (2021). Local pos -hoc explana ions o p edic i e p ocess moni o ing in manu-
ac u ing. In 29 h Eu opean Con e ence on In o ma ion Sys ems (ECIS). An Online AIS Con e ence.
Meinshausen, N. (2006). Quan ile eg ession o es s. The Jou nal o Machine Lea ning Resea ch, 7, 983–999.
Mille , T. (2019). Explana ion in a i icial in elligence: Insigh s om he social sciences. A i icial In elligence,
267, 1–38.
Mil on, F. (1939). A co ec ion: The use o anks o a oid he assump ion o no mali y implici in he analysis
o a iance. Jou nal o he Ame ican S a is ical Associa ion, 34(205), 109.
Miši´c, V. V., & Pe akis, G. (2020). Da a analy ics in ope a ions managemen : A e iew. Manu ac u ing &
Se ice Ope a ions Managemen , 22(1), 158–169.
Mi en sis, G., & Lens, H. (2022). An in e p e able p obabilis ic model o sho - e m sola powe o ecas ing
using na u al g adien boos ing. Applied Ene gy, 309, 118473.
Molina o, A. M., Simon, R., & P ei e , R. M. (2005). P edic ion e o es ima ion: A compa ison o esampling
me hods. Bioin o ma ics, 21(15), 3301–3307.
Moosbaue , J., He binge , J., Casalicchio, G., Lindaue , M., & Bischl, B. (2021). Explaining hype pa ame e
op imiza ion ia pa ial dependence plo s. Ad ances in Neu al In o ma ion P ocessing Sys ems, 34,
2280–2291.
Mula, J., Pole , R., Ga cía-Saba e , J. P., & La io, F. C. (2006). Models o p oduc ion planning unde unce -
ain y: A e iew. In e na ional Jou nal o P oduc ion Economics, 103(1), 271–285.
Nemenyi, P. B. (1963). Dis ibu ion- ee Mul iple Compa isons. Depa men o Ma hema ics, P ince on Uni-
e si y.
Neu, D. A., Lahann, J., & Fe ke, P. (2022). A sys ema ic li e a u e e iew on s a e-o - he-a deep lea ning
me hods o p ocess p edic ion. A i icial In elligence Re iew, 1, 1–27.
Pola o,M., Spe du i,A., Bu a in, A.,& deLeoni, M.(2014). Da a-awa e emaining ime p edic iono business
p ocess ins ances. In: 2014 In e na ional Join Con e ence on Neu al Ne wo ks (IJCNN) (pp. 816–823).
IEEE.
Rehse, J.-R., Mehdiye , N., & Fe ke, P. (2019). Towa ds explainable p ocess p edic ions o indus y 4.0 in
he d ki-sma -lego- ac o y. KI-Küns liche In elligenz, 33, 181–187.
Shoush, M., & Dumas, M. (2022). When o in e ene? p esc ip i e p ocess moni o ing unde unce ain y and
esou ce cons ain s. In Business P ocess Managemen Fo um: BPM 2022 Fo um, Müns e , Ge many,
Sep embe 11–16, 2022, P oceedings (pp. 207–223). Sp inge .
Sh es ha, D. L., & Soloma ine, D. P. (2006). Machine lea ning app oaches o es ima ion o p edic ion in e al
o he model ou pu . Neu al Ne wo ks, 19(2), 225–235.
Simon, H. A. (1997). The u u e o in o ma ion sys ems. Annals o Ope a ions Resea ch, 71, 3–14.
Slack,D., Hilga d,A., Singh,S., &Lakka aju, H. (2021). Reliable pos hoc explana ions:Modelingunce ain y
in explainabili y. Ad ances in Neu al In o ma ion P ocessing Sys ems, 34, 9391–9404.
Smi h, R. C. (2013). Unce ain y Quan i ica ion: Theo y, Implemen a ion, and Applica ions (Vol. 12). SIAM.
123
1030 Annals o Ope a ions Resea ch (2025) 347:991–1030
S e ens, A., De Smed , J., & Peepe ko n, J. (2022). Quan i ying explainabili y in ou come-o ien ed p edic i e
p ocess moni o ing. In P ocess Mining Wo kshops: ICPM 2021 In e na ional Wo kshops, Eindho en,
The Ne he lands, Oc obe 31–No embe 4, 2021, Re ised Selec ed Pape s (pp. 194–206). Sp inge .
Tax, N., Ve enich, I., La Rosa, M., & Dumas, M. (2017). P edic i e business p ocess moni o ing wi h LSTM
neu al ne wo ks. In Ad anced In o ma ion Sys ems Enginee ing: 29 h In e na ional Con e ence, CAiSE
2017, Essen, Ge many, June 12-16, 2017, P oceedings 29 (pp. 477–492). Sp inge .
Teinemaa, I., Dumas, M., Leon je a, A., & Maggi, F. M. (2018). Tempo al s abili y in p edic i e p ocess
moni o ing. Da a Mining and Knowledge Disco e y, 32, 1306–1338.
Teinemaa,I.,Dumas,M.,Rosa,M.L.,&Maggi,F.M.(2019).Ou come-o ien edp edic i ep ocessmoni o ing:
Re iew and benchma k. ACM T ansac ions on Knowledge Disco e y om Da a (TKDD), 13(2), 1–57.
Thomé, A. M. T., Sca a da, L. F., Fe nandez, N. S., & Sca a da, A. J. (2012). Sales and ope a ions planning:
A esea ch syn hesis. In e na ional Jou nal o P oduc ion Economics, 138(1), 1–13.
Usuga Cada id, J. P., Lamou i, S., G abo , B., Pelle in, R., & Fo in, A. (2020). Machine lea ning applied
in p oduc ion planning and con ol: A s a e-o - he-a in he e a o indus y 4.0. Jou nal o In elligen
Manu ac u ing, 31, 1531–1558.
an de Aals , W. M. P. (2022). P ocess mining: A 360 deg ee o e iew. Lec u e No es in Business In o ma ion
P ocessing. In W. M. P. an de Aals & J. Ca mona (Eds.), P ocess Mining Handbook (Vol. 448, pp.
3–34). Sp inge .
an de Waa, J., Nieuwbu g, E., C eme s, A., & Nee incx, M. (2021). E alua ing xai: A compa ison o ule-
based and example-based explana ions. A i icial In elligence, 291, 103404.
Van De Aals , W., Ad iansyah, A., De Medei os, A. K. A., A cie i, F., Baie , T., Blickle, T., Bose, J.C.,
Van Den B and, P., B and jen, R., & Buijs, J. (2012). P ocess mining mani es o. In Business P ocess
Managemen Wo kshops: BPM 2011 In e na ional Wo kshops, Cle mon -Fe and, F ance, Augus 29,
2011, Re ised Selec ed Pape s, Pa I 9 (pp. 169–194). Sp inge .
an Jaa s eld, W., & Schelle -Wol , A. (2015). Op imiza ion o indus ial-scale assemble- o-o de sys ems.
INFORMS Jou nal on Compu ing, 27(3), 544–560.
an de Aals , W. M. P. (2016). P ocess mining: Da a science in ac ion, 2nd edn.
Velmu ugan, M., Ouyang, C., Mo ei a, C., & Sindhga a, R. (2021). E alua ing ideli y o explainable me hods
o p edic i e p ocess analy ics. In In elligen In o ma ion Sys ems: CAiSE Fo um 2021, Melbou ne, VIC,
Aus alia, June 28–July 2, 2021, P oceedings (pp. 64–72). Sp inge .
Ve enich, I., Dumas, M., Rosa, M. L., Maggi, F. M., & Teinemaa, I. (2019). Su ey and c oss-benchma k
compa ison o emaining ime p edic ion me hods in business p ocess moni o ing. ACM T ansac ions
on In elligen Sys ems and Technology (TIST), 10(4), 1–34.
Vilone, G., & Longo, L. (2021). No ions o explainabili y and e alua ion app oaches o explainable a i icial
in elligence. In o ma ion Fusion, 76, 89–106.
Wey jens, H., & De Wee d , J. (2022). Lea ning unce ain y wi h a i icial neu al ne wo ks o p edic i e
p ocess moni o ing. Applied So Compu ing, 125, 109134.
Willmo , C. J., & Ma suu a, K. (2005). Ad an ages o he mean absolu e e o (MAE) o e he oo mean
squa e e o (RMSE) in assessing a e age model pe o mance. Clima e Resea ch, 30(1), 79–82.
Xue, Z., Wang, Z., & E l, M. (2016). P icing pe sonalized bundles: A new app oach and an empi ical s udy.
Manu ac u ing & Se ice Ope a ions Managemen , 18(1), 51–68.
Youn, S., Geisma , H. N., & Pinedo, M. (2022). Planning and scheduling in heal hca e o be e ca e coo -
dina ion: Cu en unde s anding, ending opics, and u u e oppo uni ies. P oduc ion and Ope a ions
Managemen , 31(12), 4407–4423.
Zhou, Y., Boo h, S., Ribei o, M. T., & Shah, J. (2022). Do ea u e a ibu ion me hods co ec ly a ibu e
ea u es? In P oceedings o he AAAI Con e ence on A i icial In elligence (Vol. 36, pp. 9623–9633).
Publishe ’s No e Sp inge Na u e emains neu al wi h ega d o ju isdic ional claims in published maps and
ins i u ional a ilia ions.
123
