Upda ing P edic ion Models
o P edic i e P ocess Moni o ing
Al onso E. M´a quez-Chamo o1,2(B), Isabel A. Nepomuceno-Chamo o1,
Manuel Resinas1,2 , and An onio Ruiz-Co ´es1,2
1I3US Ins i u e, Uni e sidad de Se illa, Se ille, Spain
{ama quez6,inepomuceno, esinas,a uiz}@us.es
2SCORE Lab, Uni e sidad de Se illa, Se ille, Spain
Abs ac . P edic i e moni o ing is a key ac i i y in some P ocess-
Awa e In o ma ion Sys ems (PAIS) such as in o ma ion sys ems o
ope a ional managemen suppo . Un o eseen ci cums ances like COVID
can in oduce changes in human beha iou , p ocesses, o compu ing
esou ces, which lead he owne o he p ocess o in o ma ion sys em o
conside whe he he quali y o he p edic ions made by he sys em (e.g.,
mean ime o solu ion) is s ill good enough, and i no , which amoun o
da a and how o en he sys em should be ained o main ain he qual-
i y o he p edic ions. To answe hese ques ions, we p opose, compa e,
and e alua e diffe en s a egies o selec ing he amoun o in o ma ion
equi ed o upda e he p edic i e model in a con ex o offline lea ning.
We pe o med an empi ical e alua ion using h ee eal-wo ld da ase s
ha span be ween 2 and 13 yea s o alida e he diffe en s a egies
which show a significan enhancemen in he p edic ion accu acy wi h
espec o a non-upda e s a egy.
Keywo ds: P edic i e p ocess moni o ing ·P ocess mining ·
P ocess-awa e in o ma ion sys ems ·P edic ion models ·Model
upda ing
1 In oduc ion
P edic i e p ocess moni o ing (PPM) p o ides p oac i e and co ec i e ac ions
o imp o e he p ocess pe o mance and mi iga e po en ial isks in eal ime.
PPM e ie es in o ma ion om P ocess-Awa e In o ma ion Sys ems (PAIS)
s o ed in e en logs o make p edic ions o e alua ion me ics, also known as
p ocess pe o mance indica o s (PPIs) [1]. A pa h ex ensi ely ollowed in he
li e a u e o p edic i e moni o ing is adap ing exis ing machine lea ning ech-
niques [2] such as decision ees, clus e ing me hods o neu al ne wo ks o ob ain
p edic i e models wi h highe accu acy. When hese app oaches a e used, he
Wo k unded by g an s RTI2018-101204-B-C21 and RTI2018-101204-B-C22 unded by
MCIN/ AEI/ 10.13039/501100011033/ and ERDF A way o making Eu ope; g an
P18-FR-2895 and US-1381595 unded by Jun a de Andaluc´ıa/ERDF, UE.
ypical p ocedu e o p edic i e moni o ing comp ises wo s eps. Fi s , a aining
s age in which he p edic i e models a e ained using da a collec ed in he e en
logs. Second, once he model is buil , i is deployed and i is used o p edic PPIs
o cu en and/o u u e p ocess execu ions.
In he absence o significan changes, ce e is pa ibus (all else being equal),
his app oach wo ks fine, bu p ocesses a e subjec o con inuous changes. Fo
ins ance, he esponse o COVID may in oduce new ways o pe o ming ac i -
i ies, use s can beha e in a diffe en way, o human o compu ing esou ces can
change o e ime. These changes may nega i ely affec he pe o mance o he
p edic i e model since he da a used o ain hem does no eflec eali y any
mo e. The e o e, he only way o keep his pe o mance o e a desi ed h eshold
is by adap ing he model o he changes.
In he machine lea ning communi y, he e a e wo main adap a ion
app oaches o ha , namely, online and offline lea ning. In online lea ning, he
p edic i e model is being upda ed con inuously om he da a i ecei es. Con-
e sely, in offline lea ning, he p edic i e model is ebuil again om he g ound.
In his pape , we decide o ocus on offline lea ning mainly o wo easons.
Fi s ly, he pace o change and he pace o new e en s in he p ocesses we a e
in e es ed in, gi es enough ime o comple ely ebuild new models. Secondly,
i s use allows one o euse a huge amoun o machine lea ning echniques ha
a e a ailable o offline lea ning, which is much mo e comp ehensi e han ha o
online lea ning. Fu he mo e, hese echniques do no need o make comp omises
o keep a easonable lea ning ime.
In his con ex , he goal o his pape is o p o ide de ails on how o ace
wo o he ques ions ha a ise in he upda e o p edic i e models: “Which da a
should be conside ed in he new model ha is being buil ?” and “How he
selec ion o da a does impac on he pe o mance o he p edic i e models?”.
By answe ing hese ques ions, we con ibu e o he s a e o he a on PPM by
p oposing six diffe en s a egies o upda ing p edic i e models (baseline, cumu-
la i e, non- cumula i e, ensemble, sampling, and concep d i ) and compa ing
hei pe o mance. Ou expe imen a ion was alida ed using h ee eal-li e e en
logs ha span be ween 2 and 13 yea s. We ha e also pe o med a compa ison o
diffe en well-known classifie s used in ela ed li e a u e.
The eminde o his pape is o ganized as ollows. Sec ion 2summa ises basic
concep s in p edic i e moni o ing. Sec ion 3p esen s he s a egies o upda ing
p edic i e models. The expe imen and he discussion o he ob ained esul s
a e p esen ed in Sec . 4. Sec ion 5summa ises he ela ed wo k. Finally, Sec . 6
concludes he wo k and p esen s possible u u e di ec ions.
2 P edic i e P ocess Moni o ing
In he ollowing, we in oduce some basic concep s o p edic i e p ocess moni-
o ing. As defined in [3], an e en log (L) is composed o a se o aces (T). Each
ace (Ti) eflec s an execu ion o a p ocess ins ance. Fo mally, we can exp ess
a ace as an o de ed lis o e en s Ti=[Ei1,...,E
im] whe e Ei1 ep esen s he
fi s e en and Eim he final e en o ace Ti. Simila ly, a log can be exp essed
as he se o aces o he ins ances ha ha e finished in an in e al o ime
L=[T1,...,T
n] whe e T1is he fi s and Tnis he las execu ed ace in he ime
in e al. Finally, an e en ep esen s he execu ion o an ac i i y o he p ocess.
Each e en con ains a se o a ibu es (a), which ep esen s in o ma ion ela ed
o such e en , e.g. imes amp, he esou ce ha execu es he ac i i y, o he
alue o some da a used h oughou he ins ance, Ej=[aj1,...,a
jo] whe e o
de e mines he o al numbe o a ibu es o he e en .
A p ocess indica o (I) is a quan ifiable me ic ocused on measu ing he
p og ess owa d a goal o s a egic objec i e. Indica o s can be classified in o
wo ypes: single-ins ance indica o s o agg ega ed indica o s. The o me is
compu ed o each ace in he log using he alues o he a ibu es o he
e en s ha compose his ace. The e o e, i can be defined as a unc ion o a
ace Ti,i.e. I(Ti). This unc ion can e u n a bina y alue, e.g a de e mined
condi ion ulfilled by he ace, o a eal alue, e.g he du a ion o an ac i i y.
Ins ead, an agg ega ed indica o is compu ed o a se aces by agg ega ing a
single-ins ance indica o using some agg ega ion unc ion, e.g. sum o a e age.
An example o his ype o indica o could be he pe cen age o inciden s sol ed
in a ce ain pe iod o ime.
A p edic i e model o an indica o Iis a unc ion PI([Eik,...,E
il]), wi h
k≤l, ha compu es a p edic ion o I om he pa ial ace [Eik,...,E
il],
whe e Eilis he las e en ha ha e occu ed in ace Tia a gi en momen .
I k= 1, hen all e en s ha ha e occu ed in he p ocess ins ance a hand a e
conside ed. Ins ead, i k=l, hen only he las e en o he p ocess ins ance is
conside ed.
In o de o ain a p edic i e model ˆ
I o a key pe o mance indica o (KPI) I,
an encoded fixed-size ep esen a ion Co all he cases C, whe e C⊆C, included
in he aining se is equi ed. This encoding, gene ally ep esen ed as a ea u e
ma ix (X), should s o e enough in o ma ion o he p ocess, and will be used as
inpu o he machine lea ning echnique employed o build he model oge he
wi h he alue o he KPI I o each case in C, which ep esen s he a ge
a iable (y). The ea u e ma ix Xis ob ained a e applying a sequence encoding
unc ion F, which ecei es a se o cases Cand e u ns a ma ix X. Each ow o
he ma ix ep esen s an e en E, i.e. he execu ion o an ac i i y o he p ocess,
o a case c∈Cand each column ep esen s he diffe en (encoded) a ibu es
ao he e en . Va ious sequence encoding echniques ha e been p oposed in
he li e a u e o his ask such as las s a e encoding [4], agg ega ion encoding
[5], o index-based encoding [6]. The o he decision is i only one classifie is
ained o he whole da ase o , on he con a y, i cases a e g ouped in o se e al
bucke s and a diffe en classifie is ained o each one. Se e al case bucke ing
echniques ha e been p oposed in he li e a u e [7]: Ze o bucke ing [5], p efix
leng h bucke ing [6], o clus e bucke ing [8]. A e hese wo decisions a e made,
a p edic i e model is buil using some machine lea ning algo i hm using he pai
(X, y) as he inpu .
Fig. 1. Upda ing models sys em in a p edic i e moni o ing p ocess.
Fig. 2. T aining s age.
Fig. 3. Run- ime moni o ing s age.
An indica o Ican ep esen diffe en issues, such as a ce ain ou come, he
nex ac i i y o he p ocess o emaining cycle ime o a gi en p ocess case. In
his wo k we ha e ocused on he ou come-based p edic ion. The e o e, we a e
p edic ing an ou come alue pe case ins ead o a alue pe each e en .
3 Upda ing P edic i e Models
Once he p edic i e models a e gene a ed ollowing he mechanisms desc ibed in
he p e ious sec ion a e deployed in o p oduc ion, hey s a making p edic ions
ha can be used o imely eac o ope a ional issues. Howe e , a e a while,
he way he p ocess was pe o med migh ha e changes; esou ces pa icipa ing
in he p ocess may come and go; e en he s uc u e o he p ocess may suffe
changes. All hese changes a e igno ed by he p edic i e model ha was deployed
some ime ago, so hese changes may nega i ely affec he pe o mance o he
p edic i e model. The e o e, i is necessa y o p o ide mechanisms o upda e he
p edic i e model. Figu e 1shows a sys em o he upda ing o p edic i e models
desc ibed below. I consis s o a aining s age depic ed in Fig. 2(which in ol es
he fil e ing o he e en log, he gene a ion o he p edic i e model and he
e alua ion o his model), he deploymen o his model, he p edic ion using his
model, and finally, a mechanism o he upda e o he dep eca ed models, named
Run- ime moni o ing p ocess as shown in Fig. 3. This mechanism can e alua e
he model in e ms o he pe o mance o he p edic ions and decide when he
p edic i e model should be upda ed acco ding o h ee possible pa ame e s: he
ime elapsed om he las deployed model, he accu acy o p edic ions and he
possible occu ence o a concep d i [9].
As men ioned in he in oduc ion, he e a e wo app oaches o upda ing
p edic i e models, namely online and offline lea ning. In his pape , we ocus
on offline lea ning because i allows one o use he huge amoun o machine
lea ning echniques o offline lea ning, which is much mo e comp ehensi e han
ha o online lea ning and, u he mo e, hese echniques do no need o make
comp omises in o de o keep a easonable lea ning ime. To he bes o ou
knowledge, any o he wo k ela ed o offline s a egies o upda ing p edic i e
models appea s in he li e a u e, so ha compa ison wi h o he pape s is no pos-
sible. In he con ex o offline lea ning, wo basic ques ions need o be answe ed
o upda e he p edic i e model:
1. When should he p edic i e model be upda ed?
2. Which da a should be conside ed in he new model ha is being buil ?
Fo he fi s ques ion, se e al s a egies can be conside ed (Run- ime mon-
i o ing s age in Fig. 3). The mos s aigh o wa d is a pe iodical upda e o he
model [10]. A easonable deadline o he change o he model can be fixed, e.g.
six-mon hly pe iodici y, and hen, a new gene a ion o he model will be ca ied
ou . A diffe en s a egy migh in ol e moni o ing he accu acy o ou p edic-
ions. When i begins o dec ease o e an unin e up ed pe iod o ime, i may
be ecommended o change he p edic i e model. A h eshold o e o can be
se , and i he p edic ion exceeds his h eshold, he p edic ion model will be
upda ed. Finally, a hi d s a egy could in ol e using he de ec ion o d i s in
p ocesses [11] o igge upda es in he p edic i e model [10]. Se e al s a egies
could be also combined o design a mo e obus sys em.
Ou ocus in his pape is, howe e , on he second ques ion. This ques ion is
ele an because o wo easons ha in ol e he quali y and cos o p edic i e
models. Conce ning he o me , i he eason o upda ing he p edic i e model
is because he p ocess has changed, i is easonable o hink ha lea ning pas
beha iou s o he p ocess may no be beneficial o he pe o mance o he p e-
dic i e model, so i migh make sense no o include he whole da a se , bu only

he mos ecen beha iou . As o he la e , he compu a ional cos o building
a p edic i e model inc eases wi h he size o he inpu da a se . The e o e, a
goal should be o achie e he bes p edic i e pe o mance by using he smalles
possible inpu da a se . In [10], au ho s p opose wo possible solu ions o he
second ques ion: e aining and inc emen al upda e o a p edic i e model, how-
e e , all p edic i e algo i hms canno lea n inc emen ally (e.g. andom o es s).
The e o e, we ha e collec ed a se o s a egies o choosing he da a se used o
building a new p edic i e model ega dless o he p edic i e algo i hm used.
A s a egy o choosing he da a se used o building a new p edic i e model
can be seen as a unc ion S ha ecei es a aining and es se pai (X, y)and
e u ns ano he aining and es se pai (X,y), such ha X⊆Xand y⊆y.
Nex we de ail se e al possible da a selec ion s a egies. We use he no a ion
X[i,j] o selec he subse o X ha is be ween iand j, whe e iand jcould be
ei he an ins an in ime such as he 7 h o Ma ch o 2019, o an ins ance numbe
since he fi s one ecei ed. Fu he mo e, i hey ake he alue 0, i e e s o he
fi s e en in he da a se and i hey ake he alue c, i e e s o he las e en
ecei ed in he da a se . The e o e, X[0,c]=X.
In he ollowing, we p esen he diffe en s a egies o he selec ion o da a.
Figu e 5shows a g aphical ep esen a ion o he diffe en s a egies desc ibed
in his sec ion. Figu e 4depic s an e en log ha will be used o explain he
diffe en s a egies in Fig. 5. This e en log is spli in o se e al in e als om 1
o n. Each in e al ep esen s all he p ocess ins ances execu ed du ing a ce ain
pe iod o ime, e.g. six mon hs.
1. Baseline s a egy (SB): In his s a egy he model is no upda ed h ough-
ou he li e o p ocess:
SB(X, y)=(X[0,c],y
[0,c])
Figu e 5a shows a g aphical ep esen a ion o he baseline s a egy. Wi h his
s a egy, a fi s in e al is selec ed as he aining se , and i is no upda ed
h oughou he li e o he p ocess. We use he es o he in e als as es se s.
2. Cumula i e s a egy (SC): This s a egy in ol es including all he cases
ha a e a ailable o aining since he beginning:
SC(X, y)=(X[0,c],y
[0,c])
Cumula i e s a egy is ep esen ed in Fig. 5b. This s a egy in ol es adding
all ins ances o a p ocess as aining se . We spli he e en log in o aining
and es se s, and we inc emen ally add each in e al om 1 o nin he
aining se and use he es o he es se .
3. Non-cumula i e s a egy (SN): This s a egy in ol es choosing only he
mos ecen cases o aining. I includes a pa ame e n ha de e mines
how many ecen cases should be included in he aining:
SN(X, y)=(X[c− n,c],y
[c− n,c])
The ad an age o his s a egy in compa ison o he cumula i e s a egy is
ha i is mo e efficien compu a ionally and i migh sol e p oblems de i ed
om using cases ha do no ollow he cu en beha iou o he p ocess. The
d awback is ha ha ing less aining ins ances migh hu he pe o mance
o he p edic i e model.
Fo he non-cumula i e s a egy (Fig. 5c), we selec an unique in e al in he
aining phase. In his manne , we only include he mos ecen cases o
aining.
4. Ensemble s a egy (SE): This s a egy is simila o he non-cumula i e
s a egy because i only includes he mos ecen s cases in aining a new
model. The diffe ence is ha , unlike he non-cumula i e s a egy, his s a egy
do no h ow away olde models, bu keep hem and combine hem using some
ensemble echnique [12]. Fo ins ance, one can use a weigh ed o ing echnique
in which he p edic ion o each model is conside ed a o e and combined using
diffe en weigh s o each model o make he final p edic ion. These weigh s
can be upda ed each ime a new model is added o he ensemble so ha olde
models ha e a lowe weigh . To his end, weigh s can be modeled using an
exponen ial decay unc ion like e−λ . Fu he mo e, besides hese weigh s, i
he las model had e y bad pe o mance, we migh be in e es ed in emo ing
i om he ensemble so ha i does no hu he o e all pe o mance. To his
end, we se a h eshold pa ame e so ha i he quali y me ic o choice, e.g.
-sco e, o he p e ious model did no mee he h eshold in he las in e al,
i is emo ed om he ensemble.
The ad an age o his s a egy is ha i has almos he same compu a ional
cos as he non-cumula i e s a egy, bu i helps o a oid disca ding all o
he old cases. Howe e , he combina ion o he diffe en models migh no be
as powe ul as a model buil using he cumula i e s a egy ha includes all
p e ious cases.
In his s a egy, depic ed in Fig. 5d, we choose he same aining and es se s
and keep olde models o combine hem using some ensemble echnique o
achie e be e p edic ions.
5. Sampling s a egy (SS): This s a egy in ol es a weigh ed sampling o all
he cases ha a e a ailable o aining since he beginning. I includes a
pa ame e s ha de e mines he numbe o samples ha mus be ob ained
om he da a:
SS(X, y)=(sampling(X[0,c], s), sampling(y[0,c], s))
Whe e sampling is a unc ion ha akes s samples om Xo y, espec-
i ely. Sampling can also be weigh ed so ha i is mo e likely o ob ain mo e
ecen samples han olde samples. A simila app oach as he one used in he
ensemble s a egy can be used he e o define hese weigh s. The ad an age
o his app oach in compa ison o he cumula i e s a egy is ha i limi s
he compu a ional cos o he new model. Fu he mo e, unlike he ensemble
s a egy i elies on he machine lea ning algo i hm ins ead o in he o ing
mechanism o combine bo h in o ma ion om old and ecen cases.
Fig. 4. Rep esen a ion o a spli e en log.
Figu e 5e shows he Sampling s a egy. We build he aining se in a inc e-
men al way using a weigh ed s a egy whe e ecen samples a e mo e likely
o be selec ed han olde samples.
6. D i s a egy (SD): This s a egy is simila o he non-cumula i e one.
I includes he mos ecen cases o aining and, when a concep d i is
de ec ed, we include as aining se , all he cases a e a ce ain ime has
passed since he d i has occu ed.
Figu e 5 shows he d i s a egy. When he concep d i is de ec ed, aining
se is buil only wi h hose cases execu ed a e he d i de ec ion.
4 Expe imen al E alua ion
As we s a ed in Sec . 1, he goal o his pape is o define he diffe en s a e-
gies o he selec ion o da a (desc ibed in Sec . 3) and compa e he p edic i e
pe o mance o he diffe en s a egies p oposed. Based on his goal, we define
a esea ch ques ion o ou expe imen a ion: Wha is he impac o he diffe en
upda ing s a egies on he accu acy o p edic ions?
The es o he sec ion is o ganized as ollows: he expe imen se up is de ailed
in Sec . 4.1. The diffe en da ase s used in he expe imen a ion a e desc ibed in
Sec . 4.2. Finally, a discussion o he ob ained esul s is p o ided in Sec . 4.3.
4.1 Expe imen Se up
As p edic i e algo i hm we ha e used andom o es [13] as seen in p e ious
wo ks in he li e a u e [14]. This echnique combines p edic o ees such ha
each ee depends on he alues o a andom ec o es ed independen ly and
wi h he same dis ibu ion o each o hem. In [14], au ho s highligh ex eme
g adien boos ing (XGBoos ) and andom o es as wo o he bes echniques in
p edic i e moni o ing. Thus, we ha e selec ed andom o es because he quali y
o esul s wi h espec o XGBoos is simila and i consumes less compu a ional
ime.
We ha e selec ed a ypical agg ega ion encoding desc ibed in [14] as one o he
mos used in he li e a u e o encode he p ocess cases and also one o he bes
pe o me s [14]. Thus, all e en s since he beginning o he case a e conside ed.
An agg ega ion unc ion is applied o he alues aken by a specific a ibu e
h oughou he case li e ime. In ou case, his unc ion is he numbe o imes
ha each specific a ibu e appea s in he case ( equency encoding). We ha e
no di ided he cases in he e en log in o diffe en bucke s. This echnique is
named Ze o bucke ing as defined in [14]. We ha e also inco po a ed he o de o
he e en s as a new a ibu e in all he logs, as well as he elapsed ime be ween
Fig. 5. G aphical ep esen a ion o diffe en p oposed s a egies o choosing he da a
se used o building a new p edic i e model.
he e en and he beginning o he case and he ime be ween he p e ious
e en and he cu en one. The de ails and he code o he expe imen a ion a e
a ailable online1.
4.2 E en Logs
Th ee diffe en eal-li e e en logs we e conside ed in ou expe imen s: IT Depa -
men o an Andalusian o ganisa ion (ITA), BPI 2015 (BPI15) [15] and T affic
fines (TRAFFIC) [16]. These logs we e chosen because hey span se e al yea s:
2, 5 and 12, espec i ely, so hey a e use ul o e alua e he effec o possible
changes on he p ocess o e ime.
1h ps://gi hub.com/isa-g oup/p edic i e-moni o ing-e olu ion.