Updating Prediction Models for Predictive Process Monitoring

Upda ing P edic ion Models o P edic i e P ocess Moni o ing Al onso E. M´a quez-Chamo o1,2(B), Isabel A. Nepomuceno-Chamo o1, Manuel Resinas1,2 , and An onio Ruiz-Co ´es1,2 1I3US Ins i u e, Uni e sidad de Se illa, Se ille, Spain {ama quez6,inepomuceno, esinas,a uiz}@us.es 2SCORE Lab, Uni e sidad de Se illa, Se ille, Spain Abs ac . P edic i e moni o ing is a key ac i i y in some P ocess- Awa e In o ma ion Sys ems (PAIS) such as in o ma ion sys ems o ope a ional managemen suppo . Un o eseen ci cums ances like COVID can in oduce changes in human beha iou , p ocesses, o compu ing esou ces, which lead he owne o he p ocess o in o ma ion sys em o conside whe he he quali y o he p edic ions made by he sys em (e.g., mean ime o solu ion) is s ill good enough, and i no , which amoun o da a and how o en he sys em should be ained o main ain he qual- i y o he p edic ions. To answe hese ques ions, we p opose, compa e, and e alua e diffe en s a egies o selec ing he amoun o in o ma ion equi ed o upda e he p edic i e model in a con ex o offline lea ning. We pe o med an empi ical e alua ion using h ee eal-wo ld da ase s ha span be ween 2 and 13 yea s o alida e he diffe en s a egies which show a significan enhancemen in he p edic ion accu acy wi h espec o a non-upda e s a egy. Keywo ds: P edic i e p ocess moni o ing ·P ocess mining · P ocess-awa e in o ma ion sys ems ·P edic ion models ·Model upda ing 1 In oduc ion P edic i e p ocess moni o ing (PPM) p o ides p oac i e and co ec i e ac ions o imp o e he p ocess pe o mance and mi iga e po en ial isks in eal ime. PPM e ie es in o ma ion om P ocess-Awa e In o ma ion Sys ems (PAIS) s o ed in e en logs o make p edic ions o e alua ion me ics, also known as p ocess pe o mance indica o s (PPIs) [1]. A pa h ex ensi ely ollowed in he li e a u e o p edic i e moni o ing is adap ing exis ing machine lea ning ech- niques [2] such as decision ees, clus e ing me hods o neu al ne wo ks o ob ain p edic i e models wi h highe accu acy. When hese app oaches a e used, he Wo k unded by g an s RTI2018-101204-B-C21 and RTI2018-101204-B-C22 unded by MCIN/ AEI/ 10.13039/501100011033/ and ERDF A way o making Eu ope; g an P18-FR-2895 and US-1381595 unded by Jun a de Andaluc´ıa/ERDF, UE. ypical p ocedu e o p edic i e moni o ing comp ises wo s eps. Fi s , a aining s age in which he p edic i e models a e ained using da a collec ed in he e en logs. Second, once he model is buil , i is deployed and i is used o p edic PPIs o cu en and/o u u e p ocess execu ions. In he absence o significan changes, ce e is pa ibus (all else being equal), his app oach wo ks fine, bu p ocesses a e subjec o con inuous changes. Fo ins ance, he esponse o COVID may in oduce new ways o pe o ming ac i - i ies, use s can beha e in a diffe en way, o human o compu ing esou ces can change o e ime. These changes may nega i ely affec he pe o mance o he p edic i e model since he da a used o ain hem does no eflec eali y any mo e. The e o e, he only way o keep his pe o mance o e a desi ed h eshold is by adap ing he model o he changes. In he machine lea ning communi y, he e a e wo main adap a ion app oaches o ha , namely, online and offline lea ning. In online lea ning, he p edic i e model is being upda ed con inuously om he da a i ecei es. Con- e sely, in offline lea ning, he p edic i e model is ebuil again om he g ound. In his pape , we decide o ocus on offline lea ning mainly o wo easons. Fi s ly, he pace o change and he pace o new e en s in he p ocesses we a e in e es ed in, gi es enough ime o comple ely ebuild new models. Secondly, i s use allows one o euse a huge amoun o machine lea ning echniques ha a e a ailable o offline lea ning, which is much mo e comp ehensi e han ha o online lea ning. Fu he mo e, hese echniques do no need o make comp omises o keep a easonable lea ning ime. In his con ex , he goal o his pape is o p o ide de ails on how o ace wo o he ques ions ha a ise in he upda e o p edic i e models: “Which da a should be conside ed in he new model ha is being buil ?” and “How he selec ion o da a does impac on he pe o mance o he p edic i e models?”. By answe ing hese ques ions, we con ibu e o he s a e o he a on PPM by p oposing six diffe en s a egies o upda ing p edic i e models (baseline, cumu- la i e, non- cumula i e, ensemble, sampling, and concep d i ) and compa ing hei pe o mance. Ou expe imen a ion was alida ed using h ee eal-li e e en logs ha span be ween 2 and 13 yea s. We ha e also pe o med a compa ison o diffe en well-known classifie s used in ela ed li e a u e. The eminde o his pape is o ganized as ollows. Sec ion 2summa ises basic concep s in p edic i e moni o ing. Sec ion 3p esen s he s a egies o upda ing p edic i e models. The expe imen and he discussion o he ob ained esul s a e p esen ed in Sec . 4. Sec ion 5summa ises he ela ed wo k. Finally, Sec . 6 concludes he wo k and p esen s possible u u e di ec ions. 2 P edic i e P ocess Moni o ing In he ollowing, we in oduce some basic concep s o p edic i e p ocess moni- o ing. As defined in [3], an e en log (L) is composed o a se o aces (T). Each ace (Ti) eflec s an execu ion o a p ocess ins ance. Fo mally, we can exp ess a ace as an o de ed lis o e en s Ti=[Ei1,...,E im] whe e Ei1 ep esen s he fi s e en and Eim he final e en o ace Ti. Simila ly, a log can be exp essed as he se o aces o he ins ances ha ha e finished in an in e al o ime L=[T1,...,T n] whe e T1is he fi s and Tnis he las execu ed ace in he ime in e al. Finally, an e en ep esen s he execu ion o an ac i i y o he p ocess. Each e en con ains a se o a ibu es (a), which ep esen s in o ma ion ela ed o such e en , e.g. imes amp, he esou ce ha execu es he ac i i y, o he alue o some da a used h oughou he ins ance, Ej=[aj1,...,a jo] whe e o de e mines he o al numbe o a ibu es o he e en . A p ocess indica o (I) is a quan ifiable me ic ocused on measu ing he p og ess owa d a goal o s a egic objec i e. Indica o s can be classified in o wo ypes: single-ins ance indica o s o agg ega ed indica o s. The o me is compu ed o each ace in he log using he alues o he a ibu es o he e en s ha compose his ace. The e o e, i can be defined as a unc ion o a ace Ti,i.e. I(Ti). This unc ion can e u n a bina y alue, e.g a de e mined condi ion ulfilled by he ace, o a eal alue, e.g he du a ion o an ac i i y. Ins ead, an agg ega ed indica o is compu ed o a se aces by agg ega ing a single-ins ance indica o using some agg ega ion unc ion, e.g. sum o a e age. An example o his ype o indica o could be he pe cen age o inciden s sol ed in a ce ain pe iod o ime. A p edic i e model o an indica o Iis a unc ion PI([Eik,...,E il]), wi h k≤l, ha compu es a p edic ion o I om he pa ial ace [Eik,...,E il], whe e Eilis he las e en ha ha e occu ed in ace Tia a gi en momen . I k= 1, hen all e en s ha ha e occu ed in he p ocess ins ance a hand a e conside ed. Ins ead, i k=l, hen only he las e en o he p ocess ins ance is conside ed. In o de o ain a p edic i e model ˆ I o a key pe o mance indica o (KPI) I, an encoded fixed-size ep esen a ion Co all he cases C, whe e C⊆C, included in he aining se is equi ed. This encoding, gene ally ep esen ed as a ea u e ma ix (X), should s o e enough in o ma ion o he p ocess, and will be used as inpu o he machine lea ning echnique employed o build he model oge he wi h he alue o he KPI I o each case in C, which ep esen s he a ge a iable (y). The ea u e ma ix Xis ob ained a e applying a sequence encoding unc ion F, which ecei es a se o cases Cand e u ns a ma ix X. Each ow o he ma ix ep esen s an e en E, i.e. he execu ion o an ac i i y o he p ocess, o a case c∈Cand each column ep esen s he diffe en (encoded) a ibu es ao he e en . Va ious sequence encoding echniques ha e been p oposed in he li e a u e o his ask such as las s a e encoding [4], agg ega ion encoding [5], o index-based encoding [6]. The o he decision is i only one classifie is ained o he whole da ase o , on he con a y, i cases a e g ouped in o se e al bucke s and a diffe en classifie is ained o each one. Se e al case bucke ing echniques ha e been p oposed in he li e a u e [7]: Ze o bucke ing [5], p efix leng h bucke ing [6], o clus e bucke ing [8]. A e hese wo decisions a e made, a p edic i e model is buil using some machine lea ning algo i hm using he pai (X, y) as he inpu . Fig. 1. Upda ing models sys em in a p edic i e moni o ing p ocess. Fig. 2. T aining s age. Fig. 3. Run- ime moni o ing s age. An indica o Ican ep esen diffe en issues, such as a ce ain ou come, he nex ac i i y o he p ocess o emaining cycle ime o a gi en p ocess case. In his wo k we ha e ocused on he ou come-based p edic ion. The e o e, we a e p edic ing an ou come alue pe case ins ead o a alue pe each e en . 3 Upda ing P edic i e Models Once he p edic i e models a e gene a ed ollowing he mechanisms desc ibed in he p e ious sec ion a e deployed in o p oduc ion, hey s a making p edic ions ha can be used o imely eac o ope a ional issues. Howe e , a e a while, he way he p ocess was pe o med migh ha e changes; esou ces pa icipa ing in he p ocess may come and go; e en he s uc u e o he p ocess may suffe changes. All hese changes a e igno ed by he p edic i e model ha was deployed some ime ago, so hese changes may nega i ely affec he pe o mance o he p edic i e model. The e o e, i is necessa y o p o ide mechanisms o upda e he p edic i e model. Figu e 1shows a sys em o he upda ing o p edic i e models desc ibed below. I consis s o a aining s age depic ed in Fig. 2(which in ol es he fil e ing o he e en log, he gene a ion o he p edic i e model and he e alua ion o his model), he deploymen o his model, he p edic ion using his model, and finally, a mechanism o he upda e o he dep eca ed models, named Run- ime moni o ing p ocess as shown in Fig. 3. This mechanism can e alua e he model in e ms o he pe o mance o he p edic ions and decide when he p edic i e model should be upda ed acco ding o h ee possible pa ame e s: he ime elapsed om he las deployed model, he accu acy o p edic ions and he possible occu ence o a concep d i [9]. As men ioned in he in oduc ion, he e a e wo app oaches o upda ing p edic i e models, namely online and offline lea ning. In his pape , we ocus on offline lea ning because i allows one o use he huge amoun o machine lea ning echniques o offline lea ning, which is much mo e comp ehensi e han ha o online lea ning and, u he mo e, hese echniques do no need o make comp omises in o de o keep a easonable lea ning ime. To he bes o ou knowledge, any o he wo k ela ed o offline s a egies o upda ing p edic i e models appea s in he li e a u e, so ha compa ison wi h o he pape s is no pos- sible. In he con ex o offline lea ning, wo basic ques ions need o be answe ed o upda e he p edic i e model: 1. When should he p edic i e model be upda ed? 2. Which da a should be conside ed in he new model ha is being buil ? Fo he fi s ques ion, se e al s a egies can be conside ed (Run- ime mon- i o ing s age in Fig. 3). The mos s aigh o wa d is a pe iodical upda e o he model [10]. A easonable deadline o he change o he model can be fixed, e.g. six-mon hly pe iodici y, and hen, a new gene a ion o he model will be ca ied ou . A diffe en s a egy migh in ol e moni o ing he accu acy o ou p edic- ions. When i begins o dec ease o e an unin e up ed pe iod o ime, i may be ecommended o change he p edic i e model. A h eshold o e o can be se , and i he p edic ion exceeds his h eshold, he p edic ion model will be upda ed. Finally, a hi d s a egy could in ol e using he de ec ion o d i s in p ocesses [11] o igge upda es in he p edic i e model [10]. Se e al s a egies could be also combined o design a mo e obus sys em. Ou ocus in his pape is, howe e , on he second ques ion. This ques ion is ele an because o wo easons ha in ol e he quali y and cos o p edic i e models. Conce ning he o me , i he eason o upda ing he p edic i e model is because he p ocess has changed, i is easonable o hink ha lea ning pas beha iou s o he p ocess may no be beneficial o he pe o mance o he p e- dic i e model, so i migh make sense no o include he whole da a se , bu only he mos ecen beha iou . As o he la e , he compu a ional cos o building a p edic i e model inc eases wi h he size o he inpu da a se . The e o e, a goal should be o achie e he bes p edic i e pe o mance by using he smalles possible inpu da a se . In [10], au ho s p opose wo possible solu ions o he second ques ion: e aining and inc emen al upda e o a p edic i e model, how- e e , all p edic i e algo i hms canno lea n inc emen ally (e.g. andom o es s). The e o e, we ha e collec ed a se o s a egies o choosing he da a se used o building a new p edic i e model ega dless o he p edic i e algo i hm used. A s a egy o choosing he da a se used o building a new p edic i e model can be seen as a unc ion S ha ecei es a aining and es se pai (X, y)and e u ns ano he aining and es se pai (X,y), such ha X⊆Xand y⊆y. Nex we de ail se e al possible da a selec ion s a egies. We use he no a ion X[i,j] o selec he subse o X ha is be ween iand j, whe e iand jcould be ei he an ins an in ime such as he 7 h o Ma ch o 2019, o an ins ance numbe since he fi s one ecei ed. Fu he mo e, i hey ake he alue 0, i e e s o he fi s e en in he da a se and i hey ake he alue c, i e e s o he las e en ecei ed in he da a se . The e o e, X[0,c]=X. In he ollowing, we p esen he diffe en s a egies o he selec ion o da a. Figu e 5shows a g aphical ep esen a ion o he diffe en s a egies desc ibed in his sec ion. Figu e 4depic s an e en log ha will be used o explain he diffe en s a egies in Fig. 5. This e en log is spli in o se e al in e als om 1 o n. Each in e al ep esen s all he p ocess ins ances execu ed du ing a ce ain pe iod o ime, e.g. six mon hs. 1. Baseline s a egy (SB): In his s a egy he model is no upda ed h ough- ou he li e o p ocess: SB(X, y)=(X[0,c],y [0,c]) Figu e 5a shows a g aphical ep esen a ion o he baseline s a egy. Wi h his s a egy, a fi s in e al is selec ed as he aining se , and i is no upda ed h oughou he li e o he p ocess. We use he es o he in e als as es se s. 2. Cumula i e s a egy (SC): This s a egy in ol es including all he cases ha a e a ailable o aining since he beginning: SC(X, y)=(X[0,c],y [0,c]) Cumula i e s a egy is ep esen ed in Fig. 5b. This s a egy in ol es adding all ins ances o a p ocess as aining se . We spli he e en log in o aining and es se s, and we inc emen ally add each in e al om 1 o nin he aining se and use he es o he es se . 3. Non-cumula i e s a egy (SN): This s a egy in ol es choosing only he mos ecen cases o aining. I includes a pa ame e n ha de e mines how many ecen cases should be included in he aining: SN(X, y)=(X[c− n,c],y [c− n,c]) The ad an age o his s a egy in compa ison o he cumula i e s a egy is ha i is mo e efficien compu a ionally and i migh sol e p oblems de i ed om using cases ha do no ollow he cu en beha iou o he p ocess. The d awback is ha ha ing less aining ins ances migh hu he pe o mance o he p edic i e model. Fo he non-cumula i e s a egy (Fig. 5c), we selec an unique in e al in he aining phase. In his manne , we only include he mos ecen cases o aining. 4. Ensemble s a egy (SE): This s a egy is simila o he non-cumula i e s a egy because i only includes he mos ecen s cases in aining a new model. The diffe ence is ha , unlike he non-cumula i e s a egy, his s a egy do no h ow away olde models, bu keep hem and combine hem using some ensemble echnique [12]. Fo ins ance, one can use a weigh ed o ing echnique in which he p edic ion o each model is conside ed a o e and combined using diffe en weigh s o each model o make he final p edic ion. These weigh s can be upda ed each ime a new model is added o he ensemble so ha olde models ha e a lowe weigh . To his end, weigh s can be modeled using an exponen ial decay unc ion like e−λ . Fu he mo e, besides hese weigh s, i he las model had e y bad pe o mance, we migh be in e es ed in emo ing i om he ensemble so ha i does no hu he o e all pe o mance. To his end, we se a h eshold pa ame e so ha i he quali y me ic o choice, e.g. -sco e, o he p e ious model did no mee he h eshold in he las in e al, i is emo ed om he ensemble. The ad an age o his s a egy is ha i has almos he same compu a ional cos as he non-cumula i e s a egy, bu i helps o a oid disca ding all o he old cases. Howe e , he combina ion o he diffe en models migh no be as powe ul as a model buil using he cumula i e s a egy ha includes all p e ious cases. In his s a egy, depic ed in Fig. 5d, we choose he same aining and es se s and keep olde models o combine hem using some ensemble echnique o achie e be e p edic ions. 5. Sampling s a egy (SS): This s a egy in ol es a weigh ed sampling o all he cases ha a e a ailable o aining since he beginning. I includes a pa ame e s ha de e mines he numbe o samples ha mus be ob ained om he da a: SS(X, y)=(sampling(X[0,c], s), sampling(y[0,c], s)) Whe e sampling is a unc ion ha akes s samples om Xo y, espec- i ely. Sampling can also be weigh ed so ha i is mo e likely o ob ain mo e ecen samples han olde samples. A simila app oach as he one used in he ensemble s a egy can be used he e o define hese weigh s. The ad an age o his app oach in compa ison o he cumula i e s a egy is ha i limi s he compu a ional cos o he new model. Fu he mo e, unlike he ensemble s a egy i elies on he machine lea ning algo i hm ins ead o in he o ing mechanism o combine bo h in o ma ion om old and ecen cases. Fig. 4. Rep esen a ion o a spli e en log. Figu e 5e shows he Sampling s a egy. We build he aining se in a inc e- men al way using a weigh ed s a egy whe e ecen samples a e mo e likely o be selec ed han olde samples. 6. D i s a egy (SD): This s a egy is simila o he non-cumula i e one. I includes he mos ecen cases o aining and, when a concep d i is de ec ed, we include as aining se , all he cases a e a ce ain ime has passed since he d i has occu ed. Figu e 5 shows he d i s a egy. When he concep d i is de ec ed, aining se is buil only wi h hose cases execu ed a e he d i de ec ion. 4 Expe imen al E alua ion As we s a ed in Sec . 1, he goal o his pape is o define he diffe en s a e- gies o he selec ion o da a (desc ibed in Sec . 3) and compa e he p edic i e pe o mance o he diffe en s a egies p oposed. Based on his goal, we define a esea ch ques ion o ou expe imen a ion: Wha is he impac o he diffe en upda ing s a egies on he accu acy o p edic ions? The es o he sec ion is o ganized as ollows: he expe imen se up is de ailed in Sec . 4.1. The diffe en da ase s used in he expe imen a ion a e desc ibed in Sec . 4.2. Finally, a discussion o he ob ained esul s is p o ided in Sec . 4.3. 4.1 Expe imen Se up As p edic i e algo i hm we ha e used andom o es [13] as seen in p e ious wo ks in he li e a u e [14]. This echnique combines p edic o ees such ha each ee depends on he alues o a andom ec o es ed independen ly and wi h he same dis ibu ion o each o hem. In [14], au ho s highligh ex eme g adien boos ing (XGBoos ) and andom o es as wo o he bes echniques in p edic i e moni o ing. Thus, we ha e selec ed andom o es because he quali y o esul s wi h espec o XGBoos is simila and i consumes less compu a ional ime. We ha e selec ed a ypical agg ega ion encoding desc ibed in [14] as one o he mos used in he li e a u e o encode he p ocess cases and also one o he bes pe o me s [14]. Thus, all e en s since he beginning o he case a e conside ed. An agg ega ion unc ion is applied o he alues aken by a specific a ibu e h oughou he case li e ime. In ou case, his unc ion is he numbe o imes ha each specific a ibu e appea s in he case ( equency encoding). We ha e no di ided he cases in he e en log in o diffe en bucke s. This echnique is named Ze o bucke ing as defined in [14]. We ha e also inco po a ed he o de o he e en s as a new a ibu e in all he logs, as well as he elapsed ime be ween Fig. 5. G aphical ep esen a ion o diffe en p oposed s a egies o choosing he da a se used o building a new p edic i e model. he e en and he beginning o he case and he ime be ween he p e ious e en and he cu en one. The de ails and he code o he expe imen a ion a e a ailable online1. 4.2 E en Logs Th ee diffe en eal-li e e en logs we e conside ed in ou expe imen s: IT Depa - men o an Andalusian o ganisa ion (ITA), BPI 2015 (BPI15) [15] and T affic fines (TRAFFIC) [16]. These logs we e chosen because hey span se e al yea s: 2, 5 and 12, espec i ely, so hey a e use ul o e alua e he effec o possible changes on he p ocess o e ime. 1h ps://gi hub.com/isa-g oup/p edic i e-moni o ing-e olu ion.