A screenshot-based task mining framework for disclosing the drivers behind variable human actions

A sc eensho -based ask mining amewo k o disclosing he d i e s behind a iable human ac ions A. Ma ínez-Rojas a,∗, A. Jiménez-Ramí ez a, J.G. En íquez a, H.A. Reije s b aDepa men o Compu e Languages and Sys ems, Uni e si y o Se ille, A enida Reina Me cedes, s/n, 41012, Se ille, Spain bDepa men o In o ma ion and Compu ing Sciences, U ech Uni e si y, P ince onplein 5, 3584 CC, U ech , The Ne he lands A R T I C L E I N F O Keywo ds: Robo ic P ocess Au oma ion UI Log Use beha io mining Task mining Decision model disco e y A B S T R A C T Robo ic P ocess Au oma ion (RPA) enables subjec ma e expe s o use he g aphical use in e ace as a means o au oma e and in eg a e sys ems. This is a as me hod o au oma e epe i i e, mundane asks. To a oid cons uc ing a so wa e obo om sc a ch, Task Mining app oaches can be used o moni o human beha io h ough a se ies o imes amped e en s, such as mouse clicks and keys okes. F om a so-called Use In e ace log (UI Log), i is possible o au oma ically disco e he p ocess model behind his beha io . Howe e , when he disco e ed p ocess model shows di e en p ocess a ian s, i is ha d o de e mine wha d i es a human’s decision o execu e one a ian o e he o he . Exis ing app oaches do analyze he UI Log in sea ch o he unde lying ules, bu neglec wha can be seen on he sc een. As a esul , a majo pa o he human decision-making emains hidden. To add ess his gap, his pape desc ibes a Task Mining amewo k ha uses he sc eensho o each e en in he UI Log as an addi ional sou ce o in o ma ion. F om such an en iched UI Log, by using image-p ocessing echniques and Machine Lea ning algo i hms, a decision ee is c ea ed, which o e s a mo e comple e explana ion o he human decision-making p ocess. The p esen ed amewo k can exp ess he decision ee g aphically, explici ly iden i ying which elemen s in he sc eensho s a e ele an o make he decision. The amewo k has been e alua ed h ough a case s udy ha in ol es a p ocess wi h eal-li e sc eensho s. The esul s indica e a sa is ac o ily high accu acy o he o e all app oach, e en i a small UI Log is used. The e alua ion also iden i ies challenges o applying he amewo k in a eal-li e se ing when a high densi y o in e ace elemen s is p esen . 1. In oduc ion In his e a o Hype au oma ion [1], o ganiza ions a e pushed o apidly iden i y and au oma e all business p ocesses whe e possible.1 This d i e has led o he widesp ead adop ion o low-code and no- code ools, alongside he inc easing popula i y o Robo ic P ocess Au- oma ion (RPA). Unlike adi ional au oma ion (e.g., API-based), RPA enables he use o g aphical use in e aces as a means o au oma e and in eg a e sys ems; by doing so, a so wa e obo ha is c ea ed in his way can closely mimic how humans in e ac wi h compu e applica ions [2]. Se e al au ho s ha e epo ed he bene i s ha RPA b ings. Since his echnology is ela i ely easy o use by subjec ma e expe s and does no necessa ily equi e IT skills o deploy i , RPA helps o sa e cos s, inc eases agili y, imp o es quali y [3–6], as well as cus ome sa is ac ion [7,8], while i s le el o in usi eness is low [9,10]. ∗Co esponding au ho . As he p e alence o RPA con inues o g ow, a ious echniques ha e eme ged o suppo and augmen so wa e au oma ion h ough- ou i s li e cycle phases, encompassing analysis, design, de elopmen , deploymen , es ing, and moni o ing [11]. Typically, au oma ion ini ia- i es s a wi h he analysis phase, in ol ing he examina ion o exis ing p ocesses o iden i y op imal candida es o au oma ion [12]. How- e e , adi ional app oaches, such as in e iews wi h subjec -ma e expe s o analysis o p ocess documen a ion, come wi h inhe en isks, including eliance on incomple e o ou da ed p ocess desc ip ions, biased in o ma ion, and o e sigh o excep ional si ua ions ha may be deemed inconsequen ial o humans bu hold signi icance o an au oma ion p ojec [13]. To mi iga e hese isks and a oid cons uc ing au oma ion om sc a ch, inno a i e app oaches like Task Mining [14, 15] and Robo ic P ocess Mining [16] ha e eme ged o lea n om he obse a ion o use s while hey a e pe o ming he p ocesses. These echniques eco d use beha io in o a Use In e ace log (UI Log, i.e., a E-mail add esses: [email p o ec ed] (A. Ma ínez-Rojas), [email p o ec ed] (A. Jiménez-Ramí ez), [email p o ec ed] (J.G. En íquez), [email p o ec ed] (H.A. Reije s). 1 Ga ne ’s de ini ion o hype au oma ion: h ps://www.ga ne .com/en/in o ma ion- echnology/glossa y/hype au oma ion h ps://doi.o g/10.1016/j.is.2023.102340 Recei ed 22 Janua y 2023; Recei ed in e ised o m 14 No embe 2023; Accep ed 18 Decembe 2023 A ailable online 21 Decembe 2023 0306-4379/© 2023 The Au ho (s). In o ma ion Sys ems 121 (2024) 102340 2 A. Ma ínez-Rojas e al. se ies o imes amped e en s like mouse clicks and keys okes) and hen analyze hem o enable he c ea ion o he au oma ion solu ion by disco e ing he p ocess model behind he log. The ield o p ocess disco e y o UI Logs has been explo ed in conside able dep h [13,17,18]. Howe e , when he disco e ed model shows di e en p ocess a ian s, he e is li le suppo o de e mine wha d i es he execu ion o one a ian o e he o he . The e o e, he a ia ions obse ed in he eco ded UI Log sugges ha he hu- man ope a o makes di e en decisions o di e en cases. Despi e his, iden i ying he speci ic ules go e ning hese decisions is gen- e ally no easible. Ne e heless, unco e ing hese ules is necessa y o model decisions, which is essen ial o e ec i ely implemen RPA so wa e [19]. Acknowledging such signi icance, se e al esea che s ha e unde - aken his challenge [17,20–22]. Thei app oaches a e based on he in o ma ion ha is in he UI Log. Howe e , a limi a ion a ises when on-sc een elemen s solely in luence decisions, as hese lea e no angible ail wi hin he log. Consequen ly, ce ain aspec s o human decision- making emain hidden, c ea ing a po en ial obs acle o au oma ion. To add ess his p oblem, he cu en pape desc ibes a amewo k ha conside s sc eensho s (i.e., sc een cap u es ha a e aken o e e y e en in he UI Log) as an addi ional sou ce o knowledge. These sc een- sho s a e analyzed o ex ac s uc u ed in o ma ion ha can ex end he da a eco ded o each e en in he log. The UI Log ex ended wi h his new in o ma ion, can hen be p ocessed o ain a Machine Lea ning algo i hm, which ies o ind he ela ions be ween his in o ma ion and he decisions made by he use . The amewo k combines hese ules wi h he ela ed sc eensho s o g aphically exp ess he ac o s ha in luence human decision-making. The amewo k ha is p esen ed in his pape builds on, in eg a es, and ex ends ou p e ious wo k, which enables us o (1) moni o use beha io ha gene a es a UI Log ha s o es one sc eensho o each e en [23], and (2) analyze hese UI Logs and he sc eensho s o disco e he unde lying p ocess model [13]. Speci ically, his pape ex ends ou p oposal in [24]. In ha pape , p omising esul s we e ob ained when dealing wi h simple sc eensho s (i.e., syn he ic mockups) and a he simple condi ions. In u n, he cu en wo k g ea ly enhances a p e ious wo k by add essing a unda- men al P ima y Resea ch Ques ion (PRQ) no conside ed so a : Can he sc een-based Task Mining app oach ex ac and ep esen he p ocess beha io obse ed in a UI Log wi hin eal-li e se ings? In ending o answe PRQ, his pape signi ican ly ex ends and im- p o es ou p e ious wo k [24] by: 1. Fo malizing and classi ying he ea u e ex ac o s ha can be applied o ex ac in o ma ion om sc eensho s. 2. P oposing da ase gene a ion me hods o each class o ea u e ex ac o . 3. Ex ending he amewo k by including a new s ep o explain he decision g aphically. This new s ep enables he connec ion o he decision ules o he eal UI elemen s on he sc eensho s. 4. E alua ing he p oposal using a case s udy ha in ol es a p o- cess wi h eal-li e sc eensho s, and analyzes he pe o mance ob ained o each class o ea u e ex ac o . The de eloped amewo k is o in e es o esea che s and p ac i ione s in he RPA ield since i explains and demons a es how o disco e au oma able decision ules ha would o he wise emain hidden. Fu - he mo e, esul s show ha he amewo k is applicable e en when only ela i ely small UI Logs a e a ailable. The es o he pape is o ganized as ollows. Sec ion 2p o ides he backg ound by desc ibing he opics o beha io moni o ing, ML, and image p ocessing. Sec ion 3desc ibes he p oposed amewo k and explains how i is able o explain decisions om a UI Log g aphically. Sec ion 4desc ibes he se up and he execu ion o he empi ical e alua- ion, which was pe o med o compa e he di e en ea u e ex ac o s. Sec ion 5 e iews simila app oaches in he li e a u e. Finally, Sec ion 6 summa izes ou wo k and desc ibes u u e esea ch lines. 2. Backg ound The app oach p esen ed in his pape builds on he opics o P o- cess Disco e y in RPA, G aphical Use In e ace (GUI) analysis, and Machine Lea ning (ML) o decision disco e y. P ocess disco e y (c . Sec ion 2.1) enables he sys ema ic analysis o business p ocesses, op imizing wo k lows o RPA in eg a ion. GUI analysis (c . Sec ion 2.2) empowe s RPA bo s o e icien ly in e ac wi h di e se so wa e in e - aces h ough image ecogni ion and objec de ec ion. Finally, ML (c . Sec ion 2.3) elucida es he human decision-making p ocess, enabling a clea unde s anding o he ules and ac o s guiding ac ions. 2.1. P ocess disco e y in RPA In he RPA ield, di e en al e na i es exis o moni o use s by ob aining e en logs om he in e ac ion be ween he use and he on -end o he in o ma ion sys ems ( he so-called UI Logs) [13,25,26]. The mos ep esen a i e one o his wo k is he UI Log concep p esen ed by Jiménez-Ramí ez e al. and Ma ínez-Rojas e al. [13,27], which, unlike he o he s, p o ides special a en ion o he sc eensho s as an in o ma ion sou ce. The UI Logs a e se s o imes amped e en s cap u ed a he ope - a ing sys em (OS) le el, independen o any speci ic p og ams. They encompass essen ial in o ma ion abou each e en , including (1) he associa ed p ocess (e.g., case o ac i i y), (2) he applica ion being used (e.g., app name o sc eensho ), and (3) he speci ic use ac ion pe o med (e.g., mouse click o keys oke). These e en s a e igge ed based on a ange o OS e en s, such as clicks, keys okes, o speci ic combina ions o hem. Task Mining o Robo ic P ocess Mining echniques can be applied o eco d hese UI Logs and disco e he ope a o p ocess model ou o hem [13,13,16,20,28–32]. Mo eo e , hese p oposals include unc ion- ali ies o clean he UI Log o i ele an in o ma ion so ha noise in he esul ing p ocess model is il e ed. In addi ion, exis ing wo ks [17,18] show how o selec a ian s/cases/ac i i ies acco ding o he equency, leng h, and o he c i e ia ha a e use ul o iden i y p ocess candida es o obo ize. The p ocess model esul ing om hese app oaches may con ain decision poin s and sepa a e b anches o di e en p ocess a ian s. To illus a e how a p ocess is disco e ed based on use moni o ing, Fig. 1 depic s a subse o he elemen a y use in e aces o egis e ing a cus ome in he con ex o a elecommunica ion company. In his egis- a ion p ocess, he human ope a o checks he email inbox and e iews he pending emails ega ding he egis a ion asks. A e opening he email, he ope a o has o alida e ha all p o ided in o ma ion ela ed o he new cus ome is co ec . Mo e p ecisely, he cus ome ID ca d is expec ed o be included as an a achmen . I i is indeed included (c . Fig. 1a), all he cus ome da a has o be egis e ed in o a CRM sys em (c . Fig. 1b). O he wise, in case he ID ca d is missing (c . Fig. 1c), an email has o be sen o he cus ome eques ing such da a (c . Fig. 1d). Rega dless o which o hese wo si ua ions occu ed (i.e., Va ian 1 o Va ian 2 o Fig. 1), he ope a o e u ns o hei inbox o p ocess he nex mail. This p ocess mus be epea ed se e al imes du ing he day o p ocess he en i e queue o emails in he ope a o ’s inbox. Fig. 2 shows an exce p o a possible UI Log esul ing om moni- o ing he ope a o while execu ing he p ocess o Fig. 1 (e.g., wi h a key logge [23]). Fig. 3 shows a possible p ocess model disco e ed om he UI Log o Fig. 2. This kind o ep esen a ion suppo s he unde s anding o he UI Log. In his example, i may be deduced ha he UI Log co ec ly depic s ha i con ains a single decision poin a e ac i i y ‘‘B’’ (i.e., a e seeing he email), whe e he p ocess b anches o in o wo di e en a ian s, i.e., Id.1 and Id.2. None heless, he model does no disclose he ac ual condi ion ha ules such a decision. Wha is mo e, i may po en ially depend on he in o ma ion ha appea s on he sc een. In o ma ion Sys ems 121 (2024) 102340 3 A. Ma ínez-Rojas e al. Fig. 1. Mockups o mo i a ing example. Fig. 2. Exce p o a UI Log ob ained om a keylogge . Fig. 3. P ocess model disco e ed om he UI Log o Fig. 2. To unco e his in o ma ion, esea che s analyze use in e ac ions [17]. T adi ional echniques o ex ac his in o ma ion om he sc een ely on ope a ing sys em APIs o accessing he Documen Objec Model (DOM), which enables sys ema ic que ying o UI elemen s [33,34]. Despi e hese possibili ies, some p ocesses, especially in he ield o ou sou cing, in ol e i ualized connec ions (e.g., using Ci ix o Team Viewe ) o enhanced secu i y [35]. This es ic s di ec access o he Windows API o he DOM ee. Consequen ly, in such scena ios, le e aging UI Logs and sc eensho s becomes he p e e ed me hod o ga he ing in o ma ion and e ec i ely au oma ing p ocesses. 2.2. GUI analysis To analyze he sc eensho s, he ield o GUI analysis allows iden- i ying he UI elemen s exis ing wi hin an image [36–38]. Besides jus loca ing he elemen on he sc een (i.e., calcula ing he bounding boxes), hey can classi y hem by he ype o elemen , e.g., image, bu - on, o ex . Op ical Cha ac e Recogni ion (OCR) echniques a e o en mo e app op ia e when dealing wi h ex s. Fo ins ance, Ke asOCR [39] allows ex ac ing wo ds and hei bounding boxes om sc eensho s. In o ma ion Sys ems 121 (2024) 102340 4 A. Ma ínez-Rojas e al. Fig. 4. Example o GUI analysis applied o a sc eensho o de ec ing bu ons. Fig. 5. F amewo k o a iabili y analysis h ough in e p e able decisions om UI Logs. Fig. 4 shows a eal UI sc eensho ela ed o ac i i y ‘‘B’’ o Fig. 3. Wha can be seen he e is how he app oach om [37] has been ailo ed o desk op sc eensho s o analyze hem and highligh he UI elemen s o a conc e e ype, i.e., bu ons. 2.3. ML o decision disco e y The ML domain p oposes supe ised lea ning algo i hms ha can lea n om a da ase [40] o explain he ules behind a decision. Da ase s a e commonly ep esen ed in abula o m whe e each column is an inpu a iable, and one o hem is he label, i.e., wha needs o be classi ied. Each ow is a membe o obse a ion o he da ase . In a classi ica ion p oblem, he algo i hm ies o ind pa e ns in he obse a ion’s inpu a iables ha help explain he labels. Decision ees a e an example o classi ica ion algo i hms ha , besides jus p o iding a classi ica ion, do so in a human-in e p e able way [41]. 3. F amewo k This sec ion desc ibes he p oposed amewo k o explaining he decisions om he da a in a UI Log using sc eensho s (c . Fig. 5). I is based on exis ing app oaches [13,25,30] ha (1) deal wi h moni o ing g aphical use in e aces o gene a e UI Logs (c . s ep 1 in Fig. 5), and (2) disco e p ocess models om such logs (c . s ep 3 in Fig. 5). As s a ed be o e, disco e ed p ocess models may con ain decision poin s whose unde s anding could be cumbe some. To amend his, he p oposed amewo k aims o ind an explana ion o each decision poin , i.e., he ules ha de e mine which b anch o ollow. Such explana ions may conside in o ma ion ha is g aphically depic ed in he use in e aces, as cap u ed in s ep 1. Mo e p ecisely, he ame- wo k includes a s ep o ex ac in o ma ion om use in e aces and inco po a e i in o he UI Log (c . s ep 2 in Fig. 5, Sec ion 3.1). Some app oaches o p ocess model disco e y, such as [13], a e bene i ed om his ex ended in o ma ion, and his is why he ea u e ex ac ion s ep is es ablished be o e he p ocess disco e y s ep in he amewo k. Once he p ocess model is disco e ed, he ex ended UI Log om s ep 2 is used as inpu o lea n he decision ee (i.e., ules) behind each decision poin (c . s ep 4 in Fig. 5, Sec ion 3.2). Finally, hese decision ees a e p ocessed o ace back hei ules o he g aphical in o ma ion in he sc eensho s (c . s ep 5 in Fig. 5, Sec ion 3.3). This helps o make sense o he decision ees. 3.1. Ex ac ing ea u es om g aphical use in e aces In gene al, a g aphical use in e ace con ains a numbe o UI elemen s ha a e ep esen ed g aphically on he sc een. To ex ac in o ma ion om a use in e ace, he amewo k i s iden i ies and classi ies he UI elemen s on each sc eensho (c . Sec ion 3.1.1), and hen ex ac s ea u es om hem (c . Sec ion 3.1.2) 3.1.1. Iden i ica ion and classi ica ion o UI elemen s The p oposed amewo k uses he app oach p oposed by Chen e al. [42] o iden i y each UI elemen on he sc een, i.e., o ob ain he bounding boxes o all he UI elemen s ha a e ound. In addi ion, he solu ion p esen ed by Mo an e al. [37] is hen used o classi y each iden i ied elemen , i.e., o ob ain he ype o each UI elemen , e.g., bu on, checkbox, e c. In o ma ion Sys ems 121 (2024) 102340 5 A. Ma ínez-Rojas e al. Fig. 6. Example o UI Elemen Loca ion and Classi ica ion (ELoC) da a o one o he bu ons on he sc een. The e o e, his s ep gene a es and s o es me ada a ela ed o each sc eensho . Ul ima ely, he me ada a would comp ehensi ely desc ibe he sc eensho in a ex ual o m in a simila way as a DOM ee does wi h a web page. Howe e , he app oach p esen ed in he amewo k simpli ies hese desc ip ions o keep jus he essen ial in o ma ion. Speci ically, his me ada a comp ises a lis o UI Elemen Loca ion and Classi ica ion da a (c . De ini ion 1) ha will be s o ed o each e en in he log. De ini ion 1. The UI Elemen Loca ion and Classi ica ion (ELoC) da a is a uple (Type, Pos, Id) wi h he ollowing componen s: •Type. This indica es he ype o he UI elemen and can ake on alues such as Tex View, ImageView, o Bu on, among o he s. This classi ica ion is based on he wo k by Mo an e al. [37] whe e a comp ehensi e lis o all possible ypes can be ound. •Pos. The posi ion o he UI elemen wi hin he sc eensho is s o ed as a uple, speci ying he op-le and bo om- igh coo dina es. This in o ma ion allows p ecise localiza ion o he UI elemen wi hin he image. •Id. This componen se es as a unique iden i ie o he UI ele- men wi hin he sc eensho . I enables e icien e e encing and e ie al o speci ic elemen s du ing he analysis. In he cu en pape , he Id o he ELoC is gene a ed using he ollowing empla e: ype-cen oidX-cen oidY. In addi ion o being pseudo-unique, his empla e can be used o ind collisions o UI el- emen s be ween di e en sc eensho s, which is necessa y o u he s eps o he amewo k. Example 1. Fo an elemen iden i ied in a sc een and classi ied as a bu on, like he ones in Fig. 6, he ELoC could be ‘‘<bu on, [{boundingBox_ opLe : (450,680)},{boundingBox_bo omRigh : (490, 750)}], bu on-470-715>’’ . 3.1.2. Fea u e ex ac ion The p e ious was cen e ed on jus de ec ing he posi ion and he ype o elemen s on he sc een. In u n, he cu en s ep goes beyond ha in o ma ion since i aims o ex ac mo e in o ma ion om he sc eensho , he so-called ea u es. The amewo k p ima ily ocuses on ex ac ing ea u es om en i e sc eensho s a he han UI elemen s sepa a ely. Howe e , wi hin his con ex , wo dis inc ypes o ea u e ex ac o s can be iden i ied (c . Fig. 7): •Single ea u e ex ac o s a e designed o conside only one speci ic UI elemen a a ime. Examples o such ea u es include he bounding box p ope y, colo , enable/disable s a us, checked/ unchecked s a us, o he cen oid (c . single FE in Fig. 7) o ex ual con en o indi idual UI elemen s. •Agg ega e ea u e ex ac o s cap u e ea u es ha may ela e o mo e han one UI elemen simul aneously. Fo ins ance, hey can measu e he numbe o bu ons o ex s p esen on a sc een (c . agg ega e FE in Fig. 7) o calcula e he bounding box ha encompasses all he ex inpu s wi hin he GUI. Rega dless o hese wo ypes, a common de ini ion can be gi en (c . De ini ion 2). De ini ion 2. AUse In e ace Fea u e Ex ac o (UIFE) is a uple (P ed, Func) whe e P ed is a p edica e o e ELoC o s a e which elemen s he UIFE can be applied o, and Func is he unc ion o be applied. This unc ion ecei es a uple (img, ELoCs) and e u ns a uple (name, alue) whe e: img is a gi en image, ELoCs is a lis o ELoC elemen s (c . De ini ion 1) con ained in img, name is he name o he ea u e ha is ex ac ed, and alue is he alue o such a ea u e. The p e ious de ini ion enables any ex ac o ha conside s a sc eensho o a speci ic c op o i (i.e., he img pa ame e ). The ex- ac o can le e age on he iden i ica ion and classi ica ion in o ma ion collec ed om he p e ious s ep (i.e., he ELoCs pa ame e ). Example 2. Algo i hm 1depic s he unc ion o a UIFE ha expec s o ecei e a c op o a single UI elemen and delega es i o an ex e nal unc ion o ex ac i s enabled s a us. I he s a us canno be ob ained (i.e., he ex e nal unc ion e u ns a null), he alue o he ea u e emains emp y. The p edica e associa ed wi h his UIFE can be ‘‘𝑒𝑙𝑜𝑐 → 𝑡𝑟𝑢𝑒’’ since i can be applied o any UI elemen . Unlike he p e ious algo i hm, Algo i hm 2le e ages he p o ided ELoCs o calcula e he numbe o bu ons p esen in a sc eensho . The p edica e associa ed wi h his UIFE can be ‘‘𝑒𝑙𝑜𝑐 →𝑒𝑙𝑜𝑐.𝑡𝑦𝑝𝑒 ==′ 𝑏𝑢𝑡𝑡𝑜𝑛′’’ since i would be in cha ge o coun ing he numbe o bu ons.2 Se e al UIFEs can be de ined o ex ac ea u es o he sc eensho s o a UI Log. Each o hem is applied o each e en as desc ibed in 2No e ha only o he sake o illus a ion he same exp ession as in he p edica e is used in he algo i hm. In o ma ion Sys ems 121 (2024) 102340 6 A. Ma ínez-Rojas e al. Fig. 7. Types o UIFEs: Single and Agg ega e. Algo i hm 1: Enabled s a us ea u e ex ac o example Inpu : Image 𝑖𝑚𝑔, Lis ⟨𝐸𝐿𝑂𝐶⟩𝑒𝑙𝑜𝑐𝑠 Ou pu : (𝑛𝑎𝑚𝑒,𝑣𝑎𝑙𝑢𝑒) 1enabled ←𝑈𝐼𝑇 𝑜𝑜𝑙𝑠.𝑔𝑒𝑡𝐸𝑛𝑎𝑏𝑙𝑒𝑑𝑆𝑡𝑎𝑡𝑢𝑠(𝑖𝑚𝑔) 2i s a us == null hen 3 alue ←‘‘’’ 4else 5 alue ←𝑠𝑡𝑎𝑡𝑢𝑠 6end 7 e u n (name = ′𝐸𝑛𝑎𝑏𝑙𝑒𝑑′, alue = 𝑣𝑎𝑙𝑢𝑒) Algo i hm 2: Coun bu ons ea u e ex ac o example Inpu : Image 𝑖𝑚𝑔, Lis ⟨𝐸𝐿𝑂𝐶⟩𝑒𝑙𝑜𝑐𝑠 Ou pu : (𝑛𝑎𝑚𝑒,𝑣𝑎𝑙𝑢𝑒) 1coun ←𝑒𝑙𝑜𝑐𝑠.𝑓𝑖𝑙𝑡𝑒𝑟(𝑒𝑙𝑜𝑐 →𝑒𝑙𝑜𝑐.𝑡𝑦𝑝𝑒 ==′𝑏𝑢𝑡𝑡𝑜𝑛′).𝑐𝑜𝑢𝑛𝑡() 2 e u n (name = ′#𝐵𝑢𝑡𝑡𝑜𝑛𝑠′, alue = 𝑐𝑜𝑢𝑛𝑡) Algo i hm 3 o gene a e an ex ended UI Log, i.e., he same UI Log wi h addi ional in o ma ion ex ac ed om he log. This algo i hm ecei es (1) he log, (2) a se o UIFEs ha a e applied o he en i e sc een (e.g., UIFE depic ed in Algo i hm 2), and (3) a se o UIFEs ha a e applied o single UI elemen s in he sc een (e.g., UIFE depic ed in Algo i hm 1). Then, on he one hand, o each e en in he log, he algo i hm ex ac s one ea u e o each UIFE ha is applied o he en i e sc een (c . lines 2 o 6 o Algo i hm 3). These UIFEs will be execu ed once and ecei e all he ELoCs ha ma ch he gi en p edica e. On he o he hand, he algo i hm will ex ac one ea u e o each ELOC ha ma ches he p edica e o each UIFE applied o a single UI elemen (c . lines 7 o 13 o Algo i hm 3). Tha is, hese UIFEs will be execu ed once pe ELoC ha ma ches he p edica e. A new column is included in he UI Log o each ex ac ed ea u e. The name o he column depends only on he name o he UIFE – in he case o he i s ype o UIFEs –, o on bo h he name o he UIFE and he id o he ELOC – in he case o he second ype o UIFEs. Example 3. Fo example, le conside s a UI Log simila o he one in Fig. 2 as he i s pa ame e o Algo i hm 3. In addi ion, le conside s Algo i hm 2as he only ea u e ex ac o in he pa ame e ex ac En- i e, and Algo i hm 1as he only ea u e ex ac o in he pa ame e ex ac Single. This la e pa ame e is also accompanied by he p edica e 𝑒𝑙𝑜𝑐 →𝑡𝑟𝑢𝑒, which s a es ha he ea u e ex ac o will be applied o any UI elemen in he sc eensho . The e o e, he esul ing ex ended log will include he same columns as he o iginal log and he ollowing columns: 1. (𝑛𝑎𝑚𝑒 =‘‘#Bu ons’’, 𝑣𝑎𝑙𝑢𝑒 = 3), as depic ed in Fig. 8. 2. (𝑛𝑎𝑚𝑒 =‘‘bu on-470-715_Enabled’’, 𝑣𝑎𝑙𝑢𝑒 =‘‘ ue’’). 3. (𝑛𝑎𝑚𝑒 =‘‘bu on-655-715_Enabled’’, 𝑣𝑎𝑙𝑢𝑒 =‘‘ ue’’). Algo i hm 3: Ex end UI Log wi h ea u es Inpu : UILog 𝑙𝑜𝑔, Se ⟨𝑈𝐼𝐹 𝐸⟩𝑒𝑥𝑡𝑟𝑎𝑐𝑡𝐸𝑛𝑡𝑖𝑟𝑒, Se ⟨𝑈 𝐼𝐹 𝐸⟩ 𝑒𝑥𝑡𝑟𝑎𝑐𝑡𝑆𝑖𝑛𝑔𝑙𝑒 Ou pu : UILog 𝐸𝑥𝑡𝑒𝑛𝑑𝑒𝑑𝑙𝑜𝑔 1newLog ←𝑐𝑜𝑝𝑦(𝑙𝑜𝑔) 2 o 𝑒𝑣𝑒𝑛𝑡 in 𝑛𝑒𝑤𝑙𝑜𝑔 do 3 o 𝑢𝑖𝑓𝑒 in 𝑒𝑥𝑡𝑟𝑎𝑐𝑡𝐸𝑛𝑡𝑖𝑟𝑒 do 4(𝑛𝑎𝑚𝑒, 𝑣𝑎𝑙𝑢𝑒)← 𝑢𝑖𝑓𝑒.𝐹 𝑢𝑛𝑐(𝑒𝑣𝑒𝑛𝑡.𝑠𝑐𝑟𝑒𝑒𝑛𝑠ℎ𝑜𝑡, 𝑒𝑣𝑒𝑛𝑡.𝑒𝑙𝑜𝑐𝑠.𝑓𝑖𝑙𝑡𝑒𝑟(𝑢𝑖𝑓𝑒.𝑃 𝑟𝑒𝑑)) 5𝑐𝑜𝑙𝑢𝑚𝑛𝑁𝑎𝑚𝑒 ←𝑛𝑎𝑚𝑒 6𝑒𝑣𝑒𝑛𝑡 + ←←←←←←←← (𝑐𝑜𝑙𝑢𝑚𝑛𝑁𝑎𝑚𝑒, 𝑣𝑎𝑙𝑢𝑒) 7end 8 o 𝑢𝑖𝑓𝑒 in 𝑒𝑥𝑡𝑟𝑎𝑐𝑡𝑆𝑖𝑛𝑔𝑙𝑒 do 9 o 𝑒𝑙𝑜𝑐 in 𝑒𝑣𝑒𝑛𝑡.𝑒𝑙𝑜𝑐𝑠.𝑓 𝑖𝑙𝑡𝑒𝑟(𝑢𝑖𝑓 𝑒.𝑃 𝑟𝑒𝑑)do 10 (𝑛𝑎𝑚𝑒, 𝑣𝑎𝑙𝑢𝑒)← 𝑢𝑖𝑓𝑒.𝑎𝑝𝑝𝑙𝑦(𝑒𝑣𝑒𝑛𝑡.𝑠𝑐𝑟𝑒𝑒𝑛𝑠ℎ𝑜𝑡.𝑐𝑟𝑜𝑝(𝑒𝑙𝑜𝑐.𝑃 𝑜𝑠), 𝑒𝑙𝑜𝑐) 11 𝑐𝑜𝑙𝑢𝑚𝑛𝑁𝑎𝑚𝑒 ←𝑒𝑙𝑜𝑐.𝐼𝑑 +‘‘_’’ +𝑛𝑎𝑚𝑒 12 𝑒𝑣𝑒𝑛𝑡 + ←←←←←←←← (𝑐𝑜𝑙𝑢𝑚𝑛𝑁𝑎𝑚𝑒, 𝑣𝑎𝑙𝑢𝑒) 13 end 14 end 15 end 16 e u n newLog 3.2. Lea ning om he log The log may con ain e en s ela ed o di e en ins ances o a business p ocess. Decision poin s appea in he p ocess model whe e hey a e spli in o al e na i e b anches. To iden i y hem and lea n which pieces o in o ma ion ule he decisions aken in such ins ances, he cu en amewo k is inspi ed by Rozina e al. [21]. Fo his, he amewo k i s con e s he UI Log o an a i ac ha ML algo i hms can use, i.e., a labeled da ase (c . Sec ion 3.2.1). The ea e , his da ase is used o ain a classi ica ion algo i hm ha can explain such a beha io (c . Sec ion 3.2.2). 3.2.1. C ea ing he labeled da ase The labeled da ase should con ain one ow o each obse a ion ha leads o a decision ela ed o a decision poin . In his line, an obse a ion is he se o e en s ha p ecedes he decision poin . In addi ion, since he aim is o ain a classi ica ion algo i hm, he decision i sel should be pa o he da ase , he so-called ’label’. Acco ding o hese equi emen s, Algo i hm 4depic s how o ans- o m a UI Log in o a da ase o a gi en decision poin . The ein, his algo i hm uns o e each case in he log (c . lines 1 o 3 o Algo i hm 4) and la ens in o a single ow all he case e en s ha a e be o e he gi en decision poin (c . lines 4 o 9 o Algo i hm 4). Each e en ’s a ibu e is included as a new column o he same ow o do his la ening. The column’s name is composed using he ac i i y’s and a ibu e’s name (c . line 7 o Algo i hm 4) o a oid collisions o he In o ma ion Sys ems 121 (2024) 102340 7 A. Ma ínez-Rojas e al. Fig. 8. Ex ended UI Log om Fig. 2 including ea u e ‘‘#Bu ons’’ ex ac ed o he sc eensho o Fig. 6. Fig. 9. Label da ase c ea ion applying Algo i hm 4 o UI Log p esen ed in Fig. 8. same a ibu es o di e en ac i i ies. Besides he la ening, he label column is also included since i s a es which decision o he decision poin has been aken by his case in he UI Log (c . line 10 o Algo i hm 4). Algo i hm 4: Gene a ion o a da ase o a decision poin Inpu : UILog 𝑙𝑜𝑔, DecicionPoin 𝑑𝑒𝑐𝑖𝑠𝑖𝑜𝑛 Ou pu : Da ase 𝑑𝑎𝑡𝑎𝑠𝑒𝑡 1𝑑𝑎𝑡𝑎𝑠𝑒𝑡 ←∅ 2𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒𝑠 ←𝑠𝑒𝑝𝑎𝑟𝑎𝑡𝑒𝐿𝑜𝑔𝐵𝑦𝐶𝑎𝑠𝑒𝑠(𝑙𝑜𝑔) 3 o 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒 in 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒𝑠 do 4𝑟𝑜𝑤 ←∅ 5 o 𝑒𝑣𝑒𝑛𝑡 in 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒.𝑒𝑣𝑒𝑛𝑡𝑠𝐵𝑒𝑓 𝑜𝑟𝑒(𝑑𝑒𝑐𝑖𝑠𝑖𝑜𝑛)do 6 o 𝑎𝑡𝑡𝑟 in 𝑒𝑣𝑒𝑛𝑡.𝑎𝑡𝑡𝑟𝑖𝑏𝑢𝑡𝑒𝑠() do 7𝑟𝑜𝑤 + ←←←←←←←← (column=𝑒𝑣𝑒𝑛𝑡.𝑎𝑐𝑡𝑖𝑣𝑖𝑡𝑦𝐼𝐷 +′_′+𝑎𝑡𝑡𝑟.𝑛𝑎𝑚𝑒 , alue=𝑎𝑡𝑡𝑟.𝑣𝑎𝑙𝑢𝑒 8end 9end 10 𝑟𝑜𝑤 + ←←←←←←←← (column=’LABEL’, alue=𝑑𝑒𝑐𝑖𝑠𝑖𝑜𝑛.𝑏𝑟𝑎𝑛𝑐ℎ(𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒)⟩ 11 𝑑𝑎𝑡𝑎𝑠𝑒𝑡 + ←←←←←←←← 𝑟𝑜𝑤 12 end 13 e u n da ase Gene ally, a se o da ase s will be gene a ed o a UI Log. Mo e p ecisely, one da ase o each decision poin ha appea s in he disco e ed p ocess model needs o be gene a ed. Example 4. Fo example, le us conside he UI Log o Fig. 8. The e is a decision poin jus a e A-B wi h wo b anches: b anch 1, which leads o ac i i y C, and b anch 2, which leads o ac i i y F. The esul ing da ase ob ained a e applying Algo i hm 4will ha e wo ows whe e he i s one has 1 as he label, and he second has 2 as he label (c . Fig. 9). 3.2.2. Gene a ing a decision ee In he li e a u e, many classi ica ion algo i hms exis ha can gen- e a e models ha y o co ec ly classi y he ows o a da ase , i.e., o ma ch each ow o i s label using he es o he columns o he da ase . This p ocess is called model aining. The amewo k desc ibed in his pape uses CHAID [43], a ee-based classi ica ion algo i hm. We use a ee-based algo i hm since, unlike o he classi ica ion algo i hms, he gene a ed models can be g aphically depic ed as a ee and a e easie o unde s and by a human. In addi ion, he CHAID algo i hm has demons a ed o ha e be e pe o mance in his con ex han o he popula algo i hms [24]. I he in o ma ion con ained in he da ase is su icien , he ained model should demons a e high aining accu acy, indica ing ha he esul ing ee can e ec i ely explain all labels (i.e., he decisions made) using he column alues (i.e., he ea u es). This ee will ha e one node pe column, each ep esen ing a c ucial ea u e o explaining he labels. The edges going ou o each node delinea e he possible alues o ha column. The lea es o he ee ep esen he labels, and he pa h om he oo o each lea de ines he conjunc ion o ules necessa y o each ha speci ic label. Howe e , achie ing aining accu acy close o 100% may aise conce ns abou o e i ing. O e i ing occu s when a model excessi ely ailo s i sel o he aining da a, cap u ing noise and ou lie s. To add ess his, we conduc a ho ough analysis o addi ional measu es, such as accu acy, ecall, o F1 sco e, as pa o ou e alua ion p ocess. Mo eo e , i is impo an o cla i y ha , as occu s when using ML me hods, achie ing a high accu acy wi h he CHAID algo i hm does no inhe en ly imply he es ablishmen o causali y be ween he ea u es and he decision ou come; a he , i signi ies co ela ion. I is essen ial o acknowledge ha he ela ionships iden i ied be ween he columns and he label could be due by chance (i.e., non- ele an ea u es ha gi e an explana ion o he decision casually), ins ead o being he ac ual chain o e en s uling he decision (i.e., he ea u es ha a e uly ele an ). While acknowledging he inhe en limi a ions o he CHAID algo- i hm, we wan o unde sco e he aluable insigh s ob ained h ough he ep esen a ion o he disco e ed ee. The esul ing ee-based model se es as a ool o e ealing ela ionships be ween ea u es and decision ou comes. I is c ucial o no e, howe e , ha he p esen ed co ela ions do no au oma ically imply causa ion. Mo eo e , ou commi men o in e p e abili y is g ounded in he belie ha models should be anspa en and comp ehensible o bo h analys s and s akeholde s. By p esen ing a clea and in e p e able decision model, we aim o enhance unde s anding o decision making du ing he p ocess examined. This, in u n, suppo s he de elopmen o he au oma ion sc ip ela ed o his p ocess. 3.3. T ace back decisions Al hough he in o ma ion gi en by he classi ica ion ee is unde - s andable by a human, he nodes (i.e., he name o he columns o he da ase ) may be di icul o associa e di ec ly wi h elemen s o he unde lying p ocess. To amend his, his s ep aims o map he ee o a se o sc eensho s whe e he conside ed ea u es a e highligh ed as desc ibed in Algo i hm 5. Mo e speci ically, o each node wi h i s co esponding ou going edges (c . line 2 o Algo i hm 5), i i s looks o he ela ed e en and ga he s i s sc eensho (lines 3 o 5 o In o ma ion Sys ems 121 (2024) 102340 8 A. Ma ínez-Rojas e al. Fig. 10. Example decision ee whe e he spli ing ules (a) a e linked o ea u es in he sc eensho (b). Algo i hm 5) Then, i looks o he ela ed ELoC – in case o single ype- o ELoCs – in case o en i e ype – in he UI Log (lines 6 o 8 and 13 o Algo i hm 5). Finally, i highligh s he bounding box o each ELoC in he sc eensho (c . lines 10 and 14 o Algo i hm 5). Algo i hm 5: T acing back ee ules o sc eensho s Inpu : UILog 𝑙𝑜𝑔, T eeNode 𝑛𝑜𝑑𝑒 Ou pu : Lis ⟨𝐼𝑚𝑎𝑔𝑒⟩𝑖𝑚𝑔𝑠 1imgs ←∅ 2 o 𝑒𝑑𝑔𝑒 in node.ou goingEdges do 3(ac i i yID, elocId, ea u e) ←decompose(node.name) 4e en ←𝑙𝑜𝑔.𝑓𝑖𝑙𝑡𝑒𝑟(𝑒→𝑒.𝑎𝑐𝑡𝑖𝑣𝑖𝑡𝑦𝐼𝐷 == 𝑎𝑐𝑡𝑖𝑣𝑖𝑡𝑦𝐼𝐷 && 𝑒.𝑎𝑡𝑡𝑟𝑖𝑏𝑢𝑡𝑒(𝑓 𝑒𝑎𝑡𝑢𝑟𝑒).𝑚𝑎𝑡𝑐ℎ(𝑒𝑑𝑔𝑒.𝑛𝑎𝑚𝑒))[0] 5sc een ←𝑐𝑜𝑝𝑦(𝑒𝑣𝑒𝑛𝑡.𝑠𝑐𝑟𝑒𝑒𝑛𝑠ℎ𝑜𝑡) 6i elocId == null hen 7ui e ←𝑈𝐼𝐹 𝐸𝐶𝑎𝑡𝑎𝑙𝑜𝑔.𝑔𝑒𝑡(𝑓𝑒𝑎𝑡𝑢𝑟𝑒) 8elocs ←𝑒𝑣𝑒𝑛𝑡.𝑒𝑙𝑜𝑐𝑠.𝑓𝑖𝑙𝑡𝑒𝑟(𝑢𝑖𝑓𝑒.𝑃 𝑟𝑒𝑑) 9 o eloc in elocs do 10 sc een.highligh (eloc.Pos) 11 end 12 else 13 eloc ←𝑒𝑣𝑒𝑛𝑡𝑠.𝑒𝑙𝑜𝑐𝑠.𝑓𝑖𝑙𝑡𝑒𝑟(𝑢→𝑢.𝐼𝑑 == 𝑒𝑙𝑜𝑐𝐼𝑑) 14 sc een.highligh (eloc.Pos) 15 end 16 imgs + ←←←←←←←← 𝑠𝑐𝑟𝑒𝑒𝑛 17 end 18 e u n imgs Example 5. Fig. 10 shows he classi ica ion ee (a) esul ing om applying he CHAID algo i hm o a da ase simila o he one o Fig. 9, and he ela ed sc eensho (b) whe e he wo ee nodes a e linked o he conc e e UI elemen s. In his example, bo h UI elemen s belong o he same sc eensho . 4. E alua ion In his sec ion, we empi ically e alua e he p oposed amewo k, aiming o answe he p ima y esea ch ques ion (PRQ) posed a he beginning o his pape : Can he sc een-based Task Mining app oach e ec i ely ex ac and ep esen p ocess beha io obse ed in a UI Log wi hin eal-li e se ings? To conduc a mo e de ailed analysis, h ee speci ic sub-ques ions (SQ) a e o mula ed. Fi s , SQ1 assesses he amewo k’s capabili y o iden i y ea u es ha in luence decisions du ing p ocess execu ion wi hin he con ex o he obse ed UI Log. •SQ1: Does he p oposed amewo k iden i y ea u es ha cap u e he ele an elemen s ha humans ake in o accoun when making decisions? Building on SQ1, SQ2 in es iga es he amewo k’s abili y o ex ac condi ions based on he iden i ied ea u es. The e alua ion speci ically assesses he choice o b anches ollowing he decision poin s in he p ocess. •SQ2: Does he p oposed amewo k co ec ly disco e he condi ions ha ep esen he human decision-making? To assess he p ac ical applicabili y o he p oposed amewo k, SQ3 add esses i s pe o mance om a ime pe spec i e. •SQ3: Does he p oposed amewo k disco e human decision- making in a ime-e icien manne ? The scope o his e alua ion ocuses on he las pa o he amewo k (i.e., s eps 4 and 5), since he i s pa (i.e., s eps 1, 2, and 3) has al eady been e alua ed in p e ious wo ks [13,24]. To ensu e igo , his e alua ion ollows he case s udy p o ocol p o- posed by B e e on e al. [44]. Sec ion 4.1 desc ibes he cha ac e is ics o he p ocess selec ed o he s udy. Sec ion 4.2 de ines he speci ic p oblem si ua ion chosen o he e alua ion. Sec ion 4.3 explains he e alua ion design, while Sec ion 4.4 de ails he execu ion en i onmen . Sec ion 4.5 encloses he da a collec ion, Sec ion 4.6 p esen s he esul s ob ained and, inally, Sec ion 4.7 discuss he h ea s o he alidi y. 4.1. Case selec ion Fo his case s udy, we selec ed a eal-wo ld p ocess iden i ied by a business expe as a ypical ope a ion o au oma e wi hin he Busi- ness P ocess Ou sou cing con ex . The chosen p ocess e ol es a ound managing unsubsc ibe eques s om clien s and closely mi o s he ad- minis a i e wo k low o a Spanish elecommunica ions company. This pa icula p ocess en ails handling a olume o 300 cases pe mon h, wi h a cus ome base o app oxima ely 1.5 million cus ome s. Fo p i acy conside a ions, we ha e eplaced speci ic da a wi h syn he ic in o ma ion. This p ocess in ques ion (c . Fig. 11) is composed o 10 ac i i ies, 1 single decision poin , and 4 a ian s. I implies dealing wi h h ee di e en eal-li e sys ems (c . Fig. 12): an email clien (i.e., Gmail), an issue acking sys em (i.e., Zendesk) and an ERP sys em (i.e., Odoo). In addi ion, he p ocess includes: (1) a UI Log wi h a case o each p ocess a ian , and (2) eal-li e sc eensho s o each o i s ac i i y e en s. No e ha he company consen ed o coope a e o he sake o his e alua ion, In o ma ion Sys ems 121 (2024) 102340 9 A. Ma ínez-Rojas e al. Fig. 11. Handling o unsubsc ip ion eques s p ocess. Fig. 12. Example o a eal-li e sc eensho (a) and i s mockup e sion (b). bu did no au ho ize us o publish he eal in o ma ion abou i s co po a e sys ems. The e o e, adap ed and anonymized e sions o he UI Log and hei sc eensho s a e used. We conside his o be a p ope case o a c i ical e alua ion since i mee s he ollowing selec ion c i e ia: (1) his p ocess highly esembles a eal p ocess wi hin a company ha we coope a e wi h in Spain, and (2) he disco e y o he decision is based on in o ma ion ha appea s on he sc eensho s, i.e., wo UI elemen s: an a achmen in an email, and a checkbox on a web o m. 4.2. Objec s The e alua ion is based on a se o syn he ic p oblems closely mimicking he elecommunica ion company’s eal use case. Fo con- iden iali y easons, we canno disclose he company’s name; we we e allowed, hough, o analyze hei p ocess in eal-li e and collec e en logs, including sc eensho s, as seeds o ou e alua ion app oach. We ha e de eloped wo ypes o p oblems (𝑃) o ou e alua ion. The i s ype in ol es c ea ing scena ios using mockups o he p ocess, while he In o ma ion Sys ems 121 (2024) 102340 16 A. Ma ínez-Rojas e al. [8] M. Laci y, L.P. Willcocks, Inno a ing in se ice: The ole and managemen o au oma ion, in: Dynamic Inno a ion in Ou sou cing, Sp inge , 2018, pp. 269–325. [9] C. F ank, In oduc ion To Robo ic P ocess Au oma ion, Ins . Robo ic P ocess Au om. (2015) 35. [10] L. Willcocks, M. Laci y, A New App oach o Au oma ing Se ices, MIT Sloan Manag. Re . 58 (1) (2016) 40–49. [11] A. Jiménez-Ramí ez, J. Chacón-Mon e o, T. Wojdynsky, J. Gonzalez En iquez, Au oma ed es ing in obo ic p ocess au oma ion p ojec s, J. So w.: E ol. P ocess (2020) e2259. [12] J.G. En íquez, A. Jiménez-Ramí ez, F. Domínguez-Mayo, J. Ga cía-Ga cía, Robo ic p ocess au oma ion: a scien i ic and indus ial sys ema ic mapping s udy, IEEE Access 8 (2020) 39113–39129. [13] A. Jimenez-Rami ez, H.A. Reije s, I. Ba ba, C. Del Valle, A me hod o imp o e he ea ly s ages o he obo ic p ocess au oma ion li ecycle, in: CAiSE, 2019, pp. 446–461. [14] W.M.P. an de Aals , P ocess Mining: Da a Science in Ac ion, Sp inge , Heidelbe g, 2016. [15] L. Reinkemeye , P ocess Mining in Ac ion. P inciples, Use Cases and Ou look, Sp inge , 2020. [16] V. Leno, A. Poly yanyy, M. Dumas, M. La Rosa, F.M. Maggi, Robo ic p ocess mining: ision and challenges, Bus. In . Sys . Eng. (2020) 1–14. [17] S. Agos inelli, M. Lupia, A. Ma ella, M. Mecella, Reac i e syn hesis o so wa e obo s in RPA om use in e ace logs, Compu . Ind. 142 (2022) 103721. [18] V. Leno, A. Augus o, M. Dumas, M. La Rosa, F.M. Maggi, A. Poly yanyy, Iden i ying candida e ou ines o obo ic p ocess au oma ion om unsegmen ed UI logs, in: ICPM, 2020, pp. 153–160. [19] M. Völke , S. Siege , M. Weske, Adding decision managemen o obo ic p ocess au oma ion, in: Business P ocess Managemen : Blockchain and Robo ic P ocess Au oma ion Fo um: BPM 2021 Blockchain and RPA Fo um, Rome, I aly, Sep embe 6–10, 2021, P oceedings 19, Sp inge , 2021, pp. 23–37. [20] V. Leno, A. A mas-Ce an es, M. Dumas, M. La Rosa, F.M. Maggi, Disco e ing p ocess maps om e en s eams, in: ICSSP, 2018, pp. 86–95. [21] A. Rozina , W.M.P. an de Aals , Decision mining in p om, in: S. Dus da , J.L. Fiadei o, A.P. She h (Eds.), BPM, Sp inge , Be lin, Heidelbe g, 2006, pp. 420–425. [22] J. Gao, S.J. an Zels , X. Lu, W.M. an de Aals , Au oma ed obo ic p ocess au oma ion: A sel -lea ning app oach, in: OTM, 2019, pp. 95–112. [23] J.M. López-Ca nice , C. del Valle, J.G. En íquez, Towa ds an OpenSou ce logge o he analysis o RPA p ojec s, in: BPM, 2020, pp. 176–184. [24] A. Ma ínez-Rojas, A. Jiménez-Ramí ez, J.G. En íquez, H.A. Reije s, Analyz- ing a iable human ac ions o obo ic p ocess au oma ion, in: In e na ional Con e ence on Business P ocess Managemen , Sp inge , 2022, pp. 75–90. [25] S. Agos inelli, M. Lupia, A. Ma ella, M. Mecella, Sma RPA: A ool o eac i ely syn hesize so wa e obo s om use in e ace logs, in: CAiSE, 2021, pp. 137–145. [26] V. Leno, S. De ia ykh, A. Poly yanyy, M. La Rosa, M. Dumas, F.M. Maggi, Robidium: Au oma ed syn hesis o obo ic p ocess au oma ion sc ip s om UI logs, in: In e na ional Con e ence on Business P ocess Managemen , CEUR Wo kshop P oceedings, 2020. [27] A. Ma ínez-Rojas, A. Jiménez-Ramí ez, J.G. En íquez, H.A. Reije s, A ool- suppo ed me hod o gene a e use in e ace logs, in: Hawaii In e na ional Con e ence on Sys em Sciences, 2023. [28] S. Agos inelli, M. Lupia, A. Ma ella, M. Mecella, Au oma ed gene a ion o execu able RPA sc ip s om use in e ace logs, in: BPM, 2020, pp. 116–131. [29] A. Augus o, R. Con o i, M. Dumas, M. La Rosa, F.M. Maggi, A. Ma ella, M. Mecella, A. Soo, Au oma ed disco e y o p ocess models om e en logs: Re iew and benchma k, IEEE T ans. Knowl. Da a Eng. 31 (4) (2018) 686–705. [30] V. Leno, A. Poly yanyy, M. La Rosa, M. Dumas, F.M. Maggi, Ac ion logge : Enabling p ocess mining o obo ic p ocess au oma ion, in: P oceedings o he Disse a ion Awa d, Doc o al Conso ium, and Demons a ion T ack A BPM 2019 Co-Loca ed wi h 17 h In e na ional Con e ence on Business P ocess Managemen (BPM 2019) Vienna, Aus ia, Sep embe 1-6, 2019, Vol. 2420, CEUR-WS, 2019, pp. 124–128. [31] J. Geye -Klingebe g, J. Nakladal, F. Baldau , F. Vei , P ocess mining and obo ic p ocess au oma ion: A pe ec ma ch., in: BPM, 2018, pp. 124–131. [32] E. Bazheno a, S. Bülow, M. Weske, Disco e ing decision models om e en logs, in: BIS, 2016, pp. 237–251. [33] H.A. Sleiman, R. Co chuelo, Tex: An e icien and e ec i e unsupe ised web in o ma ion ex ac o , Knowl.-Based Sys . 39 (2013) 109–123. [34] P. Jiménez, J.C. Roldán, R. Co chuelo, A clus e ing app oach o ex ac da a om HTML ables, In . P ocess. Manage. 58 (6) (2021) 102683. [35] C. Ike ionwu, D. Edga , Secu ed se ice deli e y model o ou sou ced se ices in a business p ocess ou sou cing ela ionship, In . J. In . Commun. Sci. 4 (1) (2019) 7–17. [36] Z. Xu, X. Baojie, W. Guoxin, Canny edge de ec ion based on open CV, in: 2017 13 h ICEMI, 2017, pp. 53–56. [37] K. Mo an, C. Be nal-Cá denas, M. Cu cio, R. Bone , D. Poshy anyk, Machine lea ning-based p o o yping o g aphical use in e aces o mobile apps, IEEE T ans. So w. Eng. 46 (2) (2018) 196–221. [38] B.P. Majumde , N. Po i, S. Ta a, J.B. Wend , Q. Zhao, M. Najo k, Rep esen a ion lea ning o in o ma ion ex ac ion om o m-like documen s, in: ACL, 2020, pp. 6495–6504. [39] Ke as OCR, 2022, A ailable a h ps://gi hub.com/ aus omo ales/ke as-oc . (Las Accessed Ma ch 2022). [40] T. Mi chell, Machine Lea ning, McG aw Hill, 1997. [41] L.H. Gilpin, D. Bau, B.Z. Yuan, A. Bajwa, M. Spec e , L. Kagal, Explaining explana ions: An o e iew o in e p e abili y o machine lea ning, in: 2018 IEEE 5 h DSAA, 2018, pp. 80–89. [42] J. Chen, M. Xie, Z. Xing, C. Chen, X. Xu, L. Zhu, G. Li, Objec de ec ion o g aphical use in e ace: Old ashioned o deep lea ning o a combina ion? in: P oceedings o he 28 h ACM Join Mee ing on Eu opean So wa e Enginee ing Con e ence and Symposium on he Founda ions o So wa e Enginee ing, 2020, pp. 1202–1214. [43] G.V. Kass, An explo a o y echnique o in es iga ing la ge quan i ies o ca ego ical da a, J. R. S a . Soc. Se . C. Appl. S a . 29 (2) (1980) 119–127. [44] P. B e e on, B. Ki chenham, D. Budgen, Z. Li, Using a p o ocol empla e o case s udy planning, in: 12 h In e na ional Con e ence on E alua ion and Assessmen in So wa e Enginee ing, (EASE) 12, 2008, pp. 1–8. [45] V. Leno, M. Dumas, M. La Rosa, F.M. Maggi, A. Poly yanyy, Au oma ed disco e y o da a ans o ma ions o obo ic p ocess au oma ion, 2020, a Xi p ep in a Xi :2001.01007.