A sc eensho -based ask mining amewo k o disclosing he d i e s behind
a iable human ac ions
A. Ma ínez-Rojas a,∗, A. Jiménez-Ramí ez a, J.G. En íquez a, H.A. Reije s b
aDepa men o Compu e Languages and Sys ems, Uni e si y o Se ille, A enida Reina Me cedes, s/n, 41012, Se ille, Spain
bDepa men o In o ma ion and Compu ing Sciences, U ech Uni e si y, P ince onplein 5, 3584 CC, U ech , The Ne he lands
A R T I C L E I N F O
Keywo ds:
Robo ic P ocess Au oma ion
UI Log
Use beha io mining
Task mining
Decision model disco e y
A B S T R A C T
Robo ic P ocess Au oma ion (RPA) enables subjec ma e expe s o use he g aphical use in e ace as a
means o au oma e and in eg a e sys ems. This is a as me hod o au oma e epe i i e, mundane asks. To
a oid cons uc ing a so wa e obo om sc a ch, Task Mining app oaches can be used o moni o human
beha io h ough a se ies o imes amped e en s, such as mouse clicks and keys okes. F om a so-called Use
In e ace log (UI Log), i is possible o au oma ically disco e he p ocess model behind his beha io . Howe e ,
when he disco e ed p ocess model shows di e en p ocess a ian s, i is ha d o de e mine wha d i es a
human’s decision o execu e one a ian o e he o he . Exis ing app oaches do analyze he UI Log in sea ch
o he unde lying ules, bu neglec wha can be seen on he sc een. As a esul , a majo pa o he human
decision-making emains hidden. To add ess his gap, his pape desc ibes a Task Mining amewo k ha uses
he sc eensho o each e en in he UI Log as an addi ional sou ce o in o ma ion. F om such an en iched
UI Log, by using image-p ocessing echniques and Machine Lea ning algo i hms, a decision ee is c ea ed,
which o e s a mo e comple e explana ion o he human decision-making p ocess. The p esen ed amewo k
can exp ess he decision ee g aphically, explici ly iden i ying which elemen s in he sc eensho s a e ele an
o make he decision. The amewo k has been e alua ed h ough a case s udy ha in ol es a p ocess wi h
eal-li e sc eensho s. The esul s indica e a sa is ac o ily high accu acy o he o e all app oach, e en i a small
UI Log is used. The e alua ion also iden i ies challenges o applying he amewo k in a eal-li e se ing when
a high densi y o in e ace elemen s is p esen .
1. In oduc ion
In his e a o Hype au oma ion [1], o ganiza ions a e pushed o
apidly iden i y and au oma e all business p ocesses whe e possible.1
This d i e has led o he widesp ead adop ion o low-code and no-
code ools, alongside he inc easing popula i y o Robo ic P ocess Au-
oma ion (RPA). Unlike adi ional au oma ion (e.g., API-based), RPA
enables he use o g aphical use in e aces as a means o au oma e
and in eg a e sys ems; by doing so, a so wa e obo ha is c ea ed
in his way can closely mimic how humans in e ac wi h compu e
applica ions [2].
Se e al au ho s ha e epo ed he bene i s ha RPA b ings. Since
his echnology is ela i ely easy o use by subjec ma e expe s and
does no necessa ily equi e IT skills o deploy i , RPA helps o sa e
cos s, inc eases agili y, imp o es quali y [3–6], as well as cus ome
sa is ac ion [7,8], while i s le el o in usi eness is low [9,10].
∗Co esponding au ho .
As he p e alence o RPA con inues o g ow, a ious echniques
ha e eme ged o suppo and augmen so wa e au oma ion h ough-
ou i s li e cycle phases, encompassing analysis, design, de elopmen ,
deploymen , es ing, and moni o ing [11]. Typically, au oma ion ini ia-
i es s a wi h he analysis phase, in ol ing he examina ion o exis ing
p ocesses o iden i y op imal candida es o au oma ion [12]. How-
e e , adi ional app oaches, such as in e iews wi h subjec -ma e
expe s o analysis o p ocess documen a ion, come wi h inhe en isks,
including eliance on incomple e o ou da ed p ocess desc ip ions,
biased in o ma ion, and o e sigh o excep ional si ua ions ha may
be deemed inconsequen ial o humans bu hold signi icance o an
au oma ion p ojec [13]. To mi iga e hese isks and a oid cons uc ing
au oma ion om sc a ch, inno a i e app oaches like Task Mining [14,
15] and Robo ic P ocess Mining [16] ha e eme ged o lea n om he
obse a ion o use s while hey a e pe o ming he p ocesses. These
echniques eco d use beha io in o a Use In e ace log (UI Log, i.e., a
E-mail add esses: [email p o ec ed] (A. Ma ínez-Rojas), [email p o ec ed] (A. Jiménez-Ramí ez), [email p o ec ed] (J.G. En íquez), [email p o ec ed]
(H.A. Reije s).
1 Ga ne ’s de ini ion o hype au oma ion: h ps://www.ga ne .com/en/in o ma ion- echnology/glossa y/hype au oma ion
h ps://doi.o g/10.1016/j.is.2023.102340
Recei ed 22 Janua y 2023; Recei ed in e ised o m 14 No embe 2023; Accep ed 18 Decembe 2023
A ailable online 21 Decembe 2023
0306-4379/© 2023 The Au ho (s).
In o ma ion Sys ems 121 (2024) 102340
2
A. Ma ínez-Rojas e al.
se ies o imes amped e en s like mouse clicks and keys okes) and
hen analyze hem o enable he c ea ion o he au oma ion solu ion
by disco e ing he p ocess model behind he log.
The ield o p ocess disco e y o UI Logs has been explo ed in
conside able dep h [13,17,18]. Howe e , when he disco e ed model
shows di e en p ocess a ian s, he e is li le suppo o de e mine
wha d i es he execu ion o one a ian o e he o he . The e o e,
he a ia ions obse ed in he eco ded UI Log sugges ha he hu-
man ope a o makes di e en decisions o di e en cases. Despi e
his, iden i ying he speci ic ules go e ning hese decisions is gen-
e ally no easible. Ne e heless, unco e ing hese ules is necessa y
o model decisions, which is essen ial o e ec i ely implemen RPA
so wa e [19].
Acknowledging such signi icance, se e al esea che s ha e unde -
aken his challenge [17,20–22]. Thei app oaches a e based on he
in o ma ion ha is in he UI Log. Howe e , a limi a ion a ises when
on-sc een elemen s solely in luence decisions, as hese lea e no angible
ail wi hin he log. Consequen ly, ce ain aspec s o human decision-
making emain hidden, c ea ing a po en ial obs acle o au oma ion.
To add ess his p oblem, he cu en pape desc ibes a amewo k
ha conside s sc eensho s (i.e., sc een cap u es ha a e aken o e e y
e en in he UI Log) as an addi ional sou ce o knowledge. These sc een-
sho s a e analyzed o ex ac s uc u ed in o ma ion ha can ex end he
da a eco ded o each e en in he log. The UI Log ex ended wi h his
new in o ma ion, can hen be p ocessed o ain a Machine Lea ning
algo i hm, which ies o ind he ela ions be ween his in o ma ion
and he decisions made by he use . The amewo k combines hese
ules wi h he ela ed sc eensho s o g aphically exp ess he ac o s ha
in luence human decision-making. The amewo k ha is p esen ed in
his pape builds on, in eg a es, and ex ends ou p e ious wo k, which
enables us o (1) moni o use beha io ha gene a es a UI Log ha
s o es one sc eensho o each e en [23], and (2) analyze hese UI Logs
and he sc eensho s o disco e he unde lying p ocess model [13].
Speci ically, his pape ex ends ou p oposal in [24]. In ha pape ,
p omising esul s we e ob ained when dealing wi h simple sc eensho s
(i.e., syn he ic mockups) and a he simple condi ions. In u n, he
cu en wo k g ea ly enhances a p e ious wo k by add essing a unda-
men al P ima y Resea ch Ques ion (PRQ) no conside ed so a : Can he
sc een-based Task Mining app oach ex ac and ep esen he p ocess
beha io obse ed in a UI Log wi hin eal-li e se ings?
In ending o answe PRQ, his pape signi ican ly ex ends and im-
p o es ou p e ious wo k [24] by:
1. Fo malizing and classi ying he ea u e ex ac o s ha can be
applied o ex ac in o ma ion om sc eensho s.
2. P oposing da ase gene a ion me hods o each class o ea u e
ex ac o .
3. Ex ending he amewo k by including a new s ep o explain he
decision g aphically. This new s ep enables he connec ion o he
decision ules o he eal UI elemen s on he sc eensho s.
4. E alua ing he p oposal using a case s udy ha in ol es a p o-
cess wi h eal-li e sc eensho s, and analyzes he pe o mance
ob ained o each class o ea u e ex ac o .
The de eloped amewo k is o in e es o esea che s and p ac i ione s
in he RPA ield since i explains and demons a es how o disco e
au oma able decision ules ha would o he wise emain hidden. Fu -
he mo e, esul s show ha he amewo k is applicable e en when
only ela i ely small UI Logs a e a ailable.
The es o he pape is o ganized as ollows. Sec ion 2p o ides
he backg ound by desc ibing he opics o beha io moni o ing, ML,
and image p ocessing. Sec ion 3desc ibes he p oposed amewo k and
explains how i is able o explain decisions om a UI Log g aphically.
Sec ion 4desc ibes he se up and he execu ion o he empi ical e alua-
ion, which was pe o med o compa e he di e en ea u e ex ac o s.
Sec ion 5 e iews simila app oaches in he li e a u e. Finally, Sec ion 6
summa izes ou wo k and desc ibes u u e esea ch lines.
2. Backg ound
The app oach p esen ed in his pape builds on he opics o P o-
cess Disco e y in RPA, G aphical Use In e ace (GUI) analysis, and
Machine Lea ning (ML) o decision disco e y. P ocess disco e y (c .
Sec ion 2.1) enables he sys ema ic analysis o business p ocesses,
op imizing wo k lows o RPA in eg a ion. GUI analysis (c . Sec ion 2.2)
empowe s RPA bo s o e icien ly in e ac wi h di e se so wa e in e -
aces h ough image ecogni ion and objec de ec ion. Finally, ML (c .
Sec ion 2.3) elucida es he human decision-making p ocess, enabling a
clea unde s anding o he ules and ac o s guiding ac ions.
2.1. P ocess disco e y in RPA
In he RPA ield, di e en al e na i es exis o moni o use s by
ob aining e en logs om he in e ac ion be ween he use and he
on -end o he in o ma ion sys ems ( he so-called UI Logs) [13,25,26].
The mos ep esen a i e one o his wo k is he UI Log concep
p esen ed by Jiménez-Ramí ez e al. and Ma ínez-Rojas e al. [13,27],
which, unlike he o he s, p o ides special a en ion o he sc eensho s
as an in o ma ion sou ce.
The UI Logs a e se s o imes amped e en s cap u ed a he ope -
a ing sys em (OS) le el, independen o any speci ic p og ams. They
encompass essen ial in o ma ion abou each e en , including (1) he
associa ed p ocess (e.g., case o ac i i y), (2) he applica ion being
used (e.g., app name o sc eensho ), and (3) he speci ic use ac ion
pe o med (e.g., mouse click o keys oke). These e en s a e igge ed
based on a ange o OS e en s, such as clicks, keys okes, o speci ic
combina ions o hem.
Task Mining o Robo ic P ocess Mining echniques can be applied
o eco d hese UI Logs and disco e he ope a o p ocess model ou o
hem [13,13,16,20,28–32]. Mo eo e , hese p oposals include unc ion-
ali ies o clean he UI Log o i ele an in o ma ion so ha noise in he
esul ing p ocess model is il e ed. In addi ion, exis ing wo ks [17,18]
show how o selec a ian s/cases/ac i i ies acco ding o he equency,
leng h, and o he c i e ia ha a e use ul o iden i y p ocess candida es
o obo ize. The p ocess model esul ing om hese app oaches may
con ain decision poin s and sepa a e b anches o di e en p ocess
a ian s.
To illus a e how a p ocess is disco e ed based on use moni o ing,
Fig. 1 depic s a subse o he elemen a y use in e aces o egis e ing a
cus ome in he con ex o a elecommunica ion company. In his egis-
a ion p ocess, he human ope a o checks he email inbox and e iews
he pending emails ega ding he egis a ion asks. A e opening he
email, he ope a o has o alida e ha all p o ided in o ma ion ela ed
o he new cus ome is co ec . Mo e p ecisely, he cus ome ID ca d is
expec ed o be included as an a achmen . I i is indeed included (c .
Fig. 1a), all he cus ome da a has o be egis e ed in o a CRM sys em
(c . Fig. 1b). O he wise, in case he ID ca d is missing (c . Fig. 1c), an
email has o be sen o he cus ome eques ing such da a (c . Fig. 1d).
Rega dless o which o hese wo si ua ions occu ed (i.e., Va ian 1 o
Va ian 2 o Fig. 1), he ope a o e u ns o hei inbox o p ocess he
nex mail. This p ocess mus be epea ed se e al imes du ing he day
o p ocess he en i e queue o emails in he ope a o ’s inbox.
Fig. 2 shows an exce p o a possible UI Log esul ing om moni-
o ing he ope a o while execu ing he p ocess o Fig. 1 (e.g., wi h a
key logge [23]).
Fig. 3 shows a possible p ocess model disco e ed om he UI
Log o Fig. 2. This kind o ep esen a ion suppo s he unde s anding
o he UI Log. In his example, i may be deduced ha he UI Log
co ec ly depic s ha i con ains a single decision poin a e ac i i y
‘‘B’’ (i.e., a e seeing he email), whe e he p ocess b anches o in o
wo di e en a ian s, i.e., Id.1 and Id.2. None heless, he model does
no disclose he ac ual condi ion ha ules such a decision. Wha is
mo e, i may po en ially depend on he in o ma ion ha appea s on
he sc een.
In o ma ion Sys ems 121 (2024) 102340
3
A. Ma ínez-Rojas e al.
Fig. 1. Mockups o mo i a ing example.
Fig. 2. Exce p o a UI Log ob ained om a keylogge .
Fig. 3. P ocess model disco e ed om he UI Log o Fig. 2.
To unco e his in o ma ion, esea che s analyze use in e ac ions
[17]. T adi ional echniques o ex ac his in o ma ion om he sc een
ely on ope a ing sys em APIs o accessing he Documen Objec Model
(DOM), which enables sys ema ic que ying o UI elemen s [33,34].
Despi e hese possibili ies, some p ocesses, especially in he ield o
ou sou cing, in ol e i ualized connec ions (e.g., using Ci ix o Team
Viewe ) o enhanced secu i y [35]. This es ic s di ec access o
he Windows API o he DOM ee. Consequen ly, in such scena ios,
le e aging UI Logs and sc eensho s becomes he p e e ed me hod o
ga he ing in o ma ion and e ec i ely au oma ing p ocesses.
2.2. GUI analysis
To analyze he sc eensho s, he ield o GUI analysis allows iden-
i ying he UI elemen s exis ing wi hin an image [36–38]. Besides
jus loca ing he elemen on he sc een (i.e., calcula ing he bounding
boxes), hey can classi y hem by he ype o elemen , e.g., image, bu -
on, o ex . Op ical Cha ac e Recogni ion (OCR) echniques a e o en
mo e app op ia e when dealing wi h ex s. Fo ins ance, Ke asOCR [39]
allows ex ac ing wo ds and hei bounding boxes om sc eensho s.
In o ma ion Sys ems 121 (2024) 102340
4
A. Ma ínez-Rojas e al.
Fig. 4. Example o GUI analysis applied o a sc eensho o de ec ing bu ons.
Fig. 5. F amewo k o a iabili y analysis h ough in e p e able decisions om UI Logs.
Fig. 4 shows a eal UI sc eensho ela ed o ac i i y ‘‘B’’ o Fig. 3.
Wha can be seen he e is how he app oach om [37] has been ailo ed
o desk op sc eensho s o analyze hem and highligh he UI elemen s
o a conc e e ype, i.e., bu ons.
2.3. ML o decision disco e y
The ML domain p oposes supe ised lea ning algo i hms ha can
lea n om a da ase [40] o explain he ules behind a decision.
Da ase s a e commonly ep esen ed in abula o m whe e each column
is an inpu a iable, and one o hem is he label, i.e., wha needs
o be classi ied. Each ow is a membe o obse a ion o he da ase .
In a classi ica ion p oblem, he algo i hm ies o ind pa e ns in he
obse a ion’s inpu a iables ha help explain he labels. Decision ees
a e an example o classi ica ion algo i hms ha , besides jus p o iding
a classi ica ion, do so in a human-in e p e able way [41].
3. F amewo k
This sec ion desc ibes he p oposed amewo k o explaining he
decisions om he da a in a UI Log using sc eensho s (c . Fig. 5). I is
based on exis ing app oaches [13,25,30] ha (1) deal wi h moni o ing
g aphical use in e aces o gene a e UI Logs (c . s ep 1 in Fig. 5), and
(2) disco e p ocess models om such logs (c . s ep 3 in Fig. 5).
As s a ed be o e, disco e ed p ocess models may con ain decision
poin s whose unde s anding could be cumbe some. To amend his, he
p oposed amewo k aims o ind an explana ion o each decision
poin , i.e., he ules ha de e mine which b anch o ollow. Such
explana ions may conside in o ma ion ha is g aphically depic ed in
he use in e aces, as cap u ed in s ep 1. Mo e p ecisely, he ame-
wo k includes a s ep o ex ac in o ma ion om use in e aces and
inco po a e i in o he UI Log (c . s ep 2 in Fig. 5, Sec ion 3.1). Some
app oaches o p ocess model disco e y, such as [13], a e bene i ed
om his ex ended in o ma ion, and his is why he ea u e ex ac ion
s ep is es ablished be o e he p ocess disco e y s ep in he amewo k.
Once he p ocess model is disco e ed, he ex ended UI Log om s ep
2 is used as inpu o lea n he decision ee (i.e., ules) behind each
decision poin (c . s ep 4 in Fig. 5, Sec ion 3.2). Finally, hese decision
ees a e p ocessed o ace back hei ules o he g aphical in o ma ion
in he sc eensho s (c . s ep 5 in Fig. 5, Sec ion 3.3). This helps o make
sense o he decision ees.
3.1. Ex ac ing ea u es om g aphical use in e aces
In gene al, a g aphical use in e ace con ains a numbe o UI
elemen s ha a e ep esen ed g aphically on he sc een. To ex ac
in o ma ion om a use in e ace, he amewo k i s iden i ies and
classi ies he UI elemen s on each sc eensho (c . Sec ion 3.1.1), and
hen ex ac s ea u es om hem (c . Sec ion 3.1.2)
3.1.1. Iden i ica ion and classi ica ion o UI elemen s
The p oposed amewo k uses he app oach p oposed by Chen
e al. [42] o iden i y each UI elemen on he sc een, i.e., o ob ain
he bounding boxes o all he UI elemen s ha a e ound. In addi ion,
he solu ion p esen ed by Mo an e al. [37] is hen used o classi y
each iden i ied elemen , i.e., o ob ain he ype o each UI elemen ,
e.g., bu on, checkbox, e c.
In o ma ion Sys ems 121 (2024) 102340
5
A. Ma ínez-Rojas e al.
Fig. 6. Example o UI Elemen Loca ion and Classi ica ion (ELoC) da a o one o he bu ons on he sc een.
The e o e, his s ep gene a es and s o es me ada a ela ed o each
sc eensho . Ul ima ely, he me ada a would comp ehensi ely desc ibe
he sc eensho in a ex ual o m in a simila way as a DOM ee does
wi h a web page. Howe e , he app oach p esen ed in he amewo k
simpli ies hese desc ip ions o keep jus he essen ial in o ma ion.
Speci ically, his me ada a comp ises a lis o UI Elemen Loca ion and
Classi ica ion da a (c . De ini ion 1) ha will be s o ed o each e en
in he log.
De ini ion 1. The UI Elemen Loca ion and Classi ica ion (ELoC)
da a is a uple (Type, Pos, Id) wi h he ollowing componen s:
•Type. This indica es he ype o he UI elemen and can ake on
alues such as Tex View, ImageView, o Bu on, among o he s.
This classi ica ion is based on he wo k by Mo an e al. [37] whe e
a comp ehensi e lis o all possible ypes can be ound.
•Pos. The posi ion o he UI elemen wi hin he sc eensho is s o ed
as a uple, speci ying he op-le and bo om- igh coo dina es.
This in o ma ion allows p ecise localiza ion o he UI elemen
wi hin he image.
•Id. This componen se es as a unique iden i ie o he UI ele-
men wi hin he sc eensho . I enables e icien e e encing and
e ie al o speci ic elemen s du ing he analysis.
In he cu en pape , he Id o he ELoC is gene a ed using he
ollowing empla e: ype-cen oidX-cen oidY. In addi ion o being
pseudo-unique, his empla e can be used o ind collisions o UI el-
emen s be ween di e en sc eensho s, which is necessa y o u he
s eps o he amewo k.
Example 1. Fo an elemen iden i ied in a sc een and classi ied
as a bu on, like he ones in Fig. 6, he ELoC could be ‘‘<bu on,
[{boundingBox_ opLe : (450,680)},{boundingBox_bo omRigh : (490,
750)}], bu on-470-715>’’ .
3.1.2. Fea u e ex ac ion
The p e ious was cen e ed on jus de ec ing he posi ion and he
ype o elemen s on he sc een. In u n, he cu en s ep goes beyond
ha in o ma ion since i aims o ex ac mo e in o ma ion om he
sc eensho , he so-called ea u es.
The amewo k p ima ily ocuses on ex ac ing ea u es om en i e
sc eensho s a he han UI elemen s sepa a ely. Howe e , wi hin his
con ex , wo dis inc ypes o ea u e ex ac o s can be iden i ied (c .
Fig. 7):
•Single ea u e ex ac o s a e designed o conside only one speci ic
UI elemen a a ime. Examples o such ea u es include he
bounding box p ope y, colo , enable/disable s a us, checked/
unchecked s a us, o he cen oid (c . single FE in Fig. 7) o ex ual
con en o indi idual UI elemen s.
•Agg ega e ea u e ex ac o s cap u e ea u es ha may ela e o
mo e han one UI elemen simul aneously. Fo ins ance, hey
can measu e he numbe o bu ons o ex s p esen on a sc een
(c . agg ega e FE in Fig. 7) o calcula e he bounding box ha
encompasses all he ex inpu s wi hin he GUI.
Rega dless o hese wo ypes, a common de ini ion can be gi en
(c . De ini ion 2).
De ini ion 2. AUse In e ace Fea u e Ex ac o (UIFE) is a uple
(P ed, Func) whe e P ed is a p edica e o e ELoC o s a e which
elemen s he UIFE can be applied o, and Func is he unc ion o be
applied. This unc ion ecei es a uple (img, ELoCs) and e u ns a uple
(name, alue) whe e: img is a gi en image, ELoCs is a lis o ELoC
elemen s (c . De ini ion 1) con ained in img, name is he name o he
ea u e ha is ex ac ed, and alue is he alue o such a ea u e.
The p e ious de ini ion enables any ex ac o ha conside s a
sc eensho o a speci ic c op o i (i.e., he img pa ame e ). The ex-
ac o can le e age on he iden i ica ion and classi ica ion in o ma ion
collec ed om he p e ious s ep (i.e., he ELoCs pa ame e ).
Example 2. Algo i hm 1depic s he unc ion o a UIFE ha expec s
o ecei e a c op o a single UI elemen and delega es i o an ex e nal
unc ion o ex ac i s enabled s a us. I he s a us canno be ob ained
(i.e., he ex e nal unc ion e u ns a null), he alue o he ea u e
emains emp y. The p edica e associa ed wi h his UIFE can be ‘‘𝑒𝑙𝑜𝑐 →
𝑡𝑟𝑢𝑒’’ since i can be applied o any UI elemen .
Unlike he p e ious algo i hm, Algo i hm 2le e ages he p o ided
ELoCs o calcula e he numbe o bu ons p esen in a sc eensho . The
p edica e associa ed wi h his UIFE can be ‘‘𝑒𝑙𝑜𝑐 →𝑒𝑙𝑜𝑐.𝑡𝑦𝑝𝑒 ==′
𝑏𝑢𝑡𝑡𝑜𝑛′’’ since i would be in cha ge o coun ing he numbe o bu ons.2
Se e al UIFEs can be de ined o ex ac ea u es o he sc eensho s
o a UI Log. Each o hem is applied o each e en as desc ibed in
2No e ha only o he sake o illus a ion he same exp ession as in he
p edica e is used in he algo i hm.

In o ma ion Sys ems 121 (2024) 102340
6
A. Ma ínez-Rojas e al.
Fig. 7. Types o UIFEs: Single and Agg ega e.
Algo i hm 1: Enabled s a us ea u e ex ac o example
Inpu : Image 𝑖𝑚𝑔, Lis ⟨𝐸𝐿𝑂𝐶⟩𝑒𝑙𝑜𝑐𝑠
Ou pu : (𝑛𝑎𝑚𝑒,𝑣𝑎𝑙𝑢𝑒)
1enabled ←𝑈𝐼𝑇 𝑜𝑜𝑙𝑠.𝑔𝑒𝑡𝐸𝑛𝑎𝑏𝑙𝑒𝑑𝑆𝑡𝑎𝑡𝑢𝑠(𝑖𝑚𝑔)
2i s a us == null hen
3 alue ←‘‘’’
4else
5 alue ←𝑠𝑡𝑎𝑡𝑢𝑠
6end
7 e u n (name = ′𝐸𝑛𝑎𝑏𝑙𝑒𝑑′, alue = 𝑣𝑎𝑙𝑢𝑒)
Algo i hm 2: Coun bu ons ea u e ex ac o example
Inpu : Image 𝑖𝑚𝑔, Lis ⟨𝐸𝐿𝑂𝐶⟩𝑒𝑙𝑜𝑐𝑠
Ou pu : (𝑛𝑎𝑚𝑒,𝑣𝑎𝑙𝑢𝑒)
1coun ←𝑒𝑙𝑜𝑐𝑠.𝑓𝑖𝑙𝑡𝑒𝑟(𝑒𝑙𝑜𝑐 →𝑒𝑙𝑜𝑐.𝑡𝑦𝑝𝑒 ==′𝑏𝑢𝑡𝑡𝑜𝑛′).𝑐𝑜𝑢𝑛𝑡()
2 e u n (name = ′#𝐵𝑢𝑡𝑡𝑜𝑛𝑠′, alue = 𝑐𝑜𝑢𝑛𝑡)
Algo i hm 3 o gene a e an ex ended UI Log, i.e., he same UI Log wi h
addi ional in o ma ion ex ac ed om he log. This algo i hm ecei es
(1) he log, (2) a se o UIFEs ha a e applied o he en i e sc een
(e.g., UIFE depic ed in Algo i hm 2), and (3) a se o UIFEs ha a e
applied o single UI elemen s in he sc een (e.g., UIFE depic ed in
Algo i hm 1). Then, on he one hand, o each e en in he log, he
algo i hm ex ac s one ea u e o each UIFE ha is applied o he en i e
sc een (c . lines 2 o 6 o Algo i hm 3). These UIFEs will be execu ed
once and ecei e all he ELoCs ha ma ch he gi en p edica e. On he
o he hand, he algo i hm will ex ac one ea u e o each ELOC ha
ma ches he p edica e o each UIFE applied o a single UI elemen (c .
lines 7 o 13 o Algo i hm 3). Tha is, hese UIFEs will be execu ed once
pe ELoC ha ma ches he p edica e. A new column is included in he
UI Log o each ex ac ed ea u e. The name o he column depends
only on he name o he UIFE – in he case o he i s ype o UIFEs –,
o on bo h he name o he UIFE and he id o he ELOC – in he case
o he second ype o UIFEs.
Example 3. Fo example, le conside s a UI Log simila o he one in
Fig. 2 as he i s pa ame e o Algo i hm 3. In addi ion, le conside s
Algo i hm 2as he only ea u e ex ac o in he pa ame e ex ac En-
i e, and Algo i hm 1as he only ea u e ex ac o in he pa ame e
ex ac Single. This la e pa ame e is also accompanied by he p edica e
𝑒𝑙𝑜𝑐 →𝑡𝑟𝑢𝑒, which s a es ha he ea u e ex ac o will be applied o
any UI elemen in he sc eensho . The e o e, he esul ing ex ended log
will include he same columns as he o iginal log and he ollowing
columns:
1. (𝑛𝑎𝑚𝑒 =‘‘#Bu ons’’, 𝑣𝑎𝑙𝑢𝑒 = 3), as depic ed in Fig. 8.
2. (𝑛𝑎𝑚𝑒 =‘‘bu on-470-715_Enabled’’, 𝑣𝑎𝑙𝑢𝑒 =‘‘ ue’’).
3. (𝑛𝑎𝑚𝑒 =‘‘bu on-655-715_Enabled’’, 𝑣𝑎𝑙𝑢𝑒 =‘‘ ue’’).
Algo i hm 3: Ex end UI Log wi h ea u es
Inpu : UILog 𝑙𝑜𝑔, Se ⟨𝑈𝐼𝐹 𝐸⟩𝑒𝑥𝑡𝑟𝑎𝑐𝑡𝐸𝑛𝑡𝑖𝑟𝑒, Se ⟨𝑈 𝐼𝐹 𝐸⟩
𝑒𝑥𝑡𝑟𝑎𝑐𝑡𝑆𝑖𝑛𝑔𝑙𝑒
Ou pu : UILog 𝐸𝑥𝑡𝑒𝑛𝑑𝑒𝑑𝑙𝑜𝑔
1newLog ←𝑐𝑜𝑝𝑦(𝑙𝑜𝑔)
2 o 𝑒𝑣𝑒𝑛𝑡 in 𝑛𝑒𝑤𝑙𝑜𝑔 do
3 o 𝑢𝑖𝑓𝑒 in 𝑒𝑥𝑡𝑟𝑎𝑐𝑡𝐸𝑛𝑡𝑖𝑟𝑒 do
4(𝑛𝑎𝑚𝑒, 𝑣𝑎𝑙𝑢𝑒)←
𝑢𝑖𝑓𝑒.𝐹 𝑢𝑛𝑐(𝑒𝑣𝑒𝑛𝑡.𝑠𝑐𝑟𝑒𝑒𝑛𝑠ℎ𝑜𝑡, 𝑒𝑣𝑒𝑛𝑡.𝑒𝑙𝑜𝑐𝑠.𝑓𝑖𝑙𝑡𝑒𝑟(𝑢𝑖𝑓𝑒.𝑃 𝑟𝑒𝑑))
5𝑐𝑜𝑙𝑢𝑚𝑛𝑁𝑎𝑚𝑒 ←𝑛𝑎𝑚𝑒
6𝑒𝑣𝑒𝑛𝑡 +
←←←←←←←← (𝑐𝑜𝑙𝑢𝑚𝑛𝑁𝑎𝑚𝑒, 𝑣𝑎𝑙𝑢𝑒)
7end
8 o 𝑢𝑖𝑓𝑒 in 𝑒𝑥𝑡𝑟𝑎𝑐𝑡𝑆𝑖𝑛𝑔𝑙𝑒 do
9 o 𝑒𝑙𝑜𝑐 in 𝑒𝑣𝑒𝑛𝑡.𝑒𝑙𝑜𝑐𝑠.𝑓 𝑖𝑙𝑡𝑒𝑟(𝑢𝑖𝑓 𝑒.𝑃 𝑟𝑒𝑑)do
10 (𝑛𝑎𝑚𝑒, 𝑣𝑎𝑙𝑢𝑒)←
𝑢𝑖𝑓𝑒.𝑎𝑝𝑝𝑙𝑦(𝑒𝑣𝑒𝑛𝑡.𝑠𝑐𝑟𝑒𝑒𝑛𝑠ℎ𝑜𝑡.𝑐𝑟𝑜𝑝(𝑒𝑙𝑜𝑐.𝑃 𝑜𝑠), 𝑒𝑙𝑜𝑐)
11 𝑐𝑜𝑙𝑢𝑚𝑛𝑁𝑎𝑚𝑒 ←𝑒𝑙𝑜𝑐.𝐼𝑑 +‘‘_’’ +𝑛𝑎𝑚𝑒
12 𝑒𝑣𝑒𝑛𝑡 +
←←←←←←←← (𝑐𝑜𝑙𝑢𝑚𝑛𝑁𝑎𝑚𝑒, 𝑣𝑎𝑙𝑢𝑒)
13 end
14 end
15 end
16 e u n newLog
3.2. Lea ning om he log
The log may con ain e en s ela ed o di e en ins ances o a
business p ocess. Decision poin s appea in he p ocess model whe e
hey a e spli in o al e na i e b anches. To iden i y hem and lea n
which pieces o in o ma ion ule he decisions aken in such ins ances,
he cu en amewo k is inspi ed by Rozina e al. [21]. Fo his, he
amewo k i s con e s he UI Log o an a i ac ha ML algo i hms
can use, i.e., a labeled da ase (c . Sec ion 3.2.1). The ea e , his
da ase is used o ain a classi ica ion algo i hm ha can explain such
a beha io (c . Sec ion 3.2.2).
3.2.1. C ea ing he labeled da ase
The labeled da ase should con ain one ow o each obse a ion
ha leads o a decision ela ed o a decision poin . In his line, an
obse a ion is he se o e en s ha p ecedes he decision poin . In
addi ion, since he aim is o ain a classi ica ion algo i hm, he decision
i sel should be pa o he da ase , he so-called ’label’.
Acco ding o hese equi emen s, Algo i hm 4depic s how o ans-
o m a UI Log in o a da ase o a gi en decision poin . The ein, his
algo i hm uns o e each case in he log (c . lines 1 o 3 o Algo i hm
4) and la ens in o a single ow all he case e en s ha a e be o e
he gi en decision poin (c . lines 4 o 9 o Algo i hm 4). Each e en ’s
a ibu e is included as a new column o he same ow o do his
la ening. The column’s name is composed using he ac i i y’s and
a ibu e’s name (c . line 7 o Algo i hm 4) o a oid collisions o he
In o ma ion Sys ems 121 (2024) 102340
7
A. Ma ínez-Rojas e al.
Fig. 8. Ex ended UI Log om Fig. 2 including ea u e ‘‘#Bu ons’’ ex ac ed o he sc eensho o Fig. 6.
Fig. 9. Label da ase c ea ion applying Algo i hm 4 o UI Log p esen ed in Fig. 8.
same a ibu es o di e en ac i i ies. Besides he la ening, he label
column is also included since i s a es which decision o he decision
poin has been aken by his case in he UI Log (c . line 10 o Algo i hm
4).
Algo i hm 4: Gene a ion o a da ase o a decision poin
Inpu : UILog 𝑙𝑜𝑔, DecicionPoin 𝑑𝑒𝑐𝑖𝑠𝑖𝑜𝑛
Ou pu : Da ase 𝑑𝑎𝑡𝑎𝑠𝑒𝑡
1𝑑𝑎𝑡𝑎𝑠𝑒𝑡 ←∅
2𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒𝑠 ←𝑠𝑒𝑝𝑎𝑟𝑎𝑡𝑒𝐿𝑜𝑔𝐵𝑦𝐶𝑎𝑠𝑒𝑠(𝑙𝑜𝑔)
3 o 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒 in 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒𝑠 do
4𝑟𝑜𝑤 ←∅
5 o 𝑒𝑣𝑒𝑛𝑡 in 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒.𝑒𝑣𝑒𝑛𝑡𝑠𝐵𝑒𝑓 𝑜𝑟𝑒(𝑑𝑒𝑐𝑖𝑠𝑖𝑜𝑛)do
6 o 𝑎𝑡𝑡𝑟 in 𝑒𝑣𝑒𝑛𝑡.𝑎𝑡𝑡𝑟𝑖𝑏𝑢𝑡𝑒𝑠() do
7𝑟𝑜𝑤 +
←←←←←←←← (column=𝑒𝑣𝑒𝑛𝑡.𝑎𝑐𝑡𝑖𝑣𝑖𝑡𝑦𝐼𝐷 +′_′+𝑎𝑡𝑡𝑟.𝑛𝑎𝑚𝑒 ,
alue=𝑎𝑡𝑡𝑟.𝑣𝑎𝑙𝑢𝑒
8end
9end
10 𝑟𝑜𝑤 +
←←←←←←←← (column=’LABEL’, alue=𝑑𝑒𝑐𝑖𝑠𝑖𝑜𝑛.𝑏𝑟𝑎𝑛𝑐ℎ(𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒)⟩
11 𝑑𝑎𝑡𝑎𝑠𝑒𝑡 +
←←←←←←←← 𝑟𝑜𝑤
12 end
13 e u n da ase
Gene ally, a se o da ase s will be gene a ed o a UI Log. Mo e
p ecisely, one da ase o each decision poin ha appea s in he
disco e ed p ocess model needs o be gene a ed.
Example 4. Fo example, le us conside he UI Log o Fig. 8. The e is a
decision poin jus a e A-B wi h wo b anches: b anch 1, which leads
o ac i i y C, and b anch 2, which leads o ac i i y F. The esul ing
da ase ob ained a e applying Algo i hm 4will ha e wo ows whe e
he i s one has 1 as he label, and he second has 2 as he label (c .
Fig. 9).
3.2.2. Gene a ing a decision ee
In he li e a u e, many classi ica ion algo i hms exis ha can gen-
e a e models ha y o co ec ly classi y he ows o a da ase , i.e., o
ma ch each ow o i s label using he es o he columns o he da ase .
This p ocess is called model aining. The amewo k desc ibed in his
pape uses CHAID [43], a ee-based classi ica ion algo i hm. We use
a ee-based algo i hm since, unlike o he classi ica ion algo i hms,
he gene a ed models can be g aphically depic ed as a ee and a e
easie o unde s and by a human. In addi ion, he CHAID algo i hm
has demons a ed o ha e be e pe o mance in his con ex han o he
popula algo i hms [24].
I he in o ma ion con ained in he da ase is su icien , he ained
model should demons a e high aining accu acy, indica ing ha he
esul ing ee can e ec i ely explain all labels (i.e., he decisions made)
using he column alues (i.e., he ea u es). This ee will ha e one
node pe column, each ep esen ing a c ucial ea u e o explaining he
labels. The edges going ou o each node delinea e he possible alues
o ha column. The lea es o he ee ep esen he labels, and he pa h
om he oo o each lea de ines he conjunc ion o ules necessa y o
each ha speci ic label.
Howe e , achie ing aining accu acy close o 100% may aise
conce ns abou o e i ing. O e i ing occu s when a model excessi ely
ailo s i sel o he aining da a, cap u ing noise and ou lie s. To
add ess his, we conduc a ho ough analysis o addi ional measu es,
such as accu acy, ecall, o F1 sco e, as pa o ou e alua ion p ocess.
Mo eo e , i is impo an o cla i y ha , as occu s when using ML
me hods, achie ing a high accu acy wi h he CHAID algo i hm does no
inhe en ly imply he es ablishmen o causali y be ween he ea u es
and he decision ou come; a he , i signi ies co ela ion. I is essen ial
o acknowledge ha he ela ionships iden i ied be ween he columns
and he label could be due by chance (i.e., non- ele an ea u es ha
gi e an explana ion o he decision casually), ins ead o being he ac ual
chain o e en s uling he decision (i.e., he ea u es ha a e uly
ele an ).
While acknowledging he inhe en limi a ions o he CHAID algo-
i hm, we wan o unde sco e he aluable insigh s ob ained h ough
he ep esen a ion o he disco e ed ee. The esul ing ee-based
model se es as a ool o e ealing ela ionships be ween ea u es and
decision ou comes. I is c ucial o no e, howe e , ha he p esen ed
co ela ions do no au oma ically imply causa ion.
Mo eo e , ou commi men o in e p e abili y is g ounded in he
belie ha models should be anspa en and comp ehensible o bo h
analys s and s akeholde s. By p esen ing a clea and in e p e able
decision model, we aim o enhance unde s anding o decision making
du ing he p ocess examined. This, in u n, suppo s he de elopmen
o he au oma ion sc ip ela ed o his p ocess.
3.3. T ace back decisions
Al hough he in o ma ion gi en by he classi ica ion ee is unde -
s andable by a human, he nodes (i.e., he name o he columns o
he da ase ) may be di icul o associa e di ec ly wi h elemen s o
he unde lying p ocess. To amend his, his s ep aims o map he ee
o a se o sc eensho s whe e he conside ed ea u es a e highligh ed
as desc ibed in Algo i hm 5. Mo e speci ically, o each node wi h
i s co esponding ou going edges (c . line 2 o Algo i hm 5), i i s
looks o he ela ed e en and ga he s i s sc eensho (lines 3 o 5 o
In o ma ion Sys ems 121 (2024) 102340
8
A. Ma ínez-Rojas e al.
Fig. 10. Example decision ee whe e he spli ing ules (a) a e linked o ea u es in he sc eensho (b).
Algo i hm 5) Then, i looks o he ela ed ELoC – in case o single ype-
o ELoCs – in case o en i e ype – in he UI Log (lines 6 o 8 and 13 o
Algo i hm 5). Finally, i highligh s he bounding box o each ELoC in
he sc eensho (c . lines 10 and 14 o Algo i hm 5).
Algo i hm 5: T acing back ee ules o sc eensho s
Inpu : UILog 𝑙𝑜𝑔, T eeNode 𝑛𝑜𝑑𝑒
Ou pu : Lis ⟨𝐼𝑚𝑎𝑔𝑒⟩𝑖𝑚𝑔𝑠
1imgs ←∅
2 o 𝑒𝑑𝑔𝑒 in node.ou goingEdges do
3(ac i i yID, elocId, ea u e) ←decompose(node.name)
4e en ←𝑙𝑜𝑔.𝑓𝑖𝑙𝑡𝑒𝑟(𝑒→𝑒.𝑎𝑐𝑡𝑖𝑣𝑖𝑡𝑦𝐼𝐷 ==
𝑎𝑐𝑡𝑖𝑣𝑖𝑡𝑦𝐼𝐷 && 𝑒.𝑎𝑡𝑡𝑟𝑖𝑏𝑢𝑡𝑒(𝑓 𝑒𝑎𝑡𝑢𝑟𝑒).𝑚𝑎𝑡𝑐ℎ(𝑒𝑑𝑔𝑒.𝑛𝑎𝑚𝑒))[0]
5sc een ←𝑐𝑜𝑝𝑦(𝑒𝑣𝑒𝑛𝑡.𝑠𝑐𝑟𝑒𝑒𝑛𝑠ℎ𝑜𝑡)
6i elocId == null hen
7ui e ←𝑈𝐼𝐹 𝐸𝐶𝑎𝑡𝑎𝑙𝑜𝑔.𝑔𝑒𝑡(𝑓𝑒𝑎𝑡𝑢𝑟𝑒)
8elocs ←𝑒𝑣𝑒𝑛𝑡.𝑒𝑙𝑜𝑐𝑠.𝑓𝑖𝑙𝑡𝑒𝑟(𝑢𝑖𝑓𝑒.𝑃 𝑟𝑒𝑑)
9 o eloc in elocs do
10 sc een.highligh (eloc.Pos)
11 end
12 else
13 eloc ←𝑒𝑣𝑒𝑛𝑡𝑠.𝑒𝑙𝑜𝑐𝑠.𝑓𝑖𝑙𝑡𝑒𝑟(𝑢→𝑢.𝐼𝑑 == 𝑒𝑙𝑜𝑐𝐼𝑑)
14 sc een.highligh (eloc.Pos)
15 end
16 imgs +
←←←←←←←← 𝑠𝑐𝑟𝑒𝑒𝑛
17 end
18 e u n imgs
Example 5. Fig. 10 shows he classi ica ion ee (a) esul ing om
applying he CHAID algo i hm o a da ase simila o he one o Fig. 9,
and he ela ed sc eensho (b) whe e he wo ee nodes a e linked o
he conc e e UI elemen s. In his example, bo h UI elemen s belong o
he same sc eensho .
4. E alua ion
In his sec ion, we empi ically e alua e he p oposed amewo k,
aiming o answe he p ima y esea ch ques ion (PRQ) posed a he
beginning o his pape : Can he sc een-based Task Mining app oach
e ec i ely ex ac and ep esen p ocess beha io obse ed in a UI Log wi hin
eal-li e se ings? To conduc a mo e de ailed analysis, h ee speci ic
sub-ques ions (SQ) a e o mula ed.
Fi s , SQ1 assesses he amewo k’s capabili y o iden i y ea u es
ha in luence decisions du ing p ocess execu ion wi hin he con ex o
he obse ed UI Log.
•SQ1: Does he p oposed amewo k iden i y ea u es ha cap u e
he ele an elemen s ha humans ake in o accoun when making
decisions?
Building on SQ1, SQ2 in es iga es he amewo k’s abili y o ex ac
condi ions based on he iden i ied ea u es. The e alua ion speci ically
assesses he choice o b anches ollowing he decision poin s in he
p ocess.
•SQ2: Does he p oposed amewo k co ec ly disco e he condi ions
ha ep esen he human decision-making?
To assess he p ac ical applicabili y o he p oposed amewo k, SQ3
add esses i s pe o mance om a ime pe spec i e.
•SQ3: Does he p oposed amewo k disco e human decision-
making in a ime-e icien manne ?
The scope o his e alua ion ocuses on he las pa o he amewo k
(i.e., s eps 4 and 5), since he i s pa (i.e., s eps 1, 2, and 3) has
al eady been e alua ed in p e ious wo ks [13,24].
To ensu e igo , his e alua ion ollows he case s udy p o ocol p o-
posed by B e e on e al. [44]. Sec ion 4.1 desc ibes he cha ac e is ics
o he p ocess selec ed o he s udy. Sec ion 4.2 de ines he speci ic
p oblem si ua ion chosen o he e alua ion. Sec ion 4.3 explains he
e alua ion design, while Sec ion 4.4 de ails he execu ion en i onmen .
Sec ion 4.5 encloses he da a collec ion, Sec ion 4.6 p esen s he esul s
ob ained and, inally, Sec ion 4.7 discuss he h ea s o he alidi y.
4.1. Case selec ion
Fo his case s udy, we selec ed a eal-wo ld p ocess iden i ied by
a business expe as a ypical ope a ion o au oma e wi hin he Busi-
ness P ocess Ou sou cing con ex . The chosen p ocess e ol es a ound
managing unsubsc ibe eques s om clien s and closely mi o s he ad-
minis a i e wo k low o a Spanish elecommunica ions company. This
pa icula p ocess en ails handling a olume o 300 cases pe mon h,
wi h a cus ome base o app oxima ely 1.5 million cus ome s. Fo
p i acy conside a ions, we ha e eplaced speci ic da a wi h syn he ic
in o ma ion.
This p ocess in ques ion (c . Fig. 11) is composed o 10 ac i i ies,
1 single decision poin , and 4 a ian s. I implies dealing wi h h ee
di e en eal-li e sys ems (c . Fig. 12): an email clien (i.e., Gmail), an
issue acking sys em (i.e., Zendesk) and an ERP sys em (i.e., Odoo). In
addi ion, he p ocess includes: (1) a UI Log wi h a case o each p ocess
a ian , and (2) eal-li e sc eensho s o each o i s ac i i y e en s. No e
ha he company consen ed o coope a e o he sake o his e alua ion,
In o ma ion Sys ems 121 (2024) 102340
9
A. Ma ínez-Rojas e al.
Fig. 11. Handling o unsubsc ip ion eques s p ocess.
Fig. 12. Example o a eal-li e sc eensho (a) and i s mockup e sion (b).
bu did no au ho ize us o publish he eal in o ma ion abou i s
co po a e sys ems. The e o e, adap ed and anonymized e sions o he
UI Log and hei sc eensho s a e used.
We conside his o be a p ope case o a c i ical e alua ion since i
mee s he ollowing selec ion c i e ia: (1) his p ocess highly esembles
a eal p ocess wi hin a company ha we coope a e wi h in Spain, and
(2) he disco e y o he decision is based on in o ma ion ha appea s
on he sc eensho s, i.e., wo UI elemen s: an a achmen in an email,
and a checkbox on a web o m.
4.2. Objec s
The e alua ion is based on a se o syn he ic p oblems closely
mimicking he elecommunica ion company’s eal use case. Fo con-
iden iali y easons, we canno disclose he company’s name; we we e
allowed, hough, o analyze hei p ocess in eal-li e and collec e en
logs, including sc eensho s, as seeds o ou e alua ion app oach. We
ha e de eloped wo ypes o p oblems (𝑃) o ou e alua ion. The i s
ype in ol es c ea ing scena ios using mockups o he p ocess, while he
In o ma ion Sys ems 121 (2024) 102340
16
A. Ma ínez-Rojas e al.
[8] M. Laci y, L.P. Willcocks, Inno a ing in se ice: The ole and managemen
o au oma ion, in: Dynamic Inno a ion in Ou sou cing, Sp inge , 2018, pp.
269–325.
[9] C. F ank, In oduc ion To Robo ic P ocess Au oma ion, Ins . Robo ic P ocess
Au om. (2015) 35.
[10] L. Willcocks, M. Laci y, A New App oach o Au oma ing Se ices, MIT Sloan
Manag. Re . 58 (1) (2016) 40–49.
[11] A. Jiménez-Ramí ez, J. Chacón-Mon e o, T. Wojdynsky, J. Gonzalez En iquez,
Au oma ed es ing in obo ic p ocess au oma ion p ojec s, J. So w.: E ol. P ocess
(2020) e2259.
[12] J.G. En íquez, A. Jiménez-Ramí ez, F. Domínguez-Mayo, J. Ga cía-Ga cía,
Robo ic p ocess au oma ion: a scien i ic and indus ial sys ema ic mapping s udy,
IEEE Access 8 (2020) 39113–39129.
[13] A. Jimenez-Rami ez, H.A. Reije s, I. Ba ba, C. Del Valle, A me hod o imp o e
he ea ly s ages o he obo ic p ocess au oma ion li ecycle, in: CAiSE, 2019, pp.
446–461.
[14] W.M.P. an de Aals , P ocess Mining: Da a Science in Ac ion, Sp inge ,
Heidelbe g, 2016.
[15] L. Reinkemeye , P ocess Mining in Ac ion. P inciples, Use Cases and Ou look,
Sp inge , 2020.
[16] V. Leno, A. Poly yanyy, M. Dumas, M. La Rosa, F.M. Maggi, Robo ic p ocess
mining: ision and challenges, Bus. In . Sys . Eng. (2020) 1–14.
[17] S. Agos inelli, M. Lupia, A. Ma ella, M. Mecella, Reac i e syn hesis o so wa e
obo s in RPA om use in e ace logs, Compu . Ind. 142 (2022) 103721.
[18] V. Leno, A. Augus o, M. Dumas, M. La Rosa, F.M. Maggi, A. Poly yanyy,
Iden i ying candida e ou ines o obo ic p ocess au oma ion om unsegmen ed
UI logs, in: ICPM, 2020, pp. 153–160.
[19] M. Völke , S. Siege , M. Weske, Adding decision managemen o obo ic
p ocess au oma ion, in: Business P ocess Managemen : Blockchain and Robo ic
P ocess Au oma ion Fo um: BPM 2021 Blockchain and RPA Fo um, Rome, I aly,
Sep embe 6–10, 2021, P oceedings 19, Sp inge , 2021, pp. 23–37.
[20] V. Leno, A. A mas-Ce an es, M. Dumas, M. La Rosa, F.M. Maggi, Disco e ing
p ocess maps om e en s eams, in: ICSSP, 2018, pp. 86–95.
[21] A. Rozina , W.M.P. an de Aals , Decision mining in p om, in: S. Dus da ,
J.L. Fiadei o, A.P. She h (Eds.), BPM, Sp inge , Be lin, Heidelbe g, 2006, pp.
420–425.
[22] J. Gao, S.J. an Zels , X. Lu, W.M. an de Aals , Au oma ed obo ic p ocess
au oma ion: A sel -lea ning app oach, in: OTM, 2019, pp. 95–112.
[23] J.M. López-Ca nice , C. del Valle, J.G. En íquez, Towa ds an OpenSou ce logge
o he analysis o RPA p ojec s, in: BPM, 2020, pp. 176–184.
[24] A. Ma ínez-Rojas, A. Jiménez-Ramí ez, J.G. En íquez, H.A. Reije s, Analyz-
ing a iable human ac ions o obo ic p ocess au oma ion, in: In e na ional
Con e ence on Business P ocess Managemen , Sp inge , 2022, pp. 75–90.
[25] S. Agos inelli, M. Lupia, A. Ma ella, M. Mecella, Sma RPA: A ool o eac i ely
syn hesize so wa e obo s om use in e ace logs, in: CAiSE, 2021, pp.
137–145.
[26] V. Leno, S. De ia ykh, A. Poly yanyy, M. La Rosa, M. Dumas, F.M. Maggi,
Robidium: Au oma ed syn hesis o obo ic p ocess au oma ion sc ip s om
UI logs, in: In e na ional Con e ence on Business P ocess Managemen , CEUR
Wo kshop P oceedings, 2020.
[27] A. Ma ínez-Rojas, A. Jiménez-Ramí ez, J.G. En íquez, H.A. Reije s, A ool-
suppo ed me hod o gene a e use in e ace logs, in: Hawaii In e na ional
Con e ence on Sys em Sciences, 2023.
[28] S. Agos inelli, M. Lupia, A. Ma ella, M. Mecella, Au oma ed gene a ion o
execu able RPA sc ip s om use in e ace logs, in: BPM, 2020, pp. 116–131.
[29] A. Augus o, R. Con o i, M. Dumas, M. La Rosa, F.M. Maggi, A. Ma ella, M.
Mecella, A. Soo, Au oma ed disco e y o p ocess models om e en logs: Re iew
and benchma k, IEEE T ans. Knowl. Da a Eng. 31 (4) (2018) 686–705.
[30] V. Leno, A. Poly yanyy, M. La Rosa, M. Dumas, F.M. Maggi, Ac ion logge :
Enabling p ocess mining o obo ic p ocess au oma ion, in: P oceedings o he
Disse a ion Awa d, Doc o al Conso ium, and Demons a ion T ack A BPM 2019
Co-Loca ed wi h 17 h In e na ional Con e ence on Business P ocess Managemen
(BPM 2019) Vienna, Aus ia, Sep embe 1-6, 2019, Vol. 2420, CEUR-WS, 2019,
pp. 124–128.
[31] J. Geye -Klingebe g, J. Nakladal, F. Baldau , F. Vei , P ocess mining and obo ic
p ocess au oma ion: A pe ec ma ch., in: BPM, 2018, pp. 124–131.
[32] E. Bazheno a, S. Bülow, M. Weske, Disco e ing decision models om e en logs,
in: BIS, 2016, pp. 237–251.
[33] H.A. Sleiman, R. Co chuelo, Tex: An e icien and e ec i e unsupe ised web
in o ma ion ex ac o , Knowl.-Based Sys . 39 (2013) 109–123.
[34] P. Jiménez, J.C. Roldán, R. Co chuelo, A clus e ing app oach o ex ac da a
om HTML ables, In . P ocess. Manage. 58 (6) (2021) 102683.
[35] C. Ike ionwu, D. Edga , Secu ed se ice deli e y model o ou sou ced se ices
in a business p ocess ou sou cing ela ionship, In . J. In . Commun. Sci. 4 (1)
(2019) 7–17.
[36] Z. Xu, X. Baojie, W. Guoxin, Canny edge de ec ion based on open CV, in: 2017
13 h ICEMI, 2017, pp. 53–56.
[37] K. Mo an, C. Be nal-Cá denas, M. Cu cio, R. Bone , D. Poshy anyk, Machine
lea ning-based p o o yping o g aphical use in e aces o mobile apps, IEEE
T ans. So w. Eng. 46 (2) (2018) 196–221.
[38] B.P. Majumde , N. Po i, S. Ta a, J.B. Wend , Q. Zhao, M. Najo k, Rep esen a ion
lea ning o in o ma ion ex ac ion om o m-like documen s, in: ACL, 2020, pp.
6495–6504.
[39] Ke as OCR, 2022, A ailable a h ps://gi hub.com/ aus omo ales/ke as-oc . (Las
Accessed Ma ch 2022).
[40] T. Mi chell, Machine Lea ning, McG aw Hill, 1997.
[41] L.H. Gilpin, D. Bau, B.Z. Yuan, A. Bajwa, M. Spec e , L. Kagal, Explaining
explana ions: An o e iew o in e p e abili y o machine lea ning, in: 2018 IEEE
5 h DSAA, 2018, pp. 80–89.
[42] J. Chen, M. Xie, Z. Xing, C. Chen, X. Xu, L. Zhu, G. Li, Objec de ec ion o
g aphical use in e ace: Old ashioned o deep lea ning o a combina ion? in:
P oceedings o he 28 h ACM Join Mee ing on Eu opean So wa e Enginee ing
Con e ence and Symposium on he Founda ions o So wa e Enginee ing, 2020,
pp. 1202–1214.
[43] G.V. Kass, An explo a o y echnique o in es iga ing la ge quan i ies o
ca ego ical da a, J. R. S a . Soc. Se . C. Appl. S a . 29 (2) (1980) 119–127.
[44] P. B e e on, B. Ki chenham, D. Budgen, Z. Li, Using a p o ocol empla e o case
s udy planning, in: 12 h In e na ional Con e ence on E alua ion and Assessmen
in So wa e Enginee ing, (EASE) 12, 2008, pp. 1–8.
[45] V. Leno, M. Dumas, M. La Rosa, F.M. Maggi, A. Poly yanyy, Au oma ed disco e y
o da a ans o ma ions o obo ic p ocess au oma ion, 2020, a Xi p ep in
a Xi :2001.01007.