A screenshot-based task mining framework for disclosing the drivers behind variable human actions

A screenshot-based task mining framework for disclosing the drivers behind variable human actions A. Martínez-Rojas a,∗, A. Jiménez-Ramírez a, J.G. Enríquez a, H.A. Reijers b aDepartment of Computer Languages and Systems, University of Seville, Avenida Reina Mercedes, s/n, 41012, Seville, Spain bDepartment of Information and Computing Sciences, Utrecht University, Princetonplein 5, 3584 CC, Utrecht, The Netherlands A R T I C L E I N F O Keywords: Robotic Process Automation UI Log User behavior mining Task mining Decision model discovery A B S T R A C T Robotic Process Automation (RPA) enables subject matter experts to use the graphical user interface as a means to automate and integrate systems. This is a fast method to automate repetitive, mundane tasks. To avoid constructing a software robot from scratch, Task Mining approaches can be used to monitor human behavior through a series of timestamped events, such as mouse clicks and keystrokes. From a so-called User Interface log (UI Log), it is possible to automatically discover the process model behind this behavior. However, when the discovered process model shows different process variants, it is hard to determine what drives a human’s decision to execute one variant over the other. Existing approaches do analyze the UI Log in search for the underlying rules, but neglect what can be seen on the screen. As a result, a major part of the human decision-making remains hidden. To address this gap, this paper describes a Task Mining framework that uses the screenshot of each event in the UI Log as an additional source of information. From such an enriched UI Log, by using image-processing techniques and Machine Learning algorithms, a decision tree is created, which offers a more complete explanation of the human decision-making process. The presented framework can express the decision tree graphically, explicitly identifying which elements in the screenshots are relevant to make the decision. The framework has been evaluated through a case study that involves a process with real-life screenshots. The results indicate a satisfactorily high accuracy of the overall approach, even if a small UI Log is used. The evaluation also identifies challenges for applying the framework in a real-life setting when a high density of interface elements is present. 1. Introduction In this era of Hyperautomation [1], organizations are pushed to rapidly identify and automate all business processes where possible.1 This drive has led to the widespread adoption of low-code and nocode tools, alongside the increasing popularity of Robotic Process Automation (RPA). Unlike traditional automation (e.g., API-based), RPA enables the use of graphical user interfaces as a means to automate and integrate systems; by doing so, a software robot that is created in this way can closely mimic how humans interact with computer applications [2]. Several authors have reported the benefits that RPA brings. Since this technology is relatively easy to use by subject matter experts and does not necessarily require IT skills to deploy it, RPA helps to save costs, increases agility, improves quality [3–6], as well as customer satisfaction [7,8], while its level of intrusiveness is low [9,10]. ∗Corresponding author. As the prevalence of RPA continues to grow, various techniques have emerged to support and augment software automation throughout its life cycle phases, encompassing analysis, design, development, deployment, testing, and monitoring [11]. Typically, automation initiatives start with the analysis phase, involving the examination of existing processes to identify optimal candidates for automation [12]. However, traditional approaches, such as interviews with subject-matter experts or analysis of process documentation, come with inherent risks, including reliance on incomplete or outdated process descriptions, biased information, and oversight of exceptional situations that may be deemed inconsequential to humans but hold significance for an automation project [13]. To mitigate these risks and avoid constructing automation from scratch, innovative approaches like Task Mining [14, 15] and Robotic Process Mining [16] have emerged to learn from the observation of users while they are performing the processes. These techniques record user behavior into a User Interface log (UI Log, i.e., a E-mail addresses: [email protected] (A. Martínez-Rojas), [email protected] (A. Jiménez-Ramírez), [email protected] (J.G. Enríquez), [email protected] (H.A. Reijers). 1 Gartner’s definition of hyperautomation: https://www.gartner.com/en/information-technology/glossary/hyperautomation https://doi.org/10.1016/j.is.2023.102340 Received 22 January 2023; Received in revised form 14 November 2023; Accepted 18 December 2023 Available online 21 December 2023 0306-4379/© 2023 The Author(s). Information Systems 121 (2024) 102340 2 A. Martínez-Rojas et al. series of timestamped events like mouse clicks and keystrokes) and then analyze them to enable the creation of the automation solution by discovering the process model behind the log. The field of process discovery for UI Logs has been explored in considerable depth [13,17,18]. However, when the discovered model shows different process variants, there is little support to determine what drives the execution of one variant over the other. Therefore, the variations observed in the recorded UI Log suggest that the human operator makes different decisions for different cases. Despite this, identifying the specific rules governing these decisions is generally not feasible. Nevertheless, uncovering these rules is necessary to model decisions, which is essential to effectively implement RPA software [19]. Acknowledging such significance, several researchers have undertaken this challenge [17,20–22]. Their approaches are based on the information that is in the UI Log. However, a limitation arises when on-screen elements solely influence decisions, as these leave no tangible trail within the log. Consequently, certain aspects of human decisionmaking remain hidden, creating a potential obstacle to automation. To address this problem, the current paper describes a framework that considers screenshots (i.e., screen captures that are taken for every event in the UI Log) as an additional source of knowledge. These screenshots are analyzed to extract structured information that can extend the data recorded for each event in the log. The UI Log extended with this new information, can then be processed to train a Machine Learning algorithm, which tries to find the relations between this information and the decisions made by the user. The framework combines these rules with the related screenshots to graphically express the factors that influence human decision-making. The framework that is presented in this paper builds on, integrates, and extends our previous work, which enables us to (1) monitor user behavior that generates a UI Log that stores one screenshot for each event [23], and (2) analyze these UI Logs and the screenshots to discover the underlying process model [13]. Specifically, this paper extends our proposal in [24]. In that paper, promising results were obtained when dealing with simple screenshots (i.e., synthetic mockups) and rather simple conditions. In turn, the current work greatly enhances a previous work by addressing a fundamental Primary Research Question (PRQ) not considered so far: Can the screen-based Task Mining approach extract and represent the process behavior observed in a UI Log within real-life settings? Intending to answer PRQ, this paper significantly extends and improves our previous work [24] by: 1. Formalizing and classifying the feature extractors that can be applied to extract information from screenshots. 2. Proposing dataset generation methods for each class of feature extractor. 3. Extending the framework by including a new step to explain the decision graphically. This new step enables the connection of the decision rules to the real UI elements on the screenshots. 4. Evaluating the proposal using a case study that involves a process with real-life screenshots, and analyzes the performance obtained for each class of feature extractor. The developed framework is of interest to researchers and practitioners in the RPA field since it explains and demonstrates how to discover automatable decision rules that would otherwise remain hidden. Furthermore, results show that the framework is applicable even when only relatively small UI Logs are available. The rest of the paper is organized as follows. Section 2provides the background by describing the topics of behavior monitoring, ML, and image processing. Section 3describes the proposed framework and explains how it is able to explain decisions from a UI Log graphically. Section 4describes the setup and the execution of the empirical evaluation, which was performed to compare the different feature extractors. Section 5reviews similar approaches in the literature. Finally, Section 6 summarizes our work and describes future research lines. 2. Background The approach presented in this paper builds on the topics of Process Discovery in RPA, Graphical User Interface (GUI) analysis, and Machine Learning (ML) for decision discovery. Process discovery (cf. Section 2.1) enables the systematic analysis of business processes, optimizing workflows for RPA integration. GUI analysis (cf. Section 2.2) empowers RPA bots to efficiently interact with diverse software interfaces through image recognition and object detection. Finally, ML (cf. Section 2.3) elucidates the human decision-making process, enabling a clear understanding of the rules and factors guiding actions. 2.1. Process discovery in RPA In the RPA field, different alternatives exist to monitor users by obtaining event logs from the interaction between the user and the front-end of the information systems (the so-called UI Logs) [13,25,26]. The most representative one for this work is the UI Log concept presented by Jiménez-Ramírez et al. and Martínez-Rojas et al. [13,27], which, unlike the others, provides special attention to the screenshots as an information source. The UI Logs are sets of timestamped events captured at the operating system (OS) level, independent of any specific programs. They encompass essential information about each event, including (1) the associated process (e.g., case or activity), (2) the application being used (e.g., app name or screenshot), and (3) the specific user action performed (e.g., mouse click or keystroke). These events are triggered based on a range of OS events, such as clicks, keystrokes, or specific combinations of them. Task Mining or Robotic Process Mining techniques can be applied to record these UI Logs and discover the operator process model out of them [13,13,16,20,28–32]. Moreover, these proposals include functionalities to clean the UI Log of irrelevant information so that noise in the resulting process model is filtered. In addition, existing works [17,18] show how to select variants/cases/activities according to the frequency, length, and other criteria that are useful to identify process candidates to robotize. The process model resulting from these approaches may contain decision points and separate branches for different process variants. To illustrate how a process is discovered based on user monitoring, Fig. 1 depicts a subset of the elementary user interfaces for registering a customer in the context of a telecommunication company. In this registration process, the human operator checks her email inbox and reviews the pending emails regarding the registration tasks. After opening the email, the operator has to validate that all provided information related to the new customer is correct. More precisely, the customer ID card is expected to be included as an attachment. If it is indeed included (cf. Fig. 1a), all the customer data has to be registered into a CRM system (cf. Fig. 1b). Otherwise, in case the ID card is missing (cf. Fig. 1c), an email has to be sent to the customer requesting such data (cf. Fig. 1d). Regardless of which of these two situations occurred (i.e., Variant 1 or Variant 2 of Fig. 1), the operator returns to their inbox to process the next mail. This process must be repeated several times during the day to process the entire queue of emails in the operator’s inbox. Fig. 2 shows an excerpt of a possible UI Log resulting from monitoring the operator while executing the process of Fig. 1 (e.g., with a key logger [23]). Fig. 3 shows a possible process model discovered from the UI Log of Fig. 2. This kind of representation supports the understanding of the UI Log. In this example, it may be deduced that the UI Log correctly depicts that it contains a single decision point after activity ‘‘B’’ (i.e., after seeing the email), where the process branches off into two different variants, i.e., Id.1 and Id.2. Nonetheless, the model does not disclose the actual condition that rules such a decision. What is more, it may potentially depend on the information that appears on the screen. Information Systems 121 (2024) 102340 3 A. Martínez-Rojas et al. Fig. 1. Mockups of motivating example. Fig. 2. Excerpt of a UI Log obtained from a keylogger. Fig. 3. Process model discovered from the UI Log of Fig. 2. To uncover this information, researchers analyze user interactions [17]. Traditional techniques to extract this information from the screen rely on operating system APIs or accessing the Document Object Model (DOM), which enables systematic querying of UI elements [33,34]. Despite these possibilities, some processes, especially in the field of outsourcing, involve virtualized connections (e.g., using Citrix or Team Viewer) for enhanced security [35]. This restricts direct access to the Windows API or the DOM tree. Consequently, in such scenarios, leveraging UI Logs and screenshots becomes the preferred method for gathering information and effectively automating processes. 2.2. GUI analysis To analyze the screenshots, the field of GUI analysis allows identifying the UI elements existing within an image [36–38]. Besides just locating the element on the screen (i.e., calculating the bounding boxes), they can classify them by the type of element, e.g., image, button, or text. Optical Character Recognition (OCR) techniques are often more appropriate when dealing with texts. For instance, KerasOCR [39] allows extracting words and their bounding boxes from screenshots. Information Systems 121 (2024) 102340 4 A. Martínez-Rojas et al. Fig. 4. Example of GUI analysis applied to a screenshot for detecting buttons. Fig. 5. Framework for variability analysis through interpretable decisions from UI Logs. Fig. 4 shows a real UI screenshot related to activity ‘‘B’’ of Fig. 3. What can be seen here is how the approach from [37] has been tailored to desktop screenshots to analyze them and highlight the UI elements of a concrete type, i.e., buttons. 2.3. ML for decision discovery The ML domain proposes supervised learning algorithms that can learn from a dataset [40] to explain the rules behind a decision. Datasets are commonly represented in tabular form where each column is an input variable, and one of them is the label, i.e., what needs to be classified. Each row is a member or observation of the dataset. In a classification problem, the algorithm tries to find patterns in the observation’s input variables that help explain the labels. Decision trees are an example of classification algorithms that, besides just providing a classification, do so in a human-interpretable way [41]. 3. Framework This section describes the proposed framework for explaining the decisions from the data in a UI Log using screenshots (cf. Fig. 5). It is based on existing approaches [13,25,30] that (1) deal with monitoring graphical user interfaces to generate UI Logs (cf. step 1 in Fig. 5), and (2) discover process models from such logs (cf. step 3 in Fig. 5). As stated before, discovered process models may contain decision points whose understanding could be cumbersome. To amend this, the proposed framework aims to find an explanation for each decision point, i.e., the rules that determine which branch to follow. Such explanations may consider information that is graphically depicted in the user interfaces, as captured in step 1. More precisely, the framework includes a step to extract information from user interfaces and incorporate it into the UI Log (cf. step 2 in Fig. 5, Section 3.1). Some approaches for process model discovery, such as [13], are benefited from this extended information, and this is why the feature extraction step is established before the process discovery step in the framework. Once the process model is discovered, the extended UI Log from step 2 is used as input to learn the decision tree (i.e., rules) behind each decision point (cf. step 4 in Fig. 5, Section 3.2). Finally, these decision trees are processed to trace back their rules to the graphical information in the screenshots (cf. step 5 in Fig. 5, Section 3.3). This helps to make sense of the decision trees. 3.1. Extracting features from graphical user interfaces In general, a graphical user interface contains a number of UI elements that are represented graphically on the screen. To extract information from a user interface, the framework first identifies and classifies the UI elements on each screenshot (cf. Section 3.1.1), and then extracts features from them (cf. Section 3.1.2) 3.1.1. Identification and classification of UI elements The proposed framework uses the approach proposed by Chen et al. [42] to identify each UI element on the screen, i.e., to obtain the bounding boxes of all the UI elements that are found. In addition, the solution presented by Moran et al. [37] is then used to classify each identified element, i.e., to obtain the type of each UI element, e.g., button, checkbox, etc. Information Systems 121 (2024) 102340 5 A. Martínez-Rojas et al. Fig. 6. Example of UI Element Location and Classification (ELoC) data for one of the buttons on the screen. Therefore, this step generates and stores metadata related to each screenshot. Ultimately, the metadata would comprehensively describe the screenshot in a textual form in a similar way as a DOM tree does with a web page. However, the approach presented in the framework simplifies these descriptions to keep just the essential information. Specifically, this metadata comprises a list of UI Element Location and Classification data (cf. Definition 1) that will be stored for each event in the log. Definition 1. The UI Element Location and Classification (ELoC) data is a tuple (Type, Pos, Id) with the following components: •Type. This indicates the type of the UI element and can take on values such as TextView, ImageView, or Button, among others. This classification is based on the work by Moran et al. [37] where a comprehensive list of all possible types can be found. •Pos. The position of the UI element within the screenshot is stored as a tuple, specifying the top-left and bottom-right coordinates. This information allows precise localization of the UI element within the image. •Id. This component serves as a unique identifier for the UI element within the screenshot. It enables efficient referencing and retrieval of specific elements during the analysis. In the current paper, the Id of the ELoC is generated using the following template: type-centroidX-centroidY. In addition to being pseudo-unique, this template can be used to find collisions of UI elements between different screenshots, which is necessary for further steps of the framework. Example 1. For an element identified in a screen and classified as a button, like the ones in Fig. 6, the ELoC could be ‘‘<button, [{boundingBox_topLeft: (450,680)},{boundingBox_bottomRight: (490, 750)}], button-470-715>’’ . 3.1.2. Feature extraction The previous was centered on just detecting the position and the type of elements on the screen. In turn, the current step goes beyond that information since it aims to extract more information from the screenshot, the so-called features. The framework primarily focuses on extracting features from entire screenshots rather than UI elements separately. However, within this context, two distinct types of feature extractors can be identified (cf. Fig. 7): •Single feature extractors are designed to consider only one specific UI element at a time. Examples of such features include the bounding box property, color, enable/disable status, checked/ unchecked status, or the centroid (cf. single FE in Fig. 7) or textual content of individual UI elements. •Aggregate feature extractors capture features that may relate to more than one UI element simultaneously. For instance, they can measure the number of buttons or texts present on a screen (cf. aggregate FE in Fig. 7) or calculate the bounding box that encompasses all the text inputs within the GUI. Regardless of these two types, a common definition can be given (cf. Definition 2). Definition 2. AUser Interface Feature Extractor (UIFE) is a tuple (Pred, Func) where Pred is a predicate over ELoC to state which elements the UIFE can be applied to, and Func is the function to be applied. This function receives a tuple (img, ELoCs) and returns a tuple (name, value) where: img is a given image, ELoCs is a list of ELoC elements (cf. Definition 1) contained in img, name is the name of the feature that is extracted, and value is the value of such a feature. The previous definition enables any extractor that considers a screenshot or a specific crop of it (i.e., the img parameter). The extractor can leverage on the identification and classification information collected from the previous step (i.e., the ELoCs parameter). Example 2. Algorithm 1depicts the function of a UIFE that expects to receive a crop of a single UI element and delegates it to an external function to extract its enabled status. If the status cannot be obtained (i.e., the external function returns a null), the value of the feature remains empty. The predicate associated with this UIFE can be ‘‘𝑒𝑙𝑜𝑐 → 𝑡𝑟𝑢𝑒’’ since it can be applied to any UI element. Unlike the previous algorithm, Algorithm 2leverages the provided ELoCs to calculate the number of buttons present in a screenshot. The predicate associated with this UIFE can be ‘‘𝑒𝑙𝑜𝑐 →𝑒𝑙𝑜𝑐.𝑡𝑦𝑝𝑒 ==′ 𝑏𝑢𝑡𝑡𝑜𝑛′’’ since it would be in charge of counting the number of buttons.2 Several UIFEs can be defined to extract features for the screenshots of a UI Log. Each of them is applied to each event as described in 2Note that only for the sake of illustration the same expression as in the predicate is used in the algorithm. Information Systems 121 (2024) 102340 6 A. Martínez-Rojas et al. Fig. 7. Types of UIFEs: Single and Aggregate. Algorithm 1: Enabled status feature extractor example Input : Image 𝑖𝑚𝑔, List⟨𝐸𝐿𝑂𝐶⟩𝑒𝑙𝑜𝑐𝑠 Output: (𝑛𝑎𝑚𝑒,𝑣𝑎𝑙𝑢𝑒) 1enabled ←𝑈𝐼𝑇 𝑜𝑜𝑙𝑠.𝑔𝑒𝑡𝐸𝑛𝑎𝑏𝑙𝑒𝑑𝑆𝑡𝑎𝑡𝑢𝑠(𝑖𝑚𝑔) 2if status == null then 3value ←‘‘’’ 4else 5value ←𝑠𝑡𝑎𝑡𝑢𝑠 6end 7return (name = ′𝐸𝑛𝑎𝑏𝑙𝑒𝑑′, value = 𝑣𝑎𝑙𝑢𝑒) Algorithm 2: Count buttons feature extractor example Input : Image 𝑖𝑚𝑔, List⟨𝐸𝐿𝑂𝐶⟩𝑒𝑙𝑜𝑐𝑠 Output: (𝑛𝑎𝑚𝑒,𝑣𝑎𝑙𝑢𝑒) 1count ←𝑒𝑙𝑜𝑐𝑠.𝑓𝑖𝑙𝑡𝑒𝑟(𝑒𝑙𝑜𝑐 →𝑒𝑙𝑜𝑐.𝑡𝑦𝑝𝑒 ==′𝑏𝑢𝑡𝑡𝑜𝑛′).𝑐𝑜𝑢𝑛𝑡() 2return (name = ′#𝐵𝑢𝑡𝑡𝑜𝑛𝑠′, value = 𝑐𝑜𝑢𝑛𝑡) Algorithm 3to generate an extended UI Log, i.e., the same UI Log with additional information extracted from the log. This algorithm receives (1) the log, (2) a set of UIFEs that are applied to the entire screen (e.g., UIFE depicted in Algorithm 2), and (3) a set of UIFEs that are applied to single UI elements in the screen (e.g., UIFE depicted in Algorithm 1). Then, on the one hand, for each event in the log, the algorithm extracts one feature for each UIFE that is applied to the entire screen (cf. lines 2 to 6 of Algorithm 3). These UIFEs will be executed once and receive all the ELoCs that match the given predicate. On the other hand, the algorithm will extract one feature for each ELOC that matches the predicate of each UIFE applied to a single UI element (cf. lines 7 to 13 of Algorithm 3). That is, these UIFEs will be executed once per ELoC that matches the predicate. A new column is included in the UI Log for each extracted feature. The name of the column depends only on the name of the UIFE – in the case of the first type of UIFEs –, or on both the name of the UIFE and the id of the ELOC – in the case of the second type of UIFEs. Example 3. For example, let considers a UI Log similar to the one in Fig. 2 as the first parameter of Algorithm 3. In addition, let considers Algorithm 2as the only feature extractor in the parameter extractEntire, and Algorithm 1as the only feature extractor in the parameter extractSingle. This latter parameter is also accompanied by the predicate 𝑒𝑙𝑜𝑐 →𝑡𝑟𝑢𝑒, which states that the feature extractor will be applied to any UI element in the screenshot. Therefore, the resulting extended log will include the same columns as the original log and the following columns: 1. (𝑛𝑎𝑚𝑒 =‘‘#Buttons’’, 𝑣𝑎𝑙𝑢𝑒 = 3), as depicted in Fig. 8. 2. (𝑛𝑎𝑚𝑒 =‘‘button-470-715_Enabled’’, 𝑣𝑎𝑙𝑢𝑒 =‘‘true’’). 3. (𝑛𝑎𝑚𝑒 =‘‘button-655-715_Enabled’’, 𝑣𝑎𝑙𝑢𝑒 =‘‘true’’). Algorithm 3: Extend UI Log with features Input : UILog 𝑙𝑜𝑔, Set⟨𝑈𝐼𝐹 𝐸⟩𝑒𝑥𝑡𝑟𝑎𝑐𝑡𝐸𝑛𝑡𝑖𝑟𝑒, Set⟨𝑈 𝐼𝐹 𝐸⟩ 𝑒𝑥𝑡𝑟𝑎𝑐𝑡𝑆𝑖𝑛𝑔𝑙𝑒 Output: UILog 𝐸𝑥𝑡𝑒𝑛𝑑𝑒𝑑𝑙𝑜𝑔 1newLog ←𝑐𝑜𝑝𝑦(𝑙𝑜𝑔) 2for 𝑒𝑣𝑒𝑛𝑡 in 𝑛𝑒𝑤𝑙𝑜𝑔 do 3for 𝑢𝑖𝑓𝑒 in 𝑒𝑥𝑡𝑟𝑎𝑐𝑡𝐸𝑛𝑡𝑖𝑟𝑒 do 4(𝑛𝑎𝑚𝑒, 𝑣𝑎𝑙𝑢𝑒)← 𝑢𝑖𝑓𝑒.𝐹 𝑢𝑛𝑐(𝑒𝑣𝑒𝑛𝑡.𝑠𝑐𝑟𝑒𝑒𝑛𝑠ℎ𝑜𝑡, 𝑒𝑣𝑒𝑛𝑡.𝑒𝑙𝑜𝑐𝑠.𝑓𝑖𝑙𝑡𝑒𝑟(𝑢𝑖𝑓𝑒.𝑃 𝑟𝑒𝑑)) 5𝑐𝑜𝑙𝑢𝑚𝑛𝑁𝑎𝑚𝑒 ←𝑛𝑎𝑚𝑒 6𝑒𝑣𝑒𝑛𝑡 + ←←←←←←←← (𝑐𝑜𝑙𝑢𝑚𝑛𝑁𝑎𝑚𝑒, 𝑣𝑎𝑙𝑢𝑒) 7end 8for 𝑢𝑖𝑓𝑒 in 𝑒𝑥𝑡𝑟𝑎𝑐𝑡𝑆𝑖𝑛𝑔𝑙𝑒 do 9for 𝑒𝑙𝑜𝑐 in 𝑒𝑣𝑒𝑛𝑡.𝑒𝑙𝑜𝑐𝑠.𝑓 𝑖𝑙𝑡𝑒𝑟(𝑢𝑖𝑓 𝑒.𝑃 𝑟𝑒𝑑)do 10 (𝑛𝑎𝑚𝑒, 𝑣𝑎𝑙𝑢𝑒)← 𝑢𝑖𝑓𝑒.𝑎𝑝𝑝𝑙𝑦(𝑒𝑣𝑒𝑛𝑡.𝑠𝑐𝑟𝑒𝑒𝑛𝑠ℎ𝑜𝑡.𝑐𝑟𝑜𝑝(𝑒𝑙𝑜𝑐.𝑃 𝑜𝑠), 𝑒𝑙𝑜𝑐) 11 𝑐𝑜𝑙𝑢𝑚𝑛𝑁𝑎𝑚𝑒 ←𝑒𝑙𝑜𝑐.𝐼𝑑 +‘‘_’’ +𝑛𝑎𝑚𝑒 12 𝑒𝑣𝑒𝑛𝑡 + ←←←←←←←← (𝑐𝑜𝑙𝑢𝑚𝑛𝑁𝑎𝑚𝑒, 𝑣𝑎𝑙𝑢𝑒) 13 end 14 end 15 end 16 return newLog 3.2. Learning from the log The log may contain events related to different instances of a business process. Decision points appear in the process model where they are split into alternative branches. To identify them and learn which pieces of information rule the decisions taken in such instances, the current framework is inspired by Rozinat et al. [21]. For this, the framework first converts the UI Log to an artifact that ML algorithms can use, i.e., a labeled dataset (cf. Section 3.2.1). Thereafter, this dataset is used to train a classification algorithm that can explain such a behavior (cf. Section 3.2.2). 3.2.1. Creating the labeled dataset The labeled dataset should contain one row for each observation that leads to a decision related to a decision point. In this line, an observation is the set of events that precedes the decision point. In addition, since the aim is to train a classification algorithm, the decision itself should be part of the dataset, the so-called ’label’. According to these requirements, Algorithm 4depicts how to transform a UI Log into a dataset for a given decision point. Therein, this algorithm runs over each case in the log (cf. lines 1 to 3 of Algorithm 4) and flattens into a single row all the case events that are before the given decision point (cf. lines 4 to 9 of Algorithm 4). Each event’s attribute is included as a new column of the same row to do this flattening. The column’s name is composed using the activity’s and attribute’s name (cf. line 7 of Algorithm 4) to avoid collisions of the Information Systems 121 (2024) 102340 7 A. Martínez-Rojas et al. Fig. 8. Extended UI Log from Fig. 2 including feature ‘‘#Buttons’’ extracted for the screenshot of Fig. 6. Fig. 9. Label dataset creation applying Algorithm 4to UI Log presented in Fig. 8. same attributes of different activities. Besides the flattening, the label column is also included since it states which decision of the decision point has been taken by this case in the UI Log (cf. line 10 of Algorithm 4). Algorithm 4: Generation of a dataset for a decision point Input : UILog 𝑙𝑜𝑔, DecicionPoint 𝑑𝑒𝑐𝑖𝑠𝑖𝑜𝑛 Output: Dataset 𝑑𝑎𝑡𝑎𝑠𝑒𝑡 1𝑑𝑎𝑡𝑎𝑠𝑒𝑡 ←∅ 2𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒𝑠 ←𝑠𝑒𝑝𝑎𝑟𝑎𝑡𝑒𝐿𝑜𝑔𝐵𝑦𝐶𝑎𝑠𝑒𝑠(𝑙𝑜𝑔) 3for 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒 in 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒𝑠 do 4𝑟𝑜𝑤 ←∅ 5for 𝑒𝑣𝑒𝑛𝑡 in 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒.𝑒𝑣𝑒𝑛𝑡𝑠𝐵𝑒𝑓 𝑜𝑟𝑒(𝑑𝑒𝑐𝑖𝑠𝑖𝑜𝑛)do 6for 𝑎𝑡𝑡𝑟 in 𝑒𝑣𝑒𝑛𝑡.𝑎𝑡𝑡𝑟𝑖𝑏𝑢𝑡𝑒𝑠() do 7𝑟𝑜𝑤 + ←←←←←←←← (column=𝑒𝑣𝑒𝑛𝑡.𝑎𝑐𝑡𝑖𝑣𝑖𝑡𝑦𝐼𝐷 +′_′+𝑎𝑡𝑡𝑟.𝑛𝑎𝑚𝑒 , value=𝑎𝑡𝑡𝑟.𝑣𝑎𝑙𝑢𝑒 8end 9end 10 𝑟𝑜𝑤 + ←←←←←←←← (column=’LABEL’, value=𝑑𝑒𝑐𝑖𝑠𝑖𝑜𝑛.𝑏𝑟𝑎𝑛𝑐ℎ(𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒)⟩ 11 𝑑𝑎𝑡𝑎𝑠𝑒𝑡 + ←←←←←←←← 𝑟𝑜𝑤 12 end 13 return dataset Generally, a set of datasets will be generated for a UI Log. More precisely, one dataset for each decision point that appears in the discovered process model needs to be generated. Example 4. For example, let us consider the UI Log of Fig. 8. There is a decision point just after A-B with two branches: branch 1, which leads to activity C, and branch 2, which leads to activity F. The resulting dataset obtained after applying Algorithm 4will have two rows where the first one has 1 as the label, and the second has 2 as the label (cf. Fig. 9). 3.2.2. Generating a decision tree In the literature, many classification algorithms exist that can generate models that try to correctly classify the rows of a dataset, i.e., to match each row to its label using the rest of the columns of the dataset. This process is called model training. The framework described in this paper uses CHAID [43], a tree-based classification algorithm. We use a tree-based algorithm since, unlike other classification algorithms, the generated models can be graphically depicted as a tree and are easier to understand by a human. In addition, the CHAID algorithm has demonstrated to have better performance in this context than other popular algorithms [24]. If the information contained in the dataset is sufficient, the trained model should demonstrate high training accuracy, indicating that the resulting tree can effectively explain all labels (i.e., the decisions made) using the column values (i.e., the features). This tree will have one node per column, each representing a crucial feature for explaining the labels. The edges going out of each node delineate the possible values of that column. The leaves of the tree represent the labels, and the path from the root to each leaf defines the conjunction of rules necessary to reach that specific label. However, achieving training accuracy close to 100% may raise concerns about overfitting. Overfitting occurs when a model excessively tailors itself to the training data, capturing noise and outliers. To address this, we conduct a thorough analysis of additional measures, such as accuracy, recall, or F1 score, as part of our evaluation process. Moreover, it is important to clarify that, as occurs when using ML methods, achieving a high accuracy with the CHAID algorithm does not inherently imply the establishment of causality between the features and the decision outcome; rather, it signifies correlation. It is essential to acknowledge that the relationships identified between the columns and the label could be due by chance (i.e., non-relevant features that give an explanation to the decision casually), instead of being the actual chain of events ruling the decision (i.e., the features that are truly relevant). While acknowledging the inherent limitations of the CHAID algorithm, we want to underscore the valuable insights obtained through the representation of the discovered tree. The resulting tree-based model serves as a tool for revealing relationships between features and decision outcomes. It is crucial to note, however, that the presented correlations do not automatically imply causation. Moreover, our commitment to interpretability is grounded in the belief that models should be transparent and comprehensible to both analysts and stakeholders. By presenting a clear and interpretable decision model, we aim to enhance understanding of decision making during the process examined. This, in turn, supports the development of the automation script related to this process. 3.3. Trace back decisions Although the information given by the classification tree is understandable by a human, the nodes (i.e., the name of the columns of the dataset) may be difficult to associate directly with elements of the underlying process. To amend this, this step aims to map the tree to a set of screenshots where the considered features are highlighted as described in Algorithm 5. More specifically, for each node with its corresponding outgoing edges (cf. line 2 of Algorithm 5), it first looks for the related event and gathers its screenshot (lines 3 to 5 of Information Systems 121 (2024) 102340 8 A. Martínez-Rojas et al. Fig. 10. Example decision tree where the splitting rules (a) are linked to features in the screenshot (b). Algorithm 5) Then, it looks for the related ELoC – in case of single type- or ELoCs – in case of entire type – in the UI Log (lines 6 to 8 and 13 of Algorithm 5). Finally, it highlights the bounding box of each ELoC in the screenshot (cf. lines 10 and 14 of Algorithm 5). Algorithm 5: Tracing back tree rules to screenshots Input : UILog 𝑙𝑜𝑔, TreeNode 𝑛𝑜𝑑𝑒 Output: List ⟨𝐼𝑚𝑎𝑔𝑒⟩𝑖𝑚𝑔𝑠 1imgs ←∅ 2for 𝑒𝑑𝑔𝑒 in node.outgoingEdges do 3(activityID, elocId, feature) ←decompose(node.name) 4event ←𝑙𝑜𝑔.𝑓𝑖𝑙𝑡𝑒𝑟(𝑒→𝑒.𝑎𝑐𝑡𝑖𝑣𝑖𝑡𝑦𝐼𝐷 == 𝑎𝑐𝑡𝑖𝑣𝑖𝑡𝑦𝐼𝐷 && 𝑒.𝑎𝑡𝑡𝑟𝑖𝑏𝑢𝑡𝑒(𝑓 𝑒𝑎𝑡𝑢𝑟𝑒).𝑚𝑎𝑡𝑐ℎ(𝑒𝑑𝑔𝑒.𝑛𝑎𝑚𝑒))[0] 5screen ←𝑐𝑜𝑝𝑦(𝑒𝑣𝑒𝑛𝑡.𝑠𝑐𝑟𝑒𝑒𝑛𝑠ℎ𝑜𝑡) 6if elocId == null then 7uife ←𝑈𝐼𝐹 𝐸𝐶𝑎𝑡𝑎𝑙𝑜𝑔.𝑔𝑒𝑡(𝑓𝑒𝑎𝑡𝑢𝑟𝑒) 8elocs ←𝑒𝑣𝑒𝑛𝑡.𝑒𝑙𝑜𝑐𝑠.𝑓𝑖𝑙𝑡𝑒𝑟(𝑢𝑖𝑓𝑒.𝑃 𝑟𝑒𝑑) 9for eloc in elocs do 10 screen.highlight(eloc.Pos) 11 end 12 else 13 eloc ←𝑒𝑣𝑒𝑛𝑡𝑠.𝑒𝑙𝑜𝑐𝑠.𝑓𝑖𝑙𝑡𝑒𝑟(𝑢→𝑢.𝐼𝑑 == 𝑒𝑙𝑜𝑐𝐼𝑑) 14 screen.highlight(eloc.Pos) 15 end 16 imgs + ←←←←←←←← 𝑠𝑐𝑟𝑒𝑒𝑛 17 end 18 return imgs Example 5. Fig. 10 shows the classification tree (a) resulting from applying the CHAID algorithm to a dataset similar to the one of Fig. 9, and the related screenshot (b) where the two tree nodes are linked to the concrete UI elements. In this example, both UI elements belong to the same screenshot. 4. Evaluation In this section, we empirically evaluate the proposed framework, aiming to answer the primary research question (PRQ) posed at the beginning of this paper: Can the screen-based Task Mining approach effectively extract and represent process behavior observed in a UI Log within real-life settings? To conduct a more detailed analysis, three specific sub-questions (SQ) are formulated. First, SQ1 assesses the framework’s capability to identify features that influence decisions during process execution within the context of the observed UI Log. •SQ1: Does the proposed framework identify features that capture the relevant elements that humans take into account when making decisions? Building on SQ1, SQ2 investigates the framework’s ability to extract conditions based on the identified features. The evaluation specifically assesses the choice of branches following the decision points in the process. •SQ2: Does the proposed framework correctly discover the conditions that represent the human decision-making? To assess the practical applicability of the proposed framework, SQ3 addresses its performance from a time perspective. •SQ3: Does the proposed framework discover human decisionmaking in a time-efficient manner? The scope of this evaluation focuses on the last part of the framework (i.e., steps 4 and 5), since the first part (i.e., steps 1, 2, and 3) has already been evaluated in previous works [13,24]. To ensure rigor, this evaluation follows the case study protocol proposed by Brereton et al. [44]. Section 4.1 describes the characteristics of the process selected for the study. Section 4.2 defines the specific problem situation chosen for the evaluation. Section 4.3 explains the evaluation design, while Section 4.4 details the execution environment. Section 4.5 encloses the data collection, Section 4.6 presents the results obtained and, finally, Section 4.7 discuss the threats to the validity. 4.1. Case selection For this case study, we selected a real-world process identified by a business expert as a typical operation to automate within the Business Process Outsourcing context. The chosen process revolves around managing unsubscribe requests from clients and closely mirrors the administrative workflow of a Spanish telecommunications company. This particular process entails handling a volume of 300 cases per month, with a customer base of approximately 1.5 million customers. For privacy considerations, we have replaced specific data with synthetic information. This process in question (cf. Fig. 11) is composed of 10 activities, 1 single decision point, and 4 variants. It implies dealing with three different real-life systems (cf. Fig. 12): an email client (i.e., Gmail), an issue tracking system (i.e., Zendesk) and an ERP system (i.e., Odoo). In addition, the process includes: (1) a UI Log with a case for each process variant, and (2) real-life screenshots for each of its activity events. Note that the company consented to cooperate for the sake of this evaluation, Information Systems 121 (2024) 102340 9 A. Martínez-Rojas et al. Fig. 11. Handling of unsubscription requests process. Fig. 12. Example of a real-life screenshot (a) and its mockup version (b). but did not authorize us to publish the real information about its corporate systems. Therefore, adapted and anonymized versions of the UI Log and their screenshots are used. We consider this to be a proper case for a critical evaluation since it meets the following selection criteria: (1) this process highly resembles a real process within a company that we cooperate with in Spain, and (2) the discovery of the decision is based on information that appears on the screenshots, i.e., two UI elements: an attachment in an email, and a checkbox on a web form. 4.2. Objects The evaluation is based on a set of synthetic problems closely mimicking the telecommunication company’s real use case. For confidentiality reasons, we cannot disclose the company’s name; we were allowed, though, to analyze their process in real-life and collect event logs, including screenshots, as seeds for our evaluation approach. We have developed two types of problems (𝑃) for our evaluation. The first type involves creating scenarios using mockups of the process, while the Information Systems 121 (2024) 102340 16 A. Martínez-Rojas et al. [8] M. Lacity, L.P. Willcocks, Innovating in service: The role and management of automation, in: Dynamic Innovation in Outsourcing, Springer, 2018, pp. 269–325. [9] C. Frank, Introduction To Robotic Process Automation, Inst. Robotic Process Autom. (2015) 35. [10] L. Willcocks, M. Lacity, A New Approach to Automating Services, MIT Sloan Manag. Rev. 58 (1) (2016) 40–49. [11] A. Jiménez-Ramírez, J. Chacón-Montero, T. Wojdynsky, J. Gonzalez Enriquez, Automated testing in robotic process automation projects, J. Softw.: Evol. Process (2020) e2259. [12] J.G. Enríquez, A. Jiménez-Ramírez, F. Domínguez-Mayo, J. García-García, Robotic process automation: a scientific and industrial systematic mapping study, IEEE Access 8 (2020) 39113–39129. [13] A. Jimenez-Ramirez, H.A. Reijers, I. Barba, C. Del Valle, A method to improve the early stages of the robotic process automation lifecycle, in: CAiSE, 2019, pp. 446–461. [14] W.M.P. van der Aalst, Process Mining: Data Science in Action, Springer, Heidelberg, 2016. [15] L. Reinkemeyer, Process Mining in Action. Principles, Use Cases and Outlook, Springer, 2020. [16] V. Leno, A. Polyvyanyy, M. Dumas, M. La Rosa, F.M. Maggi, Robotic process mining: vision and challenges, Bus. Inf. Syst. Eng. (2020) 1–14. [17] S. Agostinelli, M. Lupia, A. Marrella, M. Mecella, Reactive synthesis of software robots in RPA from user interface logs, Comput. Ind. 142 (2022) 103721. [18] V. Leno, A. Augusto, M. Dumas, M. La Rosa, F.M. Maggi, A. Polyvyanyy, Identifying candidate routines for robotic process automation from unsegmented UI logs, in: ICPM, 2020, pp. 153–160. [19] M. Völker, S. Siegert, M. Weske, Adding decision management to robotic process automation, in: Business Process Management: Blockchain and Robotic Process Automation Forum: BPM 2021 Blockchain and RPA Forum, Rome, Italy, September 6–10, 2021, Proceedings 19, Springer, 2021, pp. 23–37. [20] V. Leno, A. Armas-Cervantes, M. Dumas, M. La Rosa, F.M. Maggi, Discovering process maps from event streams, in: ICSSP, 2018, pp. 86–95. [21] A. Rozinat, W.M.P. van der Aalst, Decision mining in prom, in: S. Dustdar, J.L. Fiadeiro, A.P. Sheth (Eds.), BPM, Springer, Berlin, Heidelberg, 2006, pp. 420–425. [22] J. Gao, S.J. van Zelst, X. Lu, W.M. van der Aalst, Automated robotic process automation: A self-learning approach, in: OTM, 2019, pp. 95–112. [23] J.M. López-Carnicer, C. del Valle, J.G. Enríquez, Towards an OpenSource logger for the analysis of RPA projects, in: BPM, 2020, pp. 176–184. [24] A. Martínez-Rojas, A. Jiménez-Ramírez, J.G. Enríquez, H.A. Reijers, Analyzing variable human actions for robotic process automation, in: International Conference on Business Process Management, Springer, 2022, pp. 75–90. [25] S. Agostinelli, M. Lupia, A. Marrella, M. Mecella, SmartRPA: A tool to reactively synthesize software robots from user interface logs, in: CAiSE, 2021, pp. 137–145. [26] V. Leno, S. Deviatykh, A. Polyvyanyy, M. La Rosa, M. Dumas, F.M. Maggi, Robidium: Automated synthesis of robotic process automation scripts from UI logs, in: International Conference on Business Process Management, CEUR Workshop Proceedings, 2020. [27] A. Martínez-Rojas, A. Jiménez-Ramírez, J.G. Enríquez, H.A. Reijers, A toolsupported method to generate user interface logs, in: Hawaii International Conference on System Sciences, 2023. [28] S. Agostinelli, M. Lupia, A. Marrella, M. Mecella, Automated generation of executable RPA scripts from user interface logs, in: BPM, 2020, pp. 116–131. [29] A. Augusto, R. Conforti, M. Dumas, M. La Rosa, F.M. Maggi, A. Marrella, M. Mecella, A. Soo, Automated discovery of process models from event logs: Review and benchmark, IEEE Trans. Knowl. Data Eng. 31 (4) (2018) 686–705. [30] V. Leno, A. Polyvyanyy, M. La Rosa, M. Dumas, F.M. Maggi, Action logger: Enabling process mining for robotic process automation, in: Proceedings of the Dissertation Award, Doctoral Consortium, and Demonstration Track At BPM 2019 Co-Located with 17th International Conference on Business Process Management (BPM 2019) Vienna, Austria, September 1-6, 2019, Vol. 2420, CEUR-WS, 2019, pp. 124–128. [31] J. Geyer-Klingeberg, J. Nakladal, F. Baldauf, F. Veit, Process mining and robotic process automation: A perfect match., in: BPM, 2018, pp. 124–131. [32] E. Bazhenova, S. Bülow, M. Weske, Discovering decision models from event logs, in: BIS, 2016, pp. 237–251. [33] H.A. Sleiman, R. Corchuelo, Tex: An efficient and effective unsupervised web information extractor, Knowl.-Based Syst. 39 (2013) 109–123. [34] P. Jiménez, J.C. Roldán, R. Corchuelo, A clustering approach to extract data from HTML tables, Inf. Process. Manage. 58 (6) (2021) 102683. [35] C. Ikerionwu, D. Edgar, Secured service delivery model for outsourced services in a business process outsourcing relationship, Int. J. Inf. Commun. Sci. 4 (1) (2019) 7–17. [36] Z. Xu, X. Baojie, W. Guoxin, Canny edge detection based on open CV, in: 2017 13th ICEMI, 2017, pp. 53–56. [37] K. Moran, C. Bernal-Cárdenas, M. Curcio, R. Bonett, D. Poshyvanyk, Machine learning-based prototyping of graphical user interfaces for mobile apps, IEEE Trans. Softw. Eng. 46 (2) (2018) 196–221. [38] B.P. Majumder, N. Potti, S. Tata, J.B. Wendt, Q. Zhao, M. Najork, Representation learning for information extraction from form-like documents, in: ACL, 2020, pp. 6495–6504. [39] Keras OCR, 2022, Available at https://github.com/faustomorales/keras-ocr. (Last Accessed March 2022). [40] T. Mitchell, Machine Learning, McGraw Hill, 1997. [41] L.H. Gilpin, D. Bau, B.Z. Yuan, A. Bajwa, M. Specter, L. Kagal, Explaining explanations: An overview of interpretability of machine learning, in: 2018 IEEE 5th DSAA, 2018, pp. 80–89. [42] J. Chen, M. Xie, Z. Xing, C. Chen, X. Xu, L. Zhu, G. Li, Object detection for graphical user interface: Old fashioned or deep learning or a combination? in: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020, pp. 1202–1214. [43] G.V. Kass, An exploratory technique for investigating large quantities of categorical data, J. R. Stat. Soc. Ser. C. Appl. Stat. 29 (2) (1980) 119–127. [44] P. Brereton, B. Kitchenham, D. Budgen, Z. Li, Using a protocol template for case study planning, in: 12th International Conference on Evaluation and Assessment in Software Engineering, (EASE) 12, 2008, pp. 1–8. [45] V. Leno, M. Dumas, M. La Rosa, F.M. Maggi, A. Polyvyanyy, Automated discovery of data transformations for robotic process automation, 2020, arXiv preprint arXiv:2001.01007.