Human–computer interaction analysis for RPA support

Andrés Jiménez Ramírez, Hajo A. Reijers, and José González Enríquez 9 Human–computer interaction analysis for RPA support Framework and new horizons Abstract: Inthelastdecade, industry has wholeheartedlyembracedRPA.Alargenumber of RPA projects have been conducted, which all involve the mimicking of human behavior by software. Although vendor-specific frameworks exist to support all aspects of RPA solutions – from the analysis of the system until the enactment and control of the developed robots – most of them base their solution on workflow diagrams (e.g., BluePrism or UIPath). This chapter explores the benefits of a conceptual RPA framework that is based on log information related to the exact human–computer interaction. While the observation of employees’ computers (i.e., clicks and keystrokes) may lead to the desired data, it is by no means trivial to convert such events into a meaningful log. Our approach shows how to create a so-called user interface (UI) log based on image similarity techniques. This UI log can then be used to provide support for all stages of the RPA lifecycle. Specifically, we will describe a use case in which the UI log is used (1) to support the analysis of the underlying process and (2) to generate a test platform which checks whether or not the developed robots behave in accordance with the analyzed process. We will also discuss how other stages of the RPA lifecycle can leverage the existence of such a UI log, which leads to the identification of new research lines. Keywords: Robotic process automation, process mining, image processing 9.1 Introduction Conducting an RPA project requires following a lifecycle similar to any other software project. Variations of this lifecycle can be observed in the literature (Flechsig etal., 2019;Enríquez et al., 2020)but,in general,they havethe following activities:analysis, design,development,deployment,control and operation, and evaluation. Nowadays, industry has come up with RPA platforms, like UiPath, that are intended to cover the complete lifecycle of RPA. However, these platforms provide polished solutions mainly for the development and later stages but neglect mature sup- Acknowledgement: Research supported by the NICO project (PID2019-105455GB-C31) of the Spanish Ministry of Science, Innovation and Universities, the Trop@ project (CEI-12-TIC021) of the Junta de Andalucía, and the AIRPA (P011-19/E09) project of Spain’s CDTI. https://doi.org/10.1515/9783110676693-009 170 | A. Jiménez Ramírez et al. port for the early stage of analysis and design (Enríquez et al., 2020). Therefore, RPA projects still rely on analyzing process documentation – which may be of poor quality – or manual inspections of the information systems (ISs) – which require substantial effort – to create a basic understanding of the processes to robotize. That situation potentially leads to an inaccurate analysis, which is fatal since robots are typically deployed in production environments accessing real ISs. To prevent this scenario from happening, a common practice in RPA adoption is to keep both people and robots working in the same processes simultaneously. More precisely, human workers are the first ones who deal with the ISs and the processes. Theywillexperiencetheexceptions,undocumentedcases,etc.,andlearn howtosolve such situations, i.e., on-the-job training. The generated knowledge during these experiences is key to increase the probability of success when conducting the RPA project. Only later the digital workforce is created, which will carry out the most repetitive and mundane tasks of the processes. Nonetheless, due to the lack of proper support within commercial RPA platforms, this accumulated knowledge is typically transferred informally, e.g., through interviews or manually created reports. Against this backdrop, approaches that formally analyze the human–computer interaction (HCI) are now beginning to appear. Such approaches are referred to with the brand new term of task mining (Reinkemeyer, 2020), an adaptation of the process mining paradigm (van der Aalst, 2016). Characteristic for task mining is that the computers of the human workers need to be monitored by recording the screen, mouse, and keyboard events while the humans are interacting with the user interfaces (UIs) of the related ISs. Next, these events are processed to obtain a meaningful log, the so-called UI log. This log can then be used to discover the underlying process by using process discovery algorithms (Geyer-klingeberg and Nakladal, 2018; van der Aalst et al., 2018; Leno et al., 2018; Jimenez-Ramirez et al., 2019; Leno et al., 2020). Although the processing of the UI log is non-trivial, the benefits of using this automated discoveryovera manual approachhavebeen clearlyrecognized in recentliterature(Jimenez- Ramirez etal., 2019). Different steps in the RPA lifecycle may leverage the results of the task mining methods. First, it becomes (1) straightforward to analyze the as-is processes and (2) easier to detect the candidate processes to robotize being able to provide a to-be design for them. In addition, logged data consist of valuable information for developing robots and testing them before deployment. Moreover, the continuous monitoring of thedeployedbots maygenerateuseful metrics for controlling their performance.Also, this may generate recommendations of operational actions in the case of failures or unexpected behaviors, which supports the governance of the digital workforce. What is more, process redesign actions can be recommended in the case of changes or optimizations being detected within the processes. The objective of this chapter is twofold. First of all, it aims to describe a conceptual framework for HCI analysis and how it can be applied in an industrial environment in 9 Human–computer interaction analysis for RPA support | 171 the domain of business process outsourcing (BPO), i.e., a company that executes processes which belong to a different company. Although not limited to the BPO context, theback-officedepartmentsof BPOcompaniesareidentifiedas optimal candidatesfor RPA projects (Geyer-klingeberg and Nakladal, 2018) and, besides, they provide particularly challenging settings when strict security policies are in force. Secondly, this chapter goes through the phases of the RPA project lifecycle to discuss the application horizon of this task mining paradigm. In doing so, research lines that still need to be explored are identified. The rest of the chapter is structured as follows. Section 9.2 shows an RPA context where the HCI analysis is applicable. Section 9.3 describes the HCI approach. Section 9.4 details an application of this approach in the test phase. Section 9.5 discusses the applicability and limitations of the HCI approach in the other phases of the lifecycle. Finally, Section 9.6 concludes the chapter. 9.2 Human-first context Scenarioslikethe back-officeofa BPO company historicallyemploy onlyhuman workers to accomplish their processes. Nowadays, back-offices are becoming hybrids, i.e., some of these processes – or parts of them – are being shifted to digital workers (cf. Figure 1a and b) thanks to rapid market penetration of the RPA technology (Biscotti, 2018). Regardless of the kind of workers, the documentation of the prescribed process is a crucial ingredient. However, each kind of worker – human or digital – behaves differently in the case of non-documented cases, exceptions, or deviations. Figure 1: Early stages in back-office. On the one hand, the human workers are trained according to the documentation of the prescribed process before the real ISs are faced. Since humans are expected to perform flexibly, they can make decisions when some unexpected situation arises by applying a cognitive effort – probably just by using common sense (i. e., on-the-job 172 | A. Jiménez Ramírez et al. training). Effective improvisations might later on be shared among the rest of the team. Nonetheless, although human workers generally perform well, as already identified in the literature (Aguirre and Rodriguez, 2017), they behave poorly on repetitive, highvolume, and simple tasks. On the other hand, regarding the digital workers, constructing them requires conducting the RPA lifecycle similarly for any other software product. Analyzing and designing the desired robot implies a significant effort. The reason for this is that ambiguity and uncertainty of the prescribed process should be minimized and the selected parts to robotized need to be formalized. Once constructed, multiple instances of the same robots are deployed gradually while their behavior is controlled and tested. In the case that something occurs that is not covered by the robot’s design, the robot gets stuck and suffers the following maintenance process. First, a human operator takes control of the robot and tries to repair it either by completing the running instance or by bringing the robot to a controlled situation. Second, the robot behavior is analyzed to disclose the root cause of the error. Finally, a new RPA lifecycle starts either to include or to explicitly exclude the new behavior in the robot design – leaving it for the human workers. Whatever happens next, the new version of the robot will not get stuck again in a similar situation. As the reader may understand, this process is considerably more time consuming than the on-the-job training of human workers. Still, the digital workforce provides more work flexibility and predictability since they can work continuously and the size of the team can be altered on demand. Interestingly, companies have acknowledged that there is an efficient way to obtain their hybrid back-office: the so-called human-first context, i. e., the human backoffice works first and, when the knowledge of the process is more mature, the digital back-office is created (cf. Figure 1c). In this context, three periods can be identified regarding the workload shifting between humans and robots (cf. Figure 2). After the “human-first period,” where the human workforce is in charge of the entire process, the first versions of robots are deployed gradually to handle those first identified candidates. Itis in the “first version of robots” period in which the largest and most robust shift of workload occurs. The “maintenance and adjustments period” aims to achieve the highest level of digitization. It must be acknowledged that there is an asymptotic limit that the current state of technology cannot exceed. Eventually, only the most unstable and judgment-based parts of the process remain handled by humans. The sooner this eventual situation is reached, the more profitable the RPA project is. However, this is not an entirely stable situation since there might be changes in processes, UIs, and systems. If these changes are produced but not notified to the robots manager, they can potentially produce misbehavior of the robots, which implies that humans have to take care of the affected processes until the new version of the robots is deployed. Therefore, the human workload increases temporally. This human-first context still presents some inefficiencies regarding the duplication of duties. For example, the training of humans requires analyzing the same documentation that is required for the construction of the first version of the robots. In 9 Human–computer interaction analysis for RPA support | 173 Figure 2: Workload shift from humans to robots. addition, when humans do on-the-job training they perform a similar analysis as the one which is needed when misbehavior is detected in a robot. Therefore, the identified challenge and the focus of this chapter is to leverage the knowledge of the human team to be used by the different phases of the RPA lifecycle. 9.3 Learning from humans This section describes how the analysis of the HCI can be used to leverage the human knowledge in the context of an RPA project. This analysis is conducted from a humanfirst viewpoint. Although different approaches can be found in the literature (Leno etal., 2018; Himel and Liu, 2017; Leno et al., 2020), the basic pipeline that they follow is described here (cf. Figure 3). Figure 3: Basic pipeline for learning from HCI. 9.3.1 Behavioral monitoring This step pursues capturing as much information as possible from the computers of the human workforce during working hours. Some HCI approaches include sophisticated sensors (e.g., eye trackers or vision-based gesture recognition) (Sharma etal., 1998) to capture the broadest number of events from the physical world in a digital log. However, such sensoring is uncommon in a back-office workplace and thus, the 174 | A. Jiménez Ramírez et al. behavior that can be feasibly monitored is tightened to mouse clicks, keystrokes, and screen captures, i.e., the one produced by the most common peripheral devices. The event streams of these peripherals need to be recorded through non-intrusive monitoring software (i.e., its existence is transparent for the user) on each back-office computer and, thereafter, put all together in the same behavioral log.1The basic information that each behavioral event may include is: – computer: it identifies the physical machine where the event is registered; – timestamp: when this event occurs; – image: a screen capture taken in this computer at this timestamp; – keystrokes: the sequence (possibly empty) of keystrokes which are pushed at this timestamp; – clicktype: the type of mouse click (i. e., left, right, middle, or none) which is done at this timestamp; – clickcoords: the coordinates of the mouse at this timestamp. Every time the human performs a mouse click or a sequence of keystrokes, an event is recorded. The resulting log consists of a single long trace which may be related to more than one instance of the processes that the back-office has performed. 9.3.2 Low-level processing In the low-level processing step, the single-trace log is processed to become a more meaningful log from a process analysis perspective, i.e., a UI log (Jimenez-Ramirez etal., 2019) that can be expressed in the standard XES format (XES, 2016). For this, each behavioral event is enriched with two new attributes: – UI event: identifies the atomic process activity which is associated with each behavioral event (e. g., login into the system, or submitting a form); – trace: identifies the single execution/instance of the process that a sorted sequence of events corresponds to – also knows as the case. First, this low-level processing includes the identification of repeated UI events. Some approaches like robotic process mining (Leno etal., 2020) or desktop activity mining (Linn etal., 2018) accomplish this step relying on UI information like window names or web page frames, which, in turn, disables its application in contexts where the humans interact with the ISs through secured connections. A more general approach (Jimenez-Ramirez et al., 2019) is based on image similarity techniques. More precisely, an efficient bit-wise comparison (Gusfield, 1999) between images fingerprints (i.e., 1Since the human workers have received similar training, they are expected to behave similarly and so their individual logs. 9 Human–computer interaction analysis for RPA support | 175 short hashes which are obtained from each image in a deterministic way) (Wong et al., 2002) is used to state that two screen captures (i. e., events) are related to the same activity (i. e., UI event) according to some pre-fixed similarity threshold. Second, the one-trace log is split in different traces. For this, one UI event is selected as the starting event of each trace which uses to be the most repeated one (Jimenez-Ramirez et al., 2019) or one which is manually selected (Leno etal., 2018). Besides calculating these two new attributes, this low-level processing might be used to wipe out some unnecessary events. Since not all the activities of the human workers are relevant for the RPA development (e. g., social networking or some web surfing),some of them can be detectedat the imagelevel.Techniqueslikeopticalcharacter recognition – that looks for a text within an image – or template matching – that detects if a small image is part of a bigger image – are useful candidates to clean the behavioral log which may result in a cleaner UI log. In addition, the process mining community recognizes that the presence of these unnecessary events – noise – leads to the discovery of complex and inaccurate process models (Suriadi et al., 2017). To remedy this, they provide numerous techniques (Cheng and Kumar, 2015; Măruşter etal., 2006) to reduce such noise and enhance the quality of logs. Thus, they can be adapted to be included in this low-level processing. 9.3.3 High-level processing Thehigh-levelprocessingstepleveragesthematuretechnologyofprocessmining(van der Aalst, 2016) to analyze the UI log and discover the underlying process that the human workers perform. This discovered process is the result of high-level processing that produces a graphical process model where the considered traces and events are represented and where the variation points can be identified. At this level, a new attribute, called a variant, is included in the log to identify the process variant that each trace belongs to. In addition, several statistical properties can be extracted from each variant, trace, and event (e.g., frequency, duration, or size) which are crucial to determine which parts are relevant enough for the RPA project. Furthermore, this high-level processing enables additional mechanisms to wipe out the previous log. At this point, marginal variants or variants that contain undesired process patterns are taken out of the process (e. g., variants which present long delays between two activities may be due to the existence of some physical activity, the whole variant thus being irrelevant for RPA). As the reader may have seen, decisions which are taken in the previous step (e.g., choosing a similarity threshold or removing some events) have a clear impact on the resulting analysis. Therefore, this pipeline may require some iterations and manual fine-tuning before obtaining the desired result. Eventually, such a result represents the as-is process model that serves as the input of an RPA project. 176 | A. Jiménez Ramírez et al. 9.3.4 Preliminary evaluation The positive impact that the HCI analysis contributes to the RPA lifecycle has been establishedin a preliminaryevaluation(Jimenez-Ramirez et al., 2019). This evaluation compared the outcomes obtained by the current approach with those that resulted from the conventional analysis and design activities. Four steps were conducted during the evaluation (cf. Figure 4). First, one backoffice computer was monitored to obtain the behavioral log for a period of one week, which comprises 200 complete cases approximately. Second, the obtained behavioral log was processed by the pipeline (cf. Figure 3) to generate an initial as-is process model. Third, the analyst repeated the pipeline several times applying some cognitive effort to obtain a final as-is process model. Lastly, the two obtained process models were compared with the one obtained conventionally, i. e., based on traditional, document-oriented analysis and design phases. Figure 4: Steps in the preliminary evaluation. Thisevaluationwasappliedtotwooutsourcedprocessesofa Spanishcompany, which isactiveintheBPOdomain. Differentmeasureswereusedtocomparetheperformance of each analysis: (1) the number of process variants that were included in each model (i.e., # Var conven, #Var auto, #Var super), (2) the percentage of new variants that the supervised analysis detects where the conventional analysis missed (i. e., %New), and (3) the percentage of variants that were included in the conventional analysis but not in the supervised one (i.e., % Missed). The following conclusions can be drawn after analyzing the results of the evaluation (cf. Table 1): Table 1: Result of the preliminary evaluation. Process #Var conven #Var auto #Var super %New %Missed #1 5 46 12 7/12 (58%) 0/5 (0%) #2 9 60 13 5/13 (38 %) 1/9 (11 %) 9 Human–computer interaction analysis for RPA support | 177 – Regarding the columns #Var conven and #Var super, the supervised HCI analysis discovers a greater variety of paths than the conventional way. As observed in the column % New, this increment is above 38 % in both processes. This implies that employees had to deal with variants which were not included in the conventional analysis yet existed in the behavior of the IS. – Regarding the %Missed variants, it is relevant to note that only one was missed in the second process. This can be interpreted as follows: (1) the behavioral log was too small and did not contain all the relevant paths or (2) the path was actually not relevant for the analyst. – Finally, regarding the # Var auto, it clearly shows that the cognitive labor of the analyst is much required to tune the pipeline, since many variants were included. Nonetheless, most of them were considered noise and irrelevant to the analysis, e.g., interrupted cases or errors. 9.4 Application example: the test phase As mentioned above, testing the robots before they are deployed into production is of paramount importance to ensure the integrity of the ISs. Although proposals exist to improve the RPA test phase (Hannonen, 2020; Cewe etal., 2017), these all involve some manual work. The example application described in this section allows to do testing automatically, based on the results of the HCI analysis (Jiménez-Ramírez et al., 2020). This is done by including a testing cycle in the traditional RPA lifecycle (cf. Figure 5). Figure 5: Including automatic testing in the RPA lifecycle. 184 | A. Jiménez Ramírez et al. the development of robots, it would be valuable to work on new solutions that help to systematize and automate their development. Finally, scaling this HCI approach to a big-company level requires great computing capabilities for executing artificial vision algorithms and other image processing. Therefore, efficient techniques are in demand to detect and reduce unnecessary computations. Bibliography ABBYY (2020) State of process mining and robotic process automation 2020. Available at: www.abbyy.com/en-us/solutions/process-intelligence/research-report-2020. Last accessed: May 2020 Aguirre S, Rodriguez A (2017) Automation of a business process using robotic process automation (RPA): a case study. In: Proceedings – applied computer sciences in engineering: 4th workshop on engineering applications, WEA 2017, pp 65–71. ISBN 978-3-319-66963-2. https://doi.org/10.1007/978-3-319-66963-2_7 Biscotti F (2018) In: Gartner market share analysis: robotic process automation, Worldwide Cewe C, Koch D, Mertens R (2017) Minimal effort requirements engineering for robotic process automation with test driven development and screen recording. In: International conference on business process management. Springer, Berlin, pp 642–648 Cheng H-J, Kumar A (2015) Process mining on noisy logs – can log sanitization help to improve performance? Decis Support Syst 79:138–149 Criteria HPF (2014) Use cases and effects of information technology process automation (ITPA). Adv Robot Autom 3(3):1–11. ISSN 21689695. 10.4172/2168-9695.1000124 Enríquez JG, Jiménez-Ramírez A, Domínguez-Mayo FJ, García-García JA (2020) Robotic process automation: a scientific and industrial systematic mapping study. IEEE Access 8:39113–39129 eXtensible Event Stream (XES) for achieving interoperability in event logs and event streams (2016) IEEE Std 1849-2016, pp 1–50 Flechsig C, Lohmer J, Lasch R (2019) Realizing the full potential of robotic process automation through a combination with bpm. In: Logistics management. Springer, Berlin, pp 104–119 Geyer-klingeberg J, Nakladal J (2018) Process mining and robotic process automation: a perfect match. In: 16th international conference on business process management, industry track session, number July, pp 1–8 Gusfield D (1999) Algorithms on strings, trees, and sequences: computer science and computational biology. Cambridge University Press, Cambridge Hannonen A (2020) Automated testing for software process automation. Master’s thesis, School of Technology and Innovations, University of Vaasa Himel D, Liu Z (2017) Identifying frequent user tasks from application logs. In: Proceedings of the 22nd international conference on intelligent user interfaces, pp 263–273 Jimenez-Ramirez A, Reijers HA, Barba I, Del Valle C (2019) A method to improve the early stages of the robotic process automation lifecycle. In: International conference on advanced information systems engineering. Springer, Berlin, pp 446–461 Jiménez-Ramírez A, Chacón-Montero J, Wojdynsky T, González Enríquez J (2020) Automated testing in robotic process automation projects. J Softw Evol Process e2259. https://doi.org/10.1002/smr.2259 Leno V, Dumas M, Maggi FM, La Rosa M (2018) Multi-perspective process model discovery for robotic process automation. In: CEUR workshop proceedings, vol 2114, pp 37–45 9 Human–computer interaction analysis for RPA support | 185 Leno V, Polyvyanyy A, Dumas M, La Rosa M, Maggi FM (2020) Robotic process mining: vision and challenges. Bus Inf Syst Eng. ISSN 1867-0202. https://doi.org/10.1007/s12599-020-00641-4 Linn C, Zimmermann P, Werth D (2018) Desktop activity mining – a new level of detail in mining business processes. In: Workshops der INFORMATIK 2018 – Architekturen, Prozesse, Sicherheit und Nachhaltigkeit, pp 245–258 Măruşter L, Ton Weijters AJMM, van Der Aalst WMP, van Den Bosch A (2006) A rule-based approach for process discovery: dealing with noise and imbalance in process logs. Data Min Knowl Discov 13(1):67–87 Pinto G, Reboucas M, Castor F (2017) Inadequate testing, time pressure, and (over) confidence: a tale of continuous integration users. In: 2017 IEEE/ACM 10th international workshop on cooperative and human aspects of software engineering (CHASE), pp 74–77 Reinkemeyer L (2020) Process mining in action. Principles, use cases and outlook. Springer, Berlin. ISBN 978-3-030-40172-6 Sharma R, Pavlovic VI, Huang TS (1998) Toward multimodal human-computer interface. Proc IEEE 86(5):853–869 Staron M (2006) Adopting model driven software development in industry – a case study at two companies. In: Nierstrasz O, Whittle J, Harel D, Reggio G (eds) Model driven engineering languages and systems, Berlin, Heidelberg. Springer, Berlin, pp 57–72 Suriadi S, Andrews R, ter Hofstede AHM, Wynn MT (2017) Event log imperfection patterns for process mining: towards a systematic approach to cleaning event logs. Inf Syst 64:132–150 van der Aalst WMP (2016) Process mining: data science in action. Springer, Heidelberg van der Aalst WMP, Bichler M, Heinzl A (2018) Robotic process automation. Bus Inf Syst Eng 60(4):269–272. ISSN 1867-0202. 10.1007/s12599-018-0542-4 Wong C, Bern MW, Goldberg D (2002) An image signature for any kind of image. In: International conference on image processing, pp 409–412