A NLP-Oriented Methodology to Enhance Event Log Quality

A NLP-Oriented Methodology to Enhance Event Log Quality Bel´ en Ramos-Guti´ errez1(B),´ Angel Jes´ us Varela-Vaca1, F. Javier Ortega1,Mar ´ ıa Teresa G´ omez-L´ opez1, and Moe Thandar Wynn2 1IDEA & ITALICA Research Groups, Universidad de Sevilla, Seville, Spain {brgutierrez,ajvarela,javierortega,maytegomez}@us.es 2Queensland University of Technology (QUT), Brisbane, Australia [email protected] http://www.idea.us.es Abstract. The quality of event logs is a crucial cornerstone for the feasibility of the application of later process mining techniques. The wide variety of data that can be included in an event log refer to information about the activity, such as what, who or where. In this paper, we focus on event logs that include textual information written in a natural language that contains exhaustive descriptions of activity executions. In this context, a pre-processing step is necessary since textual information is unstructured and it can contain inaccuracies that will provoke the impracticability of process mining techniques. For this reason, we propose a methodology that applies Natural Language Processing (NLP) to raw event log by relabelling activities. The approach let the customised description of the measurement and assessment of the event log quality depending on expert requirements. Additionally, it guides the selection of the most suitable NLP techniques for use depending on the event log. The methodology has been evaluated using a real-life event log that includes detailed textual descriptions to capture the management of incidents in the aircraft assembly process in aerospace manufacturing. Keywords: Natural Language Processing ·Event log quality ·Process mining 1 Introduction Event logs include the footprints generated by an organisation’s information systems, being possible to store a wide variety of information [4,6] related to the tracked events, e.g., textual descriptions, timestamps or used resources. In general, event logs need to be adapted for a later (process mining) analysis, for instance, to discover processes. Thereby, the assessment of the quality of an event log [7] is the very first and crucial step for any subsequent analysis. The application of any process mining technique over incorrect or inaccurate event logs, e.g. process discovery, will produce incorrect or inaccurate process models [32]. Several authors have defined criteria to assess the data quality in general [9,21] and event log quality in particular [7,32], such as completeness, correctness, security and Fig. 1. Relabelling application example. trustworthiness. The Process Mining manifesto [7] introduces the quality of an event log as a quality maturity level. The imperfections than can produce a low event log quality might be improved analysing the activity labelling, timestamps, case identification, etc. In this paper, we focus on event logs that include some textual descriptions which detail what happened in various moments of process execution. We propose to first identify these activities and their inter-relations from textual constraint descriptions using NLP techniques and then relabelling activities in the event log to extract these details in an easy to handle way. Figure 1illustrates how the use of Natural Language Processing (NLP) techniques and the relabelling of activities in an event log, can improve the results of automated process discovery. For this reason, we not only adapt a general methodology to measure and assess the data quality of event logs [27], but also propose a decision-support system to assist in the selection of the most suitable NLP techniques according to the current quality level and the expected assessment. The research question is: What are the most suitable NLP techniques to use for relabelling an event log in order to improve its quality? This question does not have a simple answer, since it depends on the current quality of the event log and the dimension or dimensions that must be improved and how. To answer this research question, it is necessary to define metrics to measure the event log quality and a mechanism to assess how good is this quality level in each context. This is well-known as the fitness-per-purpose [19], where the level of quality must be customised according to the needs, as for example: (1) determining the average length of the label of the activities; (2) the level of noise allowed; and (3) the usual number of activities per trace. With this goal in mind, we propose a methodology, called LOADING-NLP, which assists in the decision-making for the application of NLP techniques over raw event logs for relabelling activities in accordance with the decision rules about data quality described by experts. A set of fitness-per-purpose metrics and dimensions are proposed to measure and assess the quality of an event log. Both, the metrics and dimensions can be customised, or extended for other examples. In this regard, the measurement and assessment can be adjusted and alternative NLP techniques can be applied. Our methodology also assists the user to select the most suitable NLP techniques. To validate the proposed methodology, we applied it to a real case study based on the management of the incidents produced during the aircraft assembly processes in aerospace manufacturing. The rest of the paper is organised as follows: Sect. 2includes the related work in the area. Section 3introduces the methodology; the measurement and the assessment of the event log quality are discussed in Sect. 4. Section 5reviews the NLP techniques that can be applied in this context and outlines to what extent they can affect the quality assessment. Section 6presents the evaluation results and Sect. 7concludes the paper. 2 Related Work In order to understand the advantages that this proposal offers, it is necessary to know the level of maturity in the following areas. Event Log Quality. The necessity to have data with suitable quality is crucial for any process and necessary for later analysis, such as process mining [10]. How to measure and assess the possibility of leveraging their quality is an important topic which has been a focus of study during the last decades. However, event logs appear in new contexts [30] and include features that make it necessary to define new metrics to measure, extend and adapt the dimensions (e.g., completeness, accuracy, simplicity) to the business process context [8,31]. Event Log Improvement. Once the data quality level can be assessed, various are the techniques that can be applied to improve the event log quality [26]. Some solutions are based on timestamp [12,13,15], case identification, and activity relabelling. Regarding the activity relabelling, the solution presented in [25] proposes to detect synonymous and polluted labels in event logs, but no techniques were proposed to improve the quality in accordance to the previous detection, and [24] uses a gamification approach to repair the labels. The types of improvements over event logs depend on the case and the later use of them [18]. Use of NLP in Business Process Management. Previous works have studied the extraction of declarative [2] and imperative business process models [3]fromtexts. In those works, the NLP techniques have been used in order to facilitate the automation of tasks that require a significant effort detecting patterns of relational order between the activities involved [1]. In addition, the detection of activities and their associated labels is crucial for further analysis and refactoring of the terms to enable an automatic analysis [17]. The text analysis in the context of the business process has also been focused on the detection of inconsistencies between the textual descriptions and the graphical representation [3], as a mechanism of misalignment detection. Use of NLP to Improve Process Discovery Results. Some works have studied the preprocessing of the event data to improve the discovery task when using real-life logs that are written in natural language. In [23], is done by automatically detecting and classifying eight different semantic roles in event data. In [14], semantic-based techniques are applied to aggregate and normalise event log text information. Other types of analysis have been made to improve the labels of the activities in a process model by detecting erroneous ways of labelling activity that lead to ambiguity and inconsistency [22]. Contrary to our proposal, in all cases, these proposals start from event data in natural language with a very process-oriented construction and a simple and correct syntax. To sum up, to the best of our knowledge, this is the first work where NLP techniques have been used to improve the event log quality according to a set of proposed metrics, guided by a decision-support system to ascertain the best techniques to apply depending on the event log and its quality level. 3 LOADING-NLP: Methodology for Assessing and Improving Event Log Quality with NLP When the labels of the events within the log include natural language texts, it is necessary to analyse and treat them to become the log useful. The NLP techniques used to this aim will cause a direct impact on the different number of labels in the log, the number of events per trace, or the similarity between the labels. Therefore, the best NLP techniques to use depends on the meaning of event log quality in each context, the event log quality before the application of the techniques, and how the experts want it to evolve after the application of the techniques. To support these three aspects, we propose the methodology presented in Fig. 2, described through a BPMN model. The first step is related to the definition of when the event log is usable (cf., Determine the usability of the event log quality), according to the measurement and the assessment described by a set of decision rules about data quality. If the event log has sufficient quality, it can be used for a process mining analysis. However, if the event log is deemed unsuitable, the expert must determine the dimension or dimensions that must be adjusted and the required assessment (cf., Introduce dimensions and assessment to achieve). Using this information, we propose a decision-support system (cf., Infer NLP techniques to apply) that provides possible NLP techniques to use for improving the event log quality according to the described decision rules about data quality and the requirements of the experts. This process can be repeated until the resulting event log achieves the required level of quality. 4 Determine the Usability of the Event Log Quality The question of when an event log has sufficient quality does not have a single answer. It will depend on the meaning of event log quality in each context. There are solutions as [27] that provide mechanisms to describe the decision rules related to the measurements and assessments adapted to each context and requirements. At first, we need to define Fig. 2. Methodology for the application of NLP techniques. the decision rules according to data quality using a set of metrics extracted from the event log. Thereby, it is necessary to define a set of metrics to evaluate the dimensions, as detailed in Subsects. 4.1 and 4.2. These measures enable us to perform an assessment according to the meaning of quality defined by the expert, as described in Subsect. 4.3. 4.1 Metrics to Measure the Quality of Event Logs Based on [11], an event log is a set of traces that represent different instances of the same process. A trace is an ordered sequence of events that represents a process instance. Every trace is associated with a unique case identifier. The execution of an activity in a business process is represented as an event in an event log. Similarly, an event is the representation of the execution of an activity in a business process. Each event is associated with a case identifier, one timestamp and can also have many other contextual attributes. Usually, each event has associated at most one timestamp, which represents the start or the end of the execution of an activity. To measure the dimensions, some metrics must be extracted from the event log [5,11], as the mentioned in [8]. In our case, the used metrics are: –Number of traces. Total number of traces in the log, and the trace jis represented by τj. –Number of events (ε). Total number of events in the log, and the event iis represented by εi. It helps to know the size of the log. –Number of different labels. Number of different labels that occur in every trace. –Number of unique labels. Number of single (unique) labels that appear in the log. 4.2 Quality Dimensions for Event Logs In general, the data quality dimensions describe the relevant aspects for a data set and typically consist of accuracy, completeness, consistency and uniqueness. However, we cannot guarantee that an event log with a high level of quality in those dimensions will produce valuable business processes. For this reason, other dimensions are included to assess the event log quality in process mining, as was defined in [31]. Those dimensions can be affected by the application of NLP techniques. Based on them, we propose the following dimensions albeit others can also be used together with our methodology: mUniqueness =Number of Unique Labels Number of Events mComplexity =Avg. Number of Events Number of T races  mRelevancy =Number of Different Labels Number of Events mConsistency = ε  i=1 |l(εi)−l(ε)| Number of Events  Uniqueness. If every label in an event log is the same, the discovered process will only include one activity. However, if each label is unique in the traces, the discovered process will have one branch per trace, thus not very useful. The uniqueness dimension, that we propose in a range between [0..1], measures the percentage of single labels regarding the total number of labels. When the values are in the extremes (i.e., 0 or 1), it implies that the process may be too simple (low uniqueness) or too complex (high uniqueness). Consistency. When the labels of activities are dissimilar, especially for textual formats, they can imply that the descriptions have different granularity. For this reason, the measurement of consistency that we propose is based on the average length1of strings in these textual descriptions, and the mean of the distance to the average. Therefore, this dimension is bounded by the length of the longest string. Relevancy. The relevancy of each label depends on the number of times that it occurs. It is important to analyse the number of different labels according to the total number. It is related to the uniqueness, but it is not exactly the same. The dimension is bound between [0..1]. Complexity. There are several metrics that can represent the complexity of an event log [26], such as the average of events per trace. We propose to measure the complexity by the mean of the number of events per trace. A higher mean implies a higher concentration of events per trace, therefore representing a more complex process. 4.3 Customising the Measurement and Assessment of Event Log Quality As commented previously, the data quality is an aspect highly related to the later use of the data, hence it must be customised according to the necessities. Following the DMN4DQ proposed in [27,28], DMN (Decision Model and Notation) [20] can be used for facilitating the description of data quality divided into measurement and assessment rules. DMN is a declarative language proposed by OMG to describe decision rules applied to a tuple of input data to obtain a tuple of outputs according to the evaluation of a set of conditions described in FEEL. A DMN table is composed of rows that describe a decision rule as an if-then condition so that, if it is satisfied, the output is returned. Also, DMN permits a hierarchical structure where the output of a DMN table can be the input of another. Using the methodology DMN4DQ, we propose to split up the measurement and assessment in two different levels for each involved dimension. Additionally, the final assessment is obtained by aggregating the assessment of every dimension, as described in Fig. 3. 1We use l() function to define the length of a label description. Fig. 3. DMN for describing the Quality Assessment. DMN tables have various types of columns (orders,inputs, and outputs). The first column establishes the order by assigning an index to each row, and includes the hit policy to determine how to act when more than one is satisfied, (cf., Fto describe that the evaluation of the condition is in order). Each input column represents the input variables that are evaluated the condition of the row. An example of DMN tables for the measurement of each dimension is detailed in Tables 1a, 1b, 1c and 1d, that illustrate the four dimensions proposed in this paper. Each dimension is described by a set of domains of the metric values, where the measurement of the metrics is the output value of the table. For example, the consistency dimension, in this case, describes that if the average of the numbers of characters is greater than 30, the measurement of the consistency will be Very low. For the measurement of each dimension, only one metric is required as input. In each row of the input column, the conditions are described in FEEL, for instance, the first row in Table 1b establishes that the valid range for the metric mUniqueness is between 0and 1. We use the metrics defined in the previous subsection as the inputs. The outputs represent the obtained values depending on the condition satisfied, for instance, if the input for mUniqueness is 0.5the outputs is High. The values and conditions established in Table 1have been adjusted according to the know-how of experts for the use case at hand. We should bear in mind that the measurement of a metric does not represent whether the metric is good or not, this is why a later assessment is necessary. Table 2includes a proposal for the assessment of each dimension, for example, both a Low and Very Low number of events will imply an Excellent assessment in the event log according to the Complexity metric. As previously commented, this assessment has been defined based on the experts’ knowledge of the event logs but other assessments can be accommodated. The assessment of each dimension needs to be aggregated to determine a global one. A possible set of decision rules for the aggregation of the assessment of the four dimensions is described in Table 3, albeit another combination can be applied according to the necessity of the organisation. Table 1. Decision tables for measuring each dimension. (a) Measurement of Uniqueness Dimension Input OutputU FmUniqueness 1[0, 0.1] Very Low 2(0.1, 0.2] Low 3(0.2, 0.4] Medium 4(0.4, 0.6] High 5(0.6, 1] Very High (b) Measurement of Consistency Dimension Input OutputCs FmConsistency 1[0, 5] Very High 2(6, 14] High 3(14, 20] Medium 4(20, 30] Low 5(30, ∞)Very Low (c) Measurement of Relevancy Dimension Input OutputR FmRelevancy 1 [0, 0.1] Very High 2 (0.1, 0.2] High 3 (0.2, 0.4] Medium 4 (0.4, 0.6] Low 5 (0.6, 1] Very Low (d) Measurement of Complexity Dimension Input OutputCx FmComplexity 1[0, 4] Very Low 2(4, 6] Low 3(7, 10] Medium 4(11, 15] High 5(16, ∞)Very High Table 2. Decision tables for the assessment of each dimension. (a) Assessment of Uniqueness Dimension Input AssessU FOutputU 1Very Low Fair 2Low ∨Medium Excellent 3High Poor 4Very High Very Poor (b) Assessment of Consistency Dimension Input AssessCs FOutputCs 1Very Low Very Poor 2Low Poor 3 Medium Fair 4High Good 5Very High Excellent (c) Assessment of Relevancy Dimension Input AssessR FOutputR 1Very Low ∨Very High Very Poor 2High ∨Medium Fair 3Low Poor (d) Assessment of Complexity Dimension Input AssessCx FOutputCx 1Very Low ∨Low Excellent 2 Medium Good 3High Poor 4Very High Very Poor Table 3is designed in such a way that, when at least 3 assessment values for the dimensions is qualified as Excellent, and the remaining one is qualified as Good or Fair, the quality outcome of the log is Excellent. Similarly, when we have three dimensions qualified as Excellent or Good and one as Fair or Poor, the quality outcome of the log will be Good. On the other hand, when we find out two dimensions as Excellent or Good and other two as Fair or Poor, the quality outcome of the log will be Fair, while, Table 3. Aggregation of the Dimensions for the Quality Assessment Inputs Output FAssessUAssessCs AssessRAssesCx Qualityassessment 1Excellent Excellent Excellent Excellent ∨Good ∨Fair Excellent 2Excellent Excellent Good ∨Fair Excellent Excellent 3Excellent Good ∨Fair Excellent Excellent Excellent 4Good ∨Fair Excellent Excellent Excellent Excellent 5Good ∨Excellent Good ∨Excellent Good ∨Excellent Good ∨Excellent Good 6Poor ∨Fair Good ∨Excellent Good ∨Excellent Good ∨Excellent Good 7Good ∨Excellent Poor ∨Fair Good ∨Excellent Good ∨Excellent Good 8Good ∨Excellent Good ∨Excellent Poor ∨Fair Good ∨Excellent Good 9Good ∨Excellent Good ∨Excellent Good ∨Excellent Poor ∨Fair Good 10 Poor ∨Fair Poor ∨Fair Good ∨Excellent Good ∨Excellent Fair 11 Good ∨Excellent Poor ∨Fair Poor ∨Fair Good ∨Excellent Fair 12 Good ∨Excellent Good ∨Excellent Poor ∨Fair Poor ∨Fair Fair 13 Poor ∨Fair Good ∨Excellent Good ∨Excellent Poor ∨Fair Fair 14 Very Poor Very Poor Very Poor Very Poor Very Poor 15 Very Poor Very Poor Very Poor ∨Poor ∨Fair Very Poor ∨Poor ∨Fair Very Poor 16 Very Poor ∨Poor Very Poor Very Poor Very Poor ∨Poor ∨Fair Very Poor 17 Very Poor ∨Poor ∨Fair Very Poor ∨Poor ∨Fair Very Poor Very Poor Very Poor 18 - - - - Poor when, at least, two dimensions are qualified as Very Poor, the quality outcome of the log will be Very Poor. In any other case, the quality outcome of the log will be Poor. 5 Improving Event Log Quality: NLP Techniques for Relabelling Activities Our proposal aims to guide selection the NLP techniques for the relabelling of activities to extract the most meaningful and representative words for each process activity, but first, we need to introduce some NLP techniques. For the sake of clarity, we take the following description of an incident as an example to show the effects of each NLP technique proposed in the paper: “WARNING: the only way lines 1, 4 and 6 are powered is by closing up 2WR, then press PAX MASKS (in LMWS or in ICP 2011VM), afterwards PAX MASKS ON is switched on. Later, 2WR is pulled out and it can be checked that lines 1, 4 and 6 are powered. Waiting for ME confirmation and validation.” The NLP techniques that are being proposed to be applied are described below: Sentence Detection. This technique splits the text into its main components (i.e., sentences) to make it easier for the next steps to extract rich information from them. For our example, the application of this technique provides the next output: 15. Fischer, D.A., Goel, K., Andrews, R., van Dun, C.G.J., Wynn, M.T., R¨ oglinger, M.: Enhancing event log quality: detecting and quantifying timestamp imperfections. In: Fahland, D., Ghidini, C., Becker, J., Dumas, M. (eds.) BPM 2020. LNCS, vol. 12168, pp. 309–326. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58666-9 18 16. Honnibal, M., Montani, I., Van Landeghem, S., Boyd, A.: spaCy: Industrial-strength Natural Language Processing in Python (2020). https://doi.org/10.5281/zenodo.1212303 17. Leopold, H., Pittke, F., Mendling, J.: Ensuring the canonicity of process models. Data Knowl. Eng. 111, 22–38 (2017) 18. Martin, N., Martinez-Millana, A., Valdivieso, B., Fern´ andez-Llatas, C.: Interactive data cleaning for process mining: a case study of an outpatient clinic’s appointment system. In: Di Francescomarino, C., Dijkman, R., Zdun, U. (eds.) BPM 2019. LNBIP, vol. 362, pp. 532– 544. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-37453-2 43 19. Mocnik, F.B., Fan, H., Zipf, A.: Data quality and fitness for purpose (2017). https://doi.org/ 10.13140/RG.2.2.13387.18726 20. OMG: Decision Model and Notation (DMN), Version 1.2 (2019). https://www.omg.org/spec/ DMN 21. Otto, B., Lee, Y.W., Caballero, I.: Information and data quality in networked business. Electron. Mark. 21(2), 79–81 (2011). https://doi.org/10.1007/s12525-011-0062-2 22. Pittke, F., Leopold, H., Mendling, J.: When language meets language: anti patterns resulting from mixing natural and modeling language. In: Fournier, F., Mendling, J. (eds.) BPM 2014. LNBIP, vol. 202, pp. 118–129. Springer, Cham (2015). https://doi.org/10.1007/978-3-319- 15895-2 11 23. Rebmann, A., van der Aalst, H.: Extracting semantic process information from the natural language in event logs. CoRR abs/2103.11761 (2021) 24. Sadeghianasl, S., ter Hofstede, A.H.M., Suriadi, S., Turkay, S.: Collaborative and interactive detection and repair of activity labels in process event logs. In: 2nd ICPM, pp. 41–48 (2020) 25. Sadeghianasl, S., ter Hofstede, A.H.M., Wynn, M.T., Suriadi, S.: A contextual approach to detecting synonymous and polluted activity labels in process event logs. In: Panetto, H., Debruyne, C., Hepp, M., Lewis, D., Ardagna, C.A., Meersman, R. (eds.) OTM 2019. LNCS, vol. 11877, pp. 76–94. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-33246- 45 26. Suriadi, S., Andrews, R., ter Hofstede, A., Wynn, M.: Event log imperfection patterns for process mining: towards a systematic approach to cleaning event logs. Inf. Syst. 64, 132– 150 (2017) 27. Valencia-Parra, A., Parody, L., Varela-Vaca, A.J., Caballero, I., G´ omez-L´ opez, M.T.: DMN4DQ: when data quality meets DMN. Decis. Support Syst. 141, 113450 (2020) 28. Valencia-Parra, ´ A., Parody, L., Varela-Vaca, ´ A.J.,Caballero,I.,G ´ omez-L´ opez, M.T.: DMN for data quality measurement and assessment. In: Di Francescomarino, C., Dijkman, R., Zdun, U. (eds.) BPM 2019. LNBIP, vol. 362, pp. 362–374. Springer, Cham (2019). https:// doi.org/10.1007/978-3-030-37453-2 30 29. Valencia-Parra, ´ A., Ramos-Guti´ errez, B., Varela-Vaca, A.J., G´ omez-L´ opez, M.T., Bernal, A.G.: Enabling process mining in aircraft manufactures: extracting event logs and discovering processes from complex data. In: Proceedings of the Industry Forum at BPM, Vienna, pp. 166–177 (2019) 30. Vanbrabant, L., Martin, N., Ramaekers, K., Braekers, K.: Quality of input data in emergency department simulations: framework and assessment techniques. Simul. Model. Pract. Theory 91, 83–101 (2019) 31. Verhulst, R.: Evaluating quality of event data within event logs: an extensible framework. Master’s thesis, Rijksuniversiteit Groningen, echnische Universiteit Eindhoven (2016) 32. Wynn, M.T., Sadiq, S.: Responsible process mining - a data quality perspective. In: Hildebrandt, T., van Dongen, B.F., R¨ oglinger, M., Mendling, J. (eds.) BPM 2019. LNCS, vol. 11675, pp. 10–15. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-26619-6 2