Measuring Performance in Knowledge Intensive Processes

Measuring Performance in Knowledge Intensive Processes BEDILIA ESTRADA-TORRES, University of Seville, Spain PEDRO HENRIQUE PICCOLI RICHETTI, Federal University of the State of Rio de Janeiro, Brazil ADELA DEL-RÍO-ORTEGA, University of Seville, Spain FERNANDA ARAUJO BAIÃO, Federal University of the State of Rio de Janeiro, Brazil MANUEL RESINAS, University of Seville, Spain FLÁVIA MARIA SANTORO, Federal University of the State of Rio de Janeiro, Brazil ANTONIO RUIZ CORTÉS, University of Seville, Spain Knowledge-Intensive Processes (KIPs) are processes whose execution is heavily dependent on knowledge workers performing various interconnected knowledge-intensive decision-making tasks. Among other characteristics, KIPs are usually non-repeatable, collaboration-oriented, unpredictable and, in many cases, driven by implicit knowledge, derived from the capabilities and previous experiences of participants. Despite the growing body of research focused on understanding KIPs and on proposing systems to support these KIPs, the research question on how to define performance measures thereon remains open. In this paper, we address this issue with a proposal to enable the performance management of KIPs. Our approach comprises an ontology that allows us to define process performance indicators (PPIs) in the context of KIPs, and a methodology that builds on the ontology and the concepts of lead and lag indicators to provide process participants with actionable guidelines that help them conduct the KIP in a way that fulfills a set of performance goals. Both the ontology and the methodology have been applied to a case study of a real organization in Brazil to manage the performance of an Incident Troubleshooting Process within an ICT (Information and Communications Technology) Outsourcing Company. CCS Concepts: • Software and its engineering → Process management ; Software performance ; Modeldriven software engineering; Abstraction, modeling and modularity; Additional Key Words and Phrases: Process performance indicators, Knowledge-intensive processes, Performance measure ∗ This work has received funding from the European Commission (FEDER), the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 645751 (RISE_BPM), Spanish and Andalusian R&D&I programmes (P12-TIC-1867 (COPAS), TIN2015-70560-R (BELI)), and Brazilien CNPq (grants 304469/2014-9 and 309069/2013-0). Authors’ addresses: Bedilia Estrada-Torres, University of Seville, Departamento de Lenguajes y Sistemas Informáticos, Escuela Técnica Superior de Ingeniería Informática. Av. Reina Mercedes s/n, Seville, 41012, Spain, [email protected]; Pedro Henrique Piccoli Richetti, Federal University of the State of Rio de Janeiro, Departamento de Informática Aplicada, Avenida Pasteur, 458 - Urca, Rio de Janeiro, Brazil, [email protected]; Adela del-Río-Ortega, University of Seville, Seville, Spain, [email protected]; Fernanda Araujo Baião, Federal University of the State of Rio de Janeiro, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil, [email protected]; Manuel Resinas, University of Seville, Seville, Spain, [email protected]; Flávia Maria Santoro, Federal University of the State of Rio de Janeiro, University of the State of Rio de Janeiro, Rio de Janeiro, Brazil, [email protected]; Antonio Ruiz Cortés, University of Seville, Seville, Spain, [email protected]. 1 INTRODUCTION Knowledge-Intensive Processes (KIPs) have been defined as a type of process that comprises sequences of activities based on intensive acquisition, sharing, storage, and (re)use of knowledge, whereby the amount of value added to the organization depends on the knowledge of the actors involved. These processes are complex, less repeatable than conventional processes, and require a lot of creativity [Isik et al . 2013]. Based on an extensive literature review, Di Ciccio et al. [Di Ciccio et al . 2015] affirmed that KIPs are processes “whose conduct and execution are heavily dependent on knowledge workers performing various interconnected knowledge-intensive decision-making tasks”. Furthermore, they derived eight key characteristics typical of KIPs: knowledge-driven; collaborationoriented; unpredictable; emergent; goal-oriented; event-driven; constraint-and rule-driven; and non-repeatable. Additionally, in [Little and Deokar 2016], the authors investigated the relevance of knowledge creation in KIPs, and sustained that the expansion and use of knowledge across organizations rely on both formal and informal social processes through effective communication. Customer support, design of new products/services, marketing, management of data quality, IT governance, and strategic planning are cited as examples of KIPs [Marjanovic and Freeze 2011]. Developing a scientific experiment, performing medical diagnosis, and controlling air traffic are other areas in which KIPs are present [Hull et al . 2016]. They observed that the way organizations deal with these kinds of processes has changed over time, for example, customer support processes in several organizations have evolved from being highly structured to being knowledge-intensive, personalized, and flexible cases. All types of business processes, knowledge intensive or not, need to be measured, in order to evaluate and continuously improve their performance [Massey et al . 2002]. The process performance is usually measured through the identification, definition, computation and evaluation of process performance indicators (PPIs). These indicators are quantifiable metrics that allow the evaluation of efficiency and effectiveness of a business process, and can be obtained from data generated within the process flow [del Río-Ortega et al. 2013, 2010; Rosenberg et al. 2010]. Performance management has been widely analyzed in the context of structured business processes. Proposals such as [Korherr and List 2007; Momm et al . 2007; Pedrinaci et al . 2008; Popova and Sharpanskykh 2010; Wetzstein et al . 2008] and [del Río-Ortega et al . 2013], provide mechanisms for the definition and monitoring of PPIs in business processes whose expected behavior is predefined. These proposals for the assessment of process performance in traditional processes provide the opportunity to recognize problems and to take corrective actions before these problems increase, while also facilitating the comparison between an organization and its competitors [Kueng 2000]. The use of general approaches for the definition of performance measures also reduces the risk of introducing differences into the operationalization of the performance information, thus avoiding erroneous analyses during decision-making tasks and inconsistencies during information exchange. In addition, these approaches also allow the automation of PPI calculation as well as easier maintenance, which are time-consuming and error-prone tasks in their absence [del Río-Ortega et al. 2017b]. However, the aforementioned proposals cannot be used “as is” in KIPs for one main reason: the different nature of these processes. Those existing proposals were developed to measure structured business processes, with a predefined order in their control flow, and a set of characteristics known a priori for the various business process elements involved. Unlike structured processes, in KIPs, new information arises at runtime, such as the explicit knowledge used in process activities or the constraints and rules that have driven actions and decision-making throughout process execution. Therefore, existing proposals need to be extended to take all this information into account during the definition of performance measures and to be able to refer to specific KIP concepts, such as performance measures related to collaboration between process participants, for example, the measurement of the interaction and messages exchanged among the team members in the context of an IT customer-support process. Certain proposals, [Chen et al . 2009;Lee et al . 2005], attempt to address the measurement of KIP information by proposing specific metrics for the assessment of knowledge management performance in specific organizations. However, to the best of our knowledge, no general proposal is available for the definition of performance measures in KIPs that can be used independently of the context and that addresses the issues described above. Furthermore, in KIPs, there are performance improvements that cannot be hard-wired into the business process model but, rather, must be translated to the participants in the form of performance goals that must be taken into account during process execution. Therefore, a performance management solution for KIPs should help to come up with useful guidelines that enable process participants to meet and improve those performance goals. In this paper, we aim to improve the performance management capabilities in KIPs. To this end, we present a twofold contribution. First, we introduce the KiPPINOT ontology, which allows the formal definition of performance indicators in this type of process. KiPPINOT is built on the basis of two ontologies, namely the Knowledge-Intensive Process Ontology (KIPO), which provides a complete and precise understanding and representation of KIPs [Santos França et al . 2015], and the PPINOT ontology [del Río-Ortega et al . 2013,2010], which is unambiguous, highly expressive and amenable to the automation of PPI definitions. The second contribution is the MPG-K methodology, a methodology that relies on KiPPINOT and the concepts of lead and lag indicators [McChesney et al . 2012] in order to provide process participants with actionable guidelines that help them to conduct the KIP in a way that fulfills a set of performance goals. Both the ontology and the methodology have been applied to a case study of a real organization in Brazil to manage the performance of an Incident Troubleshooting Process within an ICT Outsourcing Company. The structure of this paper is as follows. Section 2presents the related work. In Section 3, the two ontologies, KIPO and PPINOT, are described. Section 4explains the KiPPINOT ontology. Section 5 describes our proposed MPG-K methodology. The application of MPG-K and KiPPINOT in an ICT Outsourcing Company in Brazil is presented in Section 6. Finally, Section 7concludes the paper. 2 RELATED WORK 2.1 Performance Measurement in Business Processes Performance measurement is an active research field in management science, which has attracted interest from both academia and business [Popova and Sharpanskykh 2010]. Substantial work has been performed on the identification and classification of key performance indicators in general settings [Kaplan and Norton 1992] and those relevant for specific domains, such as logistics, production, and supply chains [Brewer and Speh 2000;Chan 2003;Krauth et al . 2005;Vaidyanathan 2005]. In the context of Process Performance Measurement, great effort has been made in the formalization of PPI definitions. These indicators are a particular case of key performance indicators that aim to specify performance requirements of business processes [del Río-Ortega et al . 2013]. As a result of this effort, a number of notations and frameworks for the description and monitoring of PPIs have been proposed [Costello and Molloy 2009;del Río-Ortega et al . 2013;González et al . 2009; Korherr and List 2007; Momm et al . 2007; Pedrinaci et al . 2008; Popova and Sharpanskykh 2010; Saldivar et al . 2016; Wetzstein et al . 2008], which differ from each other in their expressiveness, that is, the different types of PPIs that can be defined, and their features that directly support monitoring. In addition, [Van Looy and Shafagatova 2016] identified weaknesses and inadequacies concerning the definition of PPIs in a structured literature review regarding performance measurement in the business process field. Nevertheless, none of them considers the particularities of KIPs: neither in terms of other aspects of KIPs that need to be measured, such as the collaboration between process participants and the constraints and rules that drive decision making in a process execution, nor in terms of how to use these PPIs to improve process performance. 2.2 Performance Measurement of Knowledge Workers Measuring the process productivity is, in general, not trivial, but in the case of KIPs, there are even more challenges to be faced. These processes are typically based on human resources and how they perform their activities, and the result of their work is often invisible [Sturm et al . 2011]. A few approaches in the literature discuss this topic. A classification of performance indexes is proposed in [Richter von Hagen et al . 2005] to evaluate process improvement, where knowledge performance is a category addressed from four measuring views: time, value (cost), quantity, and quality. However, this proposal recognizes the calculation of performance indexes as a challenge since although there are indexes that are directly quantifiable (time, cost, and quantity), others related to quality require different techniques to obtain their values. The alignment of knowledge indicators with an organization’s goals is highlighted [Little and Deokar 2011]. According to their proposal, internal and external sources influence knowledge indicators. The former could involve human resources (e.g. experiences, training, and education level) and infrastructure (e.g. legal mechanisms and technology); while the latter may consider the general public’s reaction to the company, brand reputation and loyalty, and customer loyalty. The framework proposed aids in the identification of knowledge-asset indicators. Nevertheless, as the authors point out, how to measure these indicators has been postponed for future works. Knowledgeintensive services are addressed in [Sturm et al . 2011] as elements with a high level of complexity (high in number and with interrelated sub-tasks), variability (high rate of change in activities) and uncertainty (limited availability of resources). The proposal identifies the requirements for the measurement of productivity of those services, and the example provided allows us to infer that those services are normally the result of KIPs. The need for a flexible evaluation system that considers multiple criteria is argued in [Rannacher et al . 2013]. They investigate which variables interact with each other and influence productivity. Since the production of knowledge is the key of KIPs, the authors propose ways of measuring the cycle of knowledge management: creation, sharing, capturing, and distribution of knowledge. They relate them to the qualifications of each employee, to their training and working abilities. Furthermore, they also indicate the autonomy of the employees, which could be related to motivation, performance readiness, and a lower rate of absence. Similarly, Lee et al. [Lee et al . 2005] propose a metric for knowledge-management performance. Therefore, they assume that knowledge circulates within the organization, creating assets and influencing performance. They investigate the knowledge circulation process for organizational performance. The components of the index proposed are the measures for knowledge creation, accumulation, sharing, utilization, and internalization. Both these studies agree that measuring performance related to processes where the main concern is knowledge is complex and involves diverse variables. Isik et al. [Isik et al . 2013] pointed out the lack of methods on how to measure effectiveness and improve KIPs. Based on results obtained through a survey, the authors argued that traditional BPM methods and techniques may be inadequate for the management and/or improvement of value creation in KIPs; however, there is a need to focus on managing human interaction. In general, the aforementioned proposals agree that knowledge management issues are essential to determine the performance of KIPs. Several suggestions of variables have been discussed and there is a consensus that a simple system would be unable to address all of these variables. Proposals identified in the literature fail to present a conclusive approach on how to measure KIPs. From another perspective, the literature on Knowledge Management advocates the relevance of considering how workers deal with knowledge to establish relationships with performance indicators [Henttonen et al . 2016;Oyemomi et al . 2016;Wang et al . 2016]. Based on an empirical study, Henttonen et al. [Henttonen et al . 2016] argue that knowledge sharing has a strong relation with improved performance in the contemporary organizational context, which is mostly based on knowledge and on how it is used within companies. They highlight that, in a knowledge-based economy, the capacity to create, transfer, and adopt knowledge, rather than simply look at efficiency indicators, might regulate the long-term performance of companies. According to Wang et al. [Wang et al . 2016], knowledge sharing can convert individual knowledge into organizational knowledge, and therefore improve the performance of a company. The authors investigated innovation and intellectual capital issues as critical drivers of performance in the context of knowledge sharing. Oyemomi et al. [Oyemomi et al . 2016] also empirically tested the contribution of knowledge sharing and business processes on organizational performance. For those authors, the concept of business-knowledge processes belongs within the scenario of activities for the improvement of organizational performance. Their study took organizational performance as a measurement of productivity in view of the employees’ knowledge contributions. They investigated three organizational operation factors: leadership support, learning and training, and communication. Although there is a vast literature that investigates the relations between knowledge work and performance, no specific proposal exists for the definition of performance indicators in the context of KIPs. 3 BACKGROUND ON ONTOLOGIES: KIPO AND PPINOT An ontology is defined as “a formal explicit description of concepts in a domain of discourse" [Noy and McGuinness 2001], which includes properties to describe features or attributes of the concepts and a set of restrictions to indicate how those elements are interrelated. In this section, we introduce two ontologies, KIPO and PPINOT, which are later integrated to allow the definition of PPIs for KIPs. 3.1 KIPO: the Knowledge-Intensive Process Ontology KIPO is a task ontology that comprises the key concepts and relationships, which are relevant for understanding, describing, and managing a KIP, as proposed by França et al. [Santos França et al . 2015]. It aims to provide a common, domain-independent understanding of KIPs and, as such, it may be used as a meta-model for the structure of KIP concepts. KIPO is founded on UFO (Unified Foundational Ontology) [Guizzardi et al . 2008], which is a foundational ontology based on a number of theories from Formal Ontology, Philosophical Logics, Philosophy of Language, Linguistics, and Cognitive Psychology [Guizzardi 2005]. UFO has been used “to evaluate, re-design and integrate (meta) models of different conceptual modeling languages as well as to provide real-world semantics for their modeling constructs" [Guizzardi et al . 2008]. It is organized into three main sections: UFO-A is the core of the ontology, focusing on endurants; UFO-B is concerned with events; and UFO-C deals with social and intentional concepts [Guizzardi 2005]. Each KIPO concept is founded on one of the UFO constructs, which are formally defined in terms of meta-properties (sortability and relational dependency, among others). In short, KIPO argues that a KIP execution is driven by the agent intentions towards achieving the process objectives, and that the flow of activities (especially decision-making) within a KIP execution is deeply influenced by tacit elements from its stakeholders, such as Beliefs, Desires, Intentions, and Perceptions [Cardoso et al . 2010; Rao and Georgeff 1995]. KIPO is structured into 5 sub-ontologies, which reflect the main perspectives that characterize a KIP. Business Process Ontology (BPO) comprises elements encompassed within traditional business processes (such as activities, event flows, input/output data objects), which describe traditional parts of a KIP and serve as the basis from which specific KIP elements are specialized and enriched. Collaboration Ontology (CO) depicts concepts that explain how knowledge artifacts are exchanged between process participants, and how collaboration takes place. Decision Ontology (DO) aims to analyze the rationale of the decisions made by the process agents (i.e., the “why” and “how” decisions were made by the people involved in the process), thus allowing the motivation and outcome of each decision to be tracked. Business Rules Ontology (BRO) provides the means to describe some parts of the KIP from a declarative perspective, since describing the rules that govern a KIP execution is especially useful for describing parts of the process which are very flexible and not subject to predefined event flows. Finally, Knowledge-Intensive Process Core Ontology (KIPCO) comprises the core concepts of a KIP, mainly agents, knowledge-intensive activities, and contextual elements involved in their execution. The case study of this paper analyzes the impact of collaborative issues within a KIP execution. These issues reflect the high degree of interaction and knowledge exchange among agents, and how process evolution takes place along its timeline. The loss of this information decreases the awareness of when and how a collective action is performed, thus preventing any common understanding and effective collaboration between agents, as well as blocking adequate monitoring and improvement of a KIP. 3.2 PPINOT Ontology (PPINOT) PPINOT Ontology [del Río-Ortega et al . 2010] (hereinafter referred to as PPINOT) is an ontology for the definition of PPIs with a number of features. It has high expressiveness. It enables the definition of PPIs in a precise and unambiguous manner, thus allowing their automated processing in the different activities of the process lifecycle, including their computation and analysis. It also facilitates traceability between the business process elements and PPIs, because PPIs must be explicitly connected to business process elements during their definition, thus preventing inconsistencies and promoting their co-evolution [del Río-Ortega et al . 2017a, 2013]. The business process elements considered are those presented in an abstract business process modeling language, so that PPINOT can be integrated with any specific notation used to model structured business processes. In PPINOT, a PPI is related to a business process and it is defined by means of the following attributes: id and name , to identify the PPI; goals , indicating the relevance of the PPI; scope , which defines the subset of instances to be considered during the PPI calculation; a target value to be reached; a measure definition that specifies how the PPI is computed; the person responsible for the PPI, and people informed about it. For instance, the main attributes of the PPI “Average time spent in solving an incident" could take the following values. Goal : “to reduce the average time required to solve an incident"; scope : “all instances (all incidents) solved in a period of time (e.g., one month”); target : “less than 6 hours”; measure definition : “it is calculated as the total time from the moment when an incident is received, to the final resolution of the incident (successful or not)”; responsible: “the worker that solves the incident", and informed: “the area manager". PPINOT distinguishes between three different types of measure definitions, namely: base measures, which represent a measure definition over a single process instance; aggregated measures, which are defined by aggregating one base measure over several process instances using an aggregation function, such as sum or average; and derived measures, which represent single-instance or multi-instance measures whose value is obtained by calculating a mathematical function over other measures. Furthermore, PPINOT defines four types of base measures: Time measure measures the duration of time between two instants; Count measure measures the number of times something happens; State condition measures whether the process instance is in a certain state; and Data measure measures the value of a certain part of a data object. The first three types of base measures are connected with business process elements by means of Conditions. Time and count measures are connected using TimeInstantConditions. which represent time instants. The former uses it to indicate the starting and ending of a period of time to be considered in the calculation of a measure. The latter uses it to define what to count. State conditions are connected using StateCondition to specify the state that a business process element must achieve. Finally, Data measures are connected to data objects using DataContentSelection [del Río-Ortega et al. 2013]. 4 ONTOLOGY INTEGRATION: KIPPINOT KiPPINOT is the ontology we propose in this paper to define PPIs in the context of KIPs. The merging of pre-existing ontologies is no trivial task. An in-depth analysis of each involved ontology is required to identify potential overlaps or joint concepts between them, or to determine new concepts that should be included with the aim of providing new functionalities for the resulting ontology. In addition, it is not only a question of fitting certain pre-existing concepts with others (for example, the PPINOT Goal concept corresponds to the KIPO:ProcessGoal concept), but also of analyzing which KIP elements can be measured and which types of measures can be applied. Methodologies such as [Ganter and Stumme 2003;Stumme and Maedche 2001] and [Suárez-Figueroa et al. 2012] have been proposed for the merging of ontologies. KiPPINOT has been built on the basis of KIPO and PPINOT (see Section 3) following the NeOn Methodology [Suárez-Figueroa et al . 2012]. This methodology provides various scenarios for the integration of ontologies. In order to build KiPPINOT, we followed scenario 6, “Reusing, merging, and re-engineering ontological resources", which is focused on merging two or more ontologies to build an ontology network. Specifically, we performed the activities described below. (1) Ontology search and assessment: The first activity involves searching existing ontologies that can be used for our purpose. Although there are numerous options for defining and managing PPIs and KIPs separately, none is able to measure performance in the context of KIP. Therefore, we opted to integrate two existing ontologies. PPINOT was selected because it allows us to define PPIs in an unambiguous and complete way, it has high expressiveness, it facilitates traceability with the business process, and is independent of the language used to model the business process [del Río-Ortega et al . 2013]. The ontology known as KIPO has been selected because it is a complete ontology that comprises concepts and relationships of KIPs by means of its 5 sub-ontologies, which reflect the main perspectives that characterize KIPs; and also because it was based on the Unified Foundational Ontology (UFO). Both proposals have been previously used and validated in real scenarios, which makes them good options for the integration. (2) Ontology comparison: An analysis is required to find similarities between concepts of the two ontologies, which represent joint points. This is described in Section 4.1. Fig. 1. KiPPINOT-CORE: Core of the ontology. The figure shows the relationship between a PPI, its attributes (measure definition, process instance filter, goal, etc,) and how KIPO elements are related to them, with the aim of extending the range of application of a PPI. (3) Selection and Integration: Since PPINOT measures performance of structured business processes, not all KIPO concepts can be measured “as is" with its measures. Section 4.2 analyzes how PPINOT measures can measure KIPO concepts and also identifies new measures, their behavior, restrictions, and characteristics. (4) Ontologies aligning and merging: This activity is described in Section 4.3 and involved three main steps: (i) Extending the application range of current PPINOT measures; (ii) describing how a new PPINOT measure can help to measure KIPO concepts; and (iii) adding new elements into KiPPINOT to allow the definition of measures and a set of rules to be adopted, derived from the PPI value evaluation. KiPPINOT is presented in Figures 1,2and 3as UML diagrams. White classes represent original PPINOT elements. White classes with underlined names represent business process elements involved in the definition of PPIs: Process, DataObject and BPElement (activities, events, etc.). Dark gray classes with white letters are original KIPO concepts. Light gray classes represent new elements that do not come from the ontologies involved, but are required to connect other components or they have a new functionality in the ontology. 4.1 Comparison of Ontologies The goal of this activity is to identify possible overlap or joint points between concepts of the two ontologies. In this case, we have found the following similarities. • Process: In PPINOT, a PPI is related to a structured business Process . In KiPPINOT, the Process class is generalized by distinguishing between two types of processes where a PPI can be defined: traditional structured processes using the BusinessProcess class, and KIPs using the KIPO:KnowledgeIntensiveProcess class. • Goal: In PPINOT, the Goal is an attribute that indicates the relevance of the PPI and KIPO defines two types of goals. In KiPPINOT, Goal is extended as a class to consider the two Fig. 2. KiPPINOT-MEASURES. The figure describes in detail the complete set of measures that the ontology uses to define how a PPI needs to be calculated. A measure is related to a specific type of condition and each condition is applied over a BPElement or KIPO element. Fig. 3. KiPPINOT-ELEMENTS. The figure shows two hierarchies of concepts on which conditions are defined: KnowledgeIntensiveConcepts related to measures of time, cost and state condition and DataConcept used by DataMeasure and CountAttributeMeasure to take information from attributes of concepts. types of goals: goals related to processes, KIPO:ProcessGoal ; and goals related to activities involved in PPI definitions, KIPO:ActivityGoal. • Restrictions: In PPINOT, the Target attribute can be seen as a restriction that indicates the PPI value to be reached, while KIPO defines different types of restrictions. In KiPPINOT, target is replaced by the RestrictionOnMeasure class that agglutinates the target value and the KIPO:IntegrityRule , which is a restriction that must be true in order to achieve a goal. Fig. 8. The MPG-K Methodology workflow. solutions. Lag PPIs can also be defined based on the PPIs that are defined together with reference processes in reference frameworks developed by the industry for specific domains, such as SCOR for supply chains and ITIL for IT management. (2) Use KiPPINOT to define a set of KIP measures that are useful in the identification of different ways of executing a KIP. Alternatives of execution are common in KIPs due to unpredictable events and decisions made by users [Di Ciccio et al . 2015]. This step proposes searching for the variability in the execution of a process and determining how to measure it. For example, the communication between two participants can be carried out by means of an email or a phonecall. The way this variability can be found differs little in the way a process can be discovered. The main difference is that instead of discovering the whole process, in this case we are just interested in finding out the differences between the ways process participants perform the process. Therefore, the three classes of discovery techniques described in [Dumas et al . 2013], namely evidence-based, interview-based, workshop-based, or a combination of these techniques can be applied in this step. Evidence-based techniques are based on exploiting evidence such as process documentation, direct observation of process participants, and information system event logs for the identification of this variability. If event logs are used, one can use process mining techniques designed for non-structured processes, such as [Maggi et al . 2012], or those that discover configurable process models [Buijs et al . 2013]. Interviewbased and workshop-based techniques refer to gathering information by interviewing process participants or organizing workshops with process stakeholders, respectively. Regardless of the technique, since in a KIP the process variability can be significant, one good approach is to focus on good and bad performers in terms of the lag PPIs defined in the previous step, and then to try to find the differences between them. (3) Compute the set of lag PPIs and the KIP measures and discard those that cannot be computed. The use of KiPPINOT to define the lag PPIs and the KIP measures provide precise and unambiguous definitions. From these definitions, the computation of these PPIs and measures can be carried out in different ways. If an event log of the information system that supports the KIP is available, it is possible to use the KiPPINOT definition to directly automate its computation [del Río-Ortega et al . 2017b]. A proof-of-concept of this automation is available at the PPINOT Tool Suite 1 . If the event log is not available, then other mechanisms can be used. For instance, the lag PPIs and the KIP measures can be translated into SQL queries that are executed against the database of the information system. If there is no information system that supports the KIP or that contains the information necessary for the computation of the PPI or the KIP measure, then the process participants can be asked to collect this information [McChesney et al . 2012]. Finally, if it is not possible to gather the information required to compute a PPI or a KIP measure, or the effort necessary to gather this information is too great, then the lag PPI or KIP measure must be discarded. 1http://www.isa.us.es/ppinot (4) Perform data analysis to find correlations between them. We propose the use of correlation as the mechanism to identify relationships between the KIP measures and the lag PPIs proposed. Correlation has been widely used in the literature for similar purposes. For instance, Richard et al. [Richard et al . 2009] present a number of studies related to organizational performance measures that rely on correlations to empirically find similarities between indicators and suggest that these indicators may hold similar internal factors. In Grigori et al. [Grigori et al . 2004], correlation is used as a form of analysis of behaviors and taxonomies in Business Process Intelligence scenarios. However, correlation cannot be considered a sufficient condition to ensure the causal relationship between the KIP measures and the lag PPIs. Therefore, this correlation can be seen as a first filter that helps us to discard those KIP measures that cannot be used as lead PPIs. (5) Choose a subset of the measures that are correlated to a number of the lag PPIs as lead PPIs and define a target value for each measure. The set of KIP measures that are correlated to certain of the lag PPIs conform a set of potential lead PPIs. These potential lead PPIs must be reviewed by domain experts to analyze whether the relationship between the potential lead PPI and the lag PPI makes sense from a domain perspective. If there is an agreement that the relationship does not make sense, then the potential lead PPI should be discarded. It may also happen that there is not enough data to find correlations between a KIP measure and a lag PPI. For instance, this could be the case where the behavior we are measuring using the KIP measure is very unusual among the process participants. In this case, the KIP measure could be added as a lead PPI if there is an agreement between the domain experts that the KIP measure can be a suitable lead PPI. Once lead PPIs have been identified, a target value must be defined for them. To this end, one can use simple statistical metrics such as mean, median, and mode, or more elaborate techniques that have been proposed for the identification of thresholds in the software engineering area [Herbold et al . 2011] or in the business process area [del Río-Ortega et al. 2017a]. (6) Deploy the lead PPIs as performance goals to guide the behavior of process participants during KIP execution. The deployment of lead PPIs includes three aspects: first, the process participants must be fully aware of the performance goals defined and they must agree with their usefulness; second, the value of the lead PPIs must be continuously monitored (e.g. following the techniques discussed in Step 3) so that process participants know whether they are fulfilling the goals or not; and third, the fulfillment of the lead PPIs must be reviewed periodically so that it creates a cadence of responsibility between the process participants [McChesney et al . 2012]. The specific mechanisms for the implementation of these three aspects are organization-specific. (7) Continuously monitor whether fulfilling the lead indicators is moving the related lag indicators towards the desired goals. If not, then repeat from Step (2). For this continuous monitoring, the use of dashboards facilitates the information analysis and the specification of alerts depending on obtained and expected PPI values. In addition, an appropriate time-window should be defined for the evaluation period before considering whether PPIs are working or not. The size of the time-window depends heavily on the domain. Finally, the same techniques as in Step 3 can be used to compute both lead and lag PPIs. This step is included to account for the fact that the lead indicators that have been selected may turn out not to be predictive of the lag indicator (this could happen because correlation does not imply causation) or that something in the context changes that stops the lead indicator from being predictive of the lag indicator. In this case, a new set of lead indicators should be defined. 6 CASE STUDY The goal of this section is to understand the impact of the proposal, both of the KiPPINOT ontology and the MPG-K methodology, in a real scenario. Specifically, the following two research questions were assessed. The first question is related to the KiPPINOT ontology and the second is related to the methodology. • RQ1: How does the ontology proposed help towards improving the definition and monitoring of PPIs found in real KIP scenarios? • RQ2: How does the proposed methodology help to make evidence-based decisions for process improvement? Our objective is to demonstrate that the proposed methodology works and how it can be used in a real scenario. The case study was conducted in a real-life scenario of a Brazilian ICT outsourcing company, which has approximately a hundred contracts with various firms to provide ICT support. One of their main business processes is to solve incidents related to clients’ ICT assets, such as email server outages or network connection problems. This kind of work involves the application of technical skills, troubleshooting abilities, collaboration, and information exchange between different teams including the client. Moreover, ad-hoc decisions may be made since most of the problems are situational and, despite a number of recurring problems, there is no structured process to be followed. All these points characterize KIP aspects in such way that it would be more appropriate to manage this process as being knowledge intensive, instead of a traditional control-flow-oriented business process. The company periodically evaluates the performance of their processes and then splits the results for each client with an active contract, to generate client reports and take actions according to their service level agreements; in this way, all indicators are aggregated to the client level. The following sections describe our real scenario: an incident-troubleshooting process within an ICT Outsourcing Company. They also show how each step of the methodology is applied in this scenario. 6.1 Incident-Troubleshooting Process Characterization In order to represent the mainstream behavior of the process, Case Management Model and Notation (CMMN) was employed since problem resolution in call centers is a known application of Case Management [(OMG) 2014]. The process model is presented in Figure 9. As incident tickets can be viewed as Cases, CMMN offers sufficient flexibility to present the major case plan of the incidenttroubleshooting process and abstracts lower-level tasks that are too technical to be considered on a high-level process view. The process starts when a client informs, either via email or telephone, the ICT support team about a problem; the support team then opens an incident ticket, where all initial information about the problem is registered. A new stage then starts where a technical team is assigned and/or contact with the client is made in order to obtain more details about the incident. The next step is the troubleshooting itself, where the assigned technical team works with freedom to make decisions and to employ any action that is allowed by the company, to solve the problem. In this step, the client usually collaborates and gives feedback to the support team. Due to the many possible combinations of actions to be taken during troubleshooting, this activity is represented in CMMN as a manual collapsed stage with repetitions. If the problem remains unsolved, then another technical team can be assigned and more information can be extracted from the client prior to a new troubleshooting round. On the occurance of successful troubleshooting, the process ends with ticket closure and the case is terminated. Fig. 9. High-level CMMN model with mainstream behavior of the process. 6.2 Existing Implemented PPIs (Lag Indicators) The first step of our methodology consists of defining a set of lag PPIs that defines the expected performance of the KIP. In the performance monitoring of the KIP at hand, the following four lag PPIs are tracked. The set of lag PPIs has been modeled using the KiPPINOT ontology. For further details, see Figures 12,13,14, and 15 in the Appendix. • Average work time spent per ticket (Work Time AVG): every member of the technical teams reports how much time he or she has spent working on a ticket. This PPI allows those clients who demand more technical team effort, measured in terms of time spent, to be verified, thereby identifying those clients with a more expensive service than the average. • Average duration of tickets (Duration AVG): this measures the time from a ticket opening to its closure. This PPI considers waiting times and delays that may occur during the resolution of a ticket, and not only the effective working time of technical teams. From the customer’s point of view, it measures the time since the customer made the first contact until the final message formalizing the ticket closure. • First contact resolution ratio (First Cont. Res. %): this measures the ratio of tickets that were solved directly in the first contact with the support teams. This behavior is desired, since it implies less work time spent on the resolution of a ticket than regular tickets, thereby reflecting less costly services than the average. • Total tickets opened (Total Tickets): this PPI measures the total amount of tickets opened for each client, given a period of time. It helps management quantify resources for technical teams and negotiate with client contract values. 6.3 KIP Indicators Proposal (Lead Indicators) Following the second step (Define KIP measures) of the methodology, four KIP measures based on the concepts of the Collaboration Ontology of KIPO are proposed. These indicators are all derived from Moura et al. [Moura et al . 2015], where a collection of axioms were defined to specify which collaborative activities are present in a KIP. They were defined so that they represent different aspects of the variability that is present in collaborations. Furthermore, interview-based techniques were used to select the KIP measures that were more relevant in the determination of how a collaboration takes place. Finally, these KIP measures are good candidates for lead indicators, since they can be directly influenced by process participants. Figure 10 shows a KIPO instance of the process, where it is possible to identify the concepts related to the following proposed indicators. The set of lead PPIs has also been modeled using KiPPINOT ontology. For further details see Figures 16,17,18, and 19 in the Appendix. • Average amount of interlocutors per case (Interlocutor AVG): this denotes how many agents executed Communicative Acts during a Communicative Interaction. The information system stores all incoming and outgoing messages for each ticket registered. In this way, it is possible Fig. 10. KIPO instance of the Incident-Troubleshooting Knowledge-Intensive Process. to count how many agents exchanged messages during a troubleshooting session. A high number of agents involved in solving a problem may point towards complex problems and to cost-related issues because, with the increase of people involved, the costs are also likely to increase. • Average amount of messages exchanged per case (Messages Exch AVG): interlocutors may communicate by observing any of the following relations: one-to-one, one-to-many, and many-to-many. In order to provide means to numerically evaluate this statement, we propose counting the number of messages exchanged during each ticket resolution. This can help the company to figure out whether a higher number of messages may result in complex situations and longer duration times, since each message usually denotes some kind of analysis or action taken by an interlocutor. • Average message size per case (Message Size AVG):messageSize is an attribute of Message and denotes the extent of information present in a Message. The bigger the message is, the longer it takes to understand it, and the later the troubleshooting will start. Higher message sizes may point to verbose clients who may need some special attention, or to problems that turned complex and will therefore take more time and effort than expected. For the scope of this work, messageSize is defined as the count of characters contained in a message. • Incoming customer phone-call ratio (Phone Call Ratio): all Messages are associated with Communicative Acts, which are their Propositional Content. Audio, video and text are proposed as types of Communication Language, which is a property of a KIP message [Moura et al . 2015]. For the scope of this case study, there are no video messages involved, although text messages can be retrieved by incoming emails and audio messages by phone calls made by customers and registered in the information system of the company. This indicator may help managers to identify clients who use the audio channel most, which is a more expensive channel than that of text. 6.4 Computation and analysis of measures and indicators In Step three, the indicators are calculated. Since the company information system is not processaware, the dataset for computing the indicators was extracted by directly querying the system database. The extracted data was preprocessed by applying a data selection step to filter the incident tickets opened in a pre-defined period of analysis (the second semester of 2015) and were already solved at the time of data extraction (with a closed status); moreover, we only considered messages originating from human resources (that is, automatic messages generated by the system were discarded, since they do not involve any human effort and knowledge to be sent). The data frame analyzed comprises a total of 8,432 incident tickets, and a total of 228 distinct clients. The company experts explained that the number of clients seemed higher than the number Fig. 11. Pearson correlation coefficient of Lead (KIP indicators) and Lag (existing PPIs) indicators of contracts because on some tickets, the client name is incorrectly written with the individual email of client’s contact person, resulting in noisy information in the database. They asked only clients with the highest amounts of tickets opened to be considered, since they hold the most expensive contracts. Therefore, only the top-40 clients with the highest quantity of tickets opened in the period, which corresponded to 80% (6,781 tickets) of all tickets retrieved, were analyzed. Existing (lag) and our proposed (lead) PPIs were calculated on the basis of this information. Following the fourth step of our methodology, we considered the description of the strength of correlations using the guide proposed in [Evans 1996] for absolute values of the coefficients as 0.00-0.19: “very weak”; 0.20-0.39: “weak”; 0.40-0.59: “moderate”; 0.60-0.79: “strong”; 0.80-1.00: “very strong”. Both their existing (lag) and our proposed (lead) PPIs were compared by calculating Pearson’s correlation coefficient between each pair of indicators. Figure 11 shows the Pearson correlation coefficient for a pairwise combination of all PPIs. In order to verify whether the identified correlations represent possible influences of KIP lead indicators over existing lag PPIs, the indicators and correlation results were discussed with two company managers, who are domain experts and directly involved in the process. Furthermore, the analysis of lead-lag indicators, and correlations between lead-lead and lag-lag PPIs were calculated for a better understanding of the behavior of the existing PPIs and the proposed PPIs. As an initial suggestion of target values (Step 5) the global average was used in order to provide a benchmark from among all the indicators calculated at clients level. The following interpretations were considered relevant by the experts: • A very strong positive correlation (0.92) between Messages Exch AVG and Interlocutor AVG. This makes sense, since an increase in the people involved is likely to result in an increase of the messages exchanged. • A moderate negative correlation from Messages Exch AVG to Phone Call Ratio (-0.51). This may be explained by the fact that, during a single phone call, more information is exchanged, and sometimes not recorded, than in a single mail message. During a call, one can ask several things and immediately obtain answers and then ask again based on these answers. • A moderate negative correlation from Messages Exch AVG to First Cont. Res % (-0.52). Since the problem is solved in the first contact, there is no need to extra messages to be exchanged. • A moderate negative correlation between Interlocutor AVG and First Cont. Res % (-0.44). Since the problem is solved in the first contact, there is no need for extra people to become involved. • A moderate negative correlation between Phone Call Ratio and Duration AVG (-0.41). The experts agreed that customers whose companies communicate most by phone take less time to have their problems solved than those who communicate by email. They confirmed this behavior by citing two example clients who have dedicated technicians and the company always calls them directly. This specific contract variant bypasses the first level support contact and does not rely on email messages exchanged. In a scenario where the global average among all clients of Phone Call Ratio is 32.45% and Duration AVG is 84.00 hours, clients with a Phone Call Ratio of 78.23% and 91.26% presented a Duration AVG of 39.79 and 19.47 hours respectively. • A moderate positive correlation between Phone Call Ratio and First Cont. Res % (0.53). Since more information can be exchanged during a call than during an email, it is likely that the chance to solve non-complex problems arises in the first contact. In addition, technical teams can solve a problem more efficiently by calling the client or by directly troubleshooting the problem during a client’s first call than by sending emails. Looking at the client with the highest First Cont. Res % score, 86.23%, the experts recognized the same behavior for Phone Call Ratio as that explained above for Duration AVG, where in the situations when the client has direct phone contact to a specific technician, incidents are more likely to take less time to be solved. In the example of the client with the highest First Cont. Res % score, the Phone Call Ratio is 91.26%. • Total Tickets and Work Time AVG indicators presented only weak or very weak correlations. This may indicate that the teams have enough resources to deal with the amount of tickets or there is an opportunity to reduce costs by downsizing teams. 6.5 Deployment and continuous monitoring After having analyzed the correlations between lag and lead indicators, experts found advantages in a phone-call-oriented service desk in comparison to an email-oriented service desk. They reported that, despite the high costs of the former, it caused the reduction in the duration to solve and close tickets and the increase in first-contact resolutions, which are significant indicators of improvements in client satisfaction and retention. From the perspective of lead and lag indicators, the set of lead PPIs of Interlocutor AVG, Phone Call Ratio and Messages Exch AVG can be monitored and used to promote actions that may improve the results of the set of lag PPIs of First Cont. Res % and Duration AVG. Since the implementation of the lead indicators depends only on integrating database queries to the existing dashboard of company indicators, they can be operationalized in practice to help guide the behavior of the process participants. Generating actions to improve call-center scripts to be more assertive, training the first-level support team to deal with problems of a more complex nature, and reducing the number of interlocutors and messages exchanged, all provide examples of this (Step 6). The last step of the methodology can be exemplified by lead PPIs that have not been predictive to any lag indicator, for example, the Message Size AVG PPI, which did not contribute towards explaining the behavior of the lag PPIs analyzed. In this case, this PPI should be discarded, and in a new round of definition and revision of lead indicators for lag indicators, new KIP measures should be analyzed together with the previously selected ones. 6.6 Case Study Results Discussion The conduct of the case study and the results obtained allow us to draw conclusions related to the research questions posed in Section 6.1. With regard to RQ1, our proposed KiPPINOT ontology did provide the necessary elements and concepts to define, implement, and monitor collaborationrelated performance indicators found in a real KIP scenario. KiPPINOT inherited its characteristic of precision from PPINOT and allows the definition of PPIs in an unambiguous and precise way in a KIP scenario, it also provides the possibility of automated processing of PPI values in lifecycle phases such as computation and analysis. Finally, with regards to RQ2, we collected several pieces of evidence, perceived by experts, that showed the significance of the proposed methodology in helping them monitor lag indicators with the support of quantitative lead indicators. The experts had a feeling about this behavior, but had never been presented with data that supported their theory until the execution of this case study. The previous practice applied by the company was to monitor the existing four lag indicators without the support of quantitative lead indicators that were capable of considering KIP aspects of the process. In this way, after the analysis of the lag indicators, the process managers then based their decisions on personal observations and intuitive estimations of possible causes of undesired performance. Two limitations have been identified. First, due to the type of information registered by the company, it was not possible to define lead PPIs related to concepts such as goals and risks, because they could not be computed. Hence, we only considered communication issues in lead PPIs. These other aspects of KIPs, therefore, could not be evaluated in the case study. Finally, the application of the methodology in only one scenario constitutes the second limitation of our study. However, it also provides an opportunity to conduct future work with a long-term period of evaluation and to apply the methodology in several companies from different domains. This would allow us to compare results of several scenarios. It is also worth noting that the purpose of this paper is not to come up with new conclusions concerning the case study, but to demonstrate that the proposed methodology works. Furthermore, the advantage of this approach is that it validates this intuition with actual data, and hence enables managers to make evidence-based decisions instead of “gut”-based decisions. 7 CONCLUSIONS In this paper we propose a new mechanism to define PPIs in KIPs. Our contribution in this regard is twofold. First, we present the KiPPINOT ontology, which is the result of the alignment and integration of two existing ontologies, KIPO and PPINOT. This allows the definition of both traditional indicators, also present in structured business processes, and those specific to KIPs. The main contribution of this new ontology involves ascertaining which elements of a KIP can be measured and which type of measures can be applied to these processes (Tables 1and 2). This provides highly useful knowledge for other researchers and practitioners that want to develop their own models or ontologies to define PPIs. As advantage, KiPPINOT can be applied to both structured processes and KiPs, as well as to scenarios for which both kinds of processes can be found, such as in healthcare [Lenz and Reichert 2007]. Secondly, we propose a new methodology (the MPG-K) that builds on the KiPPINOT ontology and is based on the concepts of lead and lag indicators. This methodology provides process participants with actionable guidelines that assist them in the KIP execution in order to comply with the established performance goals. The usefulness of these two new artifacts, the ontology and the methodology, has been validated through their application in a real scenario in a Brazilian company. The insights provided by our approach were considered highly valuable by the company, which is already taking them into consideration to implement changes in the process and its execution. As a direction for future work, we plan to extend KiPPINOT to measure KIP concepts beyond the scope of the current proposal (e.g. beliefs and desires). We would also like to carry out a long-term evaluation in various scenarios in order to compare their results; and we would also consider the development of tools for the automation in the calculation and evaluation of PPIs in KIPs. REFERENCES Peter C. Brewer and Thomas W. Speh. 2000. Using the balance scorecard to measure supply chain performance. J. Bus. Logist. 21, 1 (2000), 75–93. Joos C. A. M. Buijs, Boudewijn F. van Dongen, and Wil M. P. van der Aalst. 2013. Mining Configurable Process Models from Collections of Event Logs. Berlin, Heidelberg, 33–48. Evellin C. S. Cardoso, Paulo Sérgio Santos Jr., João Paulo Andrade Almeida, Renata S. S. Guizzardi, and Giancarlo Guizzardi. 2010. Semantic integration of goal and business process modeling. In CONFENIS. Felix Tung Sun Chan. 2003. Performance measurement in a supply chain. The International Journal of Advanced Manufacturing Technology 21, 7 (2003), 534–548. Mu-Yen Chen, Mu-Jung Huang, and Yu-Chen Cheng. 2009. Measuring knowledge management performance using a competitive perspective: An empirical study. Expert Syst. Appl. 36, 4 (2009), 8449 – 8459. Claire Costello and Owen Molloy. 2009. Building a Process Performance Model for Business Activity Monitoring. In Information Systems Development, Challenges in Practice, Theory, and Education V1. 237–248. Francisco M. del-Rey-Chamorro, Rajkumar Roy, Bert van Wegen, and Andy Steele. 2003. A framework to create key performance indicators for knowledge management. J. Knowl. Manag. 7, 2 (2003), 46–62. Adela del Río-Ortega, Félix García, Manuel Resinas, Elmar Weber, Francisco Ruiz, and Antonio Ruiz-Cortés. 2017a. Enriching Decision Making with Data-Based Thresholds of Process-Related KPIs. In CAiSE Conference 2017. 193–209. Adela del Río-Ortega, Manuel Resinas, Cristina Cabanillas, and Antonio Ruiz-Cortés. 2013. On the definition and design-time analysis of process performance indicators. Inf. Syst. 38, 4 (2013), 470 – 490. Adela del Río-Ortega, Manuel Resinas, Amador Durán, Beatriz Bernárdez, Antonio Ruiz-Cortés, and Miguel Toro. 2017b. Visual PPINOT: A Graphical Notation for Process Performance Indicators. Bus. Inf. Syst. Eng. (2017). Adela del Río-Ortega, Manuel Resinas, and Antonio Ruiz-Cortés. 2010. Defining Process Performance Indicators: An Ontological Approach. In Confederated International Conferences: CoopIS, IS, DOA and ODBASE (2010). 555–572. Claudio Di Ciccio, Andrea Marrella, and Alessandro Russo. 2015. Knowledge-intensive processes: Characteristics, requirements and analysis of contemporary approaches. J. Data Semant. 4, 1 (2015), 29–57. Marlon Dumas, Marcello La Rosa, Jan Mendling, and Hajo A Reijers. 2013. Fundamentals of business process management. Vol. 1. Springer-Verlag Berlin Heidelberg. James D. Evans. 1996. Straightforward Statistics for the Behavioral Sciences. Brooks/Cole Publishing Company. Bernhard Ganter and Gerd Stumme. 2003. Creation and Merging of Ontology Top-Levels. In Conceptual Structures for Knowledge Creation and Communication. 131–145. Oscar González, Rubby Casallas, and Dirk Deridder. 2009. MMC-BPM: A Domain-Specific Language for Business Processes Analysis. In Business Information Systems Conference (BIS 2009), Vol. 21. 157–168. Daniela Grigori, Fabio Casati, Malu Castellanos, Umeshwar Dayal, Mehmet Sayal, and Ming-Chien Shan. 2004. Business Process Intelligence. Comput. Ind. 53, 3 (2004), 321 – 343. Giancarlo Guizzardi. 2005. Ontological foundations for structural conceptual models. Ph.D. Dissertation. Giancarlo Guizzardi, Ricardo de Almeida Falbo, and Renata SS Guizzardi. 2008. Grounding Software Domain Ontologies in the Unified Foundational Ontology (UFO): The case of the ODE Software Process Ontology. In Workshop Requirements Engineering and Software Environments (IDEAS 2008). 127–140. Kaisa Henttonen, Aino Kianto, and Paavo Ritala. 2016. Knowledge sharing and individual work performance: an empirical study of a public sector organisation. J. Knowl. Manag. 20, 4 (2016), 749–768. Steffen Herbold, Jens Grabowski, and Stephan Waack. 2011. Calculation and optimization of thresholds for sets of software metrics. Empirical Software Engineering 16, 6 (2011), 812–841. Richard Hull, Agnes Koschmider, Hajo A. Reijers, and William Wong. 2016. Fresh Approaches to Business Process Modeling (Dagstuhl Seminar 16191). Dagstuhl Reports 6, 5 (2016), 1–30. Öykü Isik, Willem Mertens, and Joachim Van den Bergh. 2013. Practices of knowledge intensive process management: quantitative insights. Bus. Process Manag. J. 19, 3 (2013), 515–534. Robert S. Kaplan and David P. Norton. 1992. The balanced scorecard: Measures that drive performance. Harvard Business Review 70, 1 (1992), 71–79. Birgit Korherr and Beate List. 2007. Extending the EPC and the BPMN with Business Process Goals and Performance Measures. In Int. Conf. on Enterprise Information Systems (ICEIS 2007). 287–294. Elfriede Krauth, Hans Moonen, Viara Popova, and Martijn C. Schut. 2005. Performance Measurement and Control in Logistics Service Providing. In Int. Conf. on Enterprise Information Systems. 239–247. Peter Kueng. 2000. Process performance measurement system: A tool to support process-based organizations. Total Quality Management 11, 1 (2000), 67–85. Kun Chang Lee, Sangjae Lee, and In Won Kang. 2005. KMPI: measuring knowledge management performance. Inf. Manag. 42, 3 (2005), 469–482. Richard Lenz and Manfred Reichert. 2007. IT support for healthcare processes âĂŞ premises, challenges, perspectives. Data Knowl. Eng. 61, 1 (2007), 39 – 58. Todd A. Little and Amit V. Deokar. 2011. Role of Knowledge Asset Indicators in Understanding Knowledge-Intensive Processes. In Americas Conference on Information Systems (AMCIS 2011). Todd A. Little and Amit V. Deokar. 2016. Understanding knowledge creation in the context of knowledge-intensive business processes. J. Knowl. Manag. 20, 5 (2016), 858–879. Fabrizio M. Maggi, R. P. Jagadeesh Chandra Bose, and Wil M. P. van der Aalst. 2012. Efficient Discovery of Understandable Declarative Process Models from Event Logs. In CAiSE Conference. 270–285. Olivera Marjanovic and Ronald Freeze. 2011. Knowledge Intensive Business Processes: Theoretical Foundations and Research Challenges. In Hawaii International Conference on Systems Science. 1–10. Anne P. Massey, Mitzi M. Montoya-Weiss, and Tony M. O’Driscoll. 2002. Performance-Centered Design of Knowledge- Intensive Processes. J. Manag. Inf. Syst. 18, 4 (2002), 37–58. Chris McChesney, Sean Covey, and Jim Huling. 2012. The 4 disciplines of execution: Achieving your wildly important goals. Simon and Schuster. Christof Momm, Robert Malec, and Sebastian Abeck. 2007. Towards a Model-driven Development of Monitored Processes. In Proceeding of Internationale Tagung Wirtschaftsinformatik (WI 2007). 319–336. Ednilson Veloso Moura, Flávia Maria Santoro, and Fernanda Araujo Baião. 2015. XCuteKIP: Support for Knowledge Intensive Process Activities. In Proceedings of CRIWG 2015. 164–180. Andy Neely, Huw Richards, John Mills, Ken Platts, and Mike Bourne. 1997. Designing performance measures: a structured approach. Int. J. Oper. Prod. Manag. 17, 11 (1997), 1131–1152. Natalya F. Noy and Deborah L. McGuinness. 2001. Ontology development 101: A guide to creating your first ontology. Technical Report. Object Management Group (OMG). 2014. Case Management Model and Notation, Version 1.0. Oluwafemi Oyemomi, Shaofeng Liu, Irina Neaga, and Ali Alkhuraiji. 2016. How knowledge sharing and business process contribute to organizational performance: Using the fsQCA approach. J. Bus. Res. 69, 11 (2016), 5222–5227. Carlos Pedrinaci, Dave Lambert, Branimir Wetzstein, Tammo van Lessen, Luchesar Cekov, and Marin Dimitrov. 2008. SENTINEL: A Semantic Business Process Monitoring Tool. In Proceedings of International Workshop on Ontology-supported Business Intelligence 2008. 26–30. Viara Popova and Alexei Sharpanskykh. 2010. Modeling organizational performance indicators. Inf. Syst. 35, 4 (2010), 505–527. Alexander Rannacher, Robert Stranzenbach, Flavius Sturm, Susanne Mütze-Niewöhner, and Christopher M Schlick. 2013. A System Dynamics Model for the Evaluation of the Productivity of Knowledge-intensive Services. iBusiness 5, 3B (2013), 55–58. Anand S. Rao and Michael P. Georgeff. 1995. BDI Agents: From Theory to Practice. In Proceedings of International Conference on Multi-Agent Systems (ICMAS-95). 312–319. Pierre J. Richard, Timothy M. Devinney, George S. Yip, and Gerry Johnson. 2009. Measuring Organizational Performance: Towards Methodological Best Practice. J. Manag. 35, 3 (2009), 718–804. Cornelia Richter von Hagen, Dietmar Ratz, and Roman Povalej. 2005. Towards Self-Organizing Knowledge Intensive Processes. J. Univers. Knowl. Manag. 0, 2 (2005), 148–169. Ann Rosenberg, Greg Chase, Rukhshaan Omar, James Taylor, and Mark von Rosing. 2010. Applying Real-World BPM in an SAP Environment. Galileo Press. Jorge Saldivar, Carla Vairetti, Carlos Rodríguez, Daniel Florian, Fabio Casati, and Rosa Alarcón. 2016. Analysis and improvement of business process models using spreadsheets. Inf. Syst. 57 (2016), 1 – 19. Juliana Baptista dos Santos França, Joanne Manhães Netto, Juliana do E.S.Carvalho, Flávia Maria Santoro, Fernanda Araujo Baião, and Mariano Pimentel. 2015. KIPO: the knowledge-intensive process ontology. Software & Systems Modeling 14, 3 (2015), 1127–1157. Gerd Stumme and Alexander Maedche. 2001. FCA-MERGE: Bottom-Up Merging of Ontologies. In IJCAI. Flavius Sturm, Alexander Rannacher, Lesya Straub-Bilan, Robert Stranzenbach, and Susanne Mütze-Niewöhner. 2011. Productivity measurement of knowledge-intensive services–Towards adaptable service performance evaluation systems. In Proceedings of International Conference RESER 2008. Mari Carmen Suárez-Figueroa, Asunción Gómez-Pérez, and Mariano Fernández-López. 2012. The NeOn Methodology for Ontology Engineering. In Ontology Engineering in a Networked World. 9–34. Ganesh Vaidyanathan. 2005. A Framework for Evaluating Third-party Logistics. Commun. ACM 48, 1 (2005), 89–94. Amy Van Looy and Aygun Shafagatova. 2016. Business process performance measurement: a structured literature review of indicators, measures and metrics. SpringerPlus 5, 1 (2016), 1797. Zhining Wang, Pratyush Nidhi Sharma, and Jinwei Cao. 2016. From knowledge sharing to firm performance: A predictive model comparison. J. Bus. Res. 69, 10 (2016), 4650–4658. Branimir Wetzstein, Zhilei Ma, and Frank Leymann. 2008. Towards Measuring Key Performance Indicators of Semantic Business Processes. In Business Information Systems (BIS 2008). 227–238.