Towards Modelling and Tracing Key Performance Indicators in Business Processes

Towards Modelling and Tracing Key Performance Indicators in Business Processes? Adela del-R´ıo-Ortega and Manuel Resinas Universidad de Sevilla, Spain Abstract. It is increasingly important to evaluate the performance of business processes. A key instrument in order to detect the state of current and completed processes, as well as to identify undesired behaviour, and suggest potential improvements are the key performance indicators (KPIs). The KPI lifecycle in the context of business process driven development comprises the definition, measuring, analysis and report phases. In this paper we analyse how some current proposals deal with these stages, concluding that none of them covers properly the entire cycle; we also identify the challenges which are to be faced to achieve this goal of evaluating business processes performance. 1 Introduction An important aspect in the business process lifecycle is the evaluation of business processes performance, since it helps organizations to define and measure progress towards their goals. Performance requirements on business processes are specified as Key Performance Indicators (KPIs) with target values which must be reached in a certain period. In order to define KPIs is recommended that they satisfy the SMART criteria [1]. SMART is an abbreviation for five characteristics of good KPIs: Specific (it has to be clear what the KPI exactly describes), Measurable (it has to be possible to measure a current value and to compare it to the target one), Achievable (it makes no sense to pursue a goal that will never be met), Relevant (it must be aligned with a part of the organisation’s strategy, something that really affects its performance) and Time-bounded (a KPI only has a meaning if it is known the time period in which it is measured). Target values for KPIs are usually specified in Service Level Agreements (SLAs) where provider and consumer arrange the expected service behaviour and its quality. Regarding the evaluation of KPIs we face several challenges. On the one hand there not exists any standard model to define such KPIs over business processes ?This work has been partially supported by the European Commission (FEDER), Spanish Government under the CICYT projects Web-Factories (TIN2006-00472), and SETI (TIN2009-07366); and project P07-TIC-2533 funded by the Andalusian local Government. Actas de los Talleres de las Jornadas de Ingeniería del Software y Bases de Datos, Vol. 3, No. 3, 2009 ISSN 1988-3455 SISTEDES, 2009 57 (defined for example in BPMN [2]). On the other hand there is no possibility to assure the traceability between these KPIs defined over business processes and their target values established in SLAs. In particular, this paper presents some proposals related to the different phases of the KPI lifecycle through a case study and outlines the next steps to follow to overcome the challenges previously introduced. The remainder of this paper is organised as follows. In Section 2 we present a case study. Then in Section 3 we propose a lifecycle for KPIs. Both of them will be used in Section 4 to explain some approaches and analyse them.In Section 5 we draw the conclusions from our analysis and explain the challenges we have identified. Section 6 summarizes the paper and outlines our future work. 2 Case Study We present a practical scenario developed in the context of the University of Seville. We focus on the business process within the scope of managing secondment for the teaching staff. With secondment we mean a permission requested to the university by the teaching staff in order to leave temporarily their workplace to take part on an activity to complement their training and/or research (e.g. to attend a conference), along with its funding. The process is depicted in Figure 1. It is modelled in BPMN and has three levels of abstraction. At the top level we can see the complete process, consisting of three subprocess and one task. First, the member of the teaching staff applies for her secondment approval to different superiors (the person in charge of the credit that will finance the costs, the department director, the director of the centre and, just in case of being an activity longer than 15 days, the chancellor); then she books the accommodation and transports; and after having performed the activity, she must file an application for settlement justifying the expenses of such activity, in order for the university to refund them. In the lower level we expand the first subprocess, where we divide the different superiors’ approvals in two subprocesses, and at the bottom level we detail the subprocess “Submit Secondment Application and Manage Approvals”. In this subprocess the requester must file an application with their personal and teaching data, and those related to the activity to be realized (destination, duration, etc.). This application is sent first to the department director, who will check the correctness of the teaching replacement during the secondment period, and then to the person in charge of the credit that will finance the costs, who will check the funds availability. This process ends with the secondment refusal or its partial approval. 3 KPI Lifecycle The KPI lifecycle in the context of business process driven development comprises the definition, measuring, analysis and report phases. During the definition phase, indicators that measure progress towards the achievement of certain goals Actas de los Talleres de las Jornadas de Ingeniería del Software y Bases de Datos, Vol. 3, No. 3, 2009 ISSN 1988-3455 SISTEDES, 2009 58 Fig. 1. Process of the secondment management Actas de los Talleres de las Jornadas de Ingeniería del Software y Bases de Datos, Vol. 3, No. 3, 2009 ISSN 1988-3455 SISTEDES, 2009 59 are identified and described, and target values for them are set; to this end a model or a notation is needed in order to set the value for the parameters that define a KPI and to express the way to measure the indicator. Within our case study, we can define KPIs for each of the levels. If we focus on the last subprocess, we obtain the indicators showed in Table 1. We can also define target values for these KPIs. For example we wish the duration to be less than 15 days or the percentage of refused secondments lower than 10 %. Indicator name Indicator D Duration (application fulfillment-secondment refusal or approval) TDD response Time of the Department Director TRC response Time of the Responsible for the Credit RS % of Refused Secondments RSF % of Secondments Refused because of problems with Funds RST % of Secondments Refused due to problems with Teaching replacement Table 1. List of performance indicators considered in our case study (I) After defining KPIs, their value has to be measured. For instance, if we want to measure the response time of the department director in our case study, an instrumentation would be necessary to monitor the elapsed period between the start and the end event of this process. The analysis of KPIs implies comparing KPIs values and their target values, as well as identifying causes of undesirable behaviour as violations of SLAs. For instance, in our case study we would be interested in finding the reasons or factors that drive to a percentage of refused secondment higher than 10%. The analysis phase could show that it relies on the lack of teaching replacement in certain year periods. Last, this monitoring information must be summarized and reported to the user. 4 Analysis of Related Approaches In the following we present some approaches proposed in this field applying them to our case study (if possible), and we analyse them according to the previous definition of KPI lifecycle. Popova et al. [3] In this article, the authors define a specification language for performance indicators and their relations and requirements based on ordersorted predicate logic (it employs sorts for naming sets of objects). Applying this approach to our example, once defined the KPIs (Table 1), we can give examples of relations between them: RL1: IS-INCLUDED (D,TDD,pos), RL2: IS-INCLUDED (D,TRC,pos), Actas de los Talleres de las Jornadas de Ingeniería del Software y Bases de Datos, Vol. 3, No. 3, 2009 ISSN 1988-3455 SISTEDES, 2009 60 Fig. 2. The conceptual graph for the case study RL3: INDEPENDENT (D,RS), RL4: IS-AGGREGATION-OF (RS,RSF), RL5: IS-AGGREGATION-OF (RS,RST). Finally, we can define our own preferences over the set of indicators through requirements: RQ1: Requirement (desired, Qualified-expression (min(v), has-value(RS, v))), RQ2: Requirement (desired, Qualified-expression (satisfy(v≤maxD), hasvalue(D,v))), being maxKD a constant set to 15 days, RQ3: Requirement (preferred-over, Qualified-expression (min(v), has-value(RS, v)),Qualified-expression (satisfy(v = maxD), has-value(D,v))) This language can also be used in a graphical form through conceptual graphs as shown in Figure 2 for our example. This proposal only addresses the phase of KPI definition, leaving out of its scope the rest of the lifecycle phases as well as the semantics of the language, which will be a subject of further research for the authors. Momm et al. [4] It consists of a top-down approach for developing an uniform IT support based on SOA in conjunction with the monitoring aspects required for processing the KPIs (referred to as Process Performance Indicators -PPIs- by the authors). The authors build the approach on the principles of the Model Driven Architecture (MDA) to enable the support of different SOA platforms as well as an automated generation of the required instrumentation and monitoring infrastructure. Particularly, they present a meta-model for the specification of the PPI monitoring along with an extension of the BPMN meta-model for modeling the required instrumentation for the monitoring and an outline of methodology for an automated generation of this instrumentation. In Figure 3 we can see an example where we specify the activities that shall be measured and how this specification is converted into an instrumented orchestration model to monitor the activity Check Teaching Replacement to measure its duration to calculate the response time of the department director. Actas de los Talleres de las Jornadas de Ingeniería del Software y Bases de Datos, Vol. 3, No. 3, 2009 ISSN 1988-3455 SISTEDES, 2009 61 Fig. 3. Mapping to Instrumented BPEL Orchestration Model for the Check Teaching Replacement User Task Regarding our KPI lifecycle definition, this approach deals with the first two phases, since they explain how to define a KPI and how to take the measures, but they do not conduct any analysis on the results obtained nor report them to the user. Wetzstein et al. [5] The authors present a stepwise approach for management of SLA-aware service compositions based on process performance requirements specified as Key Performance Indicators (KPIs). The idea is to use KPIs to help define the SLOs (Service Level Objectives defined in the SLAs) that must be met by partner services and IT Infrastructure. This approach consists of three phases. In the modeling phase the performance requirements on the process are gathered and KPIs with target values are specified. Then, in the configuration phase, KPIs are mapped to dependent SLOs of partner services and IT infrastructure (Figure 4 depicts an example of this mapping applied to our case study). Appropriate partner services and IT infrastructure are selected and the overall SLOs of the process are calculated and assigned to its service interface. Finally, during process execution, SLAs are monitored. In this case, the authors just partially address the analysis stage when they present the mapping depicted in Figure 4. The other phases are not covered by this proposal since the authors’ goal is to present an outline of methodology explaining the steps to accomplish but without specifying how to do it. Actas de los Talleres de las Jornadas de Ingeniería del Software y Bases de Datos, Vol. 3, No. 3, 2009 ISSN 1988-3455 SISTEDES, 2009 62 Fig. 4. Mapping of KPIs to Service Level Paramenters Mayerl et al. [6] In this paper the authors discuss how to derive metric dependency definitions from functional dependencies by applying dependency patterns. To this end, they propose a model (Class Diagram) that distinguishes between a functional part, where they define dependencies between application, service and process layers (based on concepts of BPEL and WSDL), and another part for metric dependencies, based on concepts of the CIM metrics model [7] and the QoS UML profile described in [8]. Figure 5 depicts the instantiation of the class diagram proposed in this approach, applied to our case study. The process activity CheckTeachingReplacement uses the service operation isActiveTeacher. The idea is to monitor the impact of the response time of this service operation on the duration time of the process activity. This response time is in turn influenced by the implementation of the service (components). They also introduce a mathematical formalism in order to describe dependency functions and the so-called metric characteristics or metrics calculable based on other metric values. An example is described in Figure 6 that shows a metric mdurationT to measure the duration of a process activity and also dependencies to other metrics. Finally they cover the mapping of these models to a monitoring architecture that contains functions to instrument and collect metrics, functions to aggregate and compare metrics with agreed service levels and functions to report SLA compliance and violations. In this paper, the authors go across every phase we have defined, but in some cases superficially. In particular, regarding to the definition phase they deal with the definition of metric dependencies, but do not specify how to define the metrics themselves. Besides, with respect to the analysis phase, they compare metrics values with its target ones, but do not study possible reasons for non compliance with SLAs. Castellanos et al. [9] This paper describes iBOM, a platform for business operation management developed by HP that allows users to: 1. analyse operations from a business perspective and manage them based on business goals. Actas de los Talleres de las Jornadas de Ingeniería del Software y Bases de Datos, Vol. 3, No. 3, 2009 ISSN 1988-3455 SISTEDES, 2009 63 Fig. 5. Model instantiating metric and metric dependencies Fig. 6. Metric dependency Pattern for Duration of Activities [6] Actas de los Talleres de las Jornadas de Ingeniería del Software y Bases de Datos, Vol. 3, No. 3, 2009 ISSN 1988-3455 SISTEDES, 2009 64 Fig. 7. Critical Factors analysis tree 2. Define business metrics (via a web interface), perform intelligent analysis on them to understand causes of undesired metric values, and predict future values. To analyse metrics they use data mining techniques based on decision trees. For instance, Figure 7 depicts a decision tree for our case study that helps to understand the circumstances under which a secondment is refused, and shows that the most critical is the quarter, followed by the department (SE-Software Engineering, AI-Artificial Intelligence). 3. Optimize operations to improve business metrics. The platform presented in this proposal covers the whole KPI lifecycle, but the main inconvenient when analyzing this approach is that it is not possible to check how they have addressed all these issues, since they do not give more information to this respect. 5 Challenges in modelling and tracing Key Performance Indicators in Business Processes Table 2 depicts how the proposals analysed deal with different aspects concerning KPIs: on the one hand with the four phases of the KPI lifecycle previously presented (1, 3, 4 and 7), and on the other with some particular features like the capability to identify relations between KPIs during the definition phase (2), or to check the compliance of KPIs to a SLA as part of the analysis phase(5), and the possibility to maintain the traceability between defined KPIs and their implementation (6). We use the following notation: A + sign means that the proposal successfully addresses the issue; a ∼sign indicates that it addresses it partially; and a blank indicates that it does not contemplate the issue. From the previous sections we conclude that there is neither standardized methodologies nor models to guide the KPI lifecycle in business processes. To overcome this problem we plan to develop a set of models that allows representing indicators in business processes expressed in BPMN and translations of Actas de los Talleres de las Jornadas de Ingeniería del Software y Bases de Datos, Vol. 3, No. 3, 2009 ISSN 1988-3455 SISTEDES, 2009 65