Proposing a quality model for evaluating and identifying opportunities in clinical practice guideline engines

Proposing a Quality Model for Evaluating and Identifying Opportunities in Clinical Practice Guideline Engines M. Carrero, E. Enamorado-Díaz, J.A. García-García* , and MJ Escalona Computer Languages and Systems Department. University of Seville. Sevilla. Spain. {mcarrero, eenamorado, juliangg, mjescalona}@us.es *corresponding author Abstract— Over the last decade, clinical practice guidelines (CPGs) have become an important asset for daily life in healthcare organizations. Efficient CPG management and digitization can improve the quality of patient care and healthcare by reducing variability. CPG digitization, however, is a difficult, complex task because such guidelines are usually expressed as text, and this often results in the development of partial software solutions. There are currently many CPG suites (CPGS) for managing the CPG lifecycle, but they do not all provide full support for this lifecycle, making it more difficult to choose the one which will best meet the specific needs and requirements of a healthcare organization. This paper proposes a quality model which makes it possible to compare CPGs by highlighting each phase of the lifecycle. The research was conducted using a methodology that combined a systematic literature review with quality models. The paper also discusses how the proposed model was instantiated to evaluate and compare several current CPG-based execution systems. Keywordsclinical practice guideline suites; systematic literature review; quality model; survey. I. INTRODUCTION The companies and organizations of today require innovative, flexible solutions to digitize and automate their processes [1] in conjunction with new technologies [1]. In healthcare environments, digitization has a much greater potential for disruption because it affects aspects like patient care, spiraling costs, quality and rewarding value [3]. This paper focuses on the digitization and automation of clinical practice guidelines (CPGs). Conceptually, CPGs can be defined as «statements that include healthcare processes, clinical rules and recommendations intended to optimize patient care, that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options» [4]. A well-defined CPG digitization process usually benefits both healthcare professionals and patients [5]: it makes it possible to reduce variability during clinical practice and improve the quality of clinical professionals’ performance and decision-making, while at the same time facilitating effective, reliable interventions based on empirical evidence. Many scientific initiatives, and technological proposals (CPG suites) have been published in recent years to facilitate the automation and digitization of clinical guidelines. Most of them, however, have had a limited functional scope, or their practical application has focused on treating specific pathologies in controlled environments. Consequently, such digital health innovations have not been adopted on a large scale and they are usually abandoned when they are not upscaled or kept in use over time at organization or system level [6]. The factors influencing non-adoption and abandonment are complex and include health conditions, technology, value propositions, adopters’ systems (professional staff, patients, and lay caregivers), organization(s), institutional contexts, as well as interaction and mutual adaptation between these factors over time [6]. This paper describes a quality model for comparing CPG suites according to the organizational objectives of healthcare organizations or research communities. This quality model supports each phase of the CPG lifecycle and was defined by combining well-known techniques and methodologies such as the systematic literature review (SLR) [7] and the method based on quality models and common characterization criteria proposed by the Software Engineering Group [8]. These techniques were used after considering the successful results obtained in previous studies [9]. The quality model and its characterization scheme were defined after considering the opinion of technology consultants who are experts in the application of information and communications technology (ICT) in the healthcare environment. In this regard, our research group (the ES3 group; Engineering and Science for Software Systems) has a long history of collaboration with numerous companies specializing in ICT, and this has allowed us to obtain important feedback. In fact, the results of this paper may prove very useful in helping healthcare organizations and academic communities to choose the most appropriate CPG suites for either organizational or research objectives in healthcare. The paper also illustrates the relationship between process lifecycles, clinical practice guidelines and the features supported in today's CPG suites. The results obtained could help identify possible aspects for improvement and guide future innovations and research. The rest of the paper is organized as follows. Section II describes some related work. Section III presents the methodological framework that has been followed to carry out our research. This framework also proposes a quality model based on a characterization scheme for evaluating CPG suites. Sections IV, V and VI describe and analyze how our quality model was instantiated on specific CPG suites. Finally, Section VII concludes with some lessons learned and ideas for future work. 2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS) DOI 10.1109/QRS57517.2022.00044 II. RELATED WORK When attempting to identify and analyze related work, no research publications were found which specifically featured CPG suite quality and characterization models. However, a few state-of-the-art studies did include more limited comparisons. The most relevant works in this field are briefly described below. Dadam et al. [10] carried out a state-of-the-art study to identify desirable features of process-oriented HIS supporting healthcare workflows. Here, the authors discussed and identified six features related to the expressiveness (control flow, temporal constraints and data flow), verification and consistency of models, dynamic flexibility at runtime, integration with hospital systems, and control of time dependencies between tasks. They also described how these features were applied in a real project, although no comparisons were made between systems to evaluate the features. Sim et al. [11] discussed computer-based CDS (clinical decision support) systems and their practical application, and later proposed a taxonomy to evaluate functional and design aspects of those systems. 24 characteristics were defined, grouped into 5 categories related to the clinical management context, data and knowledge sources, decision support, information delivery and workflow. However, the proposed taxonomy was presented theoretically, without quantitative evaluation mechanisms, and was not instantiated. Isern et al. [5] carried out a systematic review to identify and compare CPG suites. They analyzed 8 systems providing execution capabilities. The comparison focused on 11 aspects related to modeling aspects, technological architecture, integration and coordination of the system with external elements, type of execution engine, security aspects and the use of standard terminology. Gooch et al. [12] carried out a systematic review to identify challenges in designing and developing processoriented HIS supported by and integrated with formal models of clinical guidelines and care workflows. Having completed the review, the authors identified 25 common variables in the studies analyzed. These variables were closely related to features such as the integration of models into individual and collaborative clinical workflow systems, the possibility of checking the model with reasonable run-time behavior, the mapping of electronic health record (EHR) data to procedural tasks in the guideline or pathway, the reporting of features, flexibility, pathway adaptability at run-time, and usability. These variables, however, were presented in isolation and were not associated with the clinical guideline management lifecycle. III. METHODOLOGICAL FRAMEWORK This paper presents a quality model for identifying and comparing CPG suites. The model is based on well-known methods such as the SLR method [7] and the method based on quality models and common characterization criteria proposed by the Software Engineering Group [8]. Figure 1 shows the methodological protocol followed in this paper, which essentially involved three phases: (i) planning the review, to decide which method would be used to carry out the review and to identify and formulate the thesis that the systematic review must validate; (ii) conducting the review, which involved finding and evaluating whether many primary studies associated with the research questions were suitable and relevant enough to be possible sources for further analysis; and (iii) reporting the review, which dealt with writing up the results of the review and reporting them to potentially interested parties. These phases, shown graphically in Figure 1, are described in more detail in the following sections. Figure 1. Phases of the methodological process followed in this paper IV. PLANNING THE STUDY The planning phase established the review protocol of our survey, delimiting the context of our objectives and formulating the thesis to be proved. It defined the following aspects: 1) technical and research questions (TRQs) (the questions the study had to answer), and 2) quality assessment (QA) (a quality model or characterization scheme, like a checklist). A. Technical and research questions and search protocol The objective of this paper was to answer the following TRQs: (1) «What are the main published CPG suites currently available for managing CPG lifecycles?» and (2) «What quality model can be provided to evaluate these CPG suites?». Many search keywords could be used to address these questions, including «careflow», «care workflow», and «clinical guideline tool». These keywords were used to carry out exhaustive searches in different digital libraries (IEEExplore, ACM Digital Library, Google Scholar, Citeseer Library, ScienceDirect, and PubMed—the latter being particularly relevant in the field of Computer Science applied in the health context). B. Quality Assessment As mentioned previously, this paper proposes a quality model based on a characterization scheme which will allow CPG suites to be evaluated uniformly and which will facilitate comparative analysis. The characterization scheme will also make it possible to identify not only the needs of healthcare organizations but also new research lines. The scheme was defined after considering functional criteria in the literature and taking into account our own experience and the feedback received from our partner companies. Our research group has extensive experience in university collaboration with private companies (including FujitsuTS, Everis, Wellness Telecom and Soltel) and Spanish public organizations such as the Andalusian Regional Government’s Department of Health and Social Welfare), where we have applied computer science techniques in the health field. The abovementioned entities are experts in the design and development of process-oriented HIS capable of supporting the execution of clinical practice guidelines and this technological proposal was the result of several brainstorming sessions and interviews conducted with them. After considering the needs of these partner entities and the results of our interviews, we concluded that it was possible to establish a minimum lifecycle to strategically manage CPGs. Each phase of this CPG lifecycle is described below, together with its associated characteristics. MODELING PHASE. The objective was to represent and express any CPG, allowing it to be unequivocally interpreted and automatically processed by a software system, both structurally and formally. This phase included the following modeling (MO) criteria: •MO-01. It had to allow the use of formal modeling languages to represent elements in the CPG (activities, flows, rules, roles, etc.). •MO-02. The modeling language had to be sufficiently expressive: i.e., it had to have sufficient elements to adequately express or represent the CDG’s different concepts. •MO-03. It had to provide a modeling tool or incorporate some type of visual editor for modeling clinical processes by means of flow diagrams (flowcharts). •MO-04. It had to allow several different tasks in the clinical process to be combined/integrated to form a single task (sub-processes). The system would need to join up several tasks in the general clinical process to form a "principal" task or tasks (composite tasks, also known as subprocesses in the clinical process). •MO-05. It had to facilitate management of the medical and scientific reference documentation related to the modeled clinical process. •MO-06. It had to allow the modeling of clinical rules for establishing rule-based process flow control (for example, by means of logic expressions, decision tables, decision trees, etc.). •MO-07. It had to allow the modeling of process performance indicators (KPIs). •MO-08. It had to be compatible with standard medical terminology or vocabulary suites such as SNOMED [43], LOINC [44] or CPT [45]. DESIGN PHASE. The objective of this phase was to define a series of characteristics which, forming part of the system structure itself, would allow the subsequent execution of CPG models and facilitate communications with other systems: •DE-01. It had to allow interoperability with other medical systems and services to obtain data necessary for the clinical process. •DE-02. It had to have tools for the creation of customized graphical user interfaces (e.g., web forms, dialog boxes, etc.), through which healthcare professionals can easily interact with the system. •DE-03. It had to allow the design of mechanisms to control user access to the CPG system. •DE-04. It had to allow the design of mechanisms for importing organizational information from the healthcare institution regarding professionals, material resources, types of specialties, availability, schedules, etc. This was important for the subsequent assignment of clinical process tasks to the members of the organization and the scheduling of tasks according to available resources. •DE-05. It had to allow the design of mechanisms for assigning roles to tasks or activities. •DE-06. It had to allow the design of mechanisms for adding terms of service (SLAs) and linking with KPIs. DEPLOYMENT PHASE. This phase represented the system's ability to support flexible deployment within an organization's IT infrastructure. The following criteria were proposed to evaluate this phase: •DP-01. It had to support execution in distributed environments to guarantee its availability in case of errors or to balance the workload and avoid possible saturation. •DP-02. It had to support access to the CPG system from external machines, either from a local network or from the Internet. EXECUTION AND OPERATION PHASE. This phase included criteria for evaluating the CPG suite and the CPG model at runtime. The criteria were: •EX-01. It had to allow users to manage information about their own tasks or activities (task queries, dates for completion, own information provided in the tasks already performed, etc.). •EX-02. It had to allow direct notification or warning messages to be sent to users via e-mail, SMS, or other channels. •EX-03. It had to offer the possibility of executing multiple versions of the same CPG model, with each version presenting different changes, and controlling those changes (versioning and version control). •EX-04. It had to allow modification of a CPG model’s activity flow at runtime. For example, the user should be able to choose an alternative flow to the one proposed, alter the order of the tasks to be performed, or even add a task to the flow once the clinical process had started. •EX-05. It had to allow modification of the role assigned to a task or activity corresponding to a CPG model at runtime. •EX-06. It had to display the path being taken by a patient in the CPG model and track the clinical decisions taken in each activity. MONITORING PHASE. For the continuous improvement of the CPG model, it was necessary to monitor how the model and the CPG system behaved during the stage in which it was being executed and to keep in mind different aspects at both clinical and technical level. To this end, we proposed including the following criteria in our quality model as a means of evaluating this phase: •MC-01. The CPG suite had to provide mechanisms for monitoring aspects of the IT infrastructure such as resource consumption, availability, failures, etc. •MC-02. It had to allow the monitoring of the CPG model at runtime, displaying general technical information about clinical processes (number of instances of the CPG model, KPIs, etc.). •MC-03. It had to offer tools with which to view the execution information at runtime by means of control panels or informative graphic elements with sections dedicated to relevant data. •MC-04. It had to allow technical monitoring information to be viewed from different perspectives. The system should display different views or perspectives of information on the technical monitoring of the IT infrastructure or the clinical processes. •MC-05. It had to allow the workload to be distributed among the staff. The system should offer mechanisms for sharing the workload (tasks or activities) among the organization's personnel. This would avoid, for example, activity overload for some professionals and undertasking for others. •MC-06. It had to allow the workload to be automatically distributed among the personnel according to certain pre-established criteria. The system should offer mechanisms for automatically sharing the workload (tasks or activities) among users according to given criteria (for example, altering professionals’ task assignments during vacation periods). C. Rating method This section established a scoring method for assessing each CPG suite for each criterion and category of criteria, facilitating homogeneous comparison on three levels: Basic Scoring (BS). This was related to the basic feature score and was obtained by assigning an integer score based on a numerical scale from zero to four [0..4], where: 4 points meant that the CPG suite provided full native support; 3 points meant partial native support; 2 points meant that the CPG suite included programming interfaces that supported the evaluated feature; 1 point meant that a third party component was necessary to support the feature; and 0 points meant that CPG suite did not support the feature. Partial Score (PS). This score was associated with each category of features, and was calculated by adding up the basic scores of each feature belonging to a specific category, dividing the result by the maximum score and then multiplying the result by 10. The calculation is shown in Equation 1, where k represents a specific tool under evaluation, j is a specific category belonging to our quality model, n is the number of features in category j, BS(k,j,i) is the score for feature i (belonging to category j for tool k), and max(BS(k,j)) is the maximum basic score of category j for tool k. Final Score (FS). This score represented the final rating for each CPG suite and was calculated by adding up the partial scores for all the categories. The calculation is shown in Equation 2, where k represents a specific CPG suite under evaluation, j is a specific category belonging to our quality model, m is the number of categories, PS (k,j) is the partial score of category j for suite k, and max(PS(k)) is the total maximum score of all categories for suite k. Equation 1. Mathematical formula to calculate the score for each category belonging to our quality model Equation 2. Mathematical formula to calculate the final score for each CPG suite V. CONDUCTING THE STUDY Having established the review protocol for our study as described above, we then applied the search protocol and selected the most representative CPG suites. The selected tools were GLEE, ArdenSuite and GLARE. This section analyzes these tools considering our quality model (c.f., Section IV.B). Each tool was tested and a systematic review was carried out of all official documentation and publications by the corresponding research community. A. Evaluation of the GLEE system GLIF3 Guideline Execution Engine (GLEE) [13] is a clinical guideline execution system capable of interpreting and executing CPGs. MODELING PHASE. GLEE is capable of interpreting formal CPG models defined in GLIF3 [15], a very expressive formal language [15]. It allows the integration of several tasks in "subguides" [15], but is not natively compatible with other clinical process modeling languages [14]. Regarding tools for CPG modeling, GLEE is compatible with the files generated by the Protégé application and with the GLIF Editor tool. Both editors include graphic options for visual modeling [14][15] and checking the consistency and logical coherence of the coded guidelines [16]. The GLEE system also has mechanisms for establishing control of the clinical process flow by means of clinical rules, making it possible to manage such rules [15]. It uses a formal expression language called GEL [15], which is based on the Arden Syntax logic grammar. This system does not impose the use of any specific medical terminology standard but offers the freedom to use any standard [14]. DESIGN PHASE. GLEE allows interoperability with other clinical applications and systems using APIs [14], but it does not offer tools for the manual or automatic creation of graphical user interfaces since its function is to integrate with existing medical applications. It does, however, have an application that works locally with a graphical user interface, allowing control of the clinical process present in the guide and the carrying out of tests [14]. The GLEE architecture allows control over users’ access to the system [13], but it does not allow the importing of healthcare organizational information [14]. GLEE does not have support for assigning roles to GPC tasks or activities, since this aspect is not considered in the GLIF3 metamodel [14][15]. DEPLOYMENT PHASE. Thanks to its client-server architecture, GLEE can be used by several users simultaneously [13][14], providing defined interfaces for interaction with electronic medical records (EMRs) and with other clinical applications [13]. It also allows external medical applications to access the system through the network [13], although there is no web version of the graphical interface [14]. EXECUTION AND OPERATION PHASE. GLEE does not allow users to manage information about their own tasks [14], such as dates for completion or information about themselves provided in tasks already performed. What it does support is an event-driven execution model that can issue warnings to users, although the institution must provide a clinical event monitor to which the GLEE server can connect [15]. GLEE also allows several versions of a CPG model to be executed [14][17], permitting the user to decide the activity to be performed. However, it is not possible to add tasks or modify the model at runtime [14] [17], or to change the role assigned to a task, because roles are not contemplated in the GLIF3 model [14] [15]. GLEE displays the path followed by a patient in the clinical process in execution [14] and it also allows several clinical processes to be executed and accessed simultaneously [14]. MONITORING PHASE. Although GLEE does not offer mechanisms for the technical monitoring of the IT infrastructure [13] [14], it does provide administrative functionality for the basic control of system resources, which can be used to control the performance of the execution engine. This information, however, is provided by means of simple execution traces [14]. GLEE does not include control panels (although the system offers interfaces that would allow external systems to implement such panels [14]). Neither does it allow the workload to be distributed among the healthcare staff (since the system does not contemplate the assignment of roles [14][15] and has no options for managing staff schedules). B. Evaluation of the ArdenSuite system ArdenSuite is a CDS platform distributed by Medexter Healthcare that supports GPC execution [19]. MODELING PHASE. ArdenSuite supports the Arden Syntax modeling language [18], a formal language which represents medical knowledge in units known as medical logic modules (MLMs) [20]. ArdenSuite provides a modeling tool (ArdenSuite IDE) for modeling clinical processes by means of flowchart diagrams [20]. Based on Eclipse IDE, ArdenSuite IDE offers mechanisms for modeling and compiling MLMs and clinical processes. Regarding the possibility of integrating (combining) several tasks in a clinical process to form one single task (subprocess), ArdenSuite does not contemplate the creation of composite tasks. However, MLM modules can invoke other modules [21], which can be reused in other clinical processes. ArdenSuite also supports the management of medical and scientific reference documentation related to the clinical process represented [22], and allows the modeling of clinical rules [23] using its own formal expression language (Arden Syntax [21]). Arden Syntax encodes the rules in independent units (MLMs) [24][25]. This complicates the checking of logic consistency when several MLM units are interconnected (although it is possible to individually check the logic of each MLM). Finally, ArdenSuite does not use any specific medical vocabulary standard; in fact, it has no features which facilitate the use of controlled vocabularies or terminology standards [26]. DESIGN PHASE. ArdenSuite allows interoperability with other systems via its API web services [27] [28]. It has no tools for creating customized GUIs to allow users to interact with the system, but it does have several APIs that can facilitate access to the system by other external applications with their own GUIs [28]. ArdenSuite has its own server (ArdenSuite Server [20]), which supports MLM unit execution and interoperability with external systems via API [28] . Regarding access control, ArdenSuite Server has authentication and user management mechanisms [29] but, like ArdenSuite IDE [24][30], it offers no functionality for importing organizational information from the corresponding healthcare institution. The Arden Syntax language supports the assignment of roles to clinical process tasks [31]. DEPLOYMENT PHASE. ArdenSuite supports execution in distributed environments because the ArdenSuite Server uses web service protocols which facilitate its development [28]. ArdenSuite also provides different APIs and connectors for integration with other systems [24] and is accessible from local networks or from the Internet [19]. It also has a GUI with a web version. EXECUTION AND OPERATION PHASE. The commercial demo of ArdenSuite does not seem to contemplate the management of information from different professionals about their own tasks. The system has a clinical alert system that allows it to integrate with different EMRs that support alerts, for example by e-mail [32]. Regarding version control of a clinical process, versioning would be of individual "decision-actions" (MLMs), not of clinical processes understood as a complete activity flow [19]. Neither in the documentation nor in the commercial demo is there any mention of the possibility of making changes in the task flow of a running clinical process [22]. ArdenSuite displays the path followed by a patient in the CPG model, and tracks the clinical decisions taken in each activity [22]. MONITORING PHASE. ArdenSuite does not natively allow the technical monitoring of clinical processes [33], but its server architecture can facilitate extensions to obtain this functionality. Neither does ArdenSuite offer mechanisms for monitoring models at runtime, views or dashboards for displaying real-time execution information [33], or mechanisms for distributing the workload among the organization's personnel, either manually or automatically, according to pre-established criteria [33]. C. Evaluation of the GLARE system GLARE (Guideline Acquisition, Representation and Execution) is a clinical guidelines manager developed in collaboration with one of the largest hospitals in Italy [34]. It includes an acquisition module and an execution module. The former visually formalizes and models medical knowledge and instantiates this knowledge in a specific patient [35], while the latter allows the clinical models to be executed using a CDSS [34]. MODELING PHASE. GLARE supports its own formal modeling language, designed to strike a balance between expressiveness and complexity [36]. Thanks to its acquisition module, GLARE has a modeling tool that includes a visual editor for representing clinical processes by means of flowcharts [34]. It can also display a hierarchical, tree-like list of actions to be performed by the user [37]. GLARE makes it possible to integrate several tasks in the clinical process to form a single task, distinguishing between atomic actions and composite actions [34] and allowing the latter to be decomposed into atomic actions. With GLAREEdu [37], medical and scientific reference documentation related to the modeled clinical process can be managed and the reasons for decisions based on the guidance can be displayed . The GLARE system also allows the modeling of clinical rules, although its way of representing decision criteria differs from other systems in that it uses "scored" diagnostic decisions [37]. Instead of using a formal expression language it uses tabular representation for decision modeling [37], and it does not allow KPI modeling. It does, however, provide mechanisms for integration with medical terminologies and standards [38]. DESIGN PHASE. With regard to interoperability with other systems, GLARE is currently capable of interacting with one specific EHR. Thanks to its layered design and the use of XML, however, it could be connected to other EHRs [39]. GLARE does not have tools for creating customized graphical user interfaces that interact with the system [39], nor does it offer security mechanisms or tools for the protection of patient data or information about the system itself [39]. Its architecture includes a database that allows it to keep in mind certain resources and specific aspects of the healthcare institution [39]. It also offers support for assigning user roles to tasks [40]. Since it does not contemplate process monitoring, however, [38], it does not offer support for adding SLAs or linking with KPIs. DEPLOYMENT PHASE. GLARE supports execution in distributed environments [40] and provides mechanisms to be accessed from external machines through the network, but it does not have a GUI with a web version. EXECUTION AND OPERATION PHASE. GLARE partially supports the management of information about users’ own tasks because it allows users to consult the tasks that a given role must perform [40]. It can keep a record of the evolution of clinical guidelines [40], thus providing a means of keeping track of the different versions. The system is also prepared to support exceptions within the task flow in a running process [40]; this feature makes it possible to make changes in the flow. It also has support for optimizing the execution of the clinical process according to certain criteria. This is possible because GLARE has a database of the healthcare institution’s available resources and can therefore propose different scenarios to the healthcare professional, enabling him/her to choose the best path [36]. It is also possible to change the user role assigned to a task in a running process [40]. MONITORING PHASE. GLARE does not support the technical monitoring of the IT infrastructure, but minimal monitoring would be possible using the resource control mechanisms of the machine on which the tool is installed [38]. The system does not support the technical monitoring of running processes [38], has no mechanisms for dealing with possible technical failures [38], and cannot display technical monitoring information with different views or perspectives [38]. Finally, GLARE supports workload distribution among the staff but only at role level, not individually to each healthcare worker [40]. The workload can be adjusted/or distributed among the healthcare personnel according to the availability of the institution's human and material resources [40]. D. Threats to validity Following the recommendations published by Wohlin et al. [41], Petersen et al. [42], and Kitchenham et al. [7], this section addresses some threats to the validity of this research paper. Firstly, information contained in unindexed papers or gray literature could not be included in this research. The study may also be considered biased or unrepresentative because the researchers may have emphasized positive or negative qualitative results based on their own interests, potentially resulting in a lack of transparency. This was mitigated by involving experts in the healthcare field in the study. These experts reviewed the quantitative and qualitative assessments to ensure that the requirements regarding the quality model and its characterization scheme were met. Secondly, the quality model presented in this paper was instantiated to evaluate specific tools. The selection of these tools may constitute another threat to validity because it was influenced by the keywords used in the different digital libraries (c.f. Section 3.1.1). These keywords may have returned an insufficient set of results. This was mitigated by carrying out two iterations of the search protocol. The first iteration produced preliminary results whereas the second made it possible to refine the keywords used in the first iteration. Finally, although the quality model and its characterization scheme were considered by the authors to be a useful contribution of this research paper, some readers may see this contribution as being subjective rather than objective. This was mitigated by unifying three quality models (each researcher in the study performed his/her own quality model), which were then shared and merged. The quality model was also reviewed by ICT experts within the healthcare environment. This strategy made it possible to jointly identify as many of the features as possible. VI. ANALYSIS AND DISCUSSION This section describes the evaluation data of each CPG suite presented in the previous section. Table 1 combines the results obtained by applying the rating method defined in Section IV.C to the evaluation of each tool. Figure 2 shows the results graphically, considering the total score for each phase of the CPG lifecycle. Figure 2. Total score for each phase of the CPG lifecycle A. Modeling Phase Figure 3 shows the total score for the modeling phase of the CPG lifecycle after having quantified the rating of each feature of our quality model in each CPG suite. Each of the CPG suites supports its own formal modeling language capable of representing the main elements of CPG models. Each suite also offers visual tools for clinical process modeling with flow diagrams and mechanisms for creating composite tasks or subprocesses in the clinical process (i.e., integrating several tasks or elements into one), although each suite has its own name for this type of task. GLEE and GLARE, for example, refer to them as "subguides" and "composite actions", respectively, whereas ArdenSuite has no specific term for composite tasks because it already partially supports this functionality with its MLM units. The systems support different ways of displaying clinical guideline modeling information. The visual tool contained in GLARE, for example, shows both a flowchart and a hierarchical list with the actions to be performed by the user. Each CPG suite also provides its own mechanisms for relating medical and scientific documentation with its clinical process models. The GLARE-Edu tool and ArdenSuite, for example, allow external medical documentation to be included within its CPG models, but with GLEE it is only possible to indicate the name and a reference (i.e., a web link) to the clinical documentation used to define the CPG models. Regarding clinical rule modeling, each system uses a different strategy to carry out rule-based process flow control. GLEE and ArdenSuite, for example, use tabular representation but GLARE uses scored diagnostic decisions. No mechanisms were detected for defining KPIs or applying patient data privacy protocols during the evaluation of each tool, and none of the CPG suites were found to provide support for medical terminology standards. Table 1. Summary of evaluations of each CPG suite GLEE ARDENSUITE GLARE Feature Score Score Score Modeling Phase 7.81 7.19 8.44 MO-01 4 4 4 MO-02 4 4 4 MO-03 4 4 4 MO-04 4 3 4 MO-05 2 4 4 MO-06 4 4 4 MO-07 0 0 0 MO-08 3 0 3 Design Phase 2.08 3.75 4.58 DE-01 4 4 3 DE-02 0 1 0 DE-03 1 4 0 DE-04 0 0 4 DE-05 0 0 4 DE-06 0 0 0 Deployment Phase 7.5 7.5 10 DP-01 2 2 4 DP-02 4 4 4 Exec. & Op. Phase 5.83 4.58 5.42 EX-01 0 0 3 EX-02 3 4 0 EX-03 4 3 3 EX-04 3 0 3 EX-05 0 0 4 EX-06 4 4 0 Monitoring Phase 2.08 0.83 2.92 MC-01 1 1 1 MC-02 3 0 0 MC-03 1 1 0 MC-04 0 0 0 MC-05 0 0 3 MC-06 0 0 3 TOTAL 5.06 4.77 6.27 Figure 3. Score for modeling phase B. Design Phase Figure 4 shows the total score for the design phase of the CPG lifecycle after having quantified the rating of each feature of our quality model in each CPG suite. GLEE and ArdenSuite offer native support for connecting, in general, with medical systems or services to obtain data needed in the clinical process. GLARE partially supports this feature because it can only connect to the specific EHR for which it was programmed although, according to its developers, it could also connect to other EHRs thanks to its layered design and the use of XML. We found that the creation of graphical interfaces was another major shortcoming. In fact, this feature is only supported by ArdenSuite, which has several APIs that can facilitate access to the system by external client applications with modifiable graphical interfaces. None of the systems fully support data security mechanisms, but at least user access control is supported by ArdenSuite. GLARE is the only CPG suite that offers support for organizational data import mechanisms. It is also the only system that supports assigning roles to clinical process tasks or activities. Finally, we identified one other important deficiency. Currently, no evidence has been found to show that these CPG suites provide support for defining SLAs or KPIs. Figure 4. Score for design phase C. Deployment Phase Figure 5 shows the total score for the deployment phase of the CPG lifecycle after having quantified the rating of each feature of our quality model in each CPG suite. On the one hand, GLARE natively supports execution in distributed environments. This functionality is not implemented in the other systems, although it could be developed in GLEE (thanks to its client-server architecture) and in ArdenSuite (thanks to its service-based architecture). On the other hand, all the CPG suites allow access by other machines via the network. Figure 5. Score for deployment phase D. Execution and Operation Phase Figure 6 shows the total score for the execution and operation phase of the CPG lifecycle after having quantified the rating of each feature of our quality model in each CPG suite, . GLARE is the only CPG suite that supports, albeit partially, users’ management of information about their own tasks; the others do not offer this option. ArdenSuite has full support for sending alerts or warnings directly to users remotely, but GLEE only partially supports this function (it requires a clinical event monitor to be connected to its server). This option is not available in the other systems. All the CPG suites have some type of version control for clinical processes. GLEE and GLARE partially support modifying the task flow in a running clinical process, whereas ArdenSuite does not have this capability. GLARE is the only system analyzed that includes support for optimizing the execution of the clinical process during runtime in line with certain criteria. It is also the only system that allows the role assigned to a task to be changed in a running process. GLEE and ArdenSuite display the path followed by a patient in the running clinical process but this feature is not specified in GLARE. All the CPG suites allow several clinical processes to be executed simultaneously. Figure 6. Score for execution and operation phase E. Monitoring Phase Figure 7 shows the total score for the monitoring phase of the CPG lifecycle after having quantified the rating of each feature of our quality model in each CPG suite. After evaluating each tool, it was concluded that support for the technical monitoring of IT infrastructures was another important limitation. This function is possible, however, using the resource control mechanisms of the machine on which the system is installed. Only GLEE offers partial support for the technical monitoring of clinical processes during execution because it displays some basic technical values. The other CPG suites do not contemplate this feature. No evidence was found that any of the tools support mechanisms for displaying relevant data about running models. Likewise, none of the CPG suites allow monitoring information to be displayed from different views or perspectives. The only CPG suite that partially allows mechanisms for distributing the workload among the organization's personnel is GLARE, since this is the only one that allows the automatic distribution of the workload in line with certain preestablished criteria. Figure 7. Score for monitoring phase VII. CONCLUSION Clinical guidelines are useful tools with which to standardize existing clinical knowledge in a specific context. The main handicap of CPGs is their representation, which is usually textual. This causes ambiguity and variability when health professionals apply CPGs in clinical practice. Software systems currently exist which could help clinicians to improve CPG automation, but the use of such systems is only a very small step towards their widespread use in medical organizations. Today, these CPG suites still have shortcomings in terms of their integration into careflow management systems and the challenges they face in daily practice in healthcare institutions. This paper proposes a quality model which makes it possible to compare CPG-based execution systems, highlighting each phase of CPG lifecycle management (and thus enabling organizations to choose the most useful tool for their requirements). The proposed model was instanced to compare seven currently available systems for automatically implementing clinical guidelines, and a series of conclusions were drawn. In summary, the final scores obtained were diminished due to the lack of mechanisms or options for system monitoring. This not only affected the scoring of the monitoring phase criteria, but also had an indirect impact on other criteria in different phases, such as those associated with system design. The highest scoring criteria, however, were those associated with modeling. If the different scores are linked to characteristics, we can conclude that the GLEE system scored high in those criteria related to the process engine itself, standing out from the other systems in both its modeling and its execution of clinical processes. ArdenSuite scored well in modeling and deployment. GLARE (which obtained the highest overall rating) is the only system that allows the assignment of roles and, as an extra benefit, offers the possibility of optimizing processes in execution. With regard to future work, we plan to continue studying the main features of CPG-based execution systems to improve our quality model. We will also present new comparative studies to evaluate other possible CPG support systems, taking this paper as the point of departure from which to further explore how such systems can reduce ambiguity and variability when clinical guidelines are automated in real health contexts. ACKNOWLEDGMENTS This research was carried out as part of project GIMO-PD (RTC2019-007150-1) of the Spanish Ministry of Economy and Competitiveness, financed by European funds. It also forms part of project PID2019-105455GB-C31 (funded by MCIN/ AEI/10.13039/501100011033/ and by the European Union) and project P20_00644 (funded by the Andalusian Regional Government, Spain). REFERENCES [1] Van den Bergh, J., Thijs, S., & Viaene, S. (2014). ``Transforming Through Processes: Leading Voices on BPM, People and Technology''. Springer International Publishing. [2] Alotaibi, Y. (2016). Business process modelling challenges and solutions: a literature review''. Journal of Intelligent Manufacturing, 27(4), 701-723. [3] Kane, G. C. (2017). Digital innovation lights the fuse for better health care outcomes. MIT Sloan Management Review, 59(1). [4] Steinberg, E., Greenfield, S., Wolman, D. M., Mancher, M., & Graham, R. (Eds.). (2011). Clinical practice guidelines we can trust. National Academies Press. [5] Isern, D., & Moreno, A. (2008). Computer-based execution of clinical guidelines: a review. International journal of medical informatics, 77(12), 787-808. [6] Greenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., Hinder, S., ... & Shaw, S. (2017). Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. Journal of medical Internet research, 19(11), e8775. [7] Kitchenham B., P. Brereton. (2013). A systematic review of systematic review process research in software engineering, Information and Software Technology, Volume 55, Issue 12, Pages 2049-2075. [8] SEG (Software Enginnering Group). (2007). Guidelines Performing Systematic Literature Reviews in Software Engineering 2.3. [9] Escalona, M.J., García-García, J.A., Dominguez-Mayo, F.J., & Ramos, I. Technical Tool Surveys and Comparative Studies: A Systematical Approach. In Information Systems Development: Transforming Organisations and Society through Information Systems (ISD2014).