A prospective observational concordance study to evaluate computational model-driven clinical practice guidelines for Type 2 diabetes mellitus

A prospective observational concordance study to evaluate computational model-driven clinical practice guidelines for Type 2 diabetes mellitus Carlos Luis Parra-Calderóna,∗, Esther Román-Villarána, Celia Alvarez-Romeroa, Germán Antonio Escobar-Rodrígueza, Maria Asunción Martínez-Broccab,c, Alicia Martínez-Garcíaa, Julián Alberto García-Garcíad, María José Escalona-Cuaresmad aComputational Health Informatics’ Group. Seville Institute of Biomedicine (IbiS)/“Virgen del Rocío” University Hospital/CSIC/University of Seville, Avenida Manuel Siurot, 41013 Seville, Spain bVirgen Macarena” University Hospital, Seville, Spain cComprehensive Plan for Diabetes in Andalusia, Andalusian Health Service, Calle Doctor Fedriani, 3, 41009 Seville, Spain dComputer Languages and Systems Department, Escuela Técnica Superior de Ingeniería Informática, Avda. Reina Mercedes s/n. 41012 Seville, Spain A R T I C L E I N F O A B S T R A C T Keywords: Clinical practice guideline Evidence-based medicine Clinical decision support system Type 2 diabetes mellitus Background: Clinical Practice Guidelines (CPGs) provide healthcare professionals with performance and decision-making support during the treatment of patients. Sometimes, however, they are poorly implemented. The IDE4ICDS platform was developed and validated with CPGs for type 2 diabetes mellitus (T2DM). Objective: The main objective of this paper is to present the results of the clinical validation of the IDE4ICDS platform in a real clinical environment at two health clinics in the Andalusian Public Health System (SSPA) in the southern Spanish region of Andalusia. Methods: National and international knowledge sources on T2DM were selected and reviewed and used to define a diabetes CPG model on the IDE4ICDS platform. Once the diabetes CPG was configured and deployed, it was validated. A total of 506 patients were identified as meeting the inclusion criteria, of whom 130 could be recruited and 89 attended the appointment. Results: A concordance analysis was performed with the kappa value. Overall agreement between the recommendations provided by the system and those recorded in each patient’s EHR was good (0.61 - 0.80) with a total kappa index of 0.701, leading to the conclusion that the system provided appropriate recommendations for each patient and was therefore well-functioning. Conclusions: A series of possible improvements were identified based on the limitations for the recovery of variables related to the quality of these recolected variables, the detection of duplicate recommendations based on different input variables for the same patient, and clinical usability, such as the capacity to generate reports based on the recommendations generated. Nevertheless, the project resulted in the IDE4ICDS platform: a Clinical Decision Support System (CDSS) capable of providing appropriate recommendations for improving the management and quality of patient care and optimizing health outcomes. The result of this validation is a safe and effective pathway for developing and adopting digital transformation at the regional scale of the use of biomedical knowledge in real healthcare. 1. Introduction The need to implement CPGs in health systems was first identified in the 1990s [9]. By definition, CPGs focus on arrangements system- * Corresponding author. E-mail addresses: [email protected] (C.L. Parra-Calderón), [email protected] (E. Román-Villarán), atically developed to help healthcare professionals and their patients take decisions about appropriate care in specific clinical situations [14]. They are centered on Evidence-Based Medicine [26]. The overall aim of CPGs is to increase the quality, coherence and efficiency of health- [email protected] (C. Alvarez-Romero), [email protected] (G.A. Escobar-Rodríguez), [email protected] (M.A. Martínez-Brocca), [email protected] (A. Martínez-García), [email protected] (J.A. García-García), [email protected] (M.J. Escalona-Cuaresma). https://doi.org/10.1016/j.ijmedinf.2023.105208 Received 13 May 2023; Received in revised form 18 August 2023; Accepted 30 August 2023 2 C.L. Parra-Calderón, E. Román-Villarán, C. Alvarez-Romero et al. care, and thereby improve patient outcomes [22]. Even when CPGs are used, however, their implementation is impeded by numerous identifiable obstacles, including: (1) the subjective interpretation of CPG texts by individual health professionals, (2) a lack of time for health professionals to read CPGs, and (3) even non-familiarity with their format [14]. Murad [19]concluded that CPGs are useful in improving patients’ quality of life but that there is a significant lack of adherence to them that requires innovative solutions. Over the last years, clinical decision support systems (CDSS) have been proposed for clinical support decision making [11]. Although these systems provide potential benefits (such as increased patient safety, reduction of medical errors, and improved care, diagnostic support and improved workflows, among others [28], [29]), CDSSs are not yet fully integrated into clinical practice because of socio-cultural factors affecting healthcare professionals (such as loss of autonomy or the feeling of being replaced by a machine, among others [34][16]) and usability barriers associated with the difficulty of handling CDSS [24]. To overcome these problems, the IDE4ICDS project (Integrated Development Environment for Improving Clinical Decision Support based on Clinical Guidelines) [23]was carried out to propose a methodological framework for comprehensively and collaboratively (among software engineers and healthcare professionals) digitizing the CPG lifecycle. For this purpose, IDE4ICDS includes systematic model-driven mechanisms [27]for deploying CPGs on the IDE4ICDS platform, the architecture of which was designed as PaaS (Platform as a Service) architecture. This platform has also been designed to be adapted to the clinical practice of the Andalusian Public Healthcare System, to its mode of care, to the way data is collected, and, fundamentally, to support the integrated care model of Andalusia, which serves to overcome the main barriers identified above. One of the authors (Martinez- Brocca) is the director of the Andalusian Diabetes Care Plan. This paper aims to present the results of the clinical validation of the IDE4ICDS in a real clinical environment associated with CPGs for type 2 diabetes mellitus (T2DM) [4]. Studies confirm that 366 million people worldwide will be affected by T2DM by 2030 [33]. The high prevalence of the disease worldwide and, of course, in Andalusia, the high number of professionals involved in its clinical management and the low achievement of control targets evidenced in different healthcare systems [1]are characteristics that make this prospective observational concordance study a valuable piece of research to be replicated to evaluate CDSSs in integrated and complex healthcare settings with population coverage of more than 8 millions of people. 2. Methods 2.1. Review and selection of knowledge bases Once the IDE4ICDS platform was developed, and prior to the modeling of the T2DM CPG, a number of Spanish and international CPGs were studied. At international level, the CPG with the greatest impact in relation to T2DM is the “Standards of Medical Care in Diabetes” [10], which is updated annually by the American Diabetes Association. In Europe, one of the most important guidelines is the “QS6 - Diabetes in adults quality standard” [21], maintained by the National Institute for Health and Care Excellence (NICE). In Spain, specifically in the Andalusian Public Healthcare System, a periodically updated compendium of the reference guidelines indicated at the international and European level with a more solid scientific basis has been developed and implemented. It is called the Integrated Care Process for diabetes mellitus [15]. This document defines the data and decision processes throughout the integrated care process in the context of the Andalusian healthcare system. The coordinator of the edition of this Integrated Care Process is one of the authors of this paper (Martinez-Brocca). 2.2. Extraction of information from CPGs and design of decision rules Once the knowledge sources had been selected, a clinical researcher extracted their study variables and recommendations into a tabulated document. The material extracted included information and recommendations on anamnesis, examination, diagnosis and treatment of T2DM, and possible complications (screening, diagnosis and referrals). At the same time, decision rules were designed with the IDE4ICDS platform to display customized recommendations to the healthcare professionals on the platform during clinical practice. The rules were structured as follows (see Fig. 1): •Variable: A decision rule could comprise one or more variables. The variable illustrated in Fig. 1is “Chronic Kidney Disease”. •Value: This was assigned to the variable and had to be fulfilled for the recommendation to be displayed. It could be numeric, >/<, or TRUE/FALSE. The value illustrated in Fig. 1is TRUE. •Recommendation: This would be shown if the required variables were present with the assigned values. The recommendation illustrated in Fig. 1is “The presence of chronic kidney disease causes this patient to have high vascular risk. Statin treatment is recommended”. The rule in Fig. 1recommends that, if the patient has chronic kidney disease, a statin should be prescribed as they are at high vascular risk. The rules designed could be simple (with a single variable) or complex (with more than one variable having to occur for the recommendation to be shown). A total of 56 decision rules were designed Once the rules had been designed with the IDE4ICDS platform, they were validated by the healthcare professionals involved in the project. 2.3. Digitization of a clinical practice guideline with IDE4ICDS After identifying clinical knowledge relevant to the T2DM CPG, it was digitized using IDE4ICDS [23]. For this purpose, first, healthcare professionals formalized the T2DM CPG using the Domain Specific Languages (DSLs) that IDE4ICDS incorporates into its platform. These DSLs were designed using UML (Unified Modeling Language) extensibility mechanisms [3]. Due to space limitations in this paper, it is not possible to fully show the T2DM CPG model. However, it is possible to show a fragment of this model. Fig. 2shows the data collection process and calculation of the BMI (Body Mass Index) of the patient. This figure shows the workflow followed by the healthcare professional to calculate this clinical parameter. Later, once CPG was formalized, software engineers configure the CPG and obtain an executable version of it (based on Java and WSBPEL code) using model-driven mechanisms. This executable code is automatically deployed in the IDE4ICDS) process execution engine and its decision rule execution engine (which are based on BonitaSoft [7] technology). After configuring these execution engines, healthcare professionals can instantiate the T2DM CPG to perform care control on their patients. In this sense, Fig. 3(A) shows the user interface associated with the fragment of the model related to the calculation of the BMI parameter. Once the values of these parameters have been completed based on the information collected from the patient record by the healthcare professional, the IDE4ICDS platform automatically provides clinical recommendations (cf., Fig. 3(B)). 2.4. Clinical validation To demonstrate how well the IDE4ICDS system works, a pilot test was carried out in two health clinics managed by the Andalusian Public Healthcare System (SSPA acronym in Spanish), the largest healthcare provider in Spain, which provides free healthcare coverage to the entire population of Andalusia (the Spain’s largest region), one of the largest healthcare provider in Europe. In order to narrow the scope of the validation in a real environment, it was decided to exclude the sections of 3 C.L. Parra-Calderón, E. Román-Villarán, C. Alvarez-Romero et al. Fig. 1. BMI calculation modeling. Fig. 2. BMI calculation modeling (Fragment of the CPG Diabetes Mellitus T2). Fig. 3. Digitized fragment of the T2DM CPG associated with the calculation of the BMI clinical parameter. the T2DM CPG that referred to: prevention of DM, DM screening, diagnosis of DM, reuptake, care of hospitalized patients with DM, care of pregnant women with DM, care of children with DM, and indicators. The pilot study began in March 2019 and was conducted over a period of 8 months (until October 2019) in primary care (PC) consultations in the Cisneo Alto -Las Naciones and Carmona health clinics belonging to the SSPA. In order to participate, patients had to meet the following inclusion and exclusion criteria: •Inclusion criteria: the patient had to be suffering from DM2, aged over 35 years, in regular follow-up in PC with at least one regular visit in the last year, and registered in the PC medical records. •Exclusion criteria: ongoing pregnancy, abuse of toxic substances such as alcohol or drugs, other chronic or intercurrent medical or psychiatric illnesses unrelated to DM2 and which, in the opinion of the researcher, could lead to modifications in the planning of the patient’s follow-up and make it impossible for the patient to understand the conditions of the study and give their consent. During the piloting process, 506 patients were identified who met the inclusion criteria. Of these 506 patients, 130 could be recruited and 89 attended the appointment. The rest did not participate because: 1) they were not interested in participating, 2) they could not participate for other reasons or 3) they could not be located. The follow-up of each patient recruited was conducted as follows: •Conventional intervention: a healthcare professional treated the patient in line with the conventional protocol. 4 C.L. Parra-Calderón, E. Román-Villarán, C. Alvarez-Romero et al. Table 1 K value and concordance strength. K value Concordance strength < 0.20 Poor 0.21 - 0.40 Weak 0.41 - 0.60 Moderate 0.61 - 0.80 Good 0.81 - 1.00 Very good •Project intervention: about two weeks after the conventional intervention, the patient was called back and a clinical researcher simulated patient care using the IDE4ICDS system. •Technical incident reporting: during the piloting, incidents were reported to the project’s technical team. We highlight those incidents related to the lack of accessibility to the platform by the clinical researcher, related to the instability inherent to a research environment with a not-too-high TRL. •Decision analysis: the clinical researcher and the healthcare professional analyzed the differences between the decision taken under the conventional care procedure and the recommendation proposed by the tool: i.e. the degree of concordance between the two models. 3. Results To evaluate the fidelity of the IDE4ICDS platform, a concordance analysis was performed. The numerical parameter used was the kappa value. This kappa statistical test represented the closest measurement of concordance with reality, taking into account the fraction of concordance due to chance [2]. A kappa index (KI) equal to or greater than 0.61 is considered substantially higher than would be expected by chance and is generally accepted as a standard of high agreement. Likewise, kappa values equal to or greater than 0.81 indicate near perfect agreement. A total of 89 patients were analyzed—25 women (28.1%) and 64 men (71.9%). Once the IDE4ICDS platform validation period ended, concordance was analyzed jointly by the observers: that is to say, the information obtained by the clinical researcher with the system support (Observer 1) was compared with that recorded in the patient’s electronic health record (EHR) by the Andalusian Health Service (Observer 2). For the concordance analyses, the criteria shown in Table 1were taken into account. 3.1. Socio-demographic characteristics, lifestyle and medical history The variables with good or very good agreement were hypercholesterolemia (KI = 0.677), cancer (KI = 0.814), high blood pressure (KI = 0.883), acute myocardial infarction (KI = 0.860) and vaccinations (KI = 1.000). For smoking, 86.6% agreement was obtained, the degree of concordance between the two observers being moderate (KI = 0.564). The mismatch of 13.4% may be due to this information not having been updated in patients’ EHRs. Other characteristics with moderate agreement were allergies (KI = 0.446), heart failure (KI = 0.542), strokes (KI = 0.590) and retinopathy (KI = 0.576). Variables with low or poor agreement were liver disease (KI = 0.303), peripheral arterial disease (KI = 0.402), chronic kidney disease (KI = 0.285) and diabetic foot (KI = -0.015). The lack of concordance in these variables may be due to the patients’ own lack of knowledge regarding their pathologies. That is to say, patients with these illnesses were usually elderly and pluripathological and sometimes did not remember all their pathologies or did not know the exact name. In some cases, the information may not have been recorded in the EHR or may not have been recorded in the right place. 3.2. Recommendations Regarding the degree of agreement between the recommendations provided by the system and those entered in the EHR, 28 recommendations were considered constant because they were neither shown in the system nor recorded in the EHR. Since these recommendations were not necessary for each individual patient in the study, the degree of agreement was therefore 100%. For the 4 screening recommendations, the degree of agreement was moderate (0.41 - 0.60) with a KI of 0.574. One example of a screening recommendation was: “Annual screening for diabetic retinopathy is recommended.”. The 13 recommendations for education had a good level of agreement (KI= 0.686), an example being: “Include in structured therapeutic education programs, e.g. dietary education”. The degree of agreement between the 7 risk-related recommendations was also good (KI= 0.785), with recommendations such as: “There is a risk of fractures with pioglitazone”. For the recommendations about referral (7 in total) and treatment (21 in total), the KI were 0.913 and 0.821 respectively, indicating very good agreement (0.81 - 1.00). One example of a referral recommendation was: “Referral to ophthalmology is recommended to assess the status of diabetic retinopathy”. An example of a treatment recommendation was: “Consider treatment with liraglutide, empagliflozin or dapagliflozin”. Other recommendations on screening, additional tests and vaccinations (5 in total) had moderate agreement (KI= 0.609), an example of a recommendation for vaccinations being: “Annual vaccination against influenza and pneumococcus is recommended”. The overall agreement between the recommendations provided by the system and those recorded in each patient’s EHR was good (0.61 - 0.80), with a total KI of 0.701., and it can therefore be concluded that the system provided appropriate recommendations for each patient and is well-functioning. 4. Discussion Several reviews have been published regarding the usability and effectiveness of CDSSs. Ali et al. [5] looked at several studies and concluded that CDSSs enhance the practice and quality of clinical care moderately but effectively. Van de Velde et al. [30]conducted a review in which they emphasized that the effectiveness of recommendations can be significantly increased if they are generated automatically by CDSS tools, displayed on-screen and customized for each patient. Arditi et al. [6]also supported the use of CDSS tools, considering them potentially useful in clinical practice but concluding that further work is still needed to improve their functioning and performance. Other publications, like Middleton et al. [18], affirm that clinical practice will evolve significantly with the use of CDSS tools but that such tools should never be adopted as a substitute for action by healthcare professionals. The proposal presented in this paper offers a new approach. In the clinical sphere, it will enable us to facilitate, optimize and guarantee decision-making by healthcare professionals while, from the patient’s perspective, it can help us provide much more individualized care and achieve better results in the treatment of diseases. Unlike other tools designed for specific processes or diseases [17], [25], this proposal attempts to cover all spheres of clinical practice. Although, admittedly, some of the tools that have been designed are indeed similar to the present proposal [31], [8], they do not generate specific recommendations for each patient or accompany the healthcare professional in the course of the consultation. The proposed system has a significant degree of applicability in corporate and institutional contexts because, thanks to its potential for interpreting CPG information, improving health outcomes and reduc- 5 C.L. Parra-Calderón, E. Román-Villarán, C. Alvarez-Romero et al. ing healthcare costs, it helps reduce variability and errors in clinical practice. 4.1. Comparison with earlier work Different proposals have been put forward examining the viability of using CDSSs with CPGs. Kisser et al. [17] proposed a CPG-based screening tool for patients suffering from head or neck cancer, while Quanbeck et al. [25]focused mainly on the prescription of opioids—drugs which need to be better controlled due to the risk of overdose and addiction. The most significant difference between the IDE4ICDS system and the CDSSs described in those studies is that this proposal was designed to be scalable to all areas of clinical practice, and can be used in the future with any CPG in any health system. There have also been several proposals for tools that make it easier to develop and implement CPGs. Höhne et al. [13] presented an internet portal where CPGs can be verified, updated and developed more effectively. Van de Velde et al. [32]created a checklist for the correct implementation of CPGs. Another checklist was offered by Nan et al. [20], this time to reduce medical errors due to carelessness or ignorance. Grimmer et al. [12] prepared a list of CPG quality criteria and used it to make comparisons. These four proposals all focused on the first phase of the CPG life cycle. The IDE4ICDS system, however, covers everything from identifying a good CPG to obtaining a tool with which to assist healthcare professionals in patient care. Van de Velde et al. [31]also described the scalable development of a CPG-compatible CDSS that could be used in other areas of clinical practice. Unlike the IDE4ICDS system, however, that project did not offer personalized, individual attention to the patient. Finally, a project proposed by Boxwala et al. [8]comprised a CDSS based on a four-level framework of structured and semi-structured recommendations. The tool directly generated a series of recommendations from the data entered about the patient’s illness. All of these proposals focused exclusively on one or more phases of the CPG life cycle. The present line of research includes all of these phases and brings them together in the proposed CPG system. 4.2. Limitations of the IDE4ICDS platform During the course of the clinical validation, the clinical researcher met with patients at the health clinics to simulate clinical care consultation flows by including the CDSS in the workflow. One of the important limitations encountered when simulating clinical consultations was the lack of measurement tools like blood pressure monitors or tape measures for recording abdominal circumference. In some cases, these variables could not be measured and the information could not be recorded. The CDSS was designed and developed based on the opinion and validation of healthcare professionals, but the final results could not be reviewed by the healthcare professionals due to lack of time. It also took a long time to collect the information from patients’ EHRs that was going to be compared with that obtained in the system, because the recording of data in EHRs was very messy and it was very difficult to find the information required. In some cases, the same information was recorded in different places for each patient and in others it was not fully recorded in the patient’s EHR. Among the patients themselves, there was a feeling of mistrust with regard to their involvement in the research, and this led to a decrease in participation and to recruitment problems. 4.3. Scope for improvement detected in the IDE4ICDS platform During the process of clinical validation, possible technical improvements to the IDE4ICDS platform were identified and were left pending the future commercialization of the system: • Detection of typing errors when entering patient information. • Once the information on a variable has been entered, it will appear already filled in on the subsequent forms featuring that variable. •Some variables have numerical values that should be within a given range. If the value being entered falls outside that range, the system should detect the error. • Depending on the variables selected, a recommendation may sometimes be shown twice for the same patient. In this case, the system should recognize the duplicate recommendations and only display one. •Inclusion of variables instead of long sentences in order to improve the usability and visibility of the forms. •When presenting recommendations, a button should be included to allow each patient’s reports to be downloaded. 5. Conclusions The main objective of projects dealing with the development of CDSSs is to provide support for healthcare professionals in a specific area of healthcare or to achieve a single objective in terms of treatment, diagnosis, etc. The IDE4ICDS platform provides support throughout the entire medical consultation process, covering everything from anamnesis to diagnosis and treatment and providing each patient with personalized recommendations regarding lifestyle, disease screening, treatment changes and everything else that may influence the improvement and control of T2DM. This project resulted in a CDSS capable of providing appropriate recommendations, improving the management and quality of patient care, and optimizing health outcomes. Furthermore, possible improvements were identified during the course of clinical validation and will be implemented in a future commercialization phase. CRediT authorship contribution statement The authors declare that this work has been developed with the informed consent of the patients and under the authorization of the Biomedical Research Ethics Board of the University Hospitals Virgen Macarena and Virgen del Rocío of October 27, 2016 and whose updated patient information sheet was authorized on January 8, 2019. Declaration of competing interest The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Carlos Luis Parra-Calderón reports financial support was provided by Ministerio de Economía y Competitividad. Carlos Luis Parra-Calderon reports a relationship with Andalusian Health Service that includes: employment. Carlos Luis Parra-Calderon reports a relationship with Andalusian Public Foundation for Health Research Management in Seville that includes: non-financial support. Esther Román Villarán reports a relationship with Andalusian Public Foundation for Health Research Management in Seville that includes: employment. Celia Alvarez Romero reports a relationship with Andalusian Public Foundation for Health Research Management in Seville that includes: employment. Alicia Martínez García reports a relationship with Andalusian Public Foundation for Health Research Management in Seville that includes: employment. Germán Antonio Escobar Rodríguez reports a relationship with Andalusian Public Foundation for Health Research Management in Seville that includes: employment. Maria Asunción Martínez-Brocca reports a relationship with Andalusian Health Service that includes: employment. Julián Alberto García García reports a relationship with University of 6 C.L. Parra-Calderón, E. Román-Villarán, C. Alvarez-Romero et al. Seville that includes: employment. María José Escalona Cuaresma reports financial support was provided by Ministerio de Economía y Competitividad. María José Escalona Cuaresma reports a relationship with University of Seville that includes: employment. Acknowledgement This research has been supported by: (i) the EQUAVEL project (PID2022-137646OB-C31) from the “Ministerio de Ciencia e Innovación. Gobierno de España”; (ii) the IDE4ICDS project (RTC-2016- 5824-1) of the Spanish Ministry of Economy and Competitiveness and European Regional Development Fund (ERDF); and (iii) the Spanish Government’s Ministry of Economy and Competitiveness (reference SRTC1600C004793XV0) and ERDF funds. We would like to thank Francisco Jódar-Sánchez (https://orcid .org /0000 -0003 -1092 -9932) for his data analysis support methodology. References [1] M.A. Martínez Brocca, Diabetes Mellitus: Proceso asistencial integrado 3rd ed. Consejería de Salud, 2018. [2] V. Abraira, El índice kappa, Semergen-Medicina de Familia 27 (2001) 247–249. [3] The Unified Modeling Language User Guide, Addison-Wesley, 2005. [4] S. Alharbi, A. Alhofaian, M.M. Alaamri, Illness perception and medication adherence among adult patients with type 2 diabetes mellitus: a scoping review, Clin. Pract. 13 (2023) 71–83. [5] S.M. Ali, R. Giordano, S. Lakhani, D.M. Walker, A review of randomized controlled trials of medical record powered clinical decision support system to improve quality of diabetes care, Int. J. Med. Inform. 87 (2016) 91–100. [6] C. Arditi, M. Rège-Walther, P. Durieux, B. Burnand, Computer-Generated Reminders Delivered on Paper to Healthcare Professionals, Effects on Professional Practice and Health Care Outcomes, 2012. [7] Bonitasoft, Bonita bpm, https://es .bonitasoft .com/, 2023 [Last access: 2023]. [8] A.A. Boxwala, B.H. Rocha, S. Maviglia, V. Kashyap, S. Meltzer, J. Kim, R. Tsurikova, A. Wright, M.D. Paterno, A. Fairbanks, et al., A multi-layered framework for disseminating knowledge for computer-based decision support, J. Am. Med. Inform. Assoc. 18 (2011) i132–i139. [9] D.E. Casey, Why don’t physicians (and patients) consistently follow clinical practice guidelines?: comment on worsening trends in the management and treatment of back pain, JAMA Intern. Med. 173 (2013) 1581–1583. [10] N.A. ElSayed, G. Aleppo, V.R. Aroda, R.R. Bannuru, F.M. Brown, D. Bruemmer, B.S. Collins, K. Cusi, S.R. Das, C.H. Gibbons, et al., Introduction and Methodology: Standards of Care in, Diabetes—2023, 2023. [11] G. Paluzie-Ávila, T. Ohme, P. Aradilla, Clinical decision-making support systems in renal failure, Farm. Hosp. 38 (2014) 216–222. [12] K. Grimmer, J.M. Dizon, S. Milanese, E. King, K. Beaton, O. Thorpe, L. Lizarondo, J. Luker, Z. Machotka, S. Kumar, Efficient clinical evaluation of guideline quality: development and testing of a new tool, BMC Med. Res. Methodol. 14 (2014) 1–10. [13] W.J. Höhne, T. Karge, S.B.e. a, An internet portal for the development of clinical practice guidelines, Appl. Clin. Inf. 1 (2010) 430. [14] J.J. Gómez-Doblas, Implementación de guías clínicas, Revista espeñola de cardiología 59 (2006) 29–35. [15] Junta de Andalucía (Spain), https://www .juntadeandalucia .es /export /drupaljda / diabetes _mellitus _2018 _18 _06 _2018 .pdf, 2023 [Last access: 2023]. [16] S. Khairat, D. Marc, C.W.e. a, Reasons for physicians not adopting clinical decision support systems: critical analysis, JMIR Med. Inform. 6 (2018). [17] U. Kisser, C. Adderson-Kisser, M. Coenen, M. Stier-Jarmer, S. Becker, C. Sabariego, U. Harréus, The development of an icf-based clinical guideline and screening tool for the standardized assessment and evaluation of functioning after head and neck cancer treatment, Eur. Arch. Oto-Rhino-Laryngol. 274 (2017) 1035–1043. [18] P. Mor, Computer-interpretable clinical guidelines: a methodological review, J. Biomed. Inform. 46 (2013) 744–763. [19] M.H. Murad, Clinical practice guidelines: a primer on development and dissemination, Mayo Clin. Proc. 92 (2017) 423–433. [20] S. Nan, P. Van Gorp, X. Lu, U. Kaymak, H. Korsten, R. Vdovjak, H. Duan, A metamodel for computer executable dynamic clinical safety checklists, Impl. Sci. 13 (2018) 86. [21] Nice, https://www .nice .org .uk /guidance /ng28, 2023 [Last access: 2023]. [22] Y. Ou, I. Goldberg, C. Migdal, P.P. Lee, A critical appraisal and comparison of the quality and recommendations of glaucoma clinical practice guidelines, Ophthalmology 118 (2011) 1017–1023. [23] V. Cid de la Paz, J.A. Garcia Garcia, J.M. Ramos, et al., Diseño de un marco de trabajo dirigido por modelos y soportado por herramientas para mejorar la gestión de guías clínicas y procesos asistenciales, in: JISBD 2018: XXIII Jornadas de Ingeniería del Software y Bases de Datos, 2018. [24] A. Press, L. McCullagh, S. Khan, A. Schachter, S. Pardo, T. McGinn, Usability testing of a complex clinical decision support tool in the emergency department: lessons learned, JMIR Human Fact. 2 (2015). [25] A. Quanbeck, R.T. Brown, A.E. Zgierska, et al., Systems consultation: protocol for a novel implementation strategy designed to promote evidence-based practice in primary care, Health Res. Policy Syst. 14 (2016) 8. [26] S. Reddy, S. Herring, A. Gray, Identifying an appropriate content management system to develop clinical practice guidelines: a perspective, Health Inform. J. 23 (2017) 14–34. [27] D.C. Schmidt, Model-driven engineering, IEEE Comput. Soc. 25 (2006). [28] Reed T. Sutton, David Pincock, Daniel C. Baumgart, Daniel C. Sadowski, Richard N. Fedorak, Karen I. Kroeker, An overview of clinical decision support systems: benefits, risks, and strategies for success, npj Digit. Med. 3 (2020) 17. [29] V. Tundjungsari, A.S. Mudzakir Sofro, A. Sabiq, A. Kardiana, Investigating clinical decision support systems success factors with usability testing, Int. J. Adv. Comput. Sci. Appl. 8 (2017) 548–554. [30] S. Van de Velde, A. Heselmans, et al., A systematic review of trials evaluating success factors of interventions with computerised clinical decision support, Impl. Sci. 13 (2018) 114. [31] S. Van de Velde, P. Roshanov, T. Kortteisto, et al., Tailoring implementation strategies for evidence-based recommendations using computerised clinical decision support systems: protocol for the development of the guides tools, Impl. Sci. 11 (2015) 29. [32] S. Van de Velde, I. Kunnamo, P. Roshanov, et al., The guides checklist: development of a tool to improve the successful use of guideline-based computerised clinical decision support, BMC Med. Inform. Decis. Mak. 17 (2017) 170. [33] S. Wild, G. Roglic, A. Green, et al., Global prevalence of diabetes: estimates for the year 2000 and projections for 2030, Diabetes Care 27 (2014) 1047–1053. [34] K-H. Yu, A.L. Beam, I.S. Kohane, Artificial intelligence in healthcare, Nat. Biomed. Eng. 2018 (2) (2018) 719–731.