Design Principles of Clinical Dashboards Incorporating PROMs: Crafting and Elaborating the Potential of Clinical Dashboards Incorporating PROMs

Bischof, Anja; Salvi, Irene; Kuklinski, David; Vogel, Justus; Geissler, Alexander Research Report Design Principles of Clinical Dashboards Incorporating PROMs: Crafting and Elaborating the Potential of Clinical Dashboards Incorporating PROMs Schriftenreihe in Health Economics, Management and Policy, No. 2023-05 Provided in Cooperation with: University of St.Gallen, School of Medicine, Chair of Health Economics, Policy and Management Suggested Citation: Bischof, Anja; Salvi, Irene; Kuklinski, David; Vogel, Justus; Geissler, Alexander (2023) : Design Principles of Clinical Dashboards Incorporating PROMs: Crafting and Elaborating the Potential of Clinical Dashboards Incorporating PROMs, Schriftenreihe in Health Economics, Management and Policy, No. 2023-05, Universität St.Gallen, School of Medicine, Lehrstuhl für Management im Gesundheitswesen, St.Gallen This Version is available at: https://hdl.handle.net/10419/280743 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by-nc-nd/4.0/ Working Paper Series in Health Economics, Management and Policy 2023 – Nr. 05 Design Principles of Clinical Dashboards Incorporating PROMs: Crafting and Elaborating the Potential of Clinical Dashboards Incorporating PROMs Anja Bischof, Irene Salvi, David Kuklinski, Justus Vogel, Alexander Geissler I Working Paper Series in Health Economics, Management and Policy Editor Prof. Dr. Alexander Geissler Professor Chair of Healthcare Management School of Medicine University of St.Gallen Editorial office Anja Bischof Research assistant Chair of Healthcare Management School of Medicine University of St.Gallen The entire series of publications is available on our website at: https://med.unisg.ch/en/research/healthcare-management/publications/ © 2023. This publication is licensed by the CC license CC-BY-NC-ND 4.0 II Design Principles of Clinical Dashboards Incorporating PROMs: Crafting and Elaborating the Potential of Clinical Dashboards In-corporating PROMs Keywords: Clinical dashboards; Patient-reported outcome measures; PROMs JEL Classification: I10, I19 Authors: Anja Bischof Research assistant Chair of Healthcare Management, School of Medicine, University of St.Gallen [email protected] Irene Salvi Research assistant Chair of Healthcare Management, School of Medicine, University of St.Gallen [email protected] David Kuklinski PostDoc/ Scientific Project Leader Chair of Healthcare Management, School of Medicine, University of St.Gallen [email protected] Justus Vogel PostDoc/ Scientific Project Leader Chair of Healthcare Management, School of Medicine, University of St.Gallen [email protected] Alexander Geissler Professor Chair of Healthcare Management, School of Medicine, University of St.Gallen [email protected] Recommended citation: Bischof, Anja; Salvi, Irene; Kuklinski, David; Vogel, Justus; Geissler, Alexander (2023): Design Principles of Clinical Dashboards Incorporating PROMs, Schriftenreihe in Health Economics, Management and Policy, No. 2023-05, Universität St.Gallen, School of Medicine, Lehrstuhl für Management im Gesundheitswesen, St.Gallen University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs I Design Principles of Clinical Dashboards Incorporating PROMs Crafting and Elaborating the Potential of Clinical Dashboards Incorporating PROMs Schriftenreihe in Health Economics, Management and Policy Nr. 2023–05 Project: 301-SG (Seed grant) Anja Bischof, Irene Salvi, Dr. David Kuklinski, Dr. Justus Vogel, Prof. Dr. Alexander Geissler School of Medicine, Chair of Health Care Management St. Gallen, 16. October 2023 University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs II Funding This report was funded by the EuroQol Group. The interpretation of the results, the conclusions and the recommendations do not necessarily reflect the opinion of the EuroQol Group. The contents of the report have been prepared with the greatest possible care. However, the authors do not guarantee the absolute accuracy, completeness and timeliness of the content provided. The authors have no conflicts of interest or material involvement with the methods or products used in this report. The Chair of Management in Health Care at the University of St. Gallen has committed itself to comply with the applicable data protection law in handling the anonymized data even after the study has been completed. Acknowledgements We would like to thank Ms. Blanche Bachmann for providing her support in the scoping literature review. Furthermore, we thank Dr. Gouke Bonsel and Dr. Bas Janssen for their support during the research process and for providing valuable insights. Finally, we thank all software producers and users who took the time for an interview and transparently talked to us about their wishes and requirements for the clinical dashboards incorporating PROMs. University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs III Abstract Objectives: A clinical dashboard is a data-driven clinical decision support tool visualizing multiple key performance indicators in a single report while minimizing time and effort for data gathering. Evidence showed that including patient-reported outcome measures (PROMs) in clinical dashboards supports the clinician’s understanding of how treatments impact patients’ health status and helps identifying health-related quality of life changes at an early stage. While existing literature mainly focused on the benefits of using disease-specific PROMs in clinical dashboards, the EQ-5D has rarely been investigated despite its potential to assess the patient’s overall health status and mental well-being. To address this gap, we aimed to determine design principles for clinical dashboards incorporating generic – i.e., the EQ-5D – and disease-specific PROMs. Methods: We used a three-step approach. First, a scoping literature review summarized the evidence of relevant design principles for clinical dashboards in general. Second, insights from interviews with both software producers and users of clinical dashboards validated and enhanced the results of the literature review. Third, we built dashboard prototype using the knowledge gathered in the first two steps, which was finally evaluated by a focus group discussion. Results: We found that the design principles for clinical dashboards do not have to change between different episodes of care. The scoping literature review highlighted to incorporate various relevant design principles into clinical dashboards, such as patient data, clinical metrics, past PRO assessment scores, or peer-group comparison. Interviews showed that both software producers and users had similar views on clinical dashboard use, primarily for patient monitoring and interpretation support of PRO data. However, their opinions diverged on the key users, while users favored specialists, dashboard producer expressed their favor for a broader user base. During the focus group discussion, participants found clinical dashboards incorporating PROMs valuable, highlighting the importance getting the possibility of finally considering patients' selfreported health status during consultations. Design principles derived from literature and interviews aligned with the views expressed during focus group discussions, emphasizing the use of both generic and disease-specific PROMs. Conclusion: The chosen three-step approach permitted cross-checking the state-of-the-art in literature and detecting white spots where users and software producers showed diverging tendencies for certain design principles. Our research confirmed that the design principles for different disease areas do not differ. The past score PROM assessment and peer-group comparison were rated, by both the software producers and the users, as the most valuable design principles. Ultimately, this research aims to inform the development of clinical dashboards incorporating PROMs. University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs IV Table of Contents Abstract _____________________________________________ III List of Abbreviations ___________________________________ V List of Figures _________________________________________ V List of Tables ________________________________________ VI Appendix ___________________________________________ VI 1 Introduction _______________________________________ 1 1.1 Subject matter, objectives and structure of the study ____________________ 1 1.2 Definition of scope _____________________________________________ 2 1.2.1 Episodes of care ________________________________________ 3 1.2.2 Definition of clinical dashboards _____________________________ 3 2 Methods __________________________________________ 5 3 Results ___________________________________________ 8 3.1 Scoping Literature Review ________________________________________ 8 3.2 Interviews ___________________________________________________ 10 3.3 Focus group – Testing and evaluation of clinical dashboard prototype ______ 15 4 Discussion and implications ___________________________ 21 4.1 Practical implications | General implications __________________________ 23 4.2 Implications for EuroQol ________________________________________ 24 5 Limitations _______________________________________ 27 6 Conclusion and outlook _____________________________ 28 References __________________________________________ 29 Appendix ___________________________________________ 35 University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs V List of Abbreviations CAT COPD Assessment Test COPD Chronic obstructive pulmonary disease EMR Electronic medical record GP General practitioner HOOS Hip disability and Osteoarthritis Outcome Score HRQoL Health-related quality of life KOOS Knee injury and Osteoarthritis Outcome Score JAMAR Juvenile Arthritis Multidimensional Assessment Report EQ-5D EuroQol 5 Dimension OHS Oxford Hip Scores OKS Oxford Knee Scores PROM Patient-reported outcome measure SF-16 12-Item Short Form Survey SF-36 36-Item Short Form Survey SGRQ-C St. Georges Respiratory Questionnaire - COPD HA Hip arthroplasty KA Knee arthroplasty List of Figures Figure 1. Research approach compiling the design principles of clinical dashboards incorporating PROMs ............................................................................................................................... 5 Figure 2. Overview of the screening process ........................................................................................ 6 Figure 3. Overview of software producer and user responses on general information ............. 10 Figure 4. Overview of responses from software producers and users concerning the dashboard features ...................................................................................................................................................... 14 Figure 5. Clinical dashboard prototype according to the design principles collected in the scoping review and interviews ............................................................................................................. 15 Figure 6: Clinical dashboard prototype – subpage function test .................................................... 16 Figure 7: Clinical dashboard prototype – subpage laboratory ........................................................ 16 Figure 8: Clinical dashboard prototype – subpage imaging ............................................................ 17 Figure 9: Clinical dashboard prototype – subpage EQ-5D per dimension ................................... 17 Figure 10: Clinical dashboard prototype – subpage CAT scores per dimension ......................... 18 University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs 6 For selecting relevant papers, first, the title and abstract of each paper were screened. Next, a thorough second screening of the complete articles was conducted to evaluate relevant insights for this research. The inclusion of papers was evaluated based on the fit of title and abstract. In event of uncertainty as to whether a particular work fell in the scope of this research, the entire text was read. The two-step search through the databases yielded 212 publications (Search A n=121, Search B n=101). After having screened all papers, 44 publications were considered relevant for this review. With cross-referencing and desk research, we added 23 papers, leading to a total of 67 papers (see Figure 2, all selected papers are listed in Appendix I). Step 2. Software producer and user interviews As second step, we adopted an explorative human-centered research design. Engaging software producers and users is a crucial step towards designing user-friendly clinical dashboards that are accepted, functional, and potentially enhance the quality of care and patient outcomes [15]. Thus, we conducted interviews with software producers and users of clinical dashboards incorporating PROMs. Interviews with software producers strengthened the understanding of current products, whereas the user interviews mainly focused on identifying user needs. This allowed us to gain well-founded insights about potentials, experiences and challenges for clinical dashboards incorporating PROMs. We report the details of the study in accordance with the Consolidated Criteria for Reporting Qualitative Research (COREQ) 32-item checklist to ensure transparency and reliability (Appendix II) [40]. Figure 2. Overview of the screening process University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs 7 The recruitment of the interviewees started in our own network, and through a snowballing technique, we were able to contact software producers and users outside of the network. The interviews were semi-structured, and the relevant literature findings guided the interview questionnaire’s development. Afterward, the interview questions were iteratively adapted according to the responses of the already conducted interviews to enhance the inductive explanatory value gained. Prior to each interview, introductory slides were shown to all interviewees to ensure a common understanding of clinical dashboards, and content-related questions were clarified (see Appendix III). All interviews covered the following main areas: 1) general questions on dashboards and PROM usage, 2) questions about the market (penetration) (software producers only), 3) usage of the dashboard, 4) dashboard development and data collection, 5) feature assessment, 6) role of the patient (see Appendix IV). According to grounded theory by Corbin and Strauss [21], the conduction of interviews ended once thematic saturation was reached, i.e., when no additional information was gained from further interviews. All interviews were conducted by two authors – both research – to ensure that the same content was covered. Both interviewers had gained experience in qualitative methods as part of their doctoral research training. The interviews were conducted in English or German and digitally recorded and transcribed verbatim to perform data analysis by coding. For all interviews, also field notes were taken. According to grounded theory, the findings formed the data pool for generating explanations [22]. To identify relevant information for the design principles, first, deductive coding was used to cross-validate the findings from the scoping literature review (for deductive codes, see the codebook in Appendix V). Second, inductive coding allowed us to identify and classify information not covered by the scoping literature review. All interviews were coded by two authors independently and afterwards analyzed and compared together. In case the two authors did not find agreement for the applicability of a code, a third author joined the discussion to find consensus on unresolved aspects. All interview analyses were conducted with Atlas.ti Windows (Version 22) [23]. Step 3. Prototype and evaluation of clinical dashboard After conducting and evaluating the interviews, a clinical dashboard prototype incorporating PROMs was developed for further evaluation by physicians in a focus group discussion. Features incorporated in our prototype were based on our findings from the literature and the insights gained during the interviews and the subsequent coding. In the focus group discussion, first, the project and the previous research steps were presented. Second, the clinical dashboard prototype was introduced. Third, the participants had the chance to test the prototype on their devices. Fourth, a feedback discussion occurred which followed a semi-structured guide including six main topics: 1) positive aspects of the dashboard, 2) areas of improvement, 3) design principles that need to be overworked, 4) time of data collection, 5) alert function, 6) likelihood of collecting and using PROMs in the future. The focus group discussion was audio-recorded, and two note takers documented key insights. University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs 8 3 Results This chapter summarizes the findings from our three-step approach. The subchapters are structured accordingly. The scoping literature review and the conducted interviews served as the basis for the development of the clinical dashboard prototype and the subsequent focus group. 3.1 Scoping Literature Review Results of Scoping Literature Review Search A: Dashboard focus The scoping literature review revealed that research focuses considerably more often on dashboards for chronic diseases, such as endometriosis [21], depression [24], overactive bladder [41], cancer [25, 42–47], asthma [26], Parkinson [48], or rheumatoid arthritis [20, 49, 50] compared to one-time acute care interventions such as spine surgery [51] or HA/KA [52]. Concerning the main episodes of care in this study, only one paper dealt with HA/KA [51], while none targeted a dashboard for COPD (for the full list of episodes of care see Appendix I). Results of Scoping Literature Review Search B: PROM focus We only included papers focusing on either COPD or HA/KA to identify relevant disease-specific PROMs. Concerning COPD, the EQ-5D was included in most cases for collecting information about the patient’s health-related quality of life [53–57]. Only Smith et al. [57] additionally included the SF-36 as a generic PROM. Commonly used disease-specific PROMs for COPD are the COPD Assessment Test (CAT) [53–55], the St. Georges Respiratory Questionnaire - COPD (SGRQ- C) [53, 54, 56, 57] and the Clinical COPD Questionnaire [53]. When referring to the collection of HRQoL in HA/KA patients, several studies indicated using the EQ-5D [52, 58–66]. As an alternative to the EQ-5D, SF-12 [61, 62, 67] and SF-36 [65, 67] were used as a generic instrument. As suitable disease-specific PROMs for HA/KA patients the Western Ontario and McMaster Universities Arthritis Index (WOMAC) [58–60, 67], the Oxford Hip and Knee Scores (OHS/OKS) [52, 61, 62, 64], or the Knee injury/Hip disability and Osteoarthritis Outcome Score (KOOS/HOOS) [52] were identified. Results of Scoping Literature Review Searches A and B on design principles The scoping literature review revealed relevant insights for design principles of complementary data such as patient information [26, 46, 68] or clinical data [26, 46, 49, 50]. Further, the inclusion of additional features such as past assessment scores [20, 25, 43, 49], peer-group comparisons [42, 45, 49], goals over time [24, 49], alerts [25, 41], and free-write in features [25, 43], and dashboards customizability [24, 46, 51] were perceived as relevant. All these principles are recommended to be incorporated into a clinical dashboard to ensure a perceived additional value through its use. Based on the findings from the scoping literature review, we developed a listing of essential design principles to be considered when building a clinical dashboard incorporating PROMs (Table 1). The table includes three grouping areas: general information, data collection, and dashboard content. Each grouping area includes main design principles and potential attributes. For example, in the grouping area of “general information”, the design principle of University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs 9 “setting” covers application areas of clinical dashboards in in- or outpatient settings or a combination of both. Table 1. Overview of design principles to be tested in user and software producer interviews based on scoping literature review and expert interviews General information Sources Type of disease: chronic, acute [20, 21, 24, 26, 41, 42, 48, 51, 52] Setting: inpatient, outpatient, combination [21, 26, 43, 69, 70] Type of PROM: Disease-specific, generic, combination [52–67] Key user: Specialist, GP, all kinds of physicians, other health care professionals (e.g., physiotherapist, nurse, etc.), patient, relatives [26, 51, 71] Data collection Sources Level of reporting: micro (patient-physician communication and intra-patient comparison), meso (comparison of patient groups within departments or institutions), macro (comparison of patient groups across departments or institutions) [47] Purpose of reporting: shared decision-making (for patient and physician), better basis for decision (for physician), interpretation support of data [72–74] Data collection: digital, analog [21, 41, 47] Time of data collection: directly during appointment, before the appointment (in waiting room), independent at home [21, 41] Dashboard features Sources Patient information: 5-point Likert scale on usefulness of feature [26, 46, 68] Clinical data: 5-point Likert scale on usefulness of feature [26, 46, 49, 50] Free write-in space: 5-point Likert scale on usefulness of feature [25, 43] Past assessment PROM score: 5-point Likert scale on usefulness of feature [20, 25, 43, 49] Peer-group comparison: 5-point Likert scale on usefulness of feature [42, 45, 49] PROM-related goals: 5-point Likert scale on usefulness of feature [24, 49] Overall health-related goals: 5-point Likert scale on usefulness of feature Alerts: Immediately when critical value appears, during appointment, no [25, 41] Customizability: To individual needs, from a standard set, no [24, 46, 51] University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs 10 3.2 Interviews In total, we conducted 16 online interviews with six software producers and ten users (m= 10, f=6) until thematic saturation was reached. All software producer interviewees represented different companies and different positions. All user interviewees were chief physicians with different specialties such as pneumology, cancer care, pediatrics, or orthopedics. This allowed us to analyze distinctive needs for design principles in clinical dashboards according to the episodes of care. All interviews lasted between 20 and 45 minutes (see Appendix V). The interviews with software producers and users revealed that they perceived a differentiation in design depending on disease (one-time intervention vs. chronic disease) as not essential. Therefore, we did not distinguish between design requirements of different disease types but instead focused on the design principles for clinical dashboards incorporating PROMs in general. General information To evaluate the importance of the key design principles, we asked both interview groups to outline the type of disease, the setting, the type of PROM used, and the targeted key user for their clinical dashboard (see Figure 3). Both software producers and users preferred a dashboard for tracking the evolution of chronic diseases over time (11/16) and using it in an outpatient setting (9/16). Both interview groups mentioned using a disease-specific PROM only (6/16) or combining generic and disease-specific PROM types (10/16). However, none of the interviewees preferred to use a generic PROM such as the EQ-5D only. Concerning the key users, slight discrepancies were observed. Software producers aimed to target all kinds of physicians (3/6) and other healthcare professionals as key users (3/6), whereas the users favored the specialist as a key user (5/10). Nevertheless, we only interviewed specialized healthcare professionals instead of GPs, which might have influenced their perception of the ideal key user (for the full list of codes, see Appendix VII). Legend: The boxes’ coloring indicates the individual items’ response intensity by software producers (SP; blue) and users (U; green). The more intense the coloring, the more often it was mentioned during the interviews. Data collection The most frequently mentioned use case for data collection was for micro (i.e., patient-physi- cian communication and intra-patient comparison) (11/16), especially by users (7/10), followed by macro perspectives (i.e., comparison of patient groups across departments or institutions) for both interview groups (4/16). Furthermore, the most often mentioned purposes of reporting Figure 3. Overview of software producer and user responses on general information University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs 11 were “moderation of data through physicians” (10/16) and “better basis for decision (for physician)” (5/16) (see Table 2). The interviews also covered the point-in-time and the type of data collection. Interviewees distinguished for both categories between the first time of data collection and follow-ups. For the point-in-time of data collection, there was a slight tendency (60%) of users to fill in the questionnaires for the first time in the waiting room prior to the appointment. I2 mentioned that in this way “[patients] get an explanation by a physician or a nurse, but after that, it is on their own.” However, for the follow-ups, 75% of the interviewees preferred that the filling of the questionnaires is done independently at home. For the type of data collection, a strong tendency towards a digital collection was reported (14/16). Some users (4/10) reported that data collection is still conducted on paper and sent to patients before appointments. Table 2. Overview of software producer and user responses on data collection Data collection Level of reporting Software producer 1. Micro (4/6) 2. Macro (2/6) 3. Meso (1/6) User 1. Micro (7/10) 2. Meso (3/10) 3. Macro (2/10) Purpose of reporting Software producer 1. Moderation of data through the physician (3/6) | Better basis for decision (for physician) (3/6) 2. Shared decision making (for patient and physician) (1/6) | Interpretation support of data (1/6) | Real-time tracking (1/6) 3. Expectation management (0/6) User 1. Moderation of data through the physician (7/10) 2. Expectation management (3/10) 3. Better basis for decision (for physician) (2/10) | Interpretation support of data (2/10) 4. Shared decision making (for patient and physician) (0/10) | Real-time tracking (0/10) Data collection (1st time) Software producer 1. Digital (6/6) 2. Analog (0/6) User 1. Digital (8/10) 2. Analog (4/10) Time of data collection (1st time) Software producer 1. Independent at home (4/6) 2. Before the appointment (in the waiting room) (2/6) 3. Directly during the appointment (0/6) User 1. Before the appointment (in the waiting room) (6/10) 2. Independent at home (5/10) 3. Directly during the appointment (0/10) Data collection (follow-up) Software producer 1. Digital (6/6) User 1. Digital (8/10) University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs 12 2. Analog (0/6) 2. Analog (4/10) Time of data collection (follow-up) Software producer 1. Independent at home (4/6) 2. Before the appointment (in the waiting room) (2/6) 3. Directly during the appointment (0/6) User 1. Independent at home (8/10) 2. Before the appointment (in the waiting room) (4/10) 3. Directly during the appointment (0/10) Legend: The table shows the ranking of design principles’ attributes according to the number of mentions by software producers and users. Behind the attributes, the share of interviewees mentioning this design feature is indicated. As the interviewees could provide more than one answer for each design principle of data collection, shares can add up to more than 100%. The level of reporting represents the main purpose of the dashboard – which we distinguish into three categories: micro, meso, and macro. Micro stands for the patient-physician communication and intra-patient comparison. Meso allows for the comparison of patient groups within departments or institutions. Macro focuses on the comparison of patient groups across departments or institutions. Dashboard components Concerning the dashboard components, especially users (4/10) mentioned that some patient information (such as patient photograph, demographic information or contact details of other care team members) needs to be presented on the dashboard (see Figure 4). However, only interviewee I16 emphasized precisely what she expects the patient information to include “key events. So, surgeries need to be shown. For example, in cancer, the start of chemotherapy, completion of chemotherapy, started radiation, that you can understand what is going on in the background of those patients.” Except for I1 and I5, all other software producers rated clinical data (e.g., lab results or medication data) as meaningful information that must be included in the clinical dashboard. In contrast, users tended rather not to include clinical data in the dashboard, as I8 mentioned: “[…] and then if they [the physicians] want to have the clinical information on the patient, they just open up the EMR and check it, which is another tab in the Chrome app.” The free write-in space did not resonate well in either of the two groups. While half of the software producers replied that a free write-in space is useful for specific questions, only one user perceived this design feature as very beneficial. Nevertheless, when having such a free write-in box included, users (3/10) wanted it as an additional source of information for some specific variables. This indicates that a free write-in space should be treated as an add-on to specific variables where the user can note further information that is important for patient treatment. Software producers and users rated the past assessment PROM score (10/16) as one of the most crucial features in a clinical dashboard incorporating PROMs. From the software producer perspective, I6 mentioned: “This [past assessment PROM score] is very well received, simply the score progression up and down visually, so to speak.” A similar perception presented I9: “This [past assessment PROM score] is absolutely relevant because it is about all changes in these questionnaires that are significant. In any case, it is very important to look at the progression, not just the individual value.” Another well-perceived feature was the peer-group comparison. All software producer interviewees agreed on the inclusion of this design principle. The users acknowledged the inclusion, too (7/10). Only two users, I9 and I12, did not perceive an added value in the peer-group comparison due to the interpretation possibilities of the applied PROM (I9) and the missing University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs 13 guidance on the relevant factors to compare different patient groups concerning their PROMs (I12). Although only one user emphasized applying PROM-related goals (especially for the EQ-5D), overall health-related goals were discussed more controversially within both groups. Software producer I4 reported that their dashboard includes a feature where the physician can develop overall health goals together with the patient and check goal achievement, whereas I6 stated that this is not part of their dashboard. From the user perspective, this feature could add value to the patient-physician communication, as the goal statement makes the aim or expectations of the patients explicit (I9, I11). For alerts, software producers and users have different perceptions on whether alerts should be included in the dashboard. Software producers mentioned that an alert function needs to be included (5/6). However, they were indifferent on whether the alert is real-time (4/6) or only appears during the appointment (4/6). I4 mentioned in this context that, however, the real-time tracking leverages the potential for legal consequences, especially when sending real-time notifications on critical values because “if [a score is for] three days red or is really critical and no push notification is sent, the error is on our side. However, if a push notification is sent and the doctor does not react, the error lies with the doctor. And that is a bit of a grey area, where we still have to figure out how it's actually done.” In contrast, users rather preferred not to include alerts into the dashboard (3/10). I8 and I11 mentioned that this feature was previously built in the dashboard, but the acceptance was not high enough in their teams, which made them stop using it. Further, I12 raised the issue that additional interpretations for physicians are required to ensure that they completely understand what the deterioration or improvement in a score means. I10 favored the alerts during the appointment to highlight critical factors and to facilitate comparison over time. I13 and I16 preferred real-time alerts allowing the treating physician to react directly to the patient’s issues. The last feature to be elaborated was the degree of customizability. All interviewees agreed that customizability is required to meet the different needs. Although more users preferred customizability from a standard set (4/10) compared to individual needs (3/10) – software producers still seemed undecided whether a clinical dashboard should be adapted to individual needs (3/6) or “off the shelf” (2/6). Nevertheless, the software producers agreed that scalability is only achievable in case clinical dashboards equipped with features defined in a standard set are provided. University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs 14 Legend: The boxes’ coloring indicates the individual items’ response intensity by software producers (SP; blue) and users (U; green). The more intense the coloring, the more often it was mentioned during the interviews. Findings from inductive coding We derived the above-presented results from the deductive procedure guided by the findings from the scoping literature review. The subsequent inductive procedure allowed us to identify the potential of dashboards to enhance workflows and various barriers, reducing the impact of the clinical dashboard. Although the visualization of the PROM scores was not extensively discussed in the literature, we asked our interviewees about their preferences. Software producers indicated that the index and dimensional scores are always provided in their dashboards. Users did not indicate such a clear tendency. Exemplary, I7 voted for the visualization of dimensional scores by stating, “we must of course know the dimensionality and different aspects.” In contrast, I11 argued: “I like [index] scores better, as I said, but because we have these individual questions like there are ten questions, and you can have a summary score of it and the system plots every question on a trend. I think that is rather messy because then you have like ten different color graphs just projected over each other, and you can click them on, or off. So it's easy, but for me it's less informative.” This finding implies that the visualization of PROMs – i.e., by index or dimensional scores – is highly dependent on personal preferences. Concerning the enhanced workflow enabled through the clinical dashboard, software producers emphasized the increased efficiency (4/6), improved overview of data (3/6), and better basis for decision (2/6). Users perceived the biggest advantages of a clinical dashboard the improved overview of data (6/10), PROMs comparability such as of the EQ-5D (5/10), and increased efficiency (5/10). Potential barriers to implementing the full potential of clinical dashboards are interoperability between various systems (11/16) and the consultation of different sources (5/10). Exemplary, I11 stated: “[…] one of the problems is now that it [the clinical dashboard] feeds from multiple databases. And one of the problems is that we have research projects. They have also PROMs and they are on different data sets. And we are not able to get them out. And I know that they [the patients] completed that Figure 4. Overview of responses from software producers and users concerning the dashboard features University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs 15 JAMAR [Juvenile Arthritis Multidimensional Assessment Report], and I will not ask them [the patients] to do it again because that is silly, but then I'm not able to see it and it will ruin my trends in the clinical dashboard. […] So now I have that clinical dashboard, but I need another screen. We have two screens in our office. One is for the clinical dashboard and on the other one I open the EMR.“ Further, software producers mentioned legal consequences in case of displaying inadequate information (4/6), and the licensing of the PROM questionnaires (3/6) as potential barriers. Users rather perceived the burdensome collection of PROM data (4/10), and non-intuitive use (3/10) as additional barriers. 3.3 Design, testing, and evaluation of clinical dashboard prototype 3.3.1 Design: Prototype based on literature review and interviews The interviews revealed that the design principles of clinical dashboards do not differ by different episodes of care – except for the inclusion of the disease-specific PROM. Therefore, we developed a clinical dashboard prototype incorporating PROMs only for COPD (see Figure 5). Following our literature review, the CAT was used as the disease-specific PROM [53–55]. The dashboard contains the design principles and possible features presented and evaluated in the interviews. At the top of the dashboard, there are patient information, clinical data (categorized as “health status”), and information on medication intake. These features only include the most relevant information for the physician to get a quick overview of the patient’s status. If the physician is interested in detailed information regarding the lung function test (see Figure 6), laboratory test results (see Figure 7), or imaging (see Figure 8), a new window with the respective information will open by clicking on the corresponding box. Figure 5. Clinical dashboard prototype according to the design principles collected in the scoping review and interviews University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs 22 care when building clinical dashboards, our research has shown that design principles do not have to differ between episodes of care – only the disease-specific PROM must be adapted. The scoping literature review also revealed that various design principles must be respected when building such a clinical dashboard. However, no study analyzed in the literature review presented a conclusive list of design principles. By collecting and testing the design principles, we were able to create such a list of essential design principles and build a dashboard prototype that might serve as a point of orientation for the future development of clinical dashboards incorporating PROMs. 4.2 Research in context An enhanced visualization of the patient’s PROMs over time can facilitate the moderation role of the physician [41, 47, 75]. The displayed level of detail of PROM scores led to controversial responses on whether the index or dimensional scores should be shown. Furthermore, software producers and users indicated that the dashboard content should not vary between diseases, except that the disease-specific PROM needs to be adapted according to the episode of care. Studies show that it is common to use generic PROMs, particularly the EQ-5D, to track the health status of patients. Notably, researchers see the standardized format and content as a major benefit that facilitates its usability across different diseases and patient groups [76]. Further, the multi-lin- gual questionnaire allows for large-scale analysis [66, 77]. Not only were disease-specific PROMs complementing generic PROMs, but their combined value was also seen as even more significant than their sum in the case of the EQ-5D and CAT combination [54]. Interestingly, similar findings emerged from studies developing PROM dashboards for various episodes of care. Baeksted et al. [43] and Hassett et al. [25] found it relevant to include the cancer-care-specific PRO-CTCAE in their dashboard, and Nicolas-Boluda et al. [21] incorporated endometriosis-specific indicators into their dashboard. Also, the interviews revealed that the combination of generic and disease-specific PROMs is considered valuable (10/16). The remaining interviewees (6/16) mentioned that they only use disease-specific PROMs. These statements provide insight that generic PROMs such as the EQ-5D gain enormous value if the information is also presented on the disease-specific condition. This combination allows the physician to take a more holistic view of the patient’s health status by linking the disease-specific condition to the overall health condition. Additionally, the advantages of the dashboard and potential barriers of implementing clinical dashboards were discussed in the interviews. Software producers and users perceived the most significant benefits of using a dashboard as increased workflow efficiency and an improved overview of data. Similar facilitators for implementation are also promoted by literature [34, 78]. However, the two interview groups highlighted different barriers: Software producers focused on legal consequences in case of displaying inadequate information, whereas users considered the consultation of various data sources or the burdensome collection of PROMs as a barrier. Both groups agreed on interoperability as one of the major barriers to implementing clinical dashboards. The barrier of interoperability is also recognized in the literature [79]. According to users, the primary reason for reporting is the moderation of data through the physician. Similarly, Desantis et al. [41] found that using a clinical dashboard improves the workflow and communication of changes in the HRQoL, i.e., data moderation, between the patient and physician. The dashboard features of patient information and clinical data were included, as the literature highlighted this additional information [26, 46, 68]. However, we kept this information to University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs 23 a minimum as not all users rated these features as highly relevant. Some write-in features were placed in the dashboard represented by “today’s prescription”, “further procedure”, and “further notes”. This allows the physician to note changes in treatment, the procedure for the follow-up months, or other symptoms of the patient [25, 43]. As software producers and users rated the past assessment score and the peer-group comparison as highly relevant features in the dashboard, they were also included in the dashboard. A graphical illustration connecting the scores over time implemented the past assessment score. Baeksted et al. [43] and Watson et al. [47] proposed a similar approach. The peer-group comparison was only implemented in the EQ-5D index score as the CAT already provides an interpretation basis for the index scores. Generally, peer-group comparison should match patients in age and treatment [42] and indicate a “normal” range [49]. Both interview groups had an ambiguous tendency to include PROM-related and overall healthrelated goals in the dashboard. The overall health-related goals were often favored over the PROM-related goals. Both goal types were included in the dashboard (as a free write-in space). Cronin et al. [24] found that patients want to set and evaluate goals over time, which could be supported by a clinical dashboard. However, Liu et al. [50] warned that setting goals might further pressure the patient. As a last feature, alerts were included in the dashboard prototype. Users indicated to prefer alerts during appointments over real-time alerts. Also, the software producers favored alerts during the appointment as “inadequate” or “non-appearing” alarms in real-time might lead to legal consequences, as mentioned by I4. The challenge of including alerts is to decide on the appropriate alert level [44] and thereby avoiding “alert fatigue” of physicians [26]. Currently, thresholds for PROM alerts are often not defined yet, as they need to be data-driven and medically sound. Additionally, they often depend on patient characteristics [80], and need to be adequately sensitive. To visualize the data, the opportunity of tailoring the dashboard to user needs and preferences should exist [50]. Additionally, Engelbrecht et al. [81] formulated some guidelines for visualizing information, such as removing distracting or extraneous information, placing a minimal cognitive load on the users, or consistently applying design choices. These guidelines were considered when building the clinical dashboard prototypes. 4.3 Practical implications | General implications This research project uncovered various implications for the future development of clinical dashboards incorporating PROMs. Embedding additional patient information next to PRO scores enhances the value of clinical dashboards The trend towards patient-centered care enhances the use of clinical dashboards incorporating PROMs. Such dashboards allow treating physicians to trace the patient’s health status over time, thereby retrieving information that would not be systematically available without the use of PROMs. Furthermore, the use of clinical dashboards in daily practice further strengthens patientphysician communication [72–74] and can be used as a moderation tool. Furthermore, the scoping literature review and the interviews revealed the importance of including additional information in the clinical dashboard, such as patient information and clinical data, to provide diverse content to the user – i.e., the physician – to consider when interpreting the PROM results. Thus, the clinical University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs 24 dashboard does not provide added value if only PROM results are represented without additional information. Furthermore, no visualization difference for dashboards focusing on various episodes of care is required – except for the adaption of the disease-specific PROM. Ensuring interoperability is a key driver for the successful implementation of clinical dashboards Additionally, interviewees mentioned interoperability as a challenge. Although interoperability is not in this project’s scope, we nevertheless wanted to raise this concern. As mentioned above, certain interviewees perceived additional features such as patient information or clinical data as valuable. However, interoperability of various systems often does not allow for transferring information across systems. Hence, it is currently the case that users of clinical dashboards either manually copy information or retrieve it from another system outside of the clinical dashboard. This burdensome collection of information from various sources harms the experience of clinical dashboards and needs to be improved in the future. Customizability of dashboard features is desirable based on context The design principle of customizability was extensively discussed during the interviews. Thereby, most users preferred customizability from a standard set. In contrast, software producers were indifferent about whether clinical dashboards should be customized to individual clients’ needs or whether some standardized solutions are requested. Additionally, customizability can be tackled from various perspectives such as: Should the dashboard be customizable on department or hospital/organizational level? Which disease-specific scores are to be chosen for which episode of care? Which design features (such as patient information or clinical data, alerts) could users get rid-off in case they demand it? To what degree can the individual features on the clinical dashboards be allocated according to personal preferences? This research project did not cover these questions – however, they might guide future research. Online data collection is preferred – however, patients might need support for the first time of collection Furthermore, we also addressed the collection of data with respect to the point-in-time and type. Collecting PROMs for the first time, interviewees preferred the waiting room, as assistance might be provided by a nurse or a medical assistant. All follow-up collections, the patient should conduct individually, e.g., at home or a place of own preference. The interviewees also reflected on the trend of collecting PROMs digitally. However, depending on the patient characteristics (e.g., for older patients), this collection approach might not be appropriate. Hence, we identified some tendencies for digital data collection which may vary for the first time of collection and the followup collection. Nevertheless, implementing such a process will highly depend on the routine care process of the respective health care providers. 4.4 Implications for EuroQol EuroQol holds the potential to significantly impact clinical practice through the effective utilization of EQ-5D in clinical dashboards. By collaborating with diverse stakeholders, refining the instrument's sensitivity, and customizing its application to different medical conditions, EuroQol can empower healthcare providers with valuable patient-reported data and facilitate evidencebased decision-making for improved patient outcomes. In the following, we will further elaborate on these points: University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs 25 A combination of generic and disease-specific PROM instruments considerably strengthens a physician’s understanding of the patient’s health status EuroQol’s future research and strategic activities should emphasize the value of utilizing both generic and disease-specific PROMs in clinical dashboards. Collaborating with medical specialists identify relevant disease-specific measures and determine optimal combinations with the EQ-5D will enhance the dashboard's effectiveness in real-world data collection and clinical decision-ma- king. We found that physicians highly value the availability of a combination of generic and disease-specific PROMs. Thus, a collaboration with medical specialists can further strengthen EuroQol’s understanding of the varying needs between various diseases. Furthermore, a true added value is only achieved if the combination of PROMs reflects the needs of the physicians and provides an appropriate amount of information. An increased use of the EQ-5D in clinical dashboards can be enhanced through collaborating with stakeholders Second, EuroQol should actively collaborate with software producers to promote the integration of the EQ-5D into clinical dashboards. Providing a compelling case for using the EQ-5D and offering implementation and interpretation support to software providers will facilitate the widespread adoption of the EQ-5D in health care settings. The interviews also covered the dashboard’s degree of the customizability. Thereby, the interviewees – especially the users – indicated that they prefer to build their clinical dashboard based on features from a standard set. Furthermore, software producers agreed that scalability is only achievable if the degree of customizability is limited. Hence, if software producers provide standard sets to choose from, it must be of utmost interest for EuroQol to collaborate with these software producers to ensure the EQ-5D being part of the standard set. Furthermore, this collaboration will further provide EuroQol with important insights on what users – foremost physicians and other health care professionals – care most about in their dashboards. Refining the EQ-5Ds sensitivity reduces ceiling effects and thus broadens its applicability Third, recognizing the ceiling effect of the EQ-5D in certain situations, EuroQol should work towards resolving this limitation to ensure the EQ-5Ds practicality and sensitivity in real-world applications. By refining the instrument to cater to a broader range of patient health states, EuroQol can enhance the EQ-5D’s utility in clinical practice. Depleting the ceiling effect raises the EQ-5D’s applicability in patients with good to almost perfect health status, as changes in the general health status will become more easily detectable for physicians. Therefore, the EQ-5D gains importance in tracking patients with early-stage diseases with low impact on the generic health status, too. Developing disease-specific EQ-5D thresholds allows for better informed decisions Fourth, to achieve comparability across different medical conditions, EuroQol should establish disease-specific EQ-5D thresholds. These thresholds will enable physicians to make informed comparisons and decisions based on patient-reported outcomes for specific diseases. The interviews revealed that 8/10 users would highly appreciate peer-group comparison. Additionally, all software producers agreed they want to entail a peer-group comparison feature to enhance the physician’s information gain when consulting the dashboard. To catch up with these needs, EuroQol may guide the development of disease-specific thresholds which allow for peer-group comparison or develop these themselves. This could provide an additional competitive advantage compared to other PROM providers to get into closer contact with software producers. Providing disease-specific thresholds permitting for peer-group comparison can function as a lever to ensure University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs 26 that EuroQol and the EQ-5D remain meaningful players in gathering patients’ self-reports, and thus also being part of clinical dashboards in the future. University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs 27 5 Limitations This project faces three main limitations: First, the study primarily relied on interviews with specialists and chief physicians, potentially limiting the diversity of perspectives. Including a broader range of stakeholders, such as nurses or other medical professionals (e.g., physiotherapists), could have provided a more comprehensive understanding of the requirements and challenges related to clinical dashboards. Also, the interviewees were mainly from Western countries – especially Switzerland and Germany. Furthermore, the user interviewees were most often specialists. We did not include GPs’ perspectives in our sample. In total, we included ten male and six female interviewees. However, we did not identify an attitude-gender gap – meaning that we could not observe differences in their answering or attitudes toward clinical dashboards based on gender. Future research should further consider various dashboard stakeholders from different countries to evaluate our proposed design principles in clinical dashboards. Second, the absence of further iterations after the focus groups might restrict the opportunity to refine the dashboard based on participants' feedback. Continuous iterations will lead to a more refined and user-centered prototype. However, as the project’s aim was identifying and evaluating critical design principles for clinical dashboard – not the effective implementation – we renounced from re-designing the dashboard prototype after conducting the focus group. The insights provided during the focus group serve as a basis for recommendations on the direction of future research. Third, the prototype's implementation in PowerPoint may not have accurately reflected the actual user experience of a functioning clinical dashboard. A real application will provide more realistic insights into users' opinions and potential usability issues. Also, the non-implementa- tion of the prototype into real-world clinical settings might limit the understanding of how it integrates with existing workflows and impacts daily practice. Prototyping under live conditions will further reveal unforeseen challenges or untapped potentials. University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs 28 6 Conclusion and outlook The holistic approach of a scoping literature review, user, and software producer interviews, and developing and testing a clinical dashboard prototype allowed us to establish relevant design principles for clinical dashboards incorporating PROMs. The semi-structured interview guide was motivated by our findings from the literature. This chosen procedure permitted cross-checking the state-of-the-art in the literature and detecting white spots where users and software producers showed diverging tendencies for certain features. The last step of developing and testing the clinical dashboard in focus groups enabled us to investigate the user experience, motivating another iteration of the design principles. The contribution of this research project is threefold: First, we determined the design principles for clinical dashboards based on a scoping literature review and interviews conducted with clinical experts and clinical dashboard software producers. Interestingly, the dashboard content should not differ for treating various episodes of care. The only feature that should be case-spe- cifically adapted is the disease-specific PROM. Second, we tested the inclusion of the EQ-5D into the clinical dashboard in combination with disease-specific PROMs, an approach that was rarely considered in recent literature despite its potential to enhance the physician’s understanding of the patient’s health status and choice of treatment pathway; third, we developed a prototype and discussed it with seven focus group participants, laying the basis for building and developing “real” clinical dashboards in the future. For future research, we recommend the following: Our goal was to develop design principles of a clinical dashboard incorporating PROMs that empower healthcare providers with a holistic view of patient information, simplifying complex data and enhancing decision-making processes. While our paper’s focus centered on the clinical dashboard’s design principles, we recognize the crucial importance of two key aspects: accessibility and implementation into existing hospital information systems. Future projects should commit to elaborate on these facets to ensure the successful adoption of PROM dashboards in clinical practice. Addressing these elements will be pivotal in overcoming potential barriers and fostering widespread acceptance among healthcare professionals. Additionally, expanding the applicability of the developed clinical dashboard prototype to inpatient care presents a tremendous opportunity to revolutionize medical care within hospital settings. To achieve this goal, we recognize the indispensable role of nursing staff in the research process. By incorporating their perspectives through interviews and focus groups in future projects, their needs will be better understood and reflected to create a comprehensive solution that benefits all stakeholders. Furthermore, to better tailor the EQ-5D to relevant episodes of care, a future research project should develop disease- and patient-specific thresholds to make the EQ- 5D more actionable for physicians. Our efforts to elaborate on clinical dashboards incorporating PROMs mark a critical milestone in our quest to enhance patient care. With this project, we set the foundation for a transformative tool that will revolutionize the healthcare landscape. Through collaboration and a commitment to continuous improvement, this project has the potential to contribute to a healthier and more informed society. University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs 29 References 1. Mulley AG, Trimble C, Elwyn G (2012) Stop the silent misdiagnosis: patients’ preferences matter. BMJ 345:. https://doi.org/10.1136/BMJ.E6572 2. Barry MJ, Edgman-Levitan S (2012) Shared Decision Making — The Pinnacle of Patient- Centered Care. New England Journal of Medicine 366:780–781. https://doi.org/10.1056/NEJMP1109283/SUPPL_FILE/NEJMP1109283_DISCLO- SURES.PDF 3. Sollecito W, Johnson J (2011) McLaughlin and Kaluzny’s continuous quality improvement in health care 4. Chen J, Ou L, Hollis SJ (2013) A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting. BMC Health Serv Res 13:211. https://doi.org/10.1186/1472-6963-13-211 5. Devlin NJ, Appleby J (2010) Getting the most out of PROMs. Putting health outcomes at the heart of NHS decision making. King’s Fund 6. Greenhalgh J, Meadows K (1999) The effectiveness of the use of patient-based measures of health in routine practice in improving the process and outcomes of patient care: a literature review. J Eval Clin Pract 5:401–416. https://doi.org/10.1046/j.1365-2753.1999.00209.x 7. Hassett MJ, Cronin C, Tsou TC, et al (2022) eSyM: An electronic health record–integrated patient-reported outcomes-based cancer symptom management program used by six diverse health systems. JCO Clin Cancer Inform 1–10. https://doi.org/10.1200/cci.21.00137 8. Cronin RM, Conway D, Condon D, et al (2018) Patient and healthcare provider views on a patient-reported outcomes portal. Journal of the American Medical Informatics Association 25:1470–1480. https://doi.org/10.1093/jamia/ocy111 9. Desantis D, Baverstock RJ, Civitarese A, et al (2016) A clinical perspective on electronically collecting patient-reported outcomes at the point-of-care for overactive bladder. Canadian Urological Association Journal 10:359. https://doi.org/10.5489/cuaj.3757 10. Fautrel B, Alten R, Kirkham B, et al (2018) Call for action: how to improve use of patientreported outcomes to guide clinical decision making in rheumatoid arthritis. Rheumatol Int 38:935–947. https://doi.org/10.1007/s00296-018-4005-5 11. Detmar SB (2003) Use of HRQOL questionnaires to facilitate patient–physician communication. Expert Rev Pharmacoecon Outcomes Res 3:215–217. https://doi.org/10.1586/14737167.3.3.215 12. Cella D, Yount S, Rothrock N, et al (2007) The Patient-Reported Outcomes Measurement Information System (PROMIS). Med Care 45:S3–S11. https://doi.org/10.1097/01.mlr.0000258615.42478.55 13. Lipscomb J, Reeve BB, Clauser SB, et al (2007) Patient-Reported Outcomes Assessment in Cancer Trials: Taking Stock, Moving Forward. Journal of Clinical Oncology 25:5133–5140. https://doi.org/10.1200/JCO.2007.12.4644 14. Greenhalgh J, Gooding K, Gibbons E, et al (2018) How do patient reported outcome measures (PROMs) support clinician-patient communication and patient care? A realist synthesis. J Patient Rep Outcomes 2:42. https://doi.org/10.1186/s41687-018-0061-6 University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs 30 15. Marshall S, Haywood K, Fitzpatrick R (2006) Impact of patient-reported outcome measures on routine practice: a structured review. J Eval Clin Pract 12:559–568. https://doi.org/10.1111/j.1365-2753.2006.00650.x 16. Berry DL, Hong F, Halpenny B, et al (2014) Electronic Self-Report Assessment for Cancer and Self-Care Support: Results of a Multicenter Randomized Trial. Journal of Clinical Oncology 32:199–205. https://doi.org/10.1200/JCO.2013.48.6662 17. Berry DL, Blumenstein BA, Halpenny B, et al (2011) Enhancing Patient-Provider Communication With the Electronic Self-Report Assessment for Cancer: A Randomized Trial. Journal of Clinical Oncology 29:1029–1035. https://doi.org/10.1200/JCO.2010.30.3909 18. Khairat SS, Dukkipati A, Lauria HA, et al (2018) The Impact of Visualization Dashboards on Quality of Care and Clinician Satisfaction: Integrative Literature Review. JMIR Hum Factors 5:. https://doi.org/10.2196/HUMANFACTORS.9328 19. Dowding D, Randell R, Gardner P, et al (2015) Dashboards for improving patient care: Review of the literature. Int J Med Inform 84:87–100. https://doi.org/10.1016/j.ijmedinf.2014.10.001 20. Fautrel B, Alten R, Kirkham B, et al (2018) Call for action: How to improve use of patientreported outcomes to guide clinical decision making in rheumatoid arthritis. Rheumatol Int 38:935–947. https://doi.org/10.1007/s00296-018-4005-5 21. Nicolas-Boluda A, Oppenheimer A, Bouaziz J, Fauconnier A (2021) Patient-reported outcome measures in endometriosis. J Clin Med 10:5106. https://doi.org/10.3390/jcm10215106 22. LeRouge C, Hasselquist MB, Kellogg L, et al (2017) Using Heuristic Evaluation to Enhance the Visual Display of a Provider Dashboard for Patient-Reported Outcomes. e- GEMs (Generating Evidence & Methods to improve patient outcomes) 5:6. https://doi.org/10.13063/2327-9214.1283 23. Hartzler AL, Izard JP, Dalkin BL, et al (2016) Design and feasibility of integrating personalized PRO dashboards into prostate cancer care. Journal of the American Medical Informatics Association 23:38–47. https://doi.org/10.1093/jamia/ocv101 24. Cronin RM, Conway D, Condon D, et al (2018) Patient and healthcare provider views on a patient-reported outcomes portal. Journal of the American Medical Informatics Association 25:1470–1480. https://doi.org/10.1093/jamia/ocy111 25. Hassett MJ, Cronin C, Tsou TC, et al (2022) eSyM: An electronic health record–integrated patient-reported outcomes-based cancer symptom management program used by six diverse health systems. JCO Clin Cancer Inform 1–10. https://doi.org/10.1200/cci.21.00137 26. Rudin RS, Perez S, Rodriguez JA, et al (2021) User-centered design of a scalable, electronic health record-integrated remote symptom monitoring intervention for patients with asthma and providers in primary care. Journal of the American Medical Informatics Association 28:2433–2444. https://doi.org/10.1093/jamia/ocab157 27. Rupp M, Lau E, Kurtz SM, Alt V (2020) Projections of primary TKA and THA in Germany from 2016 through 2040. Clin Orthop Relat Res 478:1622–1633. https://doi.org/10.1097/CORR.0000000000001214 28. OECD (2019) Health at a glance 2019: OECD indicators. OECD Publishing, Paris 29. Beck M, Christen B, Zdravkovic V, Brand Ch (2021) Implantatregister SIRIS - Hüfte und Knie 30. Global Initiative for Chronic Obstructive Lung Disease (2018) Global Strategy for the diagnosis, management, and prevention of Chronic Obstructive Pulmonary Disease University of St. Gallen | School of Medicine | Chair of Health Care Management Scientific Report Design Principles of Clinical Dashboards Incorporating PROMs 31 31. Lozano R, Naghavi M, Foreman K, et al (2012) Global and regional mortality from 235 causes of death for 20 age groups in 1990 and 2010: A systematic analysis for the Global Burden of Disease Study 2010. The Lancet 380:2095–2128. https://doi.org/10.1016/S0140- 6736(12)61728-0 32. Federal Health Office (2020) Chronische Atemwegserkrankungen. https://www.bag.admin.ch/bag/de/home/krankheiten/krankheiten-im-ueberblick/chronische-atemwegser- krankungen.html 33. Wilbanks BA, Langford PA (2014) A Review of Dashboards for Data Analytics in Nursing. CIN: Computers, Informatics, Nursing 32:545–549. https://doi.org/10.1097/CIN.0000000000000106 34. Dowding D, Randell R, Gardner P, et al (2015) Dashboards for improving patient care: Review of the literature. Int J Med Inform 84:87–100. https://doi.org/10.1016/j.ijmedinf.2014.10.001 35. Al Sayah F, Lahtinen M, Bonsel GJ, et al (2021) A multi-level approach for the use of routinely collected patient-reported outcome measures (PROMs) data in healthcare systems. J Patient Rep Outcomes 5:98. https://doi.org/10.1186/s41687-021-00375-1 36. Gregor S, Kruse L, Seidel S (2020) Research perspectives: The anatomy of a design principle. J Assoc Inf Syst 21:1622–1652. https://doi.org/10.17705/1jais.00649 37. Chandra L, Seidel S, Gregor S (2015) Prescriptive knowledge in IS research: Conceptualizing design principles in terms of materiality, action, and boundary conditions. In: 2015 48th Hawaii International Conference on System Sciences. IEEE, pp 4039–4048 38. Munn Z, Peters MDJ, Stern C, et al (2018) Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol 18:143. https://doi.org/10.1186/s12874-018-0611-x 39. Tricco AC, Lillie E, Zarin W, et al (2018) PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med 169:467–473. https://doi.org/10.7326/M18-0850 40. Tong A, Sainsbury P, Craig J (2007) Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care 19:349–357. https://doi.org/10.1093/intqhc/mzm042 41. Desantis D, Baverstock RJ, Civitarese A, et al (2016) A clinical perspective on electronically collecting patient-reported outcomes at the point-of-care for overactive bladder. Canadian Urological Association Journal 10:359. https://doi.org/10.5489/cuaj.3757 42. Hartzler AL, Izard JP, Dalkin BL, et al (2016) Design and feasibility of integrating personalized PRO dashboards into prostate cancer care. Journal of the American Medical Informatics Association 23:38–47. https://doi.org/10.1093/jamia/ocv101 43. Baeksted C, Pappot H, Nissen A, et al (2017) Feasibility and acceptability of electronic symptom surveillance with clinician feedback using the Patient-Reported Outcomes version of Common Terminology Criteria for Adverse Events (PRO-CTCAE) in Danish prostate cancer patients. J Patient Rep Outcomes 1:1. https://doi.org/10.1186/s41687-017- 0005-6 44. Mooney K, Whisenant MS, Beck SL (2019) Symptom care at home: A comprehensive and pragmatic PRO system approach to improve cancer symptom . Med Care 57:S66–S72. https://doi.org/10.1097/MLR.0000000000001037 xxxviii Huber et al. 2020 1) EQ-5D biases: HRQoL of obese patients with mild to severe COPD might improve following weight reduction. For very severe COPD, a negative association of obesity and HRQoL could not be confirmed. COPD C No Yes Yes Jin et al. 2019 1) EQ-5D relevance: 5L better than 3L in THA/TKA, differentiates patients better based on their mobility THA/TKA A No Yes Yes Khairat et al. 2018 1) Visuals: colourcode to spot patients with severe symptoms 2) Challenges: Alerts may lead ot information overload N/A N/A Yes No No LeRouge et al. 2017 1) Dashboard visuals: Heuristics revealed visual characteristics, e.g., Exclude data labels from column; Unified colours, differentiated darker colour is data to emphasize; Greatest interest outcomes on top-left corner; Descriptive graph titles 2) Dashboard design: Use HCD N/A N/A Yes Yes No Leutner et al. 2021 1) Customization: different tabs, Expandable information, possibility of corrections, including and excluding diagnoses and patients, filter by time periods Rare diseases Both Yes No No Liu et al. 2020 1) Dashboard contents: Include key lab results, Some patients did not like the goal function 2) Customization: Add widgets to dashboard, ability to customize is important Rheumatoi d arthritis C Yes Yes No Liu et al. 2018 1) Use in practice: Physicians should clearly communicate the importance of PROMs to help generate “buy in” 2) Data collection: Preferences for pre-visit PROMs at home, delivered by email or text message. Technological is stilla barrier but text messages seemed to be a good bridge. The message must clearly indicate that it is healthcare-related. THA/TKA A No Yes No Lutz et al. 2022 1) Dashboard design: for success, need for clear and integrated dashboards showing relevant information only Cancer C No Yes Yes Marshall et al. 2021 1) PROMs benefits: dashboard helps patients to have realistic expectations for TKA outcomes and promote shared decisionmaking with their care provider 2) Data collection timeframe: Before surgery, 3-months, and 12-months post-surgery. Reporting on 3-month results is done in practice since most of the functional improvement is achieved by then 3) Data collection method: Critical success factors for electronic data collection are the availability of staff to assist patients with the technology, WIFI connectivity, and dedicated space for patients to complete their PROMs. THA/TKA A No Yes No Marten et al. 2021 1) Accessibility: elderly may need additional assistance to complete questionnaires N/A N/A No Yes Yes Merino et al. 2019 1) EQ-5D relevance: Uses EQ-5D to evaluate HRQL 2) EQ-5D biases: In COPD: greatest problems in mobility and pain/discomfort 3) EQ-5D biases: having suffered exacerbations in the last year, presenting a higher level of severity, being a woman, and having a low education level are related to worse HRQL in patients with COPD. COPD C No Yes Yes Molloy et al. 2020 1) Data collection method: can ensure PROM completion by sending automated text message reminders to patients (esp. For younger ones) 2) Data collection timeframe: In THA/TKA, suggest to expand the follow-up questionnaire time period to 13 months as completion increased to 46.2% through this. THA/TKA A No Yes No Mooney et al. 2019 1) Dashboard visuals: line graphs, Horizontal lines discriminating mild, moderate, and severe scores 2) Dashboard features: alerts for high severity symptoms, overlay symptom graphs to look at symptom clustering Cancer C Yes Yes No xxxix 3) Dashboard contents: 2) EHR integration: Need for integration in clinical workflow Nicolas-Boluda et al. 2021 1) PROMs benefits: patient-centred care, decision making, enable comparisons with peer-group, help to raise concerns Endometrio sis C Yes Yes Yes Nishimura et al. 2019 1) Data collection method: PROMs questionnaire is either selfadministered under supervision using a tablet computer at our outpatient clinic or as paper completed at home and returned by mail 2) Data collection method challenge: Cannot use paper and digital version interchangeably because there are big differences in scores reported COPD C No Yes No Nolan et al. 2016 1) EQ-5D-5L relevance: 5L is good for use in COPD, helps to differentiate between groups defined according to disease severity. 2) Disease-specific measures: EQ-5D-5L is correlated to disease-specific responses (and their changes) COPD C No Yes Yes Oeser et al. 2018 1) Dashboard contents: 3 classes of data to be displayed: patient, disease and therapy metrics Cancer C Yes No No Pellizzoni et al. 2020 1) Use in practice: Need for multilingual collection system in some countries (here Brazil) N/A N/A Yes Yes Yes Ragouzeos et al. 2019 1) Dashboard content: 3 types of data important for patient and physician: lab results, PROs, and medication data 2) Dashboard visuals: lab data should be placed at bottom of page, smaller than PROs because not the focus Rheumatoi d arthritis C Yes Yes No Rolfson et al. 2016 1) EQ-5D biases: Include and adjust for: age, sex, diagnosis at joint, general health status preoperatively, and joint pain and function score for THA/TKA 2) PROMs collection: Immediately before and 1 year after surgery 3) Disease-specific PROMs: should complement EQ-5D with a 1-item pain question and a single-item satisfaction outcome THA/TKA A No Yes Yes Rudin et al. 2021 1) EHR integration: integrated remote symptom monitoring 2) Dashboard features: data dashboard accessible from the EHR in 1 click + sending EHR inbox message preceding the visit. 3) Dashboard visuals: Divergent opinions between higher=better asthma control OR worse control 4) Dashboard contents: current asthma medications and refill data, recent ED visits or hospitalizations, name of asthma specialist treating patient Asthma C Yes Yes No Sen et al. 2022 1) EQ-5D biases: women have more disabilities than men in osteoarthritis knee THA/TKA A No Yes Yes Shewchuk et al. 2021 1) Dashboard features: option to highlight red flags intended to be discussed with an HCP + ease of reading for patients (eg, add a legend, increase contrast and font size, and reduce reading level) Knee OA C Yes Yes Yes Smith et al. 2019 1) PROMs dashboard benefit: improving recall COPD C No Yes Yes Spronk et al. 2021 1) Disease-specific PROMs: Adding a burn-specific item to the EQ-5D-5L is possible and has potential. Burns A No Yes Yes Strachna et al. 2021 1) Dashboard visuals: line graphs most effective to show HRQoL 2) Dashboard contents: Comparison group Cancer C Yes Yes No xl Szentes et al. 2020 1) Disease-specific PROMs: Combined use of the EQ-5D and the CAT is seen as a promising approach to best depict HRQL in COPD COPD C No Yes Yes Tai et al. 2020 1) Dashboard benefits: helps patients to understand what clinical factors explain changes in health status Falls A No Yes Yes Taxter et al. 2021 1) Dashboard visuals: Clearly labelled graphs, and vertical orientation to facilitate review and discussion 2) Dashboard content: trending data over time, personalization with patient photo and updates on life before visit JIA C Yes Partially No Tsangaris et al. 2022 1) Dashboard features: Radar allows to see overall picture, Possibility to display item-level responses, PROMs with labels (ie, up/down arrows and equal symbols) showing changes vs. previous score 2) Customization: Enables transferability to other institutions or department 3) Dashboard contents: Photograph of the patient, graph summaries, and recommendations including links to relevant resources Cancer C Yes Yes No Van Citters et al. 2020 1) PROMs dashboard benefits: supported discussions of what matters most 1) Challenges: dashboard was seen by physicians as less comprehensive and timely, more work than their EMR Cystic fibrosis C Yes Yes No Wang et al. 2021 1) EQ-5D relevance: there are other uses of EQ-5D than for economic assessments. It can be used in patient-physician communication 2) Collection frequency: In cancer, EQ-5D usually administered at each chemotherapy cycle N/A N/A No Yes Yes Watson et al. 2021 1) Dashboard contents: patient’s six most recent PROMs answers, the patient’s priority concern, and clinical actions taken in the encounter 2) Dashboard visuals: Visual flag to identify patients with high number of symptoms/concerns 3) Dashboard features: longitudinal trending and visual cues to easily differentiate mild symptoms from moderate or severe which informed the colour coded trends Cancer C Yes Yes Yes Zhou et al. 2021 1) EQ-5D biases: Discriminative ability of EQ-5D because of the variances depending on characteristics. E.g., sex, age and comorbidities COPD C No Yes Yes *A/C: Acute vs. Chronic condition. N/A is written in case there was no precise disease in the focus of the study The sections Dashboard, PROM and EQ-5D show which study was focused on which topic. Appendix II. Consolidated criteria for reporting qualitative studies (COREQ): 32-item checklist No. Item Guide questions/description Reported on Page # Domain 1: Research team and reflexivity Personal Characteristics 1. Interviewer/facilitator Which author/s conducted the interview or focus group? Page 7 2. Credentials What were the researcher’s credentials? E.g. PhD, MD Page 7 3. Occupation What was their occupation at the time of the study? Page 7 4. Gender Was the researcher male or female? n/a 5. Experience and training What experience or training did the researcher have? Page 7 Relationship with participants 6. Relationship established Was a relationship established prior to study commencement? Page 7 . 7. Participant knowledge of the interviewer What did the participants know about the researcher? e.g. personal goals, reasons for doing the research Page 7 8. Interviewer characteristics What characteristics were reported about the interviewer/facilitator? e.g. Bias, assumptions, reasons and interests in the research topic Page 7 Domain 2: study design Theoretical framework 9. Methodological orientation and Theory What methodological orientation was stated to underpin the study? e.g. grounded theory, discourse analysis, ethnography, phenomenology, content analysis Page 7 Participant selection 10. Sampling How were participants selected? e.g. purposive, convenience, consecutive, snowball Page 7 11. Method of approach How were participants approached? e.g. face- to-face, telephone, mail, email Page 10 12. Sample size How many participants were in the study? Page 10 13. Non-participation How many people refused to participate or dropped out? Reasons? n/a Setting 14. Setting of data collection Where was the data collected? e.g. home, clinic, workplace Page 10 . 15. Presence of non-partici- pants Was anyone else present besides the participants and researchers? No, only researcher and interviewee 16. Description of sample What are the important characteristics of the sample? e.g. demographic data, date Appendix VI Data collection 17. Interview guide Were questions, prompts, guides provided by the authors? Was it pilot tested? Page 7 an Appendix IV 18. Repeat interviews Were repeat interviews carried out? If yes, how many? No 19. Audio/visual recording Did the research use audio or visual recording to collect the data? Page 7 20. Field notes Were field notes made during and/or after the interview or focus group? Page 7 21. Duration What was the duration of the inter views or focus group? Appendix VI 22. Data saturation Was data saturation discussed? Page 7 23. Transcripts returned Were transcripts returned to participants for comment and/or correction? The transcripts were returned to interviewees to get their approval. Domain 3: analysis and findings Data analysis 24. Number of data coders How many data coders coded the data? Page 7 25. Description of the coding tree Did authors provide a description of the coding tree? Appendix VII 26. Derivation of themes Were themes identified in advance or derived from the data? Page 7 27. Software What software, if applicable, was used to manage the data? Page 7 28. Participant checking Did participants provide feedback on the findings? No Reporting 29. Quotations presented Were participant quotations presented to illustrate the themes/findings? Was each quotation identified? e.g. participant number Pages 11-15 30. Data and findings consistent Was there consistency between the data presented and the findings? Pages 11-15 31. Clarity of major themes Were major themes clearly presented in the findings? Appendix VII 32. Clarity of minor themes Is there a description of diverse cases or discussion of minor themes? Appendix VII Appendix III: Introduction slides presented before each interview Appendix IV. Interview guide for software producers and users 1) General questions on dashboards and PROM usage Software Producer User - Please describe the dashboard you are producing/designing. - In which countries or regions is your dashboard available? - What is the major goal of your clinical dashboard? - For what type of work (e.g., analyzing - What is your motivation for using clinical dashboards? - For what type of work (e.g., analyzing outcomes over time, communicating with the patient, comparing to other groups, getting a better overview of all data collected) do you use the outcomes over time, communicating with the patient, comparing to other groups, getting a better overview of all data collected) is the clinical dashboard designed for? - What do you think is beneficial about using clinical dashboards? - Where do you see major barriers in using clinical dashboards? - Do you incorporate PROMs in your clinical dashboard? If yes, which ones (standardized sets vs. own creation)? If no, why not? - Does the user have the possibility to choose from a set of available PROMs or is it pre-defined by you? - Do you use generic and disease-spe- cific PROMs, or just one of each? Why? - Why do you think it is beneficial to incorporate PROMs into clinical dashboards? - What do you perceive as challenging when incorporating PROMs into clinical dashboards? clinical dashboard? - What do you like about the clinical dashboard you currently use? - What don't you like about the clinical dashboard you currently use? - Do you use PROMs? If yes, which ones (standardized sets vs. own creation)? If no, why not? - Do you use generic and disease-spe- cific PROMs, or just one of each? Why? - Are PROMs already integrated into the clinical dashboard? - What is your motivation for using PROMs? - For what type of work (e.g., analyzing outcomes over time, communicating to patients, comparing to other groups) do you use PROMs? - What do you like about using PROMs? - What don't you like about using PROMs? 2) Questions about the market (penetration) Software Producer User - How many clients do already use the clinical dashboards incorporating PROMs to communicate with patients? - How do you feel about the demand for these dashboards? - What designs are requested? - What is the general feedback from customers on your solution? What do they like - what don't they like? - Is there a scientific basis for the design of your dashboards? Are physicians involved in the development process? Not asked to users. 3) Usage of the dashboard Software Producer User - In how far does your product facilitate the workflow of practitioners? - Who uses and has access to the clinical dashboard? - Do you have a special area of expertise, or can the dashboard be used at any discipline? - Can it be used in an outpatient and inpatient setting? - Can it only be used in one department/hospital or is it conceivable that it could also be used, for example, by outpatient care providers such as primary care physicians in parallel? - What type of data is available in the clinical dashboard? - Do you receive regular feedback from your users on how valuable the included data is? - Do you provide support in using the clinical dashboards? - Do your customers own or rent the software of the clinical dashboard? - Do you think different episodes of care (e.g., orthopedics and COPD, one-time intervention vs chronic disease) require different dashboard capabilities? Why? - To what extent does using the clinical dashboard make your day-to-day work easier? - Who uses and has access to the clinical dashboard? - Is it only for use in your department/hospital or is it conceivable that it could also be used, for example, by outpatient care providers such as primary care physicians? - Is all the data you need in your clinical practice included in the clinical dashboard? - What type of data is available in the clinical dashboard? - In your opinion, is there a feature missing that you would find particularly valuable? - What addition did you notice when using the clinical dashboard for the first time? - Did you feel that using a clinical dashboard was complicated and could be made easier in the future? - Do you find that using the clinical dashboard for the first time was intuitive? - Have you received any support, or have you familiarized yourself with its use? - Do you think different episodes of care (e.g., orthopedics and COPD) require different dashboard capabilities? Why? 4) Dashboard development and data collection Software Producer User - How do you develop a dashboard for a client? Do you have a basic product that is customizable to different needs, or do you develop it every time from scratch according to the needs of your client? - Who provides the dashboard you use? - How was it developed? - Is it customizable to individual needs? - How was the clinical dashboard - How would you describe your working mode? Do you work agile or in a waterfall structure? - How many iteration cycles do you go through until the delivery of the final product? - How does the roll-out of a new clinical dashboard work? - How does the data collection work? (Where, when, how, who, ...) - Where do you see difficulties in data collection? - Where is the data stored? - Do you have access to the data collected by your customers? - Is the collected data also used for other purposes than for the improvement in the patient-physician communication? - E.g., aggregation of data and comparison between hospitals, other research purposes, etc. implemented in your organization? - How does the data collection work? (Where, when, how, who, ...) - Where are difficulties in the data collection? 5) Feature Assessment Same questions for software producers and users From the literature, we have extracted some design principles that could be included in clinical dashboards - please provide your opinion on displaying these design principles and whether you already incorporate these design principles in your clinical dashboard: - Previous PROM assessments (including evolution over time). - What kind of scores does the dashboard display? Values for individual dimensions or index scores? Why? - Future PROM-related goals - Benchmarking | peer-group comparison - Alerts on symptom/symptom change warnings - If yes, at what time should the alarm/warning appear? - Patient information (patient photo, demographic information, recent health updates, and contact information for other care team members) - Clinical data (laboratory results and drug data) - What medical data do you use? - Free write-in space Software Producer User I1 I2 I3 I4 I5 I6 I7 I8 I9 I10 I11 I12 I13 I14 I15 I16 - independently at home X X X X X X X X Follow-up data collection X X X X X X X X X X X - mail with link X X X X - online/ QR code X X X - paper X X X X - telephone/ tablet/app X X X X X X Follow-up time of data collection X X X X X X X X X X - in waiting room X X X X X X - independent at home X X X X X X X X X Free write-in space X X X X X X X X X X X - none X X X X - specific question X X X X - yes X X X Goals X X X X X X X X X X X - excellent idea X - not included X X X X X - overall healthrelated goals X X X X X X Key user X X X X X X X - all kind of physicians X X X X - other health care professionals X X X X X - patient X - physician X - specialist X X Level of reporting X X X X X X X X - macro X X X X - meso X X X - micro X X X X X X past assessment score X X X X X X X X X X patient information X X X X - key events X Software Producer User I1 I2 I3 I4 I5 I6 I7 I8 I9 I10 I11 I12 I13 I14 I15 I16 Patient perspective X X X X X X X X X X X X X X X Payment X X X X X - add-on to product (for free) X X - license X X X X Peer-group comparison X X X X X X X X X X X X X X X - no X X - yes X X X X X X X X X X X - yes, but not possible X X X Personal data X X X X X X - no X X X - yes X X X Purpose of reporting X X X X X X X X X - better basis for physician's decision X X X - communication X X X X X X X X - expectation management X X X - real-time tracking X X - shared decision-making X X Roll-out X X X X X - step by step (in different clinics) X X X X - top down X Scores X X X X X X X X X X X - both X X X X - dimensional X X X X - index X X X Setting X X X X X X - in- & outpatient X X X X - inpatient - outpatient X X System support X X X X X X X X X X - in-house at corresponding institution X Software Producer User I1 I2 I3 I4 I5 I6 I7 I8 I9 I10 I11 I12 I13 I14 I15 I16 - interpretation of PROM scores X X X X X - with implementation X X X X X - none X X - remote support X X - workshops/ webinars X Type of disease X X X X - chronic X X X - combination X X - one-time intervention Type of PROM X X X X X X X X X X X X X X - combination X X X X X X X X - disease-spe- cific X X X X X X - generic - only for research X Improved workflow X X X X X X X X X X X X X - better basis for decision X X - better overview over data X X X X X X X X X - direct communication to patient X X X - efficiency X X X X X X X X X - facilitates workflow X X - inclusion into preoperative planning X - one platform for all information X - patient satisfaction X X X - program in whole institution the same X - PROMs comparability X X X X X X X - PROMs measurability X X Software Producer User I1 I2 I3 I4 I5 I6 I7 I8 I9 I10 I11 I12 I13 I14 I15 I16 - visualisation X X X Appendix VIII. ’ Participant Gender Specialty Role Qualification 1 Male Internal Medicine General practitioner Specialist in General Internal Medicine 2 Male Nephrology Chief physician Specialist in Internal Medicine and Nephrology 3 Male Pediatric and adolescent surgery Senior physician Specialist in Pediatric Surgery 4 Female Cardiology Senior physician Specialist in Cardiology 5 Female Clinical Oncology and Hematology Senior physician Specialist in Clinical Oncology and Internal Medicine 6 Female n.a. Medical student n.a. 7 Female n.a. Medical student n.a.