Methods used to assess outcome consistency in clinical studies: A literature-based evaluation

RESEARCH ARTICLE Methods used to assess outcome consistency in clinical studies: A literature-based evaluation Ewelina RogozińskaID 1,2 *, Elizabeth Gargon 3 , Rocı ´o Olmedo-Requena 4,5,6 , Amani Asour 2 , Natalie A. M. Cooper 2 , Claire L. Vale 1 , Janneke van’t Hooft 7 1Meta-Analysis Group, Institute of Clinical Trials and Methodology, MRC Clinical Trials Unit at UCL, London, England, United Kingdom, 2Women’s Health Research Unit, Queen Mary University of London, London, England, United Kingdom, 3Department of Biostatistics, University of Liverpool, Liverpool, England, United Kingdom, 4Department of Preventive Medicine and Public Health, School of Medicine, University of Granada, Granada, Spain, 5Consortium for Biomedical Research in Epidemiology and Public Health (CIBERESP), Madrid, Spain, 6Instituto de Investigacio ´n Biosanitaria ibs.GRANADA, Granada, Spain, 7Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, California, United States of America *[email protected] Abstract Evaluation studies of outcomes used in clinical research and their consistency are appearing more frequently in the literature, as a key part of the core outcome set (COS) development. Current guidance suggests such evaluation studies should use systematic review methodology as their default. We aimed to examine the methods used. We searched the Core Outcome Measures in Effectiveness Trials (COMET) database (up to May 2019) supplementing it with additional resources. We included evaluation studies of outcome consistency in clinical studies across health subjects and used a subset of A MeaSurement Tool to Assess systematic Reviews (AMSTAR) 2 (items 1–9) to assess their methods. Of 93 included evaluation studies of outcome consistency (90 full reports, three summaries), 91% (85/93) reported performing literature searches in at least one bibliographic database, and 79% (73/93) was labelled as a “systematic review”. The evaluations varied in terms of satisfying AMSTAR 2 criteria, such that 81/93 (87%) had implemented PICO in the research question, whereas only 5/93 (6%) had included the exclusions list. None of the evaluation studies explained how inconsistency of outcomes was detected, however, 80/90 (88%) concluded inconsistency in individual outcomes (66%, 55/90) or outcome domains (20%, 18/ 90). Methods used in evaluation studies of outcome consistency in clinical studies differed considerably. Despite frequent being labelled as a “systematic review”, adoption of systematic review methodology is selective. While the impact on COS development is unknown, authors of these studies should refrain from labelling them as “systematic review” and focus on ensuring that the methods used to generate the different outcomes and outcome domains are reported transparently. PLOS ONE PLOS ONE | https://doi.org/10.1371/journal.pone.0235485 July 8, 2020 1 / 12 a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 OPEN ACCESS Citation: Rogozińska E, Gargon E, Olmedo- Requena R, Asour A, Cooper NAM, Vale CL, et al. (2020) Methods used to assess outcome consistency in clinical studies: A literature-based evaluation. PLoS ONE 15(7): e0235485. https://doi. org/10.1371/journal.pone.0235485 Editor: Tim Mathes, Universitat Witten/Herdecke, GERMANY Received: March 24, 2020 Accepted: June 17, 2020 Published: July 8, 2020 Peer Review History: PLOS recognizes the benefits of transparency in the peer review process; therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. The editorial history of this article is available here: https://doi.org/10.1371/journal.pone.0235485 Copyright: ©2020 Rogozińska et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability Statement: All relevant data are within the paper and its Supporting Information files. Introduction Inconsistency (or heterogeneity) of outcomes measured in clinical studies is a widely recognised problem hindering evidence synthesis. [1–5] Core outcome sets (COS), defined as a minimum set of outcomes to be reported from all intervention trials sharing a common research objective, have been advocated as a solution to this problem. [6,7] A growing number of studies aiming to develop a COS for conditions across a range of health areas [8,9], is accompanied by reviews aiming to assess the consistency of outcomes in a formal way. [10–12] Prior to the release of the Core Outcome Measures in Effectiveness Trials (COMET) handbook [13], guidance on a COS-related methodology was mostly focused on aspects of the consensus process. [14] Review of past research appears in the handbook in the context of assessing a need for a COS—described as an optional step—and informing a list of outcomes for a consensus process. The guidance on the conduct of this type of assessments is succinct and suggests the adoption of a systematic review approach. [13–15] The method of systematically reviewing literature was introduced as a comprehensive way of summarising the evidence for the purpose of medical decision-making and identification of unanswered research questions. [16–19] The methodological rigour required for systematic review, which intends to minimise biases and provide a robust estimation of an underlying treatment effect, [16] requires considerable time and resources. [19,20] The benefits of applying the same approach to the assessment of outcome consistency or generating a long list of outcomes for Delphi survey is unclear. [21] Therefore, we set out to assess the methods adopted in evaluation studies of outcome consistency published to-date and examine their adoption of systematic review methods. Furthermore, we checked if identified evaluation studies were part of a COS project, examined methods specific to determining outcome consistency, such as type of collected outcomes, methods of their identification, and how authors assessed and presented outcome consistency or need for a COS. Materials and methods Our work was guided by a prospectively developed protocol registered with PROSPERO (CRD42018100481). Identification of relevant studies We included full texts of evaluation studies of outcome consistency in clinical studies on any health condition. We recognise that, even though, the COMET handbook refers to these type of evaluations as “systematic reviews” there is no consensus on the type of study design for this type of evaluation studies. Aiming to gain a thorough overview of the practices in this area and acknowledging lack of consensus regarding the study design, we decided to include any study design regardless of the design labelling. We searched the COMET database from its inception to June 2018; the search was updated in May 2019. [8] The COMET database is an annually updated repository of the international COS literature based on the systematic searches run in MEDLINE, SCOPUS, and Cochrane Methodology Register. [9] The search was supplemented with the resource of the Core Outcomes in Women’s and Newborn Health (CROWN) initiative [10,22] and check of references of the included studies. Systematic reviews of treatment effects with a secondary conclusion regarding outcome reporting, reviews of outcomes from non-clinical settings (registers, audits, population databases), COS protocols and reports mentioning outcome assessment but not describing them and studies focusing on a very narrow group of outcomes (e.g. only pain-related outcomes) were excluded. PLOS ONE Methods in evaluations of outcome consistency PLOS ONE | https://doi.org/10.1371/journal.pone.0235485 July 8, 2020 2 / 12 Funding: ER and CLV were supported by the UK Medical Research Council (MC_UU_12023/24). Competing interests: The authors have declared that no competing interests exist. Data collection All data were collected using a prospectively developed and piloted data collection form (S1 Text) and subsequently amalgamated into a master file in MS Excel. We collected information on publication year, medical speciality, evaluation’s aims, scope (number of included studies and their type), whether the design was labelled as a “systematic review” and was it a part of a COS project (based on study acronym or the information provided in the publication). In order to examine the adoption of systematic review methodology, all evaluation studies (regardless of declared study design) were assessed against a tailored subset of A MeaSurement Tool to Assess systematic Reviews (AMSTAR) 2 items [23] we felt were relevant (S1 Table). AMSTAR was designed as a practical critical appraisal instrument enabling rapid and reproducible quality assessments of systematic reviews. In our work, we used items covering the development of a study protocol (items 1–3), identification and selection of eligible studies (items 4–5), data collection (item 6), reporting (item 7–8) and study quality assessment (item 9). The remaining items (10 to 16) were not assessed as we felt they were not applicable to evaluation studies of outcome consistency (e.g. study funding, aspects of meta-analysis, publication bias, etc.). Information collected on the outcome-specific methods covered following elements: type of outcome (primary/secondary); a way of determining outcome type (clearly specified as primary or secondary, used in power calculation, etc.); approach to outcome extraction (any outcome reported in the publication or just those from the methods or results sections); use of any tool to assess the quality of outcome description; outcome unit as reported by the publication (individual outcomes or outcome groups); presentation of the findings (text, table, graphic format); and the conclusions. The conclusion section was examined for presence of any statement regarding inconsistency of assessed outcomes (heterogeneity, variation, etc.) or a need for a COS. Where detection of outcome inconsistency was concluded, we examined the methods and protocol of the evaluation, if available, for any description of how outcome inconsistency was defined. Where the need for a COS development was concluded, we looked for arguments given in the publications to support this conclusion. Double data extraction (AA, ROR or ER) and assessment of the subset of AMSTAR 2 items were performed on half of the included evaluations. Remaining evaluations were extracted and assessed by a single senior reviewer (ER). All discrepancies and statements supporting final judgements were recorded in the final Excel dataset. Strategy for data synthesis Extracted data were cross-tabulated and presented either as frequencies or as median with corresponding interquartile range (IQR). When examining detection of outcome consistency, we assessed only studies with a full description of outcome assessment. Fisher’s exact test was used to explore the potential difference between the evaluation studies labelled as “systematic review” and those that were not labelled as such. All descriptive summaries and graphs were computed with Excel software (MS Office 2016). The comparison of two groups—labelled and not labelled as “systematic review” was performed using Stata version 15.1. Results Study selection process and description of included studies Out of 237 records assessed, 93 evaluation studies met the inclusion criteria—90 full reports and three summaries of outcome assessment in the final report of the COS project (S1 Fig). The main reason for records exclusion was that the objective was outside of this work’s scope, PLOS ONE Methods in evaluations of outcome consistency PLOS ONE | https://doi.org/10.1371/journal.pone.0235485 July 8, 2020 3 / 12 e.g. assessment of measurements, instruments, scales or definitions (S2 Table). Included evaluation studies were mainly published between 2014 and 2019 (Fig 1), included a median of 68 studies (Table 1) and covered topics across 24 medical areas (S3 Table). Around two-thirds (63/93) of the evaluation studies set out to assess outcome consistency or need for a COS in their objectives and around half (48/93) could have been linked to a COS project. Almost 80% (73/93) described their design as a “systematic review” (Table 1). Systematic review methodology in evaluations of outcome consistency In most evaluation studies, we could identify a structured research question (87%, 81/93). Half of them declared having a protocol (50%, 46/93), of which 46% had been prospectively registered with PROSPERO (21/46) (S4 Table). In many, study identification (66%, 61/93) or data extraction (50%, 49/93) were carried out by two researchers (in duplicate). Authors rarely provided an exclusions list (5%, 5/93), a rationale behind the inclusion of specific study design (20%, 19/93), or performed quality assessment of included clinical studies (28%, 26/93) (Table 2). Even though in 91% of the evaluation studies (85/93) the literature search was performed in at least one bibliographical database (median of three databases per review) (S3 Table), we were able to classify only 33% (28/85) of them as comprehensive, as per AMSTAR 2 criteria, based on provided search details. The difference between evaluation studies labelled and not labelled as “systematic review” was statistically significant for the proportion of evaluation studies where study selection or data extraction were performed by at least two researchers, or the authors provided adequate description of included clinical studies (Table 2). Methods used to identify outcomes and their consistency The number of identified outcomes was reported in 93% (86/93) of the evaluation studies with 40% (37/93) specifying from which section of the trial publication, the outcomes were Fig 1. Number of evaluation studies of outcome consistency in clinical studies over time. https://doi.org/10.1371/journal.pone.0235485.g001 PLOS ONE Methods in evaluations of outcome consistency PLOS ONE | https://doi.org/10.1371/journal.pone.0235485 July 8, 2020 4 / 12 extracted. In two-thirds (63/93) of the evaluation studies, the authors made the distinction between primary and secondary outcomes, of which many also described how they ascertained whether the outcome was primary or not (41/63) (Table 3). Over half of the evaluation studies (55/90) looked at individual outcomes, 20% (18/90) assessed outcome domains (i.e. groups of individual outcomes referring to the same phenomena) and the remainder (19%, 17/90) applied both approaches (Table 3). Outcomes were frequently (90%, 81/90) presented in a tabulated format, and around one fifth (21%, 19/90) used a matrix to present outcome distribution (Fig 2). We found statements on the inconsistency of outcomes in conclusions of 88% (80/90) of the assessed evaluation studies. None provided a description of how this inconsistency was defined or detected. The need for a COS was declared in 82% (74/90) and frequently justified by the encountered inconsistency of assessed outcomes (76%, 68/90) (Table 3). Discussion The methods adopted in the evaluation studies of outcome consistency were variable, and none implemented all nine of the methodological expectations of systematic reviews as Table 1. Characteristics of included evaluation studies of outcome consistency. Characteristic N Descriptive Number of clinical studies per evaluation (median,Q1-3) 93 68 (34–133) Distinction between primary and subsequent publications, n (%) 93 30 (32) Objective(s) of evaluation study 93 To assess a need for COS or consistency in outcome selection, n (%) 63 (67) To identify outcomes for Delphi survey in COS development, n (%) 18 (47) To explore outcome-reporting, n (%) 13 (14) Evaluation study linked with a core outcome set project 93 48 (51) Evaluation study labelled as a “systematic review” 93 Yes, n (%) 73 (79) No, n (%) 20 (21) When the evaluation study was not labelled as “systematic review”, what other term was used to describe the design: 20 “Literature review” 11 (55) “Review of outcomes” 2 (10) “Analysis of outcome reporting” 1 (5) “Electronic database search” 1 (5) “Survey” 1 (5) “Outcome mapping” 1 (5) “Systematic exploration” 1 (5) “Analysis of studies 1 (5) “Scoping review” 1 (5) Types of research included in the evaluation studies of outcome consistency Primary research Full scale clinical study (e.g. RCTs, cohort studies), n (%) 93 59 (63) Feasibility or pilot study, n (%) 93 8 (9) Secondary research (literature review) 93 27 (29) When the literature review was included, the purpose was to: Identify outcomes, n (%) 27 10 (37) Identify clinical studies for outcome assessment, n (%) 27 15 (56) https://doi.org/10.1371/journal.pone.0235485.t001 PLOS ONE Methods in evaluations of outcome consistency PLOS ONE | https://doi.org/10.1371/journal.pone.0235485 July 8, 2020 5 / 12 Table 2. Systematic review methods as specified in the subset of AMSTAR 2 items in evaluation studies of outcome consistency. Item Assessed aspect Group Yes (n,%)� Unclear (n,%)� No (n,%)� N/A Fisher exact test (p-value) 1Did the research questions and inclusion criteria include the components of PICO? Overall 81 (87) 9 (10) 3 (3) 0 Labelled as “Syst rev” 64 (88) 8 (11) 1 (1) 0.129 Not labelled as “Syst rev” 17 (85) 1 (5) 2 (10) 2Did the report contain an explicit statement that the methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol? Overall 46 (50) 5 (5) 42 (45) 0 Labelled as “Syst rev” 42 (58) 4 (5) 27 (37) 0.06 Not labelled as “Syst rev” 4 (20) 1 (5) 15 (75) 3Did the authors explain their selection of the study designs for inclusion? Overall 19 (20) 0 74 (80) 0 Labelled as “Syst rev” 17 (23) 0 56 (77) 0.346 Not labelled as “Syst rev” 2 (10) 0 18 (90) 4Did the authors use a comprehensive literature search strategy? Overall 28 (33) 4 (5) 53 (62) 8 Labelled as “Syst rev” 23 (32) 3 (4) 43 (59) 0.200 Not labelled as “Syst rev” 5 (25) 1 (5) 10 (50) 5Did the authors perform study selection in duplicate? Overall 61 (66) 21 (23) 10 (11) 1 Labelled as “Syst rev” 56 (77) 11 (15) 5 (7) <0.001 Not labelled as “Syst rev” 5 (25) 10 (50) 5 (25) 6Did the authors perform data extraction in duplicate? Overall 49 (53) 32 (34) 12 (13) 0 Labelled as “Syst rev” 45 (62) 21 (29) 7 (10) 0.002 Not labelled as “Syst rev” 4 (20) 11 (55) 5 (25) 7Did the authors provide a list of excluded studies and justify the exclusions? Overall 5 (6) 0 84 (94) 4 Labelled as “Syst rev” 4 (5) 0 66 (90) 1.00 Not labelled as “Syst rev” 1 (5) 0 18 (90) 8Did the authors describe the included studies in adequate detail? Overall 37 (42) 2 (2) 50 (56) 4 Labelled as “Syst rev” 35 (48) 2 (3) 33 (45) 0.006 Not labelled as “Syst rev” 2 (10) 0 17 (85) 9Did the authors assess the quality of included studies? Overall 26 (29) 0 64 (71) 3 Labelled as “Syst rev” 22 (30) 0 49 (67) 0.474 Not labelled as “Syst rev” 4 (20) 0 15 (75) N/A, not applicable; Syst rev, systematic review; �Percentages calculated using as denominator number without evaluation studies in N/A category; https://doi.org/10.1371/journal.pone.0235485.t002 PLOS ONE Methods in evaluations of outcome consistency PLOS ONE | https://doi.org/10.1371/journal.pone.0235485 July 8, 2020 6 / 12 described in AMSTAR 2. The proportion of evaluation studies labelled as a “systematic review” and those that were not differed in three AMSTAR 2 items (duplicate identification of eligible studies, duplicate data extraction, and description of included clinical studies in sufficient detail). Most of the evaluations concluded that there was evidence of inconsistency in the outcomes used across clinical studies. Although none described how this inconsistency was determined, it was frequently cited to justify the need for COS development. Table 3. Methods used to identify and assess outcomes in evaluation studies of outcome consistency. Characteristic n/N�(%) Evaluation studies reporting the number of identified outcomes (individual or domains) 86/90 (96) Number of extracted outcomes (individual or domains) per review (median, Q1 to 3) 80 (43 to 158) Distinction between primary and secondary outcomes 63/93 (68) When the distinction between the type of outcome(s) was made: The primary outcome(s) was clearly specified as a primary 27/63 (43) The primary outcome(s) was clearly specified as a primary or used in the power calculation 11/63 (17) The primary outcome(s) was identified using other measures (e.g. outcome mentioned in the trial title, first reported) 3/63 (5) There were no details of how the primary outcome(s) was identified 22/63 (35) Approach to outcome extraction Outcomes mentioned anywhere in the report 22/93 (24) Outcomes mentioned only in methods 11/93 (12) Outcomes mentioned only in results 4/93 (4) Not specified or unclear 56/93 (60) Quality assessment of outcome reporting 17/93 (18) Outcomes presented Only individually 55/90 (61) Only grouped into domains�� 18/90 (20) In both ways (individually and grouped in domains) 17/90 (19) Presentation of extracted outcomes Table of outcomes (individual or domains) with the number of studies 81/90 (90) Matrix (outcomes per study) 20/90 (22) Other graphic formats (e.g. bar chart, Venn diagram, Spiral graph) 41/90 (46) Conclusion regarding outcome consistency Detection of inconsistency in outcome choice 80/90 (88) Detection of a need for COS 74/90 (82) Due to identified inconsistency in outcome choice 68/90 (76) Due to other reasons than inconsistency in outcome choice 6/90 (7) Conclusions of evaluations Wide range of identified outcomes (individual or domains) 55/90 (61) Difference in how, rather than which, the outcome was measured 52/90 (58) Infrequent reporting of outcomes relevant to patient care 37/90 (41) Impact of inconsistency in outcome choice on evidence synthesis 13/90 (14) Difference in when the outcomes were measured 12/90 (13) �Three reports (3/93) of full core outcome sets were excluded from some assessments (N = 90) as they focused only on providing a list of identified outcomes without reporting more details (see S3 Table) �� Outcome domain, constructs used to broadly classify individual outcomes referring to the same phenomena into a group https://doi.org/10.1371/journal.pone.0235485.t003 PLOS ONE Methods in evaluations of outcome consistency PLOS ONE | https://doi.org/10.1371/journal.pone.0235485 July 8, 2020 7 / 12 Our work provides the first extensive evaluation of methods used in evaluation studies of outcome consistency in clinical studies to date. It was guided by a prospectively developed protocol using broad inclusion criteria to facilitate identification of a representative sample of the studies of outcomes. Despite the benefits of publication of methodological study protocols are yet to be determined [23], their development can enhance the study’s reproducibility. [24] Our methods heavily borrow from a systematic review process; however, our work is not a systematic review and should not be judged like one. For example, we did not set out to run an extensive search across numerous databases instead of relying on the COMET database and supplemented its results with the resources of the CROWN initiative. [25] Use of such a specific source of studies might be perceived as a limiting factor leading to the omission of some relevant publications. On the other hand, the COMET database is updated periodically, using extensive search strategy, for COS-related literature. [8] In this exploratory study of published literature, we aimed to obtain a pragmatic sample of studies using relevant sources. We did not search the international prospective register of systematic reviews (PROSPERO) [26], due to lack of specific indexing of this type of evaluation studies resulting in the inability to retrieve relevant records. As only less than a quarter of the included evaluation studies registered their Fig 2. Matrix of outcomes—example. https://doi.org/10.1371/journal.pone.0235485.g002 PLOS ONE Methods in evaluations of outcome consistency PLOS ONE | https://doi.org/10.1371/journal.pone.0235485 July 8, 2020 8 / 12 protocols with PROSPERO, we believe that omission of this source did not have a substantial impact on our work. AMSTAR 2 was primarily developed to examine the robustness of systematic reviews of healthcare interventions. [27] Thus, the use of a subset of its items in this work might be contested. We decided to use AMSTAR 2 for two reasons, the COMET handbook reference to the evaluation studies of outcome consistency as “systematic reviews”, and previous evaluation of COS-related literature in the area of women’s health. [10] We selected the most relevant and applicable across a broader spectrum of study designs AMSTAR 2 items (S1 Table). Their application to all included evaluation studies allowed us to have a uniform comparison of the methods regardless of how they were labelled. Even though, a quarter of the evaluations in our sample did not self-identified as a “systematic review”, the proportion of those satisfying AMSTAR 2 criteria labelled and not labelled as “systematic review” did not differ in six out of nine evaluated items. Publication of the COMET Handbook [13] and related resources [14,15,28] increased access to guidance on robust methods for COS development. However, advice on methodology to evaluate outcomes to inform COS development is currently limited to a suggestion of adopting systematic review methodology. [13,14] Evaluation studies of outcome consistency are a relatively new phenomenon with only a handful of this type of studies published before 2010 (Fig 1). Thus, the observed differences in the adopted methods probably arise from this limited guidance. In recent years, there has been a growing body of literature focusing on the methodological aspects of medical research [29]. The methodology of methods-orientated studies is inherently heterogeneous [29] and may involve features of a systematic review [21]. Nevertheless, the authors should refrain from labelling their work as “systematic review” when this is not the case [21], ensuring that their methodological explorations are guided by a robust methodology. A well-executed systematic review is a resource-intensive and meticulous synthesis of available evidence. [16] Application of systematic review methods to assess outcome consistency raises a question over a value of applying such workload intense methodology—mainly in the form of researchers’ time. Evaluation studies included in this work frequently aimed to assess the need for a COS and, simultaneously, collect outcomes to inform a subsequent Delphi survey. Delphi process tends to comprise multiple sources of information for generating the initial list of outcomes with a literature assessment of outcome being only one of them. Delphi enables involved stakeholders to propose relevant outcomes not yet captured on the list. [30] Thus, the rationale for conducting a systematic review to inform this process is unclear. Given that the impact of the methods of discussed evaluation studies of outcome consistency on the COS development process is unknown; authors should consider whether a systematic review approach is warranted. None of the assessed evaluation studies specified how inconsistency of outcomes was detected; yet, the majority claimed in their conclusion to have identified such inconsistency. This finding highlights the need for more transparency in the design and reporting of these type of evaluations and is consistent with the conclusion of a recent COS literature. In their work, Young et al. examined number of reported outcomes, their definitions, timing, and approach to their grouping in 132 studies (development papers, protocols, and reviews) and found meaningful differences. [31] Based on their findings, they proposed a definition for a unique outcome and inconsistency of outcomes referred to in their work as outcome reporting heterogeneity (ORH). [31] The proposed definition brings attention to a difference between the number of identified outcomes (however defined) and their overlap between the clinical studies. The methods of evaluation studies of outcome consistency published to date differ in their approach to the identification of eligible studies and their outcomes. Although frequently PLOS ONE Methods in evaluations of outcome consistency PLOS ONE | https://doi.org/10.1371/journal.pone.0235485 July 8, 2020 9 / 12