Document [original]

International Journal of Technology Assessment in Health Care, 30:1 (2014), 34–43.

Cambridge University Press 2014

doi:10.1017/S026646231300072X

HOW HEALTH TECHNOLOGY ASSESSMENT

AGENCIES ADDRESS THE ISSUE OF

UNPUBLISHED DATA

Julia Kreis

Department of Health Care Management, Berlin University of Technology

Institute for Quality and Efficiency in Health Care

Dimitra Panteli,Reinhard Busse

Department of Health Care Management, Berlin University of Technology

Objectives: Reporting bias potentially threatens the validity of results in health technology assessment (HTA) reports. Our study aimed to explore policies and practices of HTA

agencies regarding strategies to include previously unpublished data in their assessments, focusing on requests to industry for unpublished data.

Methods: We included international HTA agencies with publicly available methods papers as well as HTA reports. From the methods papers and recent reports we extracted

information on requests to industry and on searches in trial registries, regulatory authority Web sites and for conference abstracts.

Results: Eighteen HTA agencies and seventy-three reports were included. Agencies’ methods papers showed variability regarding requests to industry (requests are routinely carried

out in seven cases, not mentioned in six, at the discretion of HTA authors in three, and based on manufacturer applications in two), which were reflected in the reports investigated.

As reporting of requests was limited, it often remained unclear whether unpublished data had been received. Searches in trial registries, at regulatory authorities or for conference

abstracts are described as a routine or optional part of the search strategy in the methods papers of 9, 11, and 8 included agencies, respectively. A total of 52 percent, 39 percent,

and 16 percent of reports described searches in trial registries, at regulatory agencies, and hand searching of conference proceedings.

Conclusion: International HTA agencies currently differ considerably in their efforts to address the issue of unpublished data. Requests to industry may constitute one strategy to

access and include unpublished data, while agencies can learn from each other concerning successful practice.

Keywords: Technology assessment, Biomedical (MeSH), Publication bias (MeSH), Review literature as topic (MeSH)

Evidence on Reporting Bias

Over the last few decades, evidence has been accumulating on

the existence of reporting biases in the medical literature which

“arise when the dissemination of research findings is influenced

by the nature and direction of results” (p. 298 (1)). A recent

overview found examples of reporting bias in 50 different inter-

ventions and procedures in the fields of pharmacology, surgery,

diagnostics, and prevention, thus demonstrating that this phe-

nomenon poses a major problem in the assessment of health

care interventions (2). The best known form of reporting bias is

publication bias, defined as “the publication or nonpublication

of research findings, depending on the nature and direction of

the results” (p. 298 (1)). However, even in published studies,

subsets of the original outcomes or analyses may be selectively

reported (outcome or analysis reporting bias) (3).

The authors thank Kay Dickersin, Milo Puhan, and Annette Zentner for their contributions during

the design phase of the study; Elke Hausner and Natalie McGauran for their comments on a draft

version of this paper; and Natalie McGauran for editorial support. The views expressed in this

article are those of the authors and do not necessarily represent those of the Institute for Quality

and Efficiency in Health Care or the University of Technology, Berlin.

Challenge for HTA Agencies

Reporting bias tends to overestimate benefits and underestimate

harms of health care interventions, thus leading to potentially

biased conclusions in systematic reviews and health technol-

ogy assessment (HTA) reports (2;4). This was illustrated by

a recent assessment of reboxetine by the German HTA agency

IQWiG (Institute for Quality and Efficiency in Health Care): the

inclusion of data from ten trials, which had previously been un-

published or incompletely published and were provided by the

manufacturer only after considerable public pressure, had a sig-

nificant impact on the assessment’s results. IQWiG’s conclusion

that reboxetine is “overall an ineffective and potentially harm-

ful antidepressant” (p. 8 (5)) stood in contrast to the findings

of several earlier systematic assessments that did not combine

all of the existing evidence from clinical trials (6), and ulti-

mately led to the exclusion of reboxetine from reimbursement in

Germany.

Data Accessible to HTA Agencies

Legislation requires manufacturers to grant regulatory author-

ities such as the U.S. Food and Drug Administration (FDA)

comprehensive access to both published and unpublished data

for the approval process (e.g., (7)). However, manufacturers are

not generally obliged to provide all of the relevant data to HTA

https://doi.org/10.1017/S026646231300072X

Downloaded from https://www.cambridge.org/core. Universitaetsbibliothek, on 26 Oct 2017 at 13:40:50, subject to the Cambridge Core terms of use, available at https://www.cambridge.org/core/terms.

Unpublished data in HTA reports

agencies for post-approval evaluations (8). Health policy deci-

sions, for example, on the reimbursement of drugs, are therefore

often based mainly on publicly available information with the

potential insufficiencies described above (8–10).

One way of overcoming the selective or nonpublication of

studies is their registration in public databases: in 2004, the

International Committee of Medical Journal Editors specified

that registration in a public trial registry was a prerequisite

for a study’s publication (11). In response to public concerns

about reporting biases and to legal proceedings (the paroxetine

case (12)), several individual pharmaceutical companies, as well

as industry associations such as the Pharmaceutical Research

and Manufacturers of America (PhRMA), installed their own

registries. In 2007, the FDA Amendments Act made registration

and public posting of all applicable clinical trials in the registry

ClinicalTrials.gov compulsory for the United States (13). In

Europe, the publicly accessible EU Clinical Trials Register was

launched only recently (March 2011) and contains information

on phase 2–4 adult clinical trials and any pediatric clinical trial

carried out in the European Union from 2004 onward (14).

However, existing regulations for trial registration have several

loopholes, particularly as older drugs, although widely used in

clinical practice, are not covered (8;15).

Other strategies used to identify potentially relevant unpub-

lished studies or data include searching for regulatory authority

Web sites as well as conference abstracts, as approximately half

of trials reported in conference abstracts are never fully pub-

lished, and publication is associated with positive results (16).

However, it has been argued that publicly available sources do

not necessarily contain all relevant clinical trial results for a

given healthcare intervention, and that access to these results is

still insufficient (13).

As publication is not a dichotomous event, but rather a

continuum (17), the distinction between “published” and “un-

published” is not always clear cut (2). In our study, we use the

term “published” to refer to articles published in peer-reviewed

journals and the term “unpublished” to refer both to publicly

inaccessible data owned by industry or study authors and to

“gray literature,” that is, studies not published in peer-reviewed

journals but nevertheless publicly available in sources such as

trial registries, regulatory authorities or published as conference

abstracts.

Aim of Our Study

To the best of our knowledge, no detailed analysis is currently

available as to how international HTA agencies address the

challenge of unpublished data. On the basis of an analysis of

publicly available methods papers and HTA reports, the aim

of our study was, therefore, to provide an overview of cur-

rent policies and practices with regard to searching for, in-

cluding and reporting of unpublished data. Our primary in-

terest was to explore whether and how HTA agencies carry

out requests to industry to include data in their assessments

that are not publicly available at all. We also explored to what

extent alternative strategies are pursued to identify grey litera-

ture from trial registries, regulatory authorities and conference

abstracts.

METHODS

Eligible HTA Agencies and Documents

We analyzed two different data sources: “methods papers,” that

is, documents describing the HTA organizations’ methods for

assessing the effectiveness of interventions, and “HTA reports,”

that is, full assessments of a health technology or focused as-

sessments of drugs for reimbursement and pricing decisions. We

aimed for a broad sample to capture current practice in the HTA

community and therefore identified international HTA agencies

from four different sources: (a) members of the International

Network of Agencies for Health Technology Assessment (IN-

AHTA) as of December 2010, (b) partners in the European Net-

work for Health Technology Assessment (EUnetHTA) project as

of December 2010, (c) nonprofit members in Health Technology

Assessment international (HTAi) as of May 2011, and (d) HTA

agencies included in comparative analyses published in the In-

ternational Journal of Technology Assessment in Health Care in

the years 2009 and 2010. Eligible agencies were those that pro-

duced HTA reports and for which we were able to identify both

eligible methods papers and HTA reports in English, German, or

French (for inclusion and exclusion criteria see Supplementary

Table 1: http://dx.doi.org/10.1017/S026646231300072X).

Data Collection, Extraction, and Analysis

We searched all Web sites of potentially eligible agencies at

the beginning of 2011, both to check whether they are di-

rectly involved in HTA by either commissioning or conducting

assessments and to identify methods papers as well as HTA

reports. We contacted agencies between February and May

2011 to verify the information obtained online and confirm

the completeness and currentness of the documents retrieved.

We did not contact agencies where the Web site clearly indi-

cated that they are not involved in HTA production or where

insufficient information in English, French, or German was

available.

Methods Papers.

We included methods papers developed by the re-

spective agencies that were publicly available and described

methods for assessing the effectiveness (i.e., not only cost-

effectiveness) of drug or nondrug therapeutic interventions. We

included more than one methods paper if they had different

scopes, yet only those for which eligible reports were also pub-

licly available. If further documents explicitly addressing the

inclusion and handling of unpublished data were available, they

were also included in the analysis.

35 INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 30:1, 2014

https://doi.org/10.1017/S026646231300072X

Downloaded from https://www.cambridge.org/core. Universitaetsbibliothek, on 26 Oct 2017 at 13:40:50, subject to the Cambridge Core terms of use, available at https://www.cambridge.org/core/terms.

Kreis et al.

INAHTA

members

n = 51

EUnetHTA partner

organizations

n = 42

HTAi non-profit

members

n = 60

IJTAHC

2009-2010

n = 72

Agency pool

n = 121

Duplicate organizations

n = 104

HTA agencies

included in analysis

n = 18

Excluded organizations

n = 103

- Agency ceased to exist (n = 2)

- Insufficient information available (including language of

Web site) (n = 50)

- No production of HTA (n = 15)

- Language (methods papers, reports) (n = 12)

- No own methods paper identified (n = 18)

- Reports not publicly available (n = 5)

- No reports on drug or non-drug therapeutic interventions (n = 1)

Figure 1. Flow chart of selection of HTA agencies (May 2011).

HTA Reports.

We included HTA reports that were publicly avail-

able in full (i.e., not only in summary) and reported a sys-

tematic evaluation (i.e., based on a search for literature in at

least two sources) of the effectiveness of drug or nondrug

therapeutic interventions from January 2006 onward. We ex-

cluded reports that (i) were commissioned by another agency

that was also included in our sample, (ii) were based solely

on secondary publications (e.g., “reviews of reviews” that did

not include primary studies), or (iii) were draft versions. We

included the three most recent reports on both drugs and non-

drug therapeutic interventions or fewer if less than three were

available.

To reduce the heterogeneity between reports and due to lim-

ited resources, we did not examine documents (neither methods

papers nor HTA reports) reporting the evaluation of diagnostic

procedures or population-based preventive interventions. In ad-

dition, for agencies producing different types of documents, we

restricted our analysis to one type (e.g., to multiple technology

assessments for NICE documents) to reduce the complexity of

the analysis.

We developed and pilot-tested sheets to extract infor-

mation from methods papers and HTA reports. Informa-

tion was extracted by one person (JK) and checked by a

second (DP) for the categories depicted in Supplementary

Table 2, which can be viewed online at http://dx.doi.org/

10.1017/S026646231300072X. Disagreements were solved by

discussion and by consulting the original documents again.

We provide a qualitative analysis of information obtained from

methods papers to show the range of different approaches cur-

rently in place at the agencies in our sample and descriptive

statistics for the analyses of HTA reports.

RESULTS

Agencies Included in Our Sample

A total of 121 agencies were potentially eligible and eighteen

fulfilled all inclusion criteria (see Figure 1). The agencies in-

cluded (see Table 1) are diverse with regard to their scope and

their role within the respective health care systems.

Methods Papers Describing Efforts to Include Unpublished Data

Methods papers published by eligible agencies varied consider-

ably regarding length (ranging from 9 to 294 pages), as well as

currentness, with the oldest documents having been published in

2000, and the newest ones published as drafts in 2011 (for a com-

plete list, see Supplementary Table 3, which can be viewed on-

line at http://dx.doi.org/10.1017/S026646231300072X). Some

of the agencies contacted informed us that the methods pa-

pers were being revised at the time of our research or no

longer reflected current practice and more recent, internal

guidance was being followed instead. However, in the ab-

sence of more up-to-date, publicly available documents, these

methods papers were still included. The documents also re-

flect the scope of the respective agency, thus being com-

prehensive or addressing exclusively the evaluation of drugs

or medical devices. Some of the documents are specific

INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 30:1, 2014 36

https://doi.org/10.1017/S026646231300072X

Downloaded from https://www.cambridge.org/core. Universitaetsbibliothek, on 26 Oct 2017 at 13:40:50, subject to the Cambridge Core terms of use, available at https://www.cambridge.org/core/terms.

Unpublished data in HTA reports

Table 1. Agencies Included in the Analysis and Approaches Regarding Requests to Industry

Abbreviation Agency’s name (country) Approach

AHRQ Agency for Healthcare Research and Quality (United States) 1

ASERNIP-S Australian Safety and Efficacy Register of New Interventional Procedures – Surgical (Australia) 3

CADTH Canadian Agency for Drugs and Technologies in Health (Canada) 1

CRD Centre for Reviews and Dissemination (United Kingdom) 2

DACEHTA Danish Centre for Health Technology Assessment (Denmark) 3

DERP Drug Effectiveness Review Project (United States) 1

DIMDI Deutsches Institut f¨

ur Medizinische Dokumentation und Information (Germany) 3

G-BA Gemeinsamer Bundesausschuss (Germany) 3

G¨

OG Gesundheit ¨

Osterreich GmbH (Austria) 2

HAS Haute Autorit´

edeSant

e (France) 3

HVB Hauptverband der ¨

osterreichischen Sozialversicherungstr¨

ager (Austria) 3

IQWiG Institut f¨

ur Qualit¨

at und Wirtschaftlichkeit im Gesundheitswesen (Germany) 1

KCE Belgian Health Care Knowledge Centre (Belgium) 1

LBI Ludwig Boltzmann Institut f¨

ur Health Technology Assessment (Austria) 2

MSAC Medical Services Advisory Committee (Australia) 4

NICE National Institute for Health and Clinical Excellence (United Kingdom) 1

PHARMAC Pharmaceutical Management Agency (New Zealand) 4

TLV Tandv˚

ards- och L¨

akemedelsf¨

orm˚

ansverket (Sweden) 1

Note.

Explanation: 1 =Manufacturers are routinely approached; 2 =Manufacturer requests may be carried out at

the discretion of the HTA authors; 3 =Manufacturer requests are not explicitly mentioned in the methods paper; 4 =

Manufacturers (or others) file an application with their data for the product to be assessed.

The information reflects the content of the methods papers available to us at the time of our search. Note: HAS bases its

assessments on a review of the literature and dossiers from pharmaceutical companies. The only methods papers that we were

able to identify, however, did not reflect this current practice.

guidelines for the respective agencies (e.g., DERP, LBI [for

agencies’ full names, see Table 1]), while others have a

broader target audience (e.g., DACEHTA, CRD). Where guid-

ance for obtaining data was given separately for assessing

effectiveness and adverse effects (e.g., CRD, AHRQ), we lim-

ited our analysis to the former.

Requests to Industry.

We found that HTA agencies pursue different

approaches concerning the question as to whether data are re-

quested from industry in the course of HTA production. These

can be grouped into four broad categories: (i) manufacturers are

routinely approached as part of the search for relevant literature

(seven agencies); (ii) manufacturer requests may be carried out

at the discretion of the HTA authors (three agencies); (iii) man-

ufacturer requests are not explicitly mentioned in the methods

paper (six agencies); (iv) manufacturers (or others) file an ap-

plication with their data for the product to be assessed (two

agencies). Differences between agencies’ approaches to a con-

siderable extent also reflect the respective national legal context

(e.g., when assessments are based on submissions by manufac-

turers instead of comprehensive searches carried out by HTA

agencies) (see Table 1).

Five of the seven agencies that routinely make requests to

manufacturers provide details of these processes in their meth-

ods papers; these are summarized in Table 2.

Conditions for accepting and including unpublished data in

reports vary between agencies. The following conditions were

addressed by at least on agency: completeness, adequate detail,

non-confidentiality, and the timely submission of the data (see

Table 3 for examples).

Few documents contain detailed guidance on how to re-

port requests to industry: CRD provides an example for the

documentation of a request, CADTH foresees requests to man-

ufacturers as a standard part of the documentation of the liter-

ature search, DERP explicitly states that companies providing

dossiers should be accounted for in the results section of the re-

port, and IQWiG also states that requests to industry have to be

documented in the reports. NICE publishes a list of consultees

on its Web site. Several agencies (e.g., G ¨

OG, KCE) state that

the literature search has to be documented with enough detail

to allow reproduction, without explicitly addressing requests to

industry.

Equally, only a few methods papers in our sample provide

explicit guidance on how to report previously unpublished data

37 INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 30:1, 2014

https://doi.org/10.1017/S026646231300072X

Downloaded from https://www.cambridge.org/core. Universitaetsbibliothek, on 26 Oct 2017 at 13:40:50, subject to the Cambridge Core terms of use, available at https://www.cambridge.org/core/terms.

Kreis et al.

Table 2. Standardized Formats for Requests to Industry

AHRQ

Addressee: Manufacturers of products under review (drugs, devices or other products for which a manufacturer can be identified)

Content of requests: Scientific Information Packets containing

- information on products available from the product label

- information on published and unpublished studies concerning the product

Process details: Requests are managed by the Scientific Resource Center for AHRQ on behalf of the Evidence-based Practice Centers (EPCs) “to ensure consistency in

the way Scientific Information Packets (SIPs) are requested and to ensure transparency by eliminating contact between the EPC conducting the

review and the manufacturers of products being reviewed”.

DERP

Addressee: Pharmaceutical companies that manufacture any drug included in an individual report

Content of requests: Dossiers containing

- published studies (a list of citations for all relevant published studies of drugs licensed by the respective company)

- unpublished studies (trials or observational designs; with information on, e.g., study identifier, design, indication, patient population

description, inclusion & exclusion criteria, treatment of interest and all comparisons, outcomes measured, results; sufficient amount

of details on methods to allow assessment of study quality)

- unpublished, supplementary data for published clinical studies (e.g., additional details about study methods, additional outcomes, and

results of additional subgroup analyses)

- a copy of the most recent product label

Process details: Requests are managed by the Medical Director for DERP, Center for Evidence-based Policy. As all information submitted may be made available to

the public, all statements of confidentiality are considered “null and void”.

IQWiG

Addressee: Manufacturers of the technology under review

Content of requests: For drugs, a 2-step procedure is applied:

- 1) complete list of all trials carried out with the drug under assessment from the manufacturer

- 2) detailed request on the basis of 1) for unpublished studies or for additional, unpublished information on published studies

Process details: Before submission of data, an agreement is made between IQWiG and the manufacturer involved specifying the submission process, the

requirements for documents to be submitted (e.g., information provided should comply with CONSORT), and their confidential and

non-confidential components. If a manufacturer does not agree to this contract (especially with regard to the complete transfer of all information

requested) or does not completely transfer the information requested despite this contract, no further requests are made to this manufacturer.

NICE

Addressee: Manufacturers and sponsors (organizations that market the technology under license) of the technology under review

Content of requests: Submissions, i.e., a “concise, comprehensive and structured report of all relevant information (published and unpublished) for an appraisal”,

containing

- a complete list of all studies on the technology within the indication (i.e., studies may be sponsored by manufacturers or sponsors or known

to them; NICE may request further information on studies included in the list)

- a main document including, amongst other things, the aims of treatment, approved indications and an assessment of clinical effectiveness

- appendices containing supporting information for data and analyses and the excluded evidence

Process details: At the start of a project, manufacturers and sponsors are invited to take part in the appraisal as consultees. Consultees have at least 14 weeks to

prepare their submissions which are then forwarded in full to the independent academic group preparing the evidence assessment report for NICE.

in HTA reports: DERP provides complete copies of dossiers to

the public upon request once the report is published. IQWiG

foresees the possibility to publish short summaries of previ-

ously unpublished studies in its reports, as well as any addi-

tional relevant unpublished data. The methods papers of both

agencies point out that all previously unpublished data included

in the analyses are reported. If a submission to NICE contains

previously unpublished confidential information, the manufac-

turer has to provide a second version with the confidential

information removed, which is then published on the NICE

Web site as a supplementary document together with the as-

sessment. For previously unpublished trials, NICE requests a

structured abstract for public disclosure. Information that has

been submitted in confidence is removed from the actual HTA

INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 30:1, 2014 38

https://doi.org/10.1017/S026646231300072X

Downloaded from https://www.cambridge.org/core. Universitaetsbibliothek, on 26 Oct 2017 at 13:40:50, subject to the Cambridge Core terms of use, available at https://www.cambridge.org/core/terms.

Unpublished data in HTA reports

Table 2. Continued.

TLV

Addressee: Companies concerned by a pharmaceutical reimbursement review

Content of requests: Submission containing

- general information on the drug (e.g., administration form, dosage)

- information regarding the indication (e.g., average treatment time)

- information regarding medical effects: information on the “most important and well-executed studies” that have been conducted on the

medicines (information for each of these studies in a table format, including primary and secondary effect measure, effect of treatment

[intention to treat and per protocol analysis], losses to follow up etc.); summary of the medical effect of treatment based on these studies;

summary of the effect of treatment on the patient’s quality of life and life expectancy; advantages of the medicine regarding side effects

compared to other medicines in this class of pharmaceuticals; known variations in the medical effect of the medicine (e.g., in respect to

patients’ sex and age)

- information regarding cost-effectiveness

Process details: When commencing the review of a therapeutic group, the companies responsible for marketing drugs in this group are notified by the TLV.Inthis

context, they are requested to submit information on, amongst other things, the clinical use of and cost-effectiveness of the drug or drugs

marketed by the company by filling in a standard form within 8 weeks.

Note.

The information reflects the content of the methods papers that were available to us at the time of our search (see Supplementary Table 3 for a full list of references).

Table 3 Conditions for the Acceptance of Unpublished Data (Examples)

Completeness:

IQWiG has an agreement with manufacturers containing an obligation for complete submission of all relevant data. NICE requires manufacturers and sponsors to sign a

statement declaring that all relevant material has been disclosed, and PHARMAC requests a declaration that all known unpublished clinical trials have been disclosed.

Adequate detail:

DERP requests sufficient amount of detail on methods of unpublished studies to allow for adequate assessment in order to include their data in the report. IQWiG requests

companies to comply with the CONSORT statement when submitting information. PHARMAC allows abstracts and posters to be used by manufacturers in their applications only

“as references to update information after the primary analyses or any analyses of secondary outcomes not detailed in the published report”, if they are adequately detailed.

Non-confidentiality:

DERP and IQWiG only accept data for inclusion in their analysis that may be made public; others (KCE, NICE, PHARMAC) also accept – under certain circumstances – data that

are submitted in (either academic or commercial) confidence.

Timely submission of data:

DERP specifies that unpublished data or studies cannot be submitted by pharmaceutical companies after the deadline for the dossier process (e.g., not via the public commenting

process for draft reports).

Note.

The information reflects the content of the methods papers available to us at the time of our search (see Supplementary Table 3 for a full list of references).

report by blackening the relevant passage. Several agencies

(e.g., PHARMAC, MSAC, TLV) point out that the respective

legal requirements concerning the disclosure of confidential in-

formation apply.

Searches in Publicly Available Sources.

Trial registries, regulatory au-

thorities and conference abstracts all seem to play a simi-

lar role as additional sources: routine or optional searches

in these sources are described in the methods papers

of 9, 11, and 8 of eighteen agencies, respectively (see

Supplementary Table 4, which can be viewed online at

http://dx.doi.org/10.1017/S026646231300072X). Although the

distinction between routine and optional parts of the literature

search is not always clear cut, it seems that searching for confer-

ence abstracts is less often defined as a routine element. Table 4

depicts options for searching in these sources that we identified

across agencies.

39 INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 30:1, 2014

https://doi.org/10.1017/S026646231300072X

Downloaded from https://www.cambridge.org/core. Universitaetsbibliothek, on 26 Oct 2017 at 13:40:50, subject to the Cambridge Core terms of use, available at https://www.cambridge.org/core/terms.

Kreis et al.

Table 4. Options for Searching in Publicly Available Sources

Trial registries:

- Where: ClinicalTrials.gov (www.clinicaltrials.gov); Current Controlled trials (www.controlled-trials.com); WHO International Clinical Trials Registry Platform

(www.who.int/trialsearch); Clinical Study Results (www.clinicalstudyresults.org)aand further public or commercially owned registries

- To what end: identify unpublished studies; identify additional details of published studies; identify ongoing studies; contact principle investigators for further information

Regulatory authorities:

- Where: FDA (Medical and Statistical Reviews); EMA (European Public Assessment Reports) and further national authorities

- To what end: compare data from regulatory documents with results reported in respective publications to verify or find additional data; identify unpublished studies; compare

results from published and unpublished studies

Conference abstracts:

- Where: specialized databases; conference proceedings of selected meetings via hand searching

- To what end: identify studies that are not (yet) formally published in a peer-reviewed journal or indexed by electronic databases; compare preliminary data from abstracts with

formally published results to check for possible inconsistencies or find additional data; contact abstract authors for full reports

Note.

The information reflects the content of the methods papers available to us at the time of our search (see Supplementary Table 3 for a full list of references).

a: This registry which was set up by PhRMA is no longer available.

HTA Reports Describing Efforts to Include Unpublished Data

We identified seventy-three HTA reports published between

2006 and 2011 (thirty-three on drugs and forty on nondrug

interventions [one report assessed both drug and nondrug in-

terventions and was analyzed twice]) in our analysis. Nine of

the HTA reports in our sample assessed nondrug interventions

that do not involve the application of a medical device (for ex-

ample, psychosocial interventions for people bereaved by sui-

cide); we thus excluded them from further analyses regarding

requests to industry and searches of regulatory authority Web

sites.

Requests to Industry.

Current practices described in HTA reports

broadly reflect the methods outlined in their methods papers: of

those agencies with routine requests (see above), twenty-one of

thirty-one eligible reports (68 percent) explicitly mentioned that

requests to industry were carried out. While it remained unclear

in three reports whether any response was received, eighteen

reports described the receipt of information, although in some

cases not all of the manufacturers contacted responded. Half

of these responses (n =9) apparently yielded data previously

not publicly accessible; one explicitly stated that no additional

information had been retrieved, and for eight reports it was not

clearly described.

Of those agencies where requests to industry are at the

discretion of the HTA authors, only four of nine reports (44

percent) described a corresponding request; in all cases it re-

mains unclear whether a response was received. None of the

seventeen reports of agencies where requests to industry were

not mentioned in the methods paper described a corresponding

request.

An analysis of the extent to which previously unpublished

data were included in the statistical analyses and their possible

impact on results turned out to be unfeasible in view of this

sparse reporting.

Seven reports were included of those agencies where man-

ufacturers (or others) file an application for the product to be

assessed (HAS, MSAC, PHARMAC). While three reports of

MSAC did not include unpublished data (possibly reflecting the

fact that these reports are carried out separately from the assess-

ment of the manufacturer’s application), two of the three HAS

reports included unpublished data. Data submitted to PHAR-

MAC included slides presented at conferences and further data

submitted in confidence.

Searches in Publicly Available Sources.

In thirty-eight of seventy-three re-

ports (52 percent) in our sample, trial registries were searched.

Most reports stated that this was done to identify ongoing trials,

but some also described that this served to identify both on-

going and unpublished trials. The registry most often searched

was ClinicalTrials.gov. In twenty-five of sixty-four reports (39

percent) involving the use of drugs or medical devices, regu-

latory authority Web sites (mostly the FDA) were searched to

identify further documents. In twelve of seventy-three reports

(16 percent), the literature search explicitly encompassed hand

searching of selected conference proceedings. Of the remain-

ing reports, many reported searches in specialized electronic

databases, for example, Conference Papers Index, Inside Con-

ferences or ISI Proceedings.

DISCUSSION

Policies and Practices Regarding Requests to Industry

Although the neglect of unpublished data can seriously threaten

the validity of HTA reports, international HTA agencies cur-

rently differ considerably in their efforts to address this issue,

INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 30:1, 2014 40

https://doi.org/10.1017/S026646231300072X

Downloaded from https://www.cambridge.org/core. Universitaetsbibliothek, on 26 Oct 2017 at 13:40:50, subject to the Cambridge Core terms of use, available at https://www.cambridge.org/core/terms.

Unpublished data in HTA reports

both with regard to the question as to whether and how industry

is asked for unpublished data and to policies concerning the re-

porting and publishing of data received. Current practices vary

according to the different policies implemented by the agencies,

where only a minority of agencies foresee routine data requests

to industry. Surprisingly, in those reports where manufacturers

were approached, the reporting of the request and of the data

retrieved was often sparse. It, therefore, often remained unclear

whether and which manufacturers were approached, whether

they provided data, and whether or where these were included

in the statistical analysis.

Alternative Strategies to Identify Unpublished Data

In approximately half of the reports, the search in bibliographic

databases was supplemented by a search in trial registries, al-

though it was often not clearly stated whether the main intent of

the search was to identify ongoing studies or studies not (yet)

published. Some HTA authors expended considerable effort in

identifying unpublished randomized controlled trials from trial

registries (e.g., (18;19)).

The handling of conference abstracts varied across agen-

cies. Abstract publications pose a serious problem for HTA

authors, as their inclusion may be important to reduce the influ-

ence of reporting bias, yet they may lack important information

(20). While many reports excluded abstracts, some HTA au-

thors decided to include studies on the basis of abstracts alone

to ensure that as much evidence as possible was included, as

long as enough information for appraising the study’s quality

was available (e.g., (19;21)). It is noteworthy in this context that

the methods papers of the HTA agencies often did not specify

how to proceed with conference abstracts once identified.

Although this was not the focus of our research, several

reports in our sample explicitly discussed the problem of pub-

lication bias or missing data (e.g., (22–25)), and some applied

strategies such as funnel plots or Egger’s weighted regression

statistics to investigate possible publication bias (e.g., (25;26)).

While many reports put considerable effort into trying to iden-

tify unpublished data by means of different strategies, some

HTA authors decided as a matter of principle not to search for

unpublished data to maintain the reproducibility of the data in-

cluded (e.g., (27)), or because unpublished data are difficult to

search for systematically and exhaustively (e.g., (28)).

POLICY IMPLICATIONS

As noted above, we did not assess the impact of requests to

industry for unpublished data on the conclusions of the HTA

reports as the current extent and quality of information did

not allow for such an analysis. The actual impact of requests to

industry therefore remains unclear for the reports in our sample.

However, as long as legislation does not require all trials to be

registered and the reporting of results in registries is incomplete

(29), there are strong arguments for such requests.

For those agencies or research groups that consider carry-

ing out requests to industry in the future, some lessons may

be learned from those agencies that already routinely do so.

As some methods papers highlight, it is important to ensure

that all relevant unpublished studies are disclosed upon request

(e.g., by requesting corresponding declarations from manufac-

turers): if data were provided selectively, this would possibly

even increase reporting bias. Standardized request forms fol-

lowing CONSORT guidelines (30) are a way to facilitate re-

quests to prevent selective reporting; ideally, full study reports

should be made available upon request. Efforts should also be

made to make data provided by manufacturers publicly available

as soon as possible to ensure that assessments are transparent

and the ensuing decisions open to scrutiny (31).

Given the sparse reporting of requests to industry found in

many of the reports assessed, it would be desirable to develop

reporting standards for such requests and for the information

they yield. Although reporting guidelines for systematic reviews

require all data sources to be stated (32), detailed guidance for

reporting requests to industry is currently lacking. We sug-

gest that these should include the manufacturers approached,

whether they provided data and the publication status of these

data (see also (33)).

However, some reports in our sample (e.g., (23;34)), as

well as further examples such as the reboxetine case (5) or the

Cochrane Review on Tamiflu (35), show that routine inquiries

alone are often insufficient, and despite considerable persistence

and investment of resources, all relevant data may not always

be provided, as in the case of Tamiflu (6;36).

Ideally, requests to industry will be made superfluous in

the future by legislative measures that guarantee HTA agencies

access to relevant data, as already implemented for early benefit

assessments of drugs in Germany (37). A further groundbreak-

ing initiative in the area of drug assessments is EMA’s new pol-

icy of releasing marketing authorization documents on request

(e.g., clinical study reports), after the decision-making process

for the application of the drug in question is finalized (38). This

policy has been legally challenged: in spring 2013, the General

Court, as a result of complaints filed by two pharmaceutical

companies, issued a temporary injunction ordering EMA not to

provide any documents until a final ruling is given (39). Despite

this, EMA is planning proactive publication of such information

depending on the type of data (40). For nondrug interventions,

however, the road to full availability of relevant data seems to

be much longer. Initiatives to promote the prospective public

registration of studies and disclosure of data (e.g., (41)) may,

therefore, remain more important for studies evaluating medical

devices or other nondrug interventions.

Strengths and Weaknesses of Our Study

We aimed to create a comprehensive and systematic approach

for identifying HTA agencies to obtain an unbiased picture

of the HTA community. The analysis of both methods papers

41 INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 30:1, 2014

https://doi.org/10.1017/S026646231300072X

Downloaded from https://www.cambridge.org/core. Universitaetsbibliothek, on 26 Oct 2017 at 13:40:50, subject to the Cambridge Core terms of use, available at https://www.cambridge.org/core/terms.

Kreis et al.

and recent HTA reports allowed capturing not only the policies

specified by the agencies, but also actual practices.

Challenges we encountered were the considerable inter-

twining of agencies in their work (i.e., mutual commissioning

of reports or mutual use of methods papers), the fact that sev-

eral HTA agencies produce different types of documents, and

difficulties in always clearly distinguishing between “real” HTA

reports and assessments triggered by manufacturer applications.

Our study has some limitations. Approximately a quarter

of nondrug interventions assessed in the HTA reports of our

sample did not involve the application of a medical device so that

manufacturers could not be contacted. We did not systematically

evaluate whether in these cases other possible owners of data,

e.g., research groups at universities, were contacted. Selective

publication of studies, however, may be equally (or even more)

prevalent in academic research, especially with regard to study

designs other than clinical trials. A challenge in the evaluation

of interventions not commercially owned is that it is less clear

who may have carried out studies or who may be in possession

of the data, and the diversity of relevant parties in this field may

make requests difficult.

In addition, our analysis, which was exploratory in nature,

was restricted to information available in publicly available doc-

uments, and, despite our efforts, we may have failed to identify

all relevant documents. Some methods papers may also not re-

flect current policies, as indicated by some of the respondents

to our inquiries. In the absence of other information available,

we treated methods papers equally, regardless of considerable

differences with regard to their comprehensiveness and their

currentness. As the consideration of reporting bias in HTA has

changed over time, differences between methods papers in this

regard may also be related to the time at which they were writ-

ten. Inevitably, our study does not reflect current activities or

discussions at the HTA agencies that are yet unconsidered in

official documents. Thus, future research could strive for a more

extensive participatory approach to expand our knowledge on

strategies identified in our study and agencies’ experiences.

CONCLUSIONS

It seems self-evident that HTA reports influencing health care

decisions for millions of patients should be based on all rele-

vant data. To different extents, HTA agencies search trial reg-

istries, regulatory authorities’ public documents or conference

abstracts to identify studies and data not (yet) published in peer-

reviewed journals. In future, HTA agencies should be guaran-

teed comprehensive access to relevant data either by compul-

sory registration of study protocols and comprehensive results

in public trial registries or by being granted access to clinical

study documents submitted to regulatory authorities. Until then,

requests to industry may represent one strategy to access and

include unpublished data, while agencies can learn from each

other concerning successful practice. Furthermore, the devel-

opment of reporting standards for such requests and for data

obtained from manufacturers would help to improve the trans-

parency of HTA reports.

SUPPLEMENTARY MATERIAL

Supplementary Table 1–4: http://dx.doi.org/10.1017/

S026646231300072X

CONTACT INFORMATION

Julia Kreis, Dipl.-Psych., ([email protected]), Research

Associate, Institute for Quality and Efficiency in Health Care

(IQWiG), Cologne, Germany

Dimitra Panteli, MD, MscPH, Research Fellow, Department

of Health Care Management, University of Technology, Berlin,

Germany

Reinhard Busse, MD, MPH, FFPH, Professor and Depart-

ment Head, Department of Health Care Management, Univer-

sity of Technology, Berlin, Germany

CONFLICTS OF INTEREST

Reinhard Busse has received travel support from the European

Observatory on Health Systems and Policies. Julia Kreis is

an employee of IQWiG. In order to produce unbiased health

technology assessment reports, the institute depends on access

to all of the relevant data on the topic under investigation. Kreis

therefore supports the mandatory worldwide establishment of

trial registries and public access to clinical study reports. The

other authors report they have no potential conflicts of interest.

REFERENCES

1. Sterne JAC, Egger M, Moher D, on behalf of the Cochrane Bias Meth-

ods Group. Addressing reporting biases. In: Higgins JPT, Green S, eds.

Cochrane handbook for systematic reviews of interventions. Chichester,

UK: Cochrane Collaboration and John Wiley & Sons; 2008.

2. McGauran N, Wieseler B, Kreis J, et al. Reporting bias in medical research

- A narrative review. Trials. 2010;11:37.

3. Higgins JPT, Altman DG, on behalf of the Cochrane Statistical Methods

Group and the Cochrane Bias Methods Group. Assessing risk of bias in

included studies. In: Higgins JPT, Green S, eds. Cochrane handbook for

systematic reviews of interventions. Chichester, UK: Cochrane Collabo-

ration and John Wiley & Sons; 2008.

4. Song F, Parekh S, Hooper L, et al. Dissemination and publication of

research findings: An updated review of related biases. Health Technol

Assess. 2010;14:iii, ix-xi, 1-193.

5. Eyding D, Lelgemann M, Grouven U, et al. Reboxetine for acute treatment

of major depression: Systematic review and meta-analysis of published

and unpublished placebo and selective serotonin reuptake inhibitor con-

trolled trials. BMJ. 2010;341:c4737.

6. Godlee F, Loder E. Missing clinical trial data: Setting the record straight.

BMJ. 2010;341:c5641.

7. Code of Federal Regulations. Title 21 - Food and Drugs Chapter - Food

and Drug Administration of Health and Human Services. Subchapter D

- Drugs for Human Use. Part 314 - Applications for FDA Approval to

Market a New Drug. Subpart B–ApplicationsSec. 314.50 Content and

format of an application (21CFR314.50).

INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 30:1, 2014 42

https://doi.org/10.1017/S026646231300072X

Downloaded from https://www.cambridge.org/core. Universitaetsbibliothek, on 26 Oct 2017 at 13:40:50, subject to the Cambridge Core terms of use, available at https://www.cambridge.org/core/terms.

Unpublished data in HTA reports

8. Wieseler B, McGauran N, Kaiser T. Finding studies on reboxetine: A tale

of hide and seek. BMJ. 2010;341:c4942.

9. Laurance J. Drug giants warned: Tell the truth on medicines. The Inde-

pendent. 27.02.2008.

10. Depressing research. Lancet. 2004;363:1335.

11. De Angelis C, Drazen JM, Frizelle FA, et al. Clinical trial registration: A

statement from the International Committee of Medical Journal Editors.

Lancet. 2004;364:911-912.

12. New York State Attorney General. Settlement sets new standard for

release of drug information [online]. http://www.ag.ny.gov/press-

release/settlement-sets-new-standard-release-drug-information (ac-

cessed December 2, 2012).

13. Wood AJ. Progress and deficiencies in the registration of clinical trials. N

Engl J Med. 2009;360:824-830.

14. European Medicines Agency. About EU Clinical Trials Regis-

ter [online]. https://www.clinicaltrialsregister.eu/about.html (accessed

August 29, 2013).

15. Turner EH. Closing a loophole in the FDA Amendments Act. Science.

2008;322:44-46.

16. Lefebvre C, Manheimer E, Glanville J, on behalf of the Cochrane Infor-

mation Retrieval Methods. Searching for studies. In: Higgins JPT, Green

S, eds. Cochrane handbook for systematic reviews of interventions.Chich-

ester, UK: Cochrane Collaboration and John Wiley & Sons; 2008.

17. Smith R. What is publication? BMJ. 1999;318:142.

18. Yank V, Tuohy CV, Logan AC, et al. Comparative effectiveness of in-

hospital use of recombinant factor VIIa for off-label indications vs. usual

care. Rockville (MD): Agency for Healthcare Research and Quality (US);

2010 May. Report No.: 10-EHC030-EF.

19. Fayter D, Corbett M, Heirs M, Fox D, Eastwood A. A systematic review of

photodynamic therapy in the treatment of pre-cancerous skin conditions,

Barrett’s oesophagus and cancers of the biliary tract, brain, head and neck,

lung, oesophagus and skin. Health Technol Assess. 2010;14:1-288.

20. Hopewell S, McDonald S, Clarke M, Egger M. Grey literature in meta-

analyses of randomized trials of health care interventions. Cochrane

Database Syst Rev. 2007;2:MR000010.

21.SmithB,CarsonS,FuR,etal.Drug class review: Disease-modifying

drugs for multiple sclerosis. Final update 1 report. Portland (OR): 2010.

Oregon Health & Science University; 2010 Aug.

22. Wessling A, Ramsberg J. The review of antidepressants. Solna: TLV;

2008.

23. Metcalfe S, Burgess C, Laking G, et al. Trastuzumab: Possible publication

bias. Lancet. 2008;371:1646-1648.

24. Buchberger B, Follman M, Freyer D, et al. Bedeutung von Wachstumsfak-

toren f¨

ur die Behandlung von chronischen Wunden am Beispiel des dia-

betischen Fußulcus.K

oln: DAHTA; 2010. (Schriftenreihe Health Tech-

nology Assessment).

25. Coleman CI, Baker WL, Kluger J, et al. Comparative effectiveness of an-

giotensin converting enzyme inhibitors or angiotensin II receptor blockers

added to standard medical therapy for treating stable ischemic heart dis-

ease. Rockville (MD): 2009. Agency for Healthcare Research and Quality

(US); 2009 Oct.

26. Neyt M, Van den Bruel A, Gailly J, Thiry N, Devriese S.

Tiotropium in the treatment of chronic obstructive pulmonary dis-

ease: Health technology assessment. KCE reports 108C. Brussels: Bel-

gian Health Care Knowledge Centre (KCE); 2009. (Health Technology

Assessment).

27. Gorenoi V, Sch¨

onermark MP, Hagen A. Infektionsschutz in der Knieen-

doprothetik.K

oln: DAHTA; 2010. (Schriftenreihe Health Technology

Assessment).

28. Medical Services Advisory Committee. Cryotherapy for recurrent

prostate cancer and renal cancer: Part A – Salvage cryotherapy for re-

current or persistent prostate cancer after radiotherapy. 2009. (MSAC

application 1124).

29. Prayle AP, Hurley MN, Smyth AR. Compliance with mandatory reporting

of clinical trial results on ClinicalTrials.gov: Cross sectional study. BMJ.

2012;344:d7373.

30. Moher D, Hopewell S, Schulz KF, et al. CONSORT 2010: Explanation

and elaboration; Updated guidelines for reporting parallel group ran-

domised trials. BMJ. 2010;340:c869.

31. Drummond MF, Schwartz JS, Jonsson B, et al. Key principles for the im-

proved conduct of health technology assessments for resource allocation

decisions. Int J Technol Assess Health Care. 2008;24:244-258.

32. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting

items for systematic reviews and meta-analyses: The PRISMA statement.

AnnInternMed. 2009;151:264-269, W264.

33. Niederstadt C, Droste S. Reporting and presenting information retrieval

processes: The need for optimizing common practice in health tech-

nology assessment. Int J Technol Assess Health Care. 2010;26:450-

457.

34. Institut f¨

ur Qualit¨

at und Wirtschaftlichkeit im Gesundheitswesen.

Selektive Serotonin- und Noradrenalin-Wiederaufnahmehemmer (SNRI)

bei Patienten mit Depressionen: Abschlussbericht; Auftrag A05-

20A; Version 1.1 [online]. 18.08.2010 (IQWiG-Berichte; Band 55).

http://www.iqwig.de/download/A05–20A_Abschlussbericht_SNRI_bei_

Patienten_mit_Depressionen_V1-1.pdf (accessed August 29, 2013).

35. Jefferson T, Jones MA, Doshi P, et al. Neuraminidase inhibitors for

preventing and treating influenza in healthy adults and children. Cochrane

Database Syst Rev. 2012;1:CD008965.

36. Payne D. Tamiflu: The battle for secret drug data. BMJ. 2012;345:

e7303.

37. Gesetz zur Neurordnung des Arzneimittelmarktes in der gesetzlichen

Krankenversicherung (Arzneimittelmarktneuordnungsgesetz - AMNOG)

vom 22. Dezember 2010. Bundesgesetzblatt Jahrgang. 2010 Teil I Nr. 67.

38. European Medicines Agency. European Medicines Agency widens public

access to documents [online]. 30.11.2010. http://www.ema.europa.eu/

docs/en_GB/document_library/Press_release/2010/11/

WC500099468.pdf (accessed August 29, 2013).

39. European Medicines Agency. European Medicines Agency receives

interim decisions of the General Court of the EU on access to clinical

and non-clinical information [online]. 30.04.2013. http://www.ema.

europa.eu/docs/en_GB/document_library/Press_release/2013/04/

WC500142837.pdf (accessed August 29, 2013).

40. European Medicines Agency. Publication and access to clinical-

trial data [online]. 24.06.2013. http://www.ema.europa.eu/docs/en_GB/

document_library/Other/2013/06/WC500144730.pdf (accessed August

29, 2013).

41. Godlee F. Clinical trial data for all drugs in current use. BMJ.

2012;345:e7304.

43 INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 30:1, 2014

https://doi.org/10.1017/S026646231300072X

Downloaded from https://www.cambridge.org/core. Universitaetsbibliothek, on 26 Oct 2017 at 13:40:50, subject to the Cambridge Core terms of use, available at https://www.cambridge.org/core/terms.