Developing Optimal Search Strategies for Detecting Clinically Sound Studies in MEDLINE

Journal of the American Medical Informatics Association Volume 1 Number 6 Nov / Dec 1994 Developing Optimal Search. Strategies for Detecting Clinically Sound Studies in MEDLINE Abstract Obiective let To develop optimal MEDLINE search strategies for retrieving sound clinical studies of the etiology, prognosis, diagnosis, prevention, or treatment of disorders in adult general medicine. Design: Analytic survey of operating characteristics of search strategies developed by computerized combinations of terms selected to detect studies meeting basic methodologic criteria for direct clinical use in adult general medicine. Measures: The sensitivities, specificities, precision, and accuracy of 134,264 unique combinations of search terms were determined by comparison with a manual review of all articles (the “gold standard”) in ten internal medicine and general medicine journals for 1986 and 1991. Results: Less than half of the studies of the topics of interest met basic criteria for scientific merit for testing clinical applications. Combinations of search terms reached peak sensitivities of 82% for sound studies of etiology, 92% for prognosis, 92% for diagnosis, and 99% for therapy in 1991. Compared with the best single terms, multiple terms increased sensitivity for sound studies by over 30% (absolute increase), but with some loss of specificity when sensitivity was maximized. For 1986, combinations reached peak sensitivities of 72% for etiology, 95% for prognosis, 86% for diagnosis, and 98% for therapy. When search terms were combined to maximize specificity, over 93% specificity was achieved for all purpose categories in both years. Compared with individual terms, combined terms achieved near-perfect specificity that was maintained with modest increases in sensitivity in all purpose categories except therapy. Increases in accuracy were achieved by combining terms for all purpose categories, with peak accuracies reaching over 90% for therapy in 1986 and 1991. Conclusions: The retrieval of studies of important clinical topics cited in MEDLINE can be substantially enhanced by selected combinations of indexing terms and textwords. n J Am Med Informatics Assoc. 1994;1:447-458. Clinical end-user searching of MEDLINE has risen research literature database, with only a small fracdramatically during the past five years,’ spurred by tion of articles reporting evidence that can be applied the development of user-friendly software, a prolifdirectly to clinical practice. The large number of posteration of online and compact disc formats, falling ings in MEDLINE (several million), the low prevauser charges, and advertising directed at clinicians. lence of clinically applicable studies, the well-docu- MEDLINE, however, is a general-purpose biomedical mented limitations of indexing and retrieval in Affiliation of the authors: Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario, Canada. Supported by The Ontario Ministry of Health (grant no. 02651) opment Program of Canada. Correspondence and reprints: R. Brian Haynes, MD, PhD, McMaster University Medical Centre, Room 3H7, 1200 Main Street West, Hamilton, Ontario L8N 3Z5, Canada. e-mail: and the United States National Library of Medicine (grant no. R01 LM 04696). Dr. Haynes is supported in part by a National Health Scientist Award from the National Health Research and Devel- [email protected] Received for publication: 5/12/94; accepted for publication: 7/01/94. 448 HAYNES ET AL., Optimal Clinical Searches in MEDLINE Table 1 n Formula for Calculating the Sensitivities, Specificities, Precision, and Accuracy of MEDLINE Searches for Detecting Sound Clinical Studies Manual Review Meets Does Not Meet Criteria Criteria Search Detected a b Terms Not detected C d a+c b+d Sensitivity = a/(a + c). Specificity = d/(b + d). Precision = a/(a + b + articles of other formats that are detected). Accuracy = (a + d)/(a + b + c + d).* *(a + b + c + d) = all original, review, and case reports as defined by the manual review of the literature. MEDLINE from its inception,2 and the imprecise search skills of clinical end users3 all contribute to difficulties in using MEDLINE to seek answers to clinical questions. The problems lead to both missing sound studies in searching (low sensitivity) and retrieving many citations of studies that are not sound (low specificity and low precision). A potential method for improving the detection of studies of high quality for clinical practice from MEDLINE is to include search terms that select studies that are at the most advanced stages of testing for clinical application. There are relatively few study designs for these final stages of testing. For example, the randomized controlled trial is widely accepted as the standard for testing a treatment or preventive procedure.4 Similarly, an inception cohort is required for studies of prognosis, an independent “gold standard” is required for validation of a diagnostic test, and a valid comparison group is required for studies of etiology or causation. MEDLINE indexers have had “random allocation” and “randomized controlled trials” as indexing terms for some time, recently strengthened by “publication types (pt).” For diagnostic test evaluation, “sensitivity” and “specificity, ” “predictive value of tests,” and “receiver operating characteristic (ROC) curve” are relatively recent additions to the Medical Subject Headings (MeSH) vocabulary. For prognosis, “explode cohort studies” is perhaps the closest equivalent to “inception cohort.” For etiology, there are many research design options and the indexing is similarly varied, with terms such as “risk factors” and “causality” that may be applied to the content of studies regardless of the studies’ methodologies or quality. Authors, however, may use more exact methodologic terms in their titles and abstracts, and these terms [textwords (tw)] may then provide another opportunity for retrieval. We have advocated using methodologic search filters to improve the retrieval of studies of higher quality for clinical practice5 and sought in this study to develop better methodologic search filters and to verify their validity. The information retrieval properties of combinations of terms are reported. The retrieval properties of individual terms were published previously.6 The results of this study would be of most interest to clinicians doing their own searches for clinically relevant and valid studies and to librarians involved in assisting clinicians to construct their own searches. Methods The study compared the retrieval performance of methodologic search terms and phrases in MEDLINE with a manual review of each article for each issue of ten internal medicine and general medicine journals for the two years 1986 and 1991. To evaluate MEDLINE strategies designed to retrieve studies meeting basic methodologic criteria for clinical practice, MeSH terms and textwords related to research design features were run as search strategies. These search strategies were treated as diagnostic tests for sound studies and the manual review of the literature was treated as the “gold standard.” Borrowing from the concepts of diagnostic test evaluation and library science, the sensitivities, specificities, accuracy, and precision of MEDLINE searches were determined as shown in Table 1. For example, the sensitivity of each MEDLINE search strategy was calculated as the proportion of relevant, sound citations detected by that strategy. The sample size required to detect a 20% improvement in sensitivity for the comparison of one MEDLINE search strategy with another on the same topic was 73 studies meeting the methodologic criterion in each of the purpose categories for each of the years 1986 and 1991 (a type 1 error rate of 5%, one-sided, and a type 2 error rate of 20%). Figure 1 illustrates the steps involved in the data collection and analysis stages, as detailed below. Manual Review of the Literature For the years 1986 and 1991, three research assistants assessed ten journals, the same ten in each year, for articles meeting basic methodologic criteria concerning the etiology, prognosis, diagnosis, prevention, and treatment of disease of human adults. The ten journals searched were American journal of Medicine, Journal of the American Medical Informatics Association Volume 1 Number 6 Nov / Dec 1994 449 Annnls of lnternal Medicine, Archives of Internal Medicine, BMJ (British Medical Journal in 1986), Circulation, Diabetes Care, Journal of lnternal Medicine (Acta Medica Scandinavica in 1986), Journal of the American Medical Association, Lancet, and New England journal of Medicine, including supplements. These journals were selected on the basis of high-impact factors and immediacy indexes7 and in order to provide a range of publications, including both internal medicine and Manual review of the literature (the gold standard) Classification of each article Bibliographic information captured in MEDLINE general medicine journals, and both American and European authors. Articles were classified for format, interest, purpose, and methodologic rigor. The format categories and their corresponding definitions are shown in Table 2. Items excluded from classification included bannered letters to the editor, book reviews, announcements, policy watch, editorials, brief clinical ob- Collecting search terms (the diagnostic tests) Compiling list of terms Unique identifiers captured in MEDLINE Databases linked Sensitivity, specificity, accuracy, and precision calculated Elimination of terms with < 10% sensitivity or a correlation > 0.90 Remaining terms combined in a Boolean OR relationship All combinations with <40% sensitivity or < 70% specificity were eliminated Top 30 strategies for each of sensitivity, specificity, and accuracy were retrieved by purpose category Top 10 in Strategies derived Best strategies Strategies each area in two-thirds in 1991 tested from previous for 1991 database were in 1986 research were compared tested in the Best strategies tested in the with top remaining onein 1986 tested 1991 database 10 in 1986 third in 1991 Figure 1 Steps in data collection and analysis. In the left column, the steps involved in forming the database for the gold standard (manual review of the literature) are shown. In the right column, the steps involved in forming the database for the diagnostic tests (search strategies) are shown. In the center column, the databases are linked and data pertaining to the accuracy of the search strategies for identifying sound studies as defined by the “gold standard” are calculated. In the row across the bottom, how the data are reviewed and compared is shown. HAYNES ET AL., Optimal Clinical Searches in MEDLINE servations, correspondence, news, obituaries, postgraduate and continuing-education forums, and notices. To be considered of interest in the medical care of human adults, a study had to be concerned with the understanding and management of clinical problems with clinical endpoints and recommendations for applications in human subjects, at least 50% of whom had to have been 218 years of age at study entry. Every format category was classified for interest. Articles classified as original studies, reviews, or case reports and of interest were classified for purpose. Articles could have more than one purpose and were classified for all that applied. Purpose categories and their corresponding definitions are shown in Table 3. Studies in each purpose category were evaluated for methodologic rigor by determining whether they met one key methodologic criterion specific to their purpose as shown in Table 4. These criteria were based The Format Categories and Their Corresponding Definitions Used to Classify Journal Articles Format Definition Original study Any full-text article in which the inves tigators made firsthand observations Review Any full-text article that was bannered review, that had the word review in its title or in a section heading, or that indicated in the text that the intention was to review or summarize the literature about a topic General article A general or philosophical discussion of a topic without original observation and without a statement that the purpose was to review or appraise a body of knowledge, including unbannered news items, unbannered editorials, position and opinion papers, musings, and psychosocial observations Conference report Defined as such by the journal but reclassified by us as an original article or a review article when meeting those criteria Decision analysis Dissection of the management of patients into component parts, defining routes and consequences of management based on alternatives, for the purpose of defining optimal methods of management Case report An original study involving less than ten subjects Table 3 n The Purpose Categories and Their Corresponding Definitions Used to Classify Journal Articles Classified as Original Studies, Reviews, or Case Reports* and of Interest Purpose Definition Etiology Content pertained directly to causation of a disease or condition Prognosis Content pertained directly to the prediction of the clinical course of the natural history of a disease with the disease existing at the beginning of the study Diagnosis Content pertained directly to the evaluation of a disease process, usually through comparing methods of arriving at a diagnosis Treatment or preven- Content pertained directly to therapy, tion prevention, or rehabilitation Something else Purpose of the study was something other than the above ‘The terms original study, review, and case report are explained in Table 2. on critical appraisal criteria for applied research4 but were set at a minimal level in recognition that few published studies meet the full set of criteria for unbiased clinical evaluation and that clinicians are likely to be better informed looking at the best available literature even if it falls short of perfection. For example, studies of prognosis were rated as meeting the methodologic criterion if they included a cohort of subjects who had the disease in question at baseline without the outcome of interest. This criterion would allow many more studies to qualify than would more stringent criteria such as that the studies include an inception cohort (or patients at a common and preferably early point in the courses of their disorders), with at least 80% follow-up and an objectively evaluated endpoint. Similarly, for assessment of diagnostic tests, the criterion was that data about the sensitivity and specificity of the test had to be reported; this is much less stringent than the standard of requiring independent or “blinded” interpretation of the test and the gold standard.” Interrater reliability was assessed for the classification of articles for format, interest, purpose, and methods. In every case the degree of agreement beyond chance was assessed by the kappa statistic and was greater than 0.80 (P < 0.05). The manual review of the literature served as the “gold standard” against which MEDLINE search Journal of the American Medical Informatics Association Volume 1 Number 6 NOV / Dec 1994 451 strategies (the diagnostic tests) could be tested. Results of the study apply to original and review articles that are of acceptable quality from the perspective of applicability to clinical practice. Collecting Search Terms To construct a comprehensive set of search terms, we began a list of Mesh terms and textwords and then sought input from clinicians and librarians in the United States and Canada through interviews of known searchers; requests on electronic bulletin boards and in national publications, meetings, and conferences; and requests to the National Library of Medicine and the Canada Institute for Scientific and Technical Information. Individuals were asked what terms or phrases they used when searching for studies of etiology, prognosis, diagnosis, or therapy and for related review articles. Terms could be from MeSH, including publication types (pt), check tags,. and subheadings (sh), or could be textwords (tw) denoting methodology in titles and abstracts of articles. The list, excluding incorrect MeSH terms, appears in the appendix. Some of the terms and phrases were different for the two years because new publication types were introduced in 1990 and 1991 and some of the corresponding terms changed definitions. Also, five terms retrieved no citations for the ten journals in Key Methodologic Criterion, According to Purpose Category,* Used to Determine the Methodologic Rigor of Journal Articles Classified for Purpose Purpose Key Methodologic Criterion Etiology Prognosis Diagnosis Treatment Review Formal control group: random or quasi-random allocation of participants to treatment and control groups; or a nonrandomized concurrent control trial, a cohort analytic study with matching or statistical adjustment to create comparable groups, or a case-control stud! A cohort of subjects who have the disease in question at baseline without the outcome of interest Provision of sufficient data to calculate the sensitivity and specificity of the test or likelihood ratios based on subjects who had been tested with both the test and the diagnostic standard Random or quasi-random allocation of participants to treatment and control groups Reproducible description of the methods for conducting the review (this criterion was applied to every review article regardless of the purpose for doing the review) *Purpose categories and their definitions are provided in Table 3. Classification of Original and Review Articles* That Appeared in Ten Journalst for 1986 ‘and 1991 According to Four Purpose Categories and Whether They Met the Methodoldgic Criteria for These Categories, Based on Manual Review ” Year Purpose Category* 1991 1986 Etiology 5235 531 Meeting methodologic criterion (%) ,201 (38) 155.(29) Prognosis 205 149 Meeting methodologic criterion (%) 133 (65) 106 (71) Diagnosis 412 426 Meeting methodologic criterion (%) 111 (27) 92 (22) Treatment 879 936 Meeting methodologic criterion (%) 281 (32) 270 (29) *Original and review articles are defined in Table 2. There were 3,495 original and review articles for 1991 and 3,682 for 1986. The ten journals are listed in the Methods section. Purpose categories are defined in Table 3. Their methodologic criteria are described in Table 4. Numbers other than those in parentheses are the numbers of articles. 1986 and 1991, two terms retrieved no citations in 1991, and 21 terms retrieved no citations in 1986; these terms were discarded for the respective years. Data Collection Manual ratings of articles in the ten journals for 1986 and 1991 were recorded on data collection forms, and the bibliographic information, including eight-digit unique identifiers, for these articles was captured from MEDLINE. The manual review data for each article were double-entered’ into PARADOX by two independent data clerks. Each journal title was searched in MEDLINE for 1986 and 1991 and the publication types editorial, comment, letter, and news were excluded from the search using the Boolean AND NOT operator. The MeSH terms and textwords to be tested were searched in MEDLINE for the ten journals for 1986 and 1991. The unique identifiers of retrieved citations were captured and then linked with the manual review data. Testing Strategies A computer program was written in Turbo Pascal to develop search strategies by creating all possible Boolean OR combinations of the terms in the appendix, and to determine the sensitivities, specificities, accuracy, and precision of the, combinations of terms. To make the number of combinations more tractable, 452 HAYNES ET AL., Optimal Clinical Searches in MEDLINE individual MeSH terms and textwords with a sensiwere 14 for etiology (16,383 combinations), 16 for tivity <10% were eliminated, then terms with >0.90 prognosis (65,535 combinations), nine for diagnosis correlation within each purpose category were de- (511 combinations), and 15 for treatment (32,767 comtermined and the term in the pair with the lower binations). For 1986, the numbers of terms to comsensitivity was discarded from the combination analbine were nine for etiology (511 combinations), 11 ysis. Thus, the numbers of terms to combine for 1991 for prognosis (2,047 combinations), seven for diag- Combinations of Terms (Medical Subject Headings and Textwords) with the Best Sensitivity for Detecting Sound Clinical Studies for Each Purpose Category Purpose Category* Search Strategyt Sensitivity Specificity Precision 1991 Etiology Diagnosis Treatment Exp Cohort Studies or Exp Risk or Odds (tw) and Ratio: (tw) or Relative (tw) and Risk (tw) or Case (tw) and Control: (tw) Incidence or Exp Mortality or Follow-up Studies or Mortality (sh) or Prognos: (tw) or Predict: (tw) or Course: (tw) Exp Sensitivity a#d Specificity or Diagnosis& (px) or Diagnostic Use (sh) or Sensitivity (tw) or Specificity (tw) Randomized Controlled Trial (pt) or Drug Therapy (sh) or Therapeutic Use (sh) or Random: (tw) 1986 Etiology Prognosis Diagnosis Treatment Exp Cohort Studies or Risk (tw) or Causation (tw) or Causal: (tw) Prognosis or Exp Cohort Studies or Mortality (sh) or Natural (tw) and History (tw) or Predict: (tw) or Course (tw) Diagnosis& (px) or Specificity (tw) Random Allocation or Comparative Study or Drug Therapy (sh) or Placebo: (tw) or Controlled (tw) and Trial: (tw) 0.82 0.70 0.14 (0.65) 0.92 0.73 (0.75) 0.11 0.92 (0.86) 0.99 (0.73) 0.72 (0.78) 0.95 (0.92) 0.73 0.09 0.74 0.79 0.78 0.22 0.12 0.11 0.86 0.73 0.07 (0.88) 0.98 0.71 0.18 (0.80) *Purpose categories are defined in Table 3. tw = textword; sh = subheading; px = subheading pre-explosion; pt = publication type. Sensitivity in the 1986 database. This strategy could not be tested in the 1986 database. Strategy tested with 92% sensitivity in 1991 and 75% sensitivity in 1986: “Prognosis or Exp Morbidity or Follow-up Studies or Mortality (sh) or Prognos: (tw) or Clinical (tw) and Course (tw) or Predict: (tw) or Prognostic (tw) and Factor: (tw).” This strategy could not be tested in the 1986 database. Strategy tested with 99% sensitivity in 1991 and 73% sensitivity in 1986: “Clinical Trial (pt) or Drug Therapy (sh) or Therapeutic Use (sh) or Random: (tw) or Double (tw) and Blind: (tw).” [Sensitivity in the 1991 database. Journal of the American Medical lnformatics Association Volume 1 Number 6 Nov / Dec 1994 453 Table 7 l Combinations of Terms (Medical Subject Headings and Textwords) with the Best Specificity for Detecting Sound Clinical Studies for Each Purpose Category Purpose Category* Search Strategy Sensitivity Specificity Precision 1991 Etiology Cohort Studies 0.40 0.97 0.42 or Case-control Studies (0.98) Prognosis Prognosis 0.49 0.97 0.34 or Survival Analysis (0.97) Diagnosis Exp Sensitivity a#d 0.55 0.98 0.40 Specificity (0.99) or Predictive (tw) and Value: (tw) Treatment Placebo: (tw) 0.57 0.97 0.56 or Double (tw) and Blind: (tw) (0.97)! 1986 Etiology Exp Risk 0.51 0.94 0.21 or Causation (tw) (0.88) or Causal: (tw) or Relative (tw) and Risk (tw) Prognosis Prognosis 0.58 0.97 0.31 or Natural (tw) and History (tw) (0.96) or Prognostic (tw) and Factor: (tw) Diagnosis Specificity (tw) 0.49 0.98 0.36 or Predictive (tw) and Value: (tw) (0.96)ll or False (tw) and Positive (tw) Treatment Placebo: (tw) 0.58 0.97 0.60 or Double (tw) and Blind: (tw) (0.96)ll ‘Purpose categories are defined in Table 3. tw = textword. This strategy could not be tested in the 1986 database. Strategy tested with 94% specificity in 1991 and 98% specificity in 1986: “Odds (tw) and Ratio: (tw) or Relative (tw) and Risk (tw).” This strategy could not be tested in the 1986 database. Strategy tested with 97% specificity in the two years: “Prognosis or Prognostic (tw) and Factor: (tw).” Specificity in the 1986 database. Specificity in the 1991 database. nosis (127 combinations), and 14 for treatment (16,383 combinations). Files containing the results for each individual combination (e.g., results for each of the 65,535 combinations of terms for prognosis in 1991) were sorted on each of sensitivity, specificity, and accuracy, in descending order. Using these sorted files, the 30 search strategies yielding the highest figures for each of sensitivity, specificity, and accuracy were reviewed for 1991 and for 1986 for all purpose categories. To compare search yields for 1986 and 1991, the ten strategies yielding the highest figures for each of sensitivity, specificity, and accuracy for each category for each year were compared and 95% confidence intervals (CIs) were calculated. To determine the test-retest reliability of the search strategies, searches for treatment and etiology in 1991 and for treatment in 1986 were derived using a random two-thirds of the database and tested in the remaining one-third of the database. To assess the cross-year reliability-of the searches, search strategies yielding the best values for each of sensitivity, specificity, and accuracy for each purpose category in 1991 were tested in the 1986 database and the best 1986 strategies were tested in the 1991 database. We also attempted to test search strategies described in previously published studies in our 1991 database.8-14 Results The results of the manual review of the journals appear in Table 5. Even though the journals reviewed had the highest impact and immediacy ratings for general medicine and internal medicine and the criteria for methodologic adequacy were minimal, less than half of the articles met these criteria in three of the four categories. The exception, prognosis, may have had better results because our criterion was so lenient; very few of the studies would have met a stronger methodologic requirement of including an inception cohort. 454 HAYNES ET AL., Optimal Clinical Searches in MEDLINE Table 8 n Combinations of Terms (Medical Subject Heading and Textwords) with the Best Accuracy for Detecting Sound Clinical Studies for Each Purpose Category Purpose Category* Search Strategy Sensitivity Specificity Precision Accuracy 1991 Etiology Prognosis Diagnosis Treatment 1986 Etiology Prognosis Diagnosis Treatment Cohort Studies or Exp Risk or Odds (tw) and Ratio: (tw) or Case (tw) and Control: (tw) Survival Rate or Survival Analysis or Follow-up Studies or Prognos: (tw) or Clinical (tw) and Course (tw) Exp Sensitivity a#d Specificity or Diagnosis (sh) or Diagnostic Use (sh) or Specificity (tw) or Predictive (tw) and Value: (tw) Randomized Controlled Trial (pt) or Random: (tw) Prospective Studies or Risk (tw) Prognosis or Follow-up Studies or Mortality (sh) or Natural (tw) and History (tw) or Prognostic (tw) and Factor: (tw) or Course (tw) Diagnosis & (px) or Specificity (tw) Random: (tw) or Placebo: (tw) or Double (tw) and Blind: (tw) or Controlled (tw) and Trial: (tw) 0.73 0.83 0.87 0.86 0.84 0.13 0.96 0.91 0.46 0.70 0.84 0.92 0.87 0.86 0.92 0.85 0.73 0.93 0.21 0.19 0.14 0.15 0.07 0.48 0.84 (0.51) 0.86 (0.87) 0.84 (0.82) 0.91 (0.91) 0.83 (0.72)‘” 0.87. (0.81) 0.73 (0.76) 0.93 (0.88) Purpose categories are defined in Table 3. tw = textword; sh = subheading; pt = publication type; px = subheading pre-explosion. This strategy could not be tested in the 1986 database. Strategy tested with 83% accuracy in 1991 and 51% accuracy in 1986: “Exp Risk or Etiology& (px) or Cohort (tw) or Odds (tw) and Ratio: (tw).” This strategy could not be tested in the 1986 database. Strategy tested with 82% accuracy in 1991 and 87% accuracy in 1986: “Prognosis or Exp Mortality or Follow-up Studies or Mortality (sh) or Prognos: (tw) or Clinical (tw) and Course (tw) or Prognostic (tw) and Factor: (tw).” Accuracy in the 1986 database. This strategy could not be tested in the 1986 database. Strategy tested with 91% accuracy in the two years: “Random (tw) or Placebo: (tw).” ‘Accuracy in the 1991 database. For 1991 and 1986, the combinations of terms yielding the best sensitivities for the individual purpose categories are presented in Table 6. The sensitivities, specificities, and precision among the top 30 strategies for all purpose categories and the two years were similar to the figures presented here. When comparing the top ten search strategies yielding the highest sensitivities in 1991 with the top ten search strategies for 1986, there was a 10% difference for etiology (CI, 1% to 20%), a 3%’ difference for prognosis (CI, -3% to 10%),’ a 6% difference for diagnosis (CI, -3% to 15%); and a 1% difference for treatment (CI, -1% to 4%‘). For 1991 and 1986, the combinations of terms yielding the best specificities for the individual purpose categories are presented in Table 7. ‘When reviewing the 30 search strategies yielding. the highest specificities for the individual purpose ‘categories and years, the drop in specificity across the top 30 ranged from 2% to 14%, with corresponding increases in sensitivity Journal of the American Medical lnformatics Association Volume 1 Number 6 Nov / Dec 1994 455 Table 9 n Strategies Derived From Previous Research, Their Modifications for Testing in the 1991 Database, and Their Sensitivities Author Original Strategy Modified Strategy Sensitivity* Poynard and Conn Dickersin et al.“’ Kirpalani et al.” Schuyler et al.” Gotzsche and Lange” Bernstein” Jadad and McQuayl’ {Liver Disease or Biliary Tract Disease} and (Drug Therapy (sh) or Surgery (sh) or Radiotherapy (sh) or Therapy (sh)) and (Human and {Adult or Middle Age or Aged]) and (Comparative Study or Clinical Research) {Exp Hyperbilirubinemia and Exp Infant} or Random Allocation or Exp Research Design or Clinical Trials or Random: (tw) Therapy (sh) or Prevention and Control (sh) or Exp Feeding Methods or Diet Therapy (sh) or Drug Therapy (sh) or Random Allocation or All Random: (tw) or Placebos Clinical Trials or Clinical Trial (pt) {Exp Arthritis, Rheumatoid and 1 of 17 NSAIDS} or Comparative Study or Double-blind Method or Double and Blind (tw) {Liver or Liver Disease or Biliary Tract or Biliary Tract Diseases}. and (Human and {Adult or Middle Age or Aged}) and (Random Allocation or Double-blind Method) or (Clinical Trials or Comparative Study and {Random (tw) or Double Blind (tw) or Controlled (tw))) ({Pain* or Exp Pain or Analg or Exp Analgesia or Exp Analgesics} and Human) and ({Exp Clinical ,Trials or Clinical Trials or Random* or Random Allocation or Randomized Controlled Trials or Double Blind or Double-blind Method or Meta-analysis) and Human) Drug Therapy (sh) or Therapy (sh) or Comparative study 0.85 Random Allocation or Exp Research Design or Clinical Trials or Random: (tw) Therapy (sh) or Prevention and Control (sh) or Drug Therapy (sh) or Random Allocation or Random: (tw) or Placebos Clinical Trials or Clinical Trial (pt) Comparative Study or Double-blind Method or Double (tw) and Blind: (tw) Random Allocation or Double-blind Method or Clinical Trials or Comparative Study or Random: (tw) or Double (tw) and Blind: (tw) Exp Clinical Trials or Clinical Trials or Random: (tw) or Random Allocation or Randomized Controlled Trials or Double (tw) and Blind: (tw) or Double-blind Method 0.94 0.98 0.93 0.73 0.96 0.94 *sh = subheading; tw = textword; pt = publication type; NSAIDS = nonsteroidal anti-inflammatory drug. By comparison, our most sensitive strategy for therapy (from Table 6) had a sensitivity of 99%.