An overview of the design and methods for retrieving high-quality studies for clinical care

BACKGROUND: With the information explosion, the retrieval of the best clinical evidence from large, general purpose, bibliographic databases such as MEDLINE can be difficult. Both researchers conducting systematic reviews and clinicians faced with a patient care question are confronted with the daunting task of searching for the best medical literature in electronic databases. Many have advocated the use of search filters or "hedges" to assist with the searching process. The purpose of this report is to describe the design and methods of a study that set out to develop optimal search strategies for retrieving sound clinical studies of health disorders in large electronics databases. OBJECTIVE: To describe the design and methods of a study that set out to develop optimal search strategies for retrieving sound clinical studies of health disorders in large electronic databases. DESIGN: An analytic survey comparing hand searches of 170 journals in the year 2000 with retrievals from MEDLINE, EMBASE, CINAHL, and PsycINFO for candidate search terms and combinations. The sensitivity, specificity, precision, and accuracy of unique search terms and combinations of search terms were calculated. CONCLUSION: A study design modeled after a diagnostic testing procedure with a gold standard (the hand search of the literature) and a test (the search terms) is an effective way of developing, testing, and validating search strategies for use in large electronic databases

Sample size determination for bibliographic retrieval studies

<p>Abstract</p> <p>Background</p> <p>Research for developing search strategies to retrieve high-quality clinical journal articles from MEDLINE is expensive and time-consuming. The objective of this study was to determine the minimal number of high-quality articles in a journal subset that would need to be hand-searched to update or create new MEDLINE search strategies for treatment, diagnosis, and prognosis studies.</p> <p>Methods</p> <p>The desired width of the 95% confidence intervals (W) for the lowest sensitivity among existing search strategies was used to calculate the number of high-quality articles needed to reliably update search strategies. New search strategies were derived in journal subsets formed by 2 approaches: random sampling of journals and top journals (having the most high-quality articles). The new strategies were tested in both the original large journal database and in a low-yielding journal (having few high-quality articles) subset.</p> <p>Results</p> <p>For treatment studies, if W was 10% or less for the lowest sensitivity among our existing search strategies, a subset of 15 randomly selected journals or 2 top journals were adequate for updating search strategies, based on each approach having at least 99 high-quality articles. The new strategies derived in 15 randomly selected journals or 2 top journals performed well in the original large journal database. Nevertheless, the new search strategies developed using the random sampling approach performed better than those developed using the top journal approach in a low-yielding journal subset. For studies of diagnosis and prognosis, no journal subset had enough high-quality articles to achieve the expected W (10%).</p> <p>Conclusion</p> <p>The approach of randomly sampling a small subset of journals that includes sufficient high-quality articles is an efficient way to update or create search strategies for high-quality articles on therapy in MEDLINE. The concentrations of diagnosis and prognosis articles are too low for this approach.</p

Optimal search strategies for identifying sound clinical prediction studies in EMBASE

BACKGROUND: Clinical prediction guides assist clinicians by pointing to specific elements of the patient's clinical presentation that should be considered when forming a diagnosis, prognosis or judgment regarding treatment outcome. The numbers of validated clinical prediction guides are growing in the medical literature, but their retrieval from large biomedical databases remains problematic and this presents a barrier to their uptake in medical practice. We undertook the systematic development of search strategies ("hedges") for retrieval of empirically tested clinical prediction guides from EMBASE. METHODS: An analytic survey was conducted, testing the retrieval performance of search strategies run in EMBASE against the gold standard of hand searching, using a sample of all 27,769 articles identified in 55 journals for the 2000 publishing year. All articles were categorized as original studies, review articles, general papers, or case reports. The original and review articles were then tagged as 'pass' or 'fail' for methodologic rigor in the areas of clinical prediction guides and other clinical topics. Search terms that depicted clinical prediction guides were selected from a pool of index terms and text words gathered in house and through request to clinicians, librarians and professional searchers. A total of 36,232 search strategies composed of single and multiple term phrases were trialed for retrieval of clinical prediction studies. The sensitivity, specificity, precision, and accuracy of search strategies were calculated to identify which were the best. RESULTS: 163 clinical prediction studies were identified, of which 69 (42.3%) passed criteria for scientific merit. A 3-term strategy optimized sensitivity at 91.3% and specificity at 90.2%. Higher sensitivity (97.1%) was reached with a different 3-term strategy, but with a 16% drop in specificity. The best measure of specificity (98.8%) was found in a 2-term strategy, but with a considerable fall in sensitivity to 60.9%. All single term strategies performed less well than 2- and 3-term strategies. CONCLUSION: The retrieval of sound clinical prediction studies from EMBASE is supported by several search strategies

Critical appraisal skills of family physicians in Ontario, Canada

BACKGROUND: Our primary objective in this study was to measure family physicians' knowledge of the key elements that go into assessing the validity and interpreting the results in three different types of studies: i) a randomized controlled trial (RCT); ii) a study evaluating a diagnostic test; and iii) a systematic review (SR). Our secondary objectives were to determine the relationship between the above skills and age, gender, and type of practice. METHODS: We obtained a random sample of 1000 family physicians in Ontario from the College of Family Physicians of Canada database. These physicians were sent a questionnaire in the mail with follow-up mailings to non-responders at 3 and 8 weeks. The questionnaire was designed to measure knowledge and understanding of the basic concepts of critical appraisal skills. Based on the responses to the questions an Evidence Based Medicine (EBM) Knowledge Score was determined for each physician. RESULTS: A response rate of 30.2% was achieved. The respondents were younger and more likely to be recent graduates than the population of Ontario Family Physicians as a whole. This was an expected outcome. Just over 50% of respondents were able to answer questions concerning the critical appraisal of methods and the interpretation of results of research articles satisfactorily. The average score on the 12-point EBM Knowledge Scale was 6.4. The younger physicians scored higher than the older physicians, and academic physicians scored higher than community-based physicians. Scores of male and female physicians did not differ. CONCLUSIONS: We have shown that in a population of physicians which is younger than the general population of physicians, about 50% have reasonable knowledge regarding the critical appraisal of the methods and the interpretation of results of a research article. In general, younger physicians were more knowledgeable than were older physicians. EBM principles were felt to be important to the practice of medicine by 95% of respondents

Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: Methods of a decision-maker-researcher partnership systematic review

<p>Abstract</p> <p>Background</p> <p>Computerized clinical decision support systems are information technology-based systems designed to improve clinical decision-making. As with any healthcare intervention with claims to improve process of care or patient outcomes, decision support systems should be rigorously evaluated before widespread dissemination into clinical practice. Engaging healthcare providers and managers in the review process may facilitate knowledge translation and uptake. The objective of this research was to form a partnership of healthcare providers, managers, and researchers to review randomized controlled trials assessing the effects of computerized decision support for six clinical application areas: primary preventive care, therapeutic drug monitoring and dosing, drug prescribing, chronic disease management, diagnostic test ordering and interpretation, and acute care management; and to identify study characteristics that predict benefit.</p> <p>Methods</p> <p>The review was undertaken by the Health Information Research Unit, McMaster University, in partnership with Hamilton Health Sciences, the Hamilton, Niagara, Haldimand, and Brant Local Health Integration Network, and pertinent healthcare service teams. Following agreement on information needs and interests with decision-makers, our earlier systematic review was updated by searching Medline, EMBASE, EBM Review databases, and Inspec, and reviewing reference lists through 6 January 2010. Data extraction items were expanded according to input from decision-makers. Authors of primary studies were contacted to confirm data and to provide additional information. Eligible trials were organized according to clinical area of application. We included randomized controlled trials that evaluated the effect on practitioner performance or patient outcomes of patient care provided with a computerized clinical decision support system compared with patient care without such a system.</p> <p>Results</p> <p>Data will be summarized using descriptive summary measures, including proportions for categorical variables and means for continuous variables. Univariable and multivariable logistic regression models will be used to investigate associations between outcomes of interest and study specific covariates. When reporting results from individual studies, we will cite the measures of association and p-values reported in the studies. If appropriate for groups of studies with similar features, we will conduct meta-analyses.</p> <p>Conclusion</p> <p>A decision-maker-researcher partnership provides a model for systematic reviews that may foster knowledge translation and uptake.</p

Teaching clinical informatics to third-year medical students: negative results from two controlled trials

BACKGROUND: Prior educational interventions to increase seeking evidence by medical students have been unsuccessful. METHODS: We report two quasirandomized controlled trials to increase seeking of medical evidence by third-year medical students. In the first trial (1997–1998), we placed computers in clinical locations and taught their use in a 6-hour course. Based on negative results, we created SUMSearch(TM), an Internet site that automates searching for medical evidence by simultaneous meta-searching of MEDLINE and other sites. In the second trial (1999–2000), we taught SUMSearch's use in a 5½-hour course. Both courses were taught during the medicine clerkship. For each trial, we surveyed the entire third-year class at 6 months, after half of the students had taken the course (intervention group). The students who had not received the intervention were the control group. We measured self-report of search frequency and satisfaction with search quality and speed. RESULTS: The proportion of all students who reported searching at least weekly for medical evidence significantly increased from 19% (1997–1998) to 42% (1999–2000). The proportion of all students who were satisfied with their search results increased significantly between study years. However, in neither study year did the interventions increase searching or satisfaction with results. Satisfaction with the speed of searching was 27% in 1999–2000. This did not increase between studies years and was not changed by the interventions. CONCLUSION: None of our interventions affected searching habits. Even with automated searching, students report low satisfaction with search speed. We are concerned that students using current strategies for seeking medical evidence will be less likely to seek and appraise original studies when they enter medical practice and have less time

What kind of evidence is it that Evidence-Based Medicine advocates want health care providers and consumers to pay attention to?

BACKGROUND: In 1992, Evidence-Based Medicine advocates proclaimed a "new paradigm", in which evidence from health care research is the best basis for decisions for individual patients and health systems. Hailed in New York Times Magazine in 2001 as one of the most influential ideas of the year, this approach was initially and provocatively pitted against the traditional teaching of medicine, in which the key elements of knowing for clinical purposes are understanding of basic pathophysiologic mechanisms of disease coupled with clinical experience. This paper reviews the origins, aspirations, philosophical limitations, and practical challenges of evidence-based medicine. DISCUSSION: EBM has long since evolved beyond its initial (mis)conception, that EBM might replace traditional medicine. EBM is now attempting to augment rather than replace individual clinical experience and understanding of basic disease mechanisms. EBM must continue to evolve, however, to address a number of issues including scientific underpinnings, moral stance and consequences, and practical matters of dissemination and application. For example, accelerating the transfer of research findings into clinical practice is often based on incomplete evidence from selected groups of people, who experience a marginal benefit from an expensive technology, raising issues of the generalizability of the findings, and increasing problems with how many and who can afford the new innovations in care. SUMMARY: Advocates of evidence-based medicine want clinicians and consumers to pay attention to the best findings from health care research that are both valid and ready for clinical application. Much remains to be done to reach this goal

Distances from the Correlation between Galaxy Luminosities and Rotation Rates

A large luminosity--linewidth template sample is now available, improved absorption corrections have been derived, and there are a statistically significant number of galaxies with well determined distances to supply the zero point. A revised estimate of the Hubble Constant is H_0=77 +-4 km/s/Mpc where the error is the 95% probability statistical error. Systematic uncertainties are potentially twice as large.Comment: 21 pages, 9 figures. Invited chapter for the book `Post-Hipparcos Cosmic Candles', Eds. F. Caputo and A. Heck (Kluwer Academic Publishers, Dordrecht

Developing search strategies for clinical practice guidelines in SUMSearch and Google Scholar and assessing their retrieval performance

<p>Abstract</p> <p>Background</p> <p>Information overload, increasing time constraints, and inappropriate search strategies complicate the detection of clinical practice guidelines (CPGs). The aim of this study was to provide clinicians with recommendations for search strategies to efficiently identify relevant CPGs in SUMSearch and Google Scholar.</p> <p>Methods</p> <p>We compared the retrieval efficiency (retrieval performance) of search strategies to identify CPGs in SUMSearch and Google Scholar. For this purpose, a two-term GLAD (GuideLine And Disease) strategy was developed, combining a defined CPG term with a specific disease term (MeSH term). We used three different CPG terms and nine MeSH terms for nine selected diseases to identify the most efficient GLAD strategy for each search engine. The retrievals for the nine diseases were pooled. To compare GLAD strategies, we used a manual review of all retrievals as a reference standard. The CPGs detected had to fulfil predefined criteria, e.g., the inclusion of therapeutic recommendations. Retrieval performance was evaluated by calculating so-called diagnostic parameters (sensitivity, specificity, and "Number Needed to Read" [NNR]) for search strategies.</p> <p>Results</p> <p>The search yielded a total of 2830 retrievals; 987 (34.9%) in Google Scholar and 1843 (65.1%) in SUMSearch. Altogether, we found 119 unique and relevant guidelines for nine diseases (reference standard). Overall, the GLAD strategies showed a better retrieval performance in SUMSearch than in Google Scholar. The performance pattern between search engines was similar: search strategies including the term "guideline" yielded the highest sensitivity (SUMSearch: 81.5%; Google Scholar: 31.9%), and search strategies including the term "practice guideline" yielded the highest specificity (SUMSearch: 89.5%; Google Scholar: 95.7%), and the lowest NNR (SUMSearch: 7.0; Google Scholar: 9.3).</p> <p>Conclusion</p> <p>SUMSearch is a useful tool to swiftly gain an overview of available CPGs. Its retrieval performance is superior to that of Google Scholar, where a search is more time consuming, as substantially more retrievals have to be reviewed to detect one relevant CPG. In both search engines, the CPG term "guideline" should be used to obtain a comprehensive overview of CPGs, and the term "practice guideline" should be used if a less time consuming approach for the detection of CPGs is desired.</p