Interpreting systematic reviews: are we ready to make our own conclusions? A cross-sectional study

<p>Abstract</p> <p>Background</p> <p>Independent evaluation of clinical evidence is advocated in evidence-based medicine (EBM). However, authors' conclusions are often appealing for readers who look for quick messages. We assessed how well a group of Malaysian hospital practitioners and medical students derived their own conclusions from systematic reviews (SRs) and to what extent these were influenced by their prior beliefs and the direction of the study results.</p> <p>Methods</p> <p>We conducted two cross-sectional studies: one with hospital practitioners (<it>n </it>= 150) attending an EBM course in June 2008 in a tertiary hospital and one with final-year medical students (<it>n </it>= 35) in November 2008. We showed our participants four Cochrane SR abstracts without the authors' conclusions. For each article, the participants chose a conclusion from among six options comprising different combinations of the direction of effect and the strength of the evidence. We predetermined the single option that best reflected the actual authors' conclusions and labelled this as our best conclusion. We compared the participants' choices with our predetermined best conclusions. Two chosen reviews demonstrated that the intervention was beneficial ("positive"), and two others did not ("negative"). We also asked the participants their prior beliefs about the intervention.</p> <p>Results</p> <p>Overall, 60.3% correctly identified the direction of effect, and 30.1% chose the best conclusions, having identified both the direction of effect and the strength of evidence. More students (48.2%) than practitioners (22.2%) chose the best conclusions (<it>P </it>< 0.001). Fewer than one-half (47%) correctly identified the direction of effect against their prior beliefs. "Positive" SRs were more likely than "negative" SRs to change the participants' beliefs about the effect of the intervention (relative risk (RR) 1.8, 95% confidence interval 1.3 to 2.6) and "convert" those who were previously unsure by making them choose the appropriate direction of effect (RR 1.9, 95% confidence interval 1.3 to 2.8).</p> <p>Conclusions</p> <p>The majority of our participants could not generate appropriate conclusions from SRs independently. Judicious direction from the authors' conclusions still appears crucial to guiding our health care practitioners in identifying appropriate messages from research. Authors, editors and reviewers should ensure that the conclusions of a paper accurately reflect the results. Similar studies should be conducted in other settings where awareness and application of EBM are different.</p> <p>Please see Commentary: <url>http://www.biomedcentral.com/1741-7015/9/31/</url>.</p

Current sample size conventions: Flaws, harms, and alternatives

<p>Abstract</p> <p>Background</p> <p>The belief remains widespread that medical research studies must have statistical power of at least 80% in order to be scientifically sound, and peer reviewers often question whether power is high enough.</p> <p>Discussion</p> <p>This requirement and the methods for meeting it have severe flaws. Notably, the true nature of how sample size influences a study's projected scientific or practical value precludes any meaningful blanket designation of <80% power as "inadequate". In addition, standard calculations are inherently unreliable, and focusing only on power neglects a completed study's most important results: estimates and confidence intervals. Current conventions harm the research process in many ways: promoting misinterpretation of completed studies, eroding scientific integrity, giving reviewers arbitrary power, inhibiting innovation, perverting ethical standards, wasting effort, and wasting money. Medical research would benefit from alternative approaches, including established <it>value of information </it>methods, simple choices based on cost or feasibility that have recently been justified, sensitivity analyses that examine a meaningful array of possible findings, and following previous analogous studies. To promote more rational approaches, research training should cover the issues presented here, peer reviewers should be extremely careful before raising issues of "inadequate" sample size, and reports of completed studies should not discuss power.</p> <p>Summary</p> <p>Common conventions and expectations concerning sample size are deeply flawed, cause serious harm to the research process, and should be replaced by more rational alternatives.</p

A philosophical analysis of the evidence-based medicine debate

BACKGROUND: The term "evidence-based medicine" (or EBM) was introduced about ten years ago, and there has been considerable debate about the value of EBM. However, this debate has sometimes been obscured by a lack of conceptual clarity concerning the nature and status of EBM. DISCUSSION: First, we note that EBM proponents have obscured the current debate by defining EBM in an overly broad, indeed almost vacuous, manner; we offer a clearer account of EBM and its relation to the alternative approaches to medicine. Second, while EBM proponents commonly cite the philosophical work of Thomas Kuhn and claim that EBM is a Kuhnian 'paradigm shift,' we argue that such claims are seriously mistaken and unduly polarize the EBM debate. Third, we suggest that it is much more fruitful to understand the relationship between EBM and its alternatives in light of a different philosophical metaphor: W.V. Quine's metaphor of the web of belief. Seen in this way, we argue that EBM is an approach to medical practice that is indeed importantly different from the alternatives. SUMMARY: We can have a more productive debate about the value of EBM by being clearer about the nature of EBM and its relationship to alternative approaches to medicine

The association between histamine 2 receptor antagonist use and Clostridium difficile infection: a systematic review and meta-analysis.

Background Clostridium difficile infection (CDI) is a major health problem. Epidemiological evidence suggests that there is an association between acid suppression therapy and development of CDI. Purpose We sought to systematically review the literature that examined the association between histamine 2 receptor antagonists (H2RAs) and CDI. Data source We searched Medline, Current Contents, Embase, ISI Web of Science and Elsevier Scopus from 1990 to 2012 for all analytical studies that examined the association between H2RAs and CDI. Study selection Two authors independently reviewed the studies for eligibility. Data extraction Data about studies characteristics, adjusted effect estimates and quality were extracted. Data synthesis Thirty-five observations from 33 eligible studies that included 201834 participants were analyzed. Studies were performed in 6 countries and nine of them were multicenter. Most studies did not specify the type or duration of H2RAs therapy. The pooled effect estimate was 1.44, 95% CI (1.22–1.7), I2 = 70.5%. This association was consistent across different subgroups (by study design and country) and there was no evidence of publication bias. The pooled effect estimate for high quality studies was 1.39 (1.15–1.68), I2 = 72.3%. Meta-regression analysis of 10 study-level variables did not identify sources of heterogeneity. In a speculative analysis, the number needed to harm (NNH) with H2RAs at 14 days after hospital admission in patients receiving antibiotics or not was 58, 95% CI (37, 115) and 425, 95% CI (267, 848), respectively. For the general population, the NNH at 1 year was 4549, 95% CI (2860, 9097). Conclusion In this rigorous systematic review and meta-analysis, we observed an association between H2RAs and CDI. The absolute risk of CDI associated with H2RAs is highest in hospitalized patients receiving antibiotics

Relationships between Specific Airway Resistance and Forced Expiratory Flows in Asthmatic Children

. The first aim was to assess the relationships between forced expiratory flows and sRaw in a large group of asthmatic children in a transversal study. We then performed a longitudinal study in order to determine whether sRaw of preschool children could predict subsequent impairment of forced expiratory flows at school age.Pulmonary function tests (sRaw and forced expiratory flows) of 2193 asthmatic children were selected for a transversal analysis, while 365 children were retrospectively selected for longitudinal assessment from preschool to school age. (% predicted) (−0.09, 95% CI, −0.20 to 0). and could be used in preschool children to predict subsequent mild airflow limitation

Norwegian Physicians' Knowledge of and Opinions about Evidence-Based Medicine: Cross-Sectional Study

Objective: To answer five research questions: Do Norwegian physicians know about the three important aspects of EBM? Do they use EBM methods in their clinical practice? What are their attitudes towards EBM? Has EBM in their opinion changed medical practice during the last 10 years? Do they use EBM based information sources? Design: Cross sectional survey in 2006. Setting: Norway. Participants: 966 doctors who responded to a questionnaire (70% response rate). Results: In total 87% of the physicians mentioned the use of randomised clinical trials as a key aspect of EBM, while 53% of them mentioned use of clinical expertise and only 19% patients' values. 40% of the respondents reported that their practice had always been evidence-based. Many respondents experienced difficulties in using EBM principles in their clinical practice because of lack of time and difficulties in searching EBM based literature. 80% agreed that EBM helps physicians towards better practice and 52% that it improves patients' health. As reasons for changes in medical practice 86% of respondents mentioned medical progress, but only 39% EBM. Conclusions: The results of the study indicate that Norwegian physicians have a limited knowledge of the key aspects of EBM but a positive attitude towards the concept. They had limited experience in the practice of EBM and were rather indifferent to the impact of EBM on medical practice. For solving a patient problem, physicians would rather consult a colleague than searching evidence based resources such as the Cochrane Library

Core components for effective infection prevention and control programmes: new WHO evidence-based recommendations

Abstract Health care-associated infections (HAI) are a major public health problem with a significant impact on morbidity, mortality and quality of life. They represent also an important economic burden to health systems worldwide. However, a large proportion of HAI are preventable through effective infection prevention and control (IPC) measures. Improvements in IPC at the national and facility level are critical for the successful containment of antimicrobial resistance and the prevention of HAI, including outbreaks of highly transmissible diseases through high quality care within the context of universal health coverage. Given the limited availability of IPC evidence-based guidance and standards, the World Health Organization (WHO) decided to prioritize the development of global recommendations on the core components of effective IPC programmes both at the national and acute health care facility level, based on systematic literature reviews and expert consensus. The aim of the guideline development process was to identify the evidence and evaluate its quality, consider patient values and preferences, resource implications, and the feasibility and acceptability of the recommendations. As a result, 11 recommendations and three good practice statements are presented here, including a summary of the supporting evidence, and form the substance of a new WHO IPC guideline

What kind of evidence is it that Evidence-Based Medicine advocates want health care providers and consumers to pay attention to?

BACKGROUND: In 1992, Evidence-Based Medicine advocates proclaimed a "new paradigm", in which evidence from health care research is the best basis for decisions for individual patients and health systems. Hailed in New York Times Magazine in 2001 as one of the most influential ideas of the year, this approach was initially and provocatively pitted against the traditional teaching of medicine, in which the key elements of knowing for clinical purposes are understanding of basic pathophysiologic mechanisms of disease coupled with clinical experience. This paper reviews the origins, aspirations, philosophical limitations, and practical challenges of evidence-based medicine. DISCUSSION: EBM has long since evolved beyond its initial (mis)conception, that EBM might replace traditional medicine. EBM is now attempting to augment rather than replace individual clinical experience and understanding of basic disease mechanisms. EBM must continue to evolve, however, to address a number of issues including scientific underpinnings, moral stance and consequences, and practical matters of dissemination and application. For example, accelerating the transfer of research findings into clinical practice is often based on incomplete evidence from selected groups of people, who experience a marginal benefit from an expensive technology, raising issues of the generalizability of the findings, and increasing problems with how many and who can afford the new innovations in care. SUMMARY: Advocates of evidence-based medicine want clinicians and consumers to pay attention to the best findings from health care research that are both valid and ready for clinical application. Much remains to be done to reach this goal

Multivariable risk prediction can greatly enhance the statistical power of clinical trial subgroup analysis

BACKGROUND: When subgroup analyses of a positive clinical trial are unrevealing, such findings are commonly used to argue that the treatment's benefits apply to the entire study population; however, such analyses are often limited by poor statistical power. Multivariable risk-stratified analysis has been proposed as an important advance in investigating heterogeneity in treatment benefits, yet no one has conducted a systematic statistical examination of circumstances influencing the relative merits of this approach vs. conventional subgroup analysis. METHODS: Using simulated clinical trials in which the probability of outcomes in individual patients was stochastically determined by the presence of risk factors and the effects of treatment, we examined the relative merits of a conventional vs. a "risk-stratified" subgroup analysis under a variety of circumstances in which there is a small amount of uniformly distributed treatment-related harm. The statistical power to detect treatment-effect heterogeneity was calculated for risk-stratified and conventional subgroup analysis while varying: 1) the number, prevalence and odds ratios of individual risk factors for risk in the absence of treatment, 2) the predictiveness of the multivariable risk model (including the accuracy of its weights), 3) the degree of treatment-related harm, and 5) the average untreated risk of the study population. RESULTS: Conventional subgroup analysis (in which single patient attributes are evaluated "one-at-a-time") had at best moderate statistical power (30% to 45%) to detect variation in a treatment's net relative risk reduction resulting from treatment-related harm, even under optimal circumstances (overall statistical power of the study was good and treatment-effect heterogeneity was evaluated across a major risk factor [OR = 3]). In some instances a multi-variable risk-stratified approach also had low to moderate statistical power (especially when the multivariable risk prediction tool had low discrimination). However, a multivariable risk-stratified approach can have excellent statistical power to detect heterogeneity in net treatment benefit under a wide variety of circumstances, instances under which conventional subgroup analysis has poor statistical power. CONCLUSION: These results suggest that under many likely scenarios, a multivariable risk-stratified approach will have substantially greater statistical power than conventional subgroup analysis for detecting heterogeneity in treatment benefits and safety related to previously unidentified treatment-related harm. Subgroup analyses must always be well-justified and interpreted with care, and conventional subgroup analyses can be useful under some circumstances; however, clinical trial reporting should include a multivariable risk-stratified analysis when an adequate externally-developed risk prediction tool is available