From the Trenches: A Cross-Sectional Study Applying the GRADE Tool in Systematic Reviews of Healthcare Interventions

Background: GRADE was developed to address shortcomings of tools to rate the quality of a body of evidence. While much has been published about GRADE, there are few empirical and systematic evaluations. Objective: To assess GRADE for systematic reviews (SRs) in terms of inter-rater agreement and identify areas of uncertainty. Design: Cross-sectional, descriptive study. Methods: We applied GRADE to three SRs (n = 48, 66, and 75 studies, respectively) with 29 comparisons and 12 outcomes overall. Two reviewers graded evidence independently for outcomes deemed clinically important a priori. Inter-rater reliability was assessed using kappas for four main domains (risk of bias, consistency, directness, and precision) and overall quality of evidence. Results: For the first review, reliability was: k = 0.41 for risk of bias; 0.84 consistency; 0.18 precision; and 0.44 overall quality. Kappa could not be calculated for directness as one rater assessed all items as direct; assessors agreed in 41 % of cases. For the second review reliability was: 0.37 consistency and 0.19 precision. Kappa could not be assessed for other items; assessors agreed in 33 % of cases for risk of bias; 100 % directness; and 58 % overall quality. For the third review, reliability was: 0.06 risk of bias; 0.79 consistency; 0.21 precision; and 0.18 overall quality. Assessors agreed in 100 % of cases for directness. Precision created the most uncertainty due to difficulties in identifying ‘‘optimal’ ’ information size and ‘‘clinica

Cautionary tales in the interpretation of clinical studies involving older persons

The care of patients 65 years or older presents a challenge for evidence-based medicine. Such patients are underrepresented in clinical trials, are more vulnerable to treatment-induced harm, and often are unable to fully participate in treatment decisions. We outline several cautionary themes in the interpretation of clinical studies of therapeutic interventions involving older persons as they apply to processes of everyday clinical decision making. In particular, we focus on issues of study design and quality of evidence, choice of outcome measures, missing outcome data, assessment of potential harm, quantifying treatment effects in individual patients (and adjusting these for effect modifiers and reduced life expectancy), eliciting patient values and preferences, prioritizing therapeutic goals and selection of treatments, and assisting patients in adhering to agreed therapeutic regimens. ©2010 American Medical Association. All rights reserved

Interventions for behaviour change and self-management in stroke secondary prevention: protocol for an overview of reviews

Abstract Background Stroke secondary prevention guidelines recommend medication prescription and adherence, active education and behavioural counselling regarding lifestyle risk factors. To impact on recurrent vascular events, positive behaviour/s must be adopted and sustained as a lifestyle choice, requiring theoretically informed behaviour change and self-management interventions. A growing number of systematic reviews have addressed complex interventions in stroke secondary prevention. Differing terminology, inclusion criteria and overlap of studies between reviews makes the mechanism/s that affect positive change difficult to identify or replicate clinically. Adopting a two-phase approach, this overview will firstly comprehensively summarise systematic reviews in this area and secondly identify and synthesise primary studies in these reviews which provide person-centred, theoretically informed interventions for stroke secondary prevention. Methods An overview of reviews will be conducted using a systematic search strategy across the Cochrane Database of Systematic Reviews, PubMed and Epistomonikas. Inclusion criteria: systematic reviews where the population comprises individuals post-stroke or TIA and where data relating to person-centred risk reduction are synthesised for evidence of efficacy when compared to standard care or no intervention. Primary outcomes of interest include mortality, recurrent stroke and other cardiovascular events. In phase 1, two reviewers will independently (1) assess the eligibility of identified reviews for inclusion; (2) rate the quality of included reviews using the ROBIS tool; (3) identify unique primary studies and overlap between reviews; (4) summarise the published evidence supporting person-centred behavioural change and self-management interventions in stroke secondary prevention and (5) identify evidence gaps in this field. In phase 2, two independent reviewers will (1) examine person-centred, primary studies in each review using the Template for Intervention Description and Replication (TIDieR checklist), itemising, where present, theoretical frameworks underpinning interventions; (2) group studies employing theoretically informed interventions by the intervention delivered and by the outcomes reported (3) apply GRADE quality of evidence for each intervention by outcome/s identified from theoretically informed primary studies. Disagreement between reviewers at each process stage will be discussed and a third reviewer consulted. Discussion This overview will comprehensively bring together the best available evidence supporting person-centred, stroke secondary prevention strategies in an accessible format, identifying current knowledge gaps

Impact of quality of evidence on the strength of recommendations: an empirical study

<p>Abstract</p> <p>Background</p> <p>Evidence is necessary but not sufficient for decision-making, such as making recommendations by clinical practice guideline panels. However, the fundamental premise of evidence-based medicine (EBM) rests on the assumed link between the quality of evidence and "truth" and/or correctness in making guideline recommendations. If this assumption is accurate, then the quality of evidence ought to play a key role in making guideline recommendations. Surprisingly, and despite the widespread penetration of EBM in health care, there has been no empirical research to date investigating the impact of quality of evidence on the strength of recommendations made by guidelines panels.</p> <p>Methods</p> <p>The American Association of Blood Banking (AABB) has recently convened a 12 member panel to develop clinical practice guidelines (CPG) for the use of fresh-frozen plasma (FFP) for 6 different clinical indications. The panel was instructed that 4 factors should play a role in making recommendation: quality of evidence, uncertainty about the balance between desirable (benefits) and undesirable effects (harms), uncertainty or variability in values and preferences, and uncertainty about whether the intervention represents a wise use of resources (costs). Each member of the panel was asked to make his/her final judgments on the strength of recommendation and the overall quality of the body of evidence. "Voting" was anonymous and was based on the use of GRADE (Grading quality of evidence and strength of recommendations) system, which clearly distinguishes between quality of evidence and strength of recommendations.</p> <p>Results</p> <p>Despite the fact that many factors play role in formulating CPG recommendations, we show that when the quality of evidence is higher, the probability of making a strong recommendation for or against an intervention dramatically increases. Probability of making strong recommendation was 62% when evidence is "moderate", while it was only 23% and 13% when evidence was "low" or "very low", respectively.</p> <p>Conclusion</p> <p>We report the first empirical evaluation of the relationship between quality of evidence pertinent to a clinical question and strength of the corresponding guideline recommendations. Understanding the relationship between quality of evidence and probability of making (strong) recommendation has profound implications for the science of quality measurement in health care.</p

Methodological challenges in appraising evidence on diagnostic testing for WHO guidelines on hepatitis B and hepatitis C virus infection

Abstract Linking persons with hepatitis B (HBV) and hepatitis C (HCV) infection with appropriate prevention and treatment requires that they first be diagnosed. The World Health Organization (WHO) has developed its first guidelines on testing for chronic HBV and HCV infection, using a framework based on methods from the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) Working Group for the formulation of recommendations, including determining the strength of recommendations and quality of evidence. Recommendations were formulated based on the overall quality of the evidence, in addition to other considerations, including the balance between benefits and harms, values and preferences, feasibility and resource implications. This article summarizes methodological challenges and additional considerations encountered in applying these procedures to diagnostic testing for viral hepatitis, and strategies to address these. Direct evidence on the effects of tests and test strategies on clinical outcomes was not available. Given the availability of effective treatments for HBV and HCV that are generally acceptable to patients, the Guidelines Development Group (GDG) considered diagnostic accuracy a reasonable surrogate for clinical outcomes. In order to increase the number of patients identified with chronic HBV and HCV infection who could benefit from treatments, the GDG determined that tests and testing strategies associated with slightly lower diagnostic accuracy could be recommended when associated with lower costs; increased testing access, uptake, and linkage to care; greater feasibility; or if preferred by patients