Improving the use of research evidence in guideline development: 9. Grading evidence and recommendations

BACKGROUND: The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the ninth of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this. OBJECTIVES: We reviewed the literature on grading evidence and recommendations in guidelines. METHODS: We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct a full systematic review ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments. KEY QUESTIONS AND ANSWERS: Should WHO grade the quality of evidence and the strength of recommendations? • Users of recommendations need to know how much confidence they can place in the underlying evidence and the recommendations. The degree of confidence depends on a number of factors and requires complex judgments. These judgments should be made explicitly in WHO recommendations. A systematic and explicit approach to making judgments about the quality of evidence and the strength of recommendations can help to prevent errors, facilitate critical appraisal of these judgments, and can help to improve communication of this information. What criteria should be used to grade evidence and recommendations? • Both the quality of evidence and the strength of recommendations should be graded. The criteria used to grade the strength of recommendations should include the quality of the underlying evidence, but should not be limited to that. • The approach to grading should be one that has wide international support and is suitable for a wide range of different types of recommendations. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach, which is currently suggested in the Guidelines for WHO Guidelines, is being used by an increasing number of other organizations internationally. It should be used more consistently by WHO. Further developments of this approach should ensure its wide applicability. Should WHO use the same grading system for all of its recommendations? • Although there are arguments for and against using the same grading system across a wide range of different types of recommendations, WHO should use a uniform grading system to prevent confusion for developers and users of recommendations

Improving the use of research evidence in guideline development: 13. Applicability, transferability and adaptation

BACKGROUND: The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the thirteenth of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this. OBJECTIVES: We reviewed the literature on applicability, transferability, and adaptation of guidelines. METHODS: We searched five databases for existing systematic reviews and relevant primary methodological research. We reviewed the titles of all citations and retrieved abstracts and full text articles if the citations appeared relevant to the topic. We checked the reference lists of articles relevant to the questions and used snowballing as a technique to obtain additional information. We used the definition "coming from, concerning or belonging to at least two or all nations" for the term international. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments. KEY QUESTIONS AND ANSWERS: We did not identify systematic reviews addressing the key questions. We found individual studies and projects published in the peer reviewed literature and on the Internet. Should WHO develop international recommendations? • Resources for developing high quality recommendations are limited. Internationally developed recommendations can facilitate access to and pooling of resources, reduce unnecessary duplication, and involve international scientists. • Priority should be given to international health problems and problems that are important in low and middle-income countries, where these advantages are likely to be greatest. • Factors that influence the transferability of recommendations across different settings should be considered systematically and flagged, including modifying factors, important variation in needs, values, costs and the availability of resources. What should be done centrally and locally? • The preparation of systematic reviews and evidence profiles should be coordinated centrally, in collaboration with organizations that produce systematic reviews. Centrally developed evidence profiles should be adaptable to specific local circumstances. • Consideration should be given to models that involve central coordination with work being undertaken by centres located throughout the world. • While needs, availability of resources, costs, the presence of modifying factors and values need to be assessed locally, support for undertaking these assessments may be needed to make guidelines applicable. • WHO should provide local support for adapting and implementing recommendations by developing tools, building capacity, learning from international experience, and through international networks that support evidence-informed health policies, such as the Evidence-informed Policy Network (EVIPNet). How should recommendations be adapted? • WHO should provide detailed guidance for adaptation of international recommendations. • Local adaptation processes should be systematic and transparent, they should involve stakeholders, and they should report the key factors that influence decisions, including those flagged in international guidelines, and the reasons for any modifications that are made

Chapter 7: Grading a Body of Evidence on Diagnostic Tests

10.1007/s11606-012-2021-9Journal of General Internal Medicine27SUPPL.1S47-S55JGIM

The GRADE System for Rating Clinical Guidelines

Brian Kavanagh critiques the GRADE system of grading guidelines, arguing that even though it has evolved through the Evidence-Based Medicine movement, there is no evidence that GRADE itself is reliable

From the Trenches: A Cross-Sectional Study Applying the GRADE Tool in Systematic Reviews of Healthcare Interventions

Background: GRADE was developed to address shortcomings of tools to rate the quality of a body of evidence. While much has been published about GRADE, there are few empirical and systematic evaluations. Objective: To assess GRADE for systematic reviews (SRs) in terms of inter-rater agreement and identify areas of uncertainty. Design: Cross-sectional, descriptive study. Methods: We applied GRADE to three SRs (n = 48, 66, and 75 studies, respectively) with 29 comparisons and 12 outcomes overall. Two reviewers graded evidence independently for outcomes deemed clinically important a priori. Inter-rater reliability was assessed using kappas for four main domains (risk of bias, consistency, directness, and precision) and overall quality of evidence. Results: For the first review, reliability was: k = 0.41 for risk of bias; 0.84 consistency; 0.18 precision; and 0.44 overall quality. Kappa could not be calculated for directness as one rater assessed all items as direct; assessors agreed in 41 % of cases. For the second review reliability was: 0.37 consistency and 0.19 precision. Kappa could not be assessed for other items; assessors agreed in 33 % of cases for risk of bias; 100 % directness; and 58 % overall quality. For the third review, reliability was: 0.06 risk of bias; 0.79 consistency; 0.21 precision; and 0.18 overall quality. Assessors agreed in 100 % of cases for directness. Precision created the most uncertainty due to difficulties in identifying ‘‘optimal’ ’ information size and ‘‘clinica

Prognostic role of EGFR gene copy number and KRAS mutation in patients with locally advanced rectal cancer treated with preoperative chemoradiotherapy

Epidermal growth factor receptor (EGFR), evaluated by immunohistochemistry, has been shown to have prognostic significance in patients with colorectal cancer. Gene copy number (GCN) of EGFR and KRAS status predict response and outcome in patients treated with anti-EGFR therapy, but their prognostic significance in colorectal cancer patients is still unclear.We have retrospectively reviewed the baseline EGFR GCN, KRAS status and clinical outcome of 146 locally advanced rectal cancer (LARC) patients treated with preoperative chemoradiotherapy. Pathological response evaluated by Dworak's tumour regression grade (TRG), disease-free survival (DFS) and overall survival (OS) were analysed.Tumour regression grade 4 and TRG3-4 were achieved in 14.4 and 30.8\% of the patients respectively. Twenty-nine (19.9\%) and 33 patients (19.2\%) had an EGFR/nuclei ratio >2.9 and CEP7 polisomy >50\% respectively; 28 patients (19.2\%) had a KRAS mutation. Neither EGFR GCN nor KRAS status was statistically correlated to TRG. 5-year DFS and OS were 63.3 and 71.5\%, respectively, and no significant relation with EGFR GCN or KRAS status was found.Our data show that EGFR GCN and KRAS status are not prognostic factors in LARC treated with preoperative chemoradiation

Quality and Reporting of Diagnostic Accuracy Studies in TB, HIV and Malaria: Evaluation Using QUADAS and STARD Standards

BackgroundPoor methodological quality and reporting are known concerns with diagnostic accuracy studies. In 2003, the QUADAS tool and the STARD standards were published for evaluating the quality and improving the reporting of diagnostic studies, respectively. However, it is unclear whether these tools have been applied to diagnostic studies of infectious diseases. We performed a systematic review on the methodological and reporting quality of diagnostic studies in TB, malaria and HIV.MethodsWe identified diagnostic accuracy studies of commercial tests for TB, malaria and HIV through a systematic search of the literature using PubMed and EMBASE (2004–2006). Original studies that reported sensitivity and specificity data were included. Two reviewers independently extracted data on study characteristics and diagnostic accuracy, and used QUADAS and STARD to evaluate the quality of methods and reporting, respectively.FindingsNinety (38%) of 238 articles met inclusion criteria. All studies had design deficiencies. Study quality indicators that were met in less than 25% of the studies included adequate description of[...] and description of the team executing the test and management of indeterminate/outlier results (both 17%). The use of STARD was not explicitly mentioned in any study. Only 22% of 46 journals that published the studies included in this review required authors to use STARD

Phase II study of continuous daily sunitinib dosing in patients with previously treated advanced non-small cell lung cancer

Background:Sunitinib malate (SUTENT) has promising single-agent activity given on Schedule 4/2 (4 weeks on treatment followed by 2 weeks off treatment) in advanced non-small cell lung cancer (NSCLC).Methods:We examined the activity of sunitinib on a continuous daily dosing (CDD) schedule in an open-label, multicentre phase II study in patients with previously treated, advanced NSCLC. Patients ⩾18 years with stage IIIB/IV NSCLC after failure with platinum-based chemotherapy, received sunitinib 37.5 mg per day. The primary end point was objective response rate (ORR). Secondary end points included progression-free survival (PFS), overall survival (OS), 1-year survival rate, and safety.Results:Of 47 patients receiving sunitinib, one patient achieved a confirmed partial response (ORR 2.1% (95% confidence interval (CI) 0.1, 11.3)) and 11 (23.4%) had stable disease (SD) ⩾8 weeks. Five patients had SD>6 months. Median PFS was 11.9 weeks (95% CI 8.6, 14.1) and median OS was 37.1 weeks (95% CI 31.1, 69.7). The 1-year survival probability was 38.4% (95% CI 24.2, 52.5). Treatment was generally well tolerated.Conclusions:The safety profile and time-to-event analyses, albeit relatively low response rate of 2%, suggest single-agent sunitinib on a CDD schedule may be a potential therapeutic agent for patients with advanced, refractory NSCLC