Quality and methods of developing practice guidelines

BACKGROUND: It is not known whether there are differences in the quality and recommendations between evidence-based (EB) and consensus-based (CB) guidelines. We used breast cancer guidelines as a case study to assess for these differences. METHODS: Five different instruments to evaluate the quality of guidelines were identified by a literature search. We also searched MEDLINE and the Internet to locate 8 breast cancer guidelines. These guidelines were classified in three categories: evidence based, consensus based and consensus based with no explicit consideration of evidence (CB-EB). Each guideline was evaluated by three of the authors using each of the instruments. For each guideline we assessed the agreement among 14 decision points which were selected from the NCCN (National Cancer Comprehensive Network) guidelines algorithm. For each decision point we recorded the level of the quality of the information used to support it. A regression analysis was performed to assess if the percentage of high quality evidence used in the guidelines development was related to the overall quality of the guidelines. RESULTS: Three guidelines were classified as EB, three as CB-EB and two as CB. The EB guidelines scored better than CB, with the CB-EB scoring in the middle among all instruments for guidelines quality assessment. No major disagreement in recommendations was detected among the guidelines regardless of the method used for development, but the EB guidelines had a better agreement with the benchmark guideline for any decision point. When the source of evidence used to support decision were of high quality, we found a higher level of full agreement among the guidelines' recommendations. Up to 94% of variation in the quality score among guidelines could be explained by the quality of evidence used for guidelines development. CONCLUSION: EB guidelines have a better quality than CB guidelines and CB-EB guidelines. Explicit use of high quality evidence can lead to a better agreement among recommendations. However, no major disagreement among guidelines was noted regardless of the method for their development

Guidelines on Chemotherapy in Advanced Stage Gynecological Malignancies: An Evaluation of 224 Professional Societies and Organizations

BACKGROUND: Clinical practice guidelines are important for guiding practice, but it is unclear if they are commensurate with the available evidence. METHODS: We examined guidelines produced by cancer and gynecological societies and organizations and evaluated their coverage of and stance towards chemotherapy for advanced stage disease among 4 gynecological malignancies (breast, ovarian, cervical, endometrial cancer) where the evidence for the use of chemotherapy is very different (substantial and conclusive for breast and ovarian cancer, limited and suggesting no major benefit for cervical and endometrial cancer). Eligible societies and organizations were identified through systematic internet searches (last update June 2009). Pertinent websites were scrutinized for presence of clinical practice guidelines, and relative guidelines were analyzed. RESULTS: Among 224 identified eligible societies and organizations, 69 (31%) provided any sort of guidelines, while recommendations for chemotherapy on advanced stage gynecological malignancies were available in 20 of them. Only 14 had developed their own guideline, and only 5 had developed guidelines for all 4 malignancies. Use of levels of evidence and grades of recommendations, and aspects of the production, implementation, and timeliness of the guidelines did not differ significantly across malignancies. Guidelines on breast and ovarian cancer utilized significantly more randomized trials and meta-analyses. Guidelines differed across malignancies on their coverage of disease-free survival (p = 0.033), response rates (p = 0.024), symptoms relief (p = 0.005), quality of life (p = 0.001) and toxicity (p = 0.039), with breast and ovarian cancer guidelines typically covering more frequently these outcomes. All guidelines explicitly or implicitly endorsed the use of chemotherapy. CONCLUSIONS: Clinical practice guidelines are provided by the minority of professional societies and organizations. Available guidelines tend to recommend chemotherapy even for diseases where the effect of chemotherapy is controversial and recommendations are based on scant evidence

Maternal and perinatal guideline development in hospitals in South East Asia: the experience of the SEA-ORCHID project

<p>Abstract</p> <p>Background</p> <p>Clinical practice guidelines (CPGs) are commonly used to support practitioners to improve practice. However many studies have raised concerns about guideline quality. The reasons why guidelines are not developed following the established development methods are not clear.</p> <p>The SEA-ORCHID project aims to increase the generation and use of locally relevant research and improve clinical practice in maternal and perinatal care in four countries in South East Asia. Baseline data highlighted that development of evidence-based CPGs according to recommended processes was very rare in the SEA-ORCHID hospitals. The project investigators suggested that there were aspects of the recommended development process that made it very difficult in the participating hospitals.</p> <p>We therefore aimed to explore the experience of guideline development and particularly the enablers of and barriers to developing evidence-based guidelines in the nine hospitals in South East Asia participating in the SEA-ORCHID project, so as to better understand how evidence-based guideline development could be facilitated in these settings.</p> <p>Methods</p> <p>Semi-structured, face-to-face interviews were undertaken with senior and junior healthcare providers (nurses, midwives, doctors) from the maternal and neonatal services at each of the nine participating hospitals. Interviews were audio-recorded, transcribed and a thematic analysis undertaken.</p> <p>Results</p> <p>Seventy-five individual, 25 pair and eleven group interviews were conducted. Participants clearly valued evidence-based guidelines. However they also identified several major barriers to guideline development including time, lack of awareness of process, difficulties searching for evidence and arranging guideline development group meetings, issues with achieving multi-disciplinarity and consumer involvement. They also highlighted the central importance of keeping guidelines up-to-date.</p> <p>Conclusion</p> <p>Healthcare providers in the SEA-ORCHID hospitals face a series of barriers to developing evidence-based guidelines. At present, in many hospitals, several of these barriers are insurmountable, and as a result, rigorous, evidence-based guidelines are not being developed. Given the acknowledged benefits of evidence-based guidelines, perhaps a new approach to supporting their development in these contexts is needed.</p

Improving the use of research evidence in guideline development: 16. Evaluation

BACKGROUND: The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the last of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this. OBJECTIVES: We reviewed the literature on evaluating guidelines and recommendations, including their quality, whether they are likely to be up-to-date, and their implementation. We also considered the role of guideline developers in undertaking evaluations that are needed to inform recommendations. METHODS: We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments. KEY QUESTIONS AND ANSWERS: Our answers to these questions were informed by a review of instruments for evaluating guidelines, several studies of the need for updating guidelines, discussions of the pros and cons of different research designs for evaluating the implementation of guidelines, and consideration of the use of uncertainties identified in systematic reviews to set research priorities. How should the quality of guidelines or recommendations be appraised? • WHO should put into place processes to ensure that both internal and external review of guidelines is undertaken routinely. • A checklist, such as the AGREE instrument, should be used. • The checklist should be adapted and tested to ensure that it is suitable to the broad range of recommendations that WHO produces, including public health and health policy recommendations, and that it includes questions about equity and other items that are particularly important for WHO guidelines. When should guidelines or recommendations be updated? • Processes should be put into place to ensure that guidelines are monitored routinely to determine if they are in need of updating. • People who are familiar with the topic, such as Cochrane review groups, should do focused, routine searches for new research that would require revision of the guideline. • Periodic review of guidelines by experts not involved in developing the guidelines should also be considered. • Consideration should be given to establishing guideline panels that are ongoing, to facilitate routine updating, with members serving fixed periods with a rotating membership. How should the impact of guidelines or recommendations be evaluated? • WHO headquarters and regional offices should support member states and those responsible for policy decisions and implementation to evaluate the impact of their decisions and actions by providing advice regarding impact assessment, practical support and coordination of efforts. • Before-after evaluations should be used cautiously and when there are important uncertainties regarding the effects of a policy or its implementation, randomised evaluations should be used when possible. What responsibility should WHO take for ensuring that important uncertainties are addressed by future research when the evidence needed to inform recommendations is lacking? • Guideline panels should routinely identify important uncertainties and research priorities. This source of potential priorities for research should be used systematically to inform priority-setting processes for global research

Clinical practice guidelines for the foot and ankle in rheumatoid arthritis: a critical appraisal

Background: Clinical practice guidelines are recommendations systematically developed to assist clinical decision-making and inform healthcare. In current rheumatoid arthritis (RA) guidelines, management of the foot and ankle is under-represented and the quality of recommendation is uncertain. This study aimed to identify and critically appraise clinical practice guidelines for foot and ankle management in RA. Methods: Guidelines were identified electronically and through hand searching. Search terms 'rheumatoid arthritis', 'clinical practice guidelines' and related synonyms were used. Critical appraisal and quality rating were conducted using the Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument. Results: Twenty-four guidelines were included. Five guidelines were high quality and recommended for use. Five high quality and seven low quality guidelines were recommended for use with modifications. Seven guidelines were low quality and not recommended for use. Five early and twelve established RA guidelines were recommended for use. Only two guidelines were foot and ankle specific. Five recommendation domains were identified in both early and established RA guidelines. These were multidisciplinary team care, foot healthcare access, foot health assessment/review, orthoses/insoles/splints, and therapeutic footwear. Established RA guidelines also had an 'other foot care treatments' domain. Conclusions: Foot and ankle management for RA features in many clinical practice guidelines recommended for use. Unfortunately, supporting evidence in the guidelines is low quality. Agreement levels are predominantly 'expert opinion' or 'good clinical practice'. More research investigating foot and ankle management for RA is needed prior to inclusion in clinical practice guidelines

Evidence-informed health policy 1 – Synthesis of findings from a multi-method study of organizations that support the use of research evidence

Background: Organizations have been established in many countries and internationally to support the use of research evidence by producing clinical practice guidelines, undertaking health technology assessments, and/or directly supporting the use of research evidence in developing health policy on an international, national, and state or provincial level. Learning from these organizations can reduce the need to 'reinvent the wheel' and inform decisions about how best to organize support for such organizations, particularly in low- and middle-income countries (LMICs). Methods: We undertook a multi-method study in three phases – a survey, interviews, and case descriptions that drew on site visits – and in each of the second and third phases we focused on a purposive sample of those involved in the previous phase. We used the seven main recommendations that emerged from the advice offered in the interviews to organize much of the synthesis of findings across phases and methods. We used a constant comparative method to identify themes from across phases and methods. Results: Seven recommendations emerged for those involved in establishing or leading organizations that support the use of research evidence in developing health policy: 1) collaborate with other organizations; 2) establish strong links with policymakers and involve stakeholders in the work; 3) be independent and manage conflicts of interest among those involved in the work; 4) build capacity among those working in the organization; 5) use good methods and be transparent in the work; 6) start small, have a clear audience and scope, and address important questions; and 7) be attentive to implementation considerations, even if implementation is not a remit. Four recommendations emerged for the World Health Organization (WHO) and other international organizations and networks: 1) support collaborations among organizations; 2) support local adaptation efforts; 3) mobilize support; and 4) create global public goods. Conclusion: This synthesis of findings from a multi-method study, along with the more detailed findings from each of the three phases of the study (which are reported in the three following articles in the series), provide a strong basis on which researchers, policymakers, international organizations (and networks) like WHO can respond to the growing chorus of voices calling for efforts to support the use of research evidence in developing health policy

Development of evidence-based clinical practice guidelines (CPGs): comparing approaches

<p>Abstract</p> <p>Background</p> <p>While the potential of clinical practice guidelines (CPGs) to support implementation of evidence has been demonstrated, it is not currently being achieved. CPGs are both poorly developed and ineffectively implemented. To improve clinical practice and health outcomes, both well-developed CPGs and effective methods of CPG implementation are needed. We sought to establish whether there is agreement on the fundamental characteristics of an evidence-based CPG development process and to explore whether the level of guidance provided in CPG development handbooks is sufficient for people using these handbooks to be able to apply it.</p> <p>Methods</p> <p>CPG development handbooks were identified through a broad search of published and grey literature. Documents published in English produced by national or international organisations purporting to support development of evidence-based CPGs were included. A list of 14 key elements of a CPG development process was developed. Two authors read each handbook. For each handbook a judgement was made as to how it addressed each element; assigned as: 'mentioned and clear guidance provided', 'mentioned but limited practical detail provided ', or 'not mentioned'.</p> <p>Results</p> <p>Six CPG development handbooks were included. These were produced by the Council of Europe, the National Health and Medical Research Council of Australia, the National Institute for Health and Clinical Excellence in the UK, the New Zealand Guidelines Group, the Scottish Intercollegiate Guideline Network, and the World Health Organization (WHO).</p> <p>There was strong concordance between the handbooks on the key elements of an evidence-based CPG development process. All six of the handbooks require and provide guidance on establishment of a multidisciplinary guideline development group, involvement of consumers, identification of clinical questions or problems, systematic searches for and appraisal of research evidence, a process for drafting recommendations, consultation with others beyond the guideline development group, and ongoing review and updating of the CPG.</p> <p>Conclusion</p> <p>The key elements of an evidence-based CPG development process are addressed with strong concordance by existing CPG development handbooks. Further research is required to determine why these key elements are often not addressed by CPG developers.</p

Reporting of conflicts of interest in guidelines of preventive and therapeutic interventions

BACKGROUND: Guidelines published in major medical journals are very influential in determining clinical practice. It would be essential to evaluate whether conflicts of interests are disclosed in these publications. We evaluated the reporting of conflicts of interest and the factors that may affect such disclosure in a sample of 191 guidelines on therapeutic and/or preventive measures published in 6 major clinical journals (Annals of Internal Medicine, BMJ, JAMA, Lancet, New England Journal of Medicine, Pediatrics) in 1979, 1984, 1989, 1994 and 1999. RESULTS: Only 7 guidelines (3.7%) mentioned conflicts of interest and all were published in 1999 (17.5% (7/40) of guidelines published in 1999 alone). Reporting of conflicts of interest differed significantly by journal (p=0.026), availability of disclosure policy by the journal (p=0.043), source of funding (p < 0.001) and number of authors (p=0.004). In the entire database of 191 guidelines, a mere 18 authors disclosed a total of 24 potential conflicts of interest and most pertained to minor issues. CONCLUSIONS: Despite some recent improvement, reporting of conflicts of interest in clinical guidelines published in influential journals is largely neglected