Improving the use of research evidence in guideline development: 16. Evaluation

BACKGROUND: The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the last of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this. OBJECTIVES: We reviewed the literature on evaluating guidelines and recommendations, including their quality, whether they are likely to be up-to-date, and their implementation. We also considered the role of guideline developers in undertaking evaluations that are needed to inform recommendations. METHODS: We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments. KEY QUESTIONS AND ANSWERS: Our answers to these questions were informed by a review of instruments for evaluating guidelines, several studies of the need for updating guidelines, discussions of the pros and cons of different research designs for evaluating the implementation of guidelines, and consideration of the use of uncertainties identified in systematic reviews to set research priorities. How should the quality of guidelines or recommendations be appraised? • WHO should put into place processes to ensure that both internal and external review of guidelines is undertaken routinely. • A checklist, such as the AGREE instrument, should be used. • The checklist should be adapted and tested to ensure that it is suitable to the broad range of recommendations that WHO produces, including public health and health policy recommendations, and that it includes questions about equity and other items that are particularly important for WHO guidelines. When should guidelines or recommendations be updated? • Processes should be put into place to ensure that guidelines are monitored routinely to determine if they are in need of updating. • People who are familiar with the topic, such as Cochrane review groups, should do focused, routine searches for new research that would require revision of the guideline. • Periodic review of guidelines by experts not involved in developing the guidelines should also be considered. • Consideration should be given to establishing guideline panels that are ongoing, to facilitate routine updating, with members serving fixed periods with a rotating membership. How should the impact of guidelines or recommendations be evaluated? • WHO headquarters and regional offices should support member states and those responsible for policy decisions and implementation to evaluate the impact of their decisions and actions by providing advice regarding impact assessment, practical support and coordination of efforts. • Before-after evaluations should be used cautiously and when there are important uncertainties regarding the effects of a policy or its implementation, randomised evaluations should be used when possible. What responsibility should WHO take for ensuring that important uncertainties are addressed by future research when the evidence needed to inform recommendations is lacking? • Guideline panels should routinely identify important uncertainties and research priorities. This source of potential priorities for research should be used systematically to inform priority-setting processes for global research

Clinical practice guidelines for the foot and ankle in rheumatoid arthritis: a critical appraisal

Background: Clinical practice guidelines are recommendations systematically developed to assist clinical decision-making and inform healthcare. In current rheumatoid arthritis (RA) guidelines, management of the foot and ankle is under-represented and the quality of recommendation is uncertain. This study aimed to identify and critically appraise clinical practice guidelines for foot and ankle management in RA. Methods: Guidelines were identified electronically and through hand searching. Search terms 'rheumatoid arthritis', 'clinical practice guidelines' and related synonyms were used. Critical appraisal and quality rating were conducted using the Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument. Results: Twenty-four guidelines were included. Five guidelines were high quality and recommended for use. Five high quality and seven low quality guidelines were recommended for use with modifications. Seven guidelines were low quality and not recommended for use. Five early and twelve established RA guidelines were recommended for use. Only two guidelines were foot and ankle specific. Five recommendation domains were identified in both early and established RA guidelines. These were multidisciplinary team care, foot healthcare access, foot health assessment/review, orthoses/insoles/splints, and therapeutic footwear. Established RA guidelines also had an 'other foot care treatments' domain. Conclusions: Foot and ankle management for RA features in many clinical practice guidelines recommended for use. Unfortunately, supporting evidence in the guidelines is low quality. Agreement levels are predominantly 'expert opinion' or 'good clinical practice'. More research investigating foot and ankle management for RA is needed prior to inclusion in clinical practice guidelines

Improving the use of research evidence in guideline development: 11. Incorporating considerations of cost-effectiveness, affordability and resource implications

BACKGROUND: The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the 11(th )of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this. OBJECTIVES: We reviewed the literature on incorporating considerations of cost-effectiveness, affordability and resource implications in guidelines and recommendations. METHODS: We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments. KEY QUESTIONS AND ANSWERS: When is it important to incorporate cost-effectiveness, resource implications and affordability considerations in WHO guidelines (which topics)? • For cost-effectiveness: The need for cost/effectiveness information should be dictated by the specific question, of which several may be addressed in a single guideline. It is proposed that the indications for undertaking a cost-effectiveness analysis (CEA) could be a starting point for determining which recommendation(s) in the guideline would benefit from such analysis. • For resource implications/affordability: The resource implications of each individual recommendation need to be considered when implementation issues are being discussed. How can cost-effectiveness, resource implications and affordability be explicitly taken into account in WHO guidelines? • For cost-effectiveness: ∘ If data are available, the ideal time to consider cost-effectiveness is during the evidence gathering and synthesizing stage. However, because of the inconsistent availability of CEAs and the procedural difficulty associated with adjusting results from different CEAs to make them comparable, it is also possible for cost-effectiveness to be considered during the stage of developing recommendations. ∘ Depending on the quantity and quality and relevance of the data available, such data can be considered in a qualitative way or in a quantitative way, ranging from a listing of the costs to a modelling exercise. At the very least, a qualitative approach like a commentary outlining the economic issues that need to be considered is necessary. If a quantitative approach is to be used, the full model should be transparent and comprehensive. • For resource implications/affordability: ∘ Resource implications, including health system changes, for each recommendation in a WHO guideline should be explored. At the minimum, a qualitative description that can serve as a gross indicator of the amount of resources needed, relative to current practice, should be provided. How does one provide guidance in contextualizing guideline recommendations at the country level based on considerations of cost-effectiveness, resource implications and affordability? • All models should be made available and ideally are designed to allow for analysts to make changes in key parameters and reapply results in their own country. • In the global guidelines, scenarios and extensive sensitivity/uncertainty analysis can be applied. Resource implications for WHO • From the above, it is clear that guidelines development groups will need a health economist. There is need to ensure that this is included in the budget for guidelines and that there is in-house support for this as well

Twelve years of clinical practice guideline development, dissemination and evaluation in Canada (1994 to 2005)

Background - Despite the growing availability of clinical practice guidelines since the early 1990's, little is known about how guideline development and dissemination may have changed over time in Canada. This study compares Canadian guideline development, dissemination, and evaluation in two six year periods from 1994–1999 and 2000–2005. // Methods - Survey of guideline developers who submitted their clinical practice guidelines to the Canadian Medical Association Infobase (a Canadian guideline repository) between 1994 and 2005. Survey items included information about the developers, aspects of guideline development, and dissemination and evaluation activities. // Results - Surveys were sent to the developers of 2341 guidelines in the CMA Infobase over the 12 year period, 1664 surveys were returned (response rate 71%). Of these, 730 unique guidelines were released from 1994–1999, and 630 were released from 2000–2005. Compared to the earlier period, more recent guidelines were being produced in English only. There has been little change in the type of organizations developing guidelines with most developed by provincial and national organizations. In the recent period, developers were more likely to report using computerized search strategies (94% versus 88%), publishing the search strategy (42% versus 34%), reaching consensus using open discussion (95% versus 78%), and evaluating effectiveness of the dissemination strategies (12% versus 6%) and the impact of the CPGs on health outcomes (24% versus 5%). Recent guidelines were less likely to be based on literature reviews (94% versus 99.6%) and were disseminated using fewer strategies (mean 4.78 versus 4.12). // Conclusion - Given that guideline development processes have improved in some areas over the past 12 years yet not in others, ongoing monitoring of guideline quality is required. Guidelines produced more recently in Canada are less likely to be based on a review of the evidence and only about half discuss levels of evidence underlying recommendations. Guideline dissemination and implementation activities have actually decreased. Unfortunately, the potential positive impact on patient health outcomes will not be realized until the recommendations are adopted and acted upon

An interdisciplinary guideline development process: the Clinic on Low-back pain in Interdisciplinary Practice (CLIP) low-back pain guidelines

<p>Abstract</p> <p>Background</p> <p>Evaluation of low-back pain guidelines using Appraisal of Guidelines Research and Evaluation (AGREE) criteria has shown weaknesses, particularly in stakeholder involvement and applicability of recommendations. The objectives of this project were to: 1) develop a primary care interdisciplinary clinical practice guideline aimed at preventing prolonged disability from low-back pain, using a community of practice approach, and 2) assess the participants' impressions with the process, and evaluate the relationship between participant characteristics and their participation.</p> <p>Methods</p> <p>Ten stakeholder representatives recruited 136 clinicians to participate in this community of practice. Clinicians were drawn from the following professions: physiotherapists (46%), occupational therapists (37%), and family physicians (17%). Using previously published guidelines, systematic reviews, and meta-analyses, a first draft of the guidelines was presented to the community of practice. Four communication tools were provided for discussion and exchanges with experts: a web-based discussion forum, an anonymous comment form, meetings, and a symposium. Participants were prompted for comments on interpretation, clarity, and applicability of the recommendations. Clinical management recommendations were revised following these exchanges. At the end of the project, a questionnaire was sent to the participants to assess satisfaction towards the guidelines and the development process.</p> <p>Results</p> <p>Twelve clinical management recommendations on management of low-back pain and persistent disability were initially developed. These were discussed through 188 comments posted on the discussion forum and 103 commentary forms submitted. All recommendations were modified following input of the participants. A clinical algorithm summarizing the guidelines was also developed. A response rate of 75% was obtained for the satisfaction questionnaire. The majority of respondents appreciated the development process and agreed with the guideline content. Most participants thought recommendations improved between versions, and that participant comments contributed to this improvement. All stakeholders officially endorsed the guidelines.</p> <p>Conclusion</p> <p>The community of practice approach was a successful method to develop guidelines on low-back pain, with participants providing information to improve guideline recommendations. The information technology infrastructure that was developed remains for continuous interdisciplinary exchanges and updating of the guidelines.</p

An educational game for teaching clinical practice guidelines to Internal Medicine residents: development, feasibility and acceptability

<p>Abstract</p> <p>Background</p> <p>Adherence to Clinical Practice Guidelines (CPGs) remains suboptimal among internal medicine trainees. Educational games are of growing interest and have the potential to improve adherence to CPGs. The objectives of this study were to develop an educational game to teach CPGs in Internal Medicine residency programs and to evaluate its feasibility and acceptability.</p> <p>Methods</p> <p>We developed the Guide-O-Game^©in the format of a TV game show with questions based on recommendations of CPGs. The development of the Guide-O-Game^©consisted of the creation of a multimedia interactive tool, the development of recommendation-based questions, and the definition of the game's rules. We evaluated its feasibility through pilot testing and its acceptability through a qualitative process.</p> <p>Results</p> <p>The multimedia interactive tool uses a Macromedia Flash web application and consists of a manager interface and a user interface. The user interface allows the choice of two game styles. We created so far 16 sets of questions relating to 9 CPGs. The pilot testing proved that the game was feasible. The qualitative evaluation showed that residents considered the game to be acceptable.</p> <p>Conclusion</p> <p>We developed an educational game to teach CPGs to Internal Medicine residents that is both feasible and acceptable. Future work should evaluate its impact on educational outcomes.</p

The systematic guideline review: method, rationale, and test on chronic heart failure

Background: Evidence-based guidelines have the potential to improve healthcare. However, their de-novo-development requires substantial resources-especially for complex conditions, and adaptation may be biased by contextually influenced recommendations in source guidelines. In this paper we describe a new approach to guideline development-the systematic guideline review method (SGR), and its application in the development of an evidence-based guideline for family physicians on chronic heart failure (CHF). Methods: A systematic search for guidelines was carried out. Evidence-based guidelines on CHF management in adults in ambulatory care published in English or German between the years 2000 and 2004 were included. Guidelines on acute or right heart failure were excluded. Eligibility was assessed by two reviewers, methodological quality of selected guidelines was appraised using the AGREE instrument, and a framework of relevant clinical questions for diagnostics and treatment was derived. Data were extracted into evidence tables, systematically compared by means of a consistency analysis and synthesized in a preliminary draft. Most relevant primary sources were re-assessed to verify the cited evidence. Evidence and recommendations were summarized in a draft guideline. Results: Of 16 included guidelines five were of good quality. A total of 35 recommendations were systematically compared: 25/35 were consistent, 9/35 inconsistent, and 1/35 un-rateable (derived from a single guideline). Of the 25 consistencies, 14 were based on consensus, seven on evidence and four differed in grading. Major inconsistencies were found in 3/9 of the inconsistent recommendations. We re-evaluated the evidence for 17 recommendations (evidence-based, differing evidence levels and minor inconsistencies) - the majority was congruent. Incongruity was found where the stated evidence could not be verified in the cited primary sources, or where the evaluation in the source guidelines focused on treatment benefits and underestimated the risks. The draft guideline was completed in 8.5 man-months. The main limitation to this study was the lack of a second reviewer. Conclusion: The systematic guideline review including framework development, consistency analysis and validation is an effective, valid, and resource saving-approach to the development of evidence-based guidelines

Incorporating a gender perspective into the development of clinical guidelines: a training course for guideline developers

<p>Abstract</p> <p>Background</p> <p>Dutch guideline-developing organizations do not focus systematically on differences between men and women when developing guidelines, even though there is increasing evidence that being male or female may have an effect on health and health outcomes. In collaboration with two prominent Dutch guideline-developing organizations, we designed a training course to encourage systematic attention to sex differences in guideline development procedures.</p> <p>Methods</p> <p>The course is targeted towards guideline developers. Its aims are to improve awareness concerning the relevance of considering sex differences in the guideline development process, as well as the competence and skills necessary for putting this into practice. The design and teaching methods of the course are based on adult learning styles and principles of changing provider behaviour. It was adjusted to the working methods of guideline organizations. The course was taught to, and evaluated by, a group of staff members from two guideline organizations in the Netherlands.</p> <p>Results</p> <p>The course consists of five modules, each of which corresponds to a key step in the guideline development process. The participants rated the training course positively on content, programme, and trainers. Their written comments suggest that the course met its objectives.</p> <p>Conclusion</p> <p>The training course is the first to address sex differences in guideline development. Results from the pilot test suggest that the course achieved its objectives. Because its modules and teaching methods of the course are widely transferable, the course could be useful for many organizations that are involved in developing guidelines. Follow-up studies are needed to assess the long-term effect of the course on the actions of guideline developers and its utility in other settings.</p

Evidence-based guidelines in the evaluation of work disability: an international survey and a comparison of quality of development

<p>Abstract</p> <p>Background</p> <p>In social insurance, the evaluation of work disability is becoming stricter as priority is given to the resumption of work, which calls for a guarantee of quality for these evaluations. Evidence-based guidelines have become a major instrument in the quality control of health care, and the quality of these guidelines' development can be assessed using the AGREE instrument. In social insurance medicine, such guidelines are relatively new. We were interested to know what guidelines have been developed to support the medical evaluation of work disability and the quality of these guidelines.</p> <p>Methods</p> <p>Five European countries that were reported to use guidelines were approached, using a recent inventory of evaluations of work disability in Europe. We focused on guidelines that are disease-oriented and formally prescribed in social insurance medicine. Using the AGREE instrument, these guidelines were appraised by two researchers. We asked two experts involved in guideline development to indicate if they agreed with our results and to provide explanations for insufficient scores.</p> <p>Results</p> <p>We found six German and sixteen Dutch sets of disease-oriented guidelines in official use. The AGREE instrument was applicable, requiring minor adaptations. The appraisers reached consensus on all items. Each guideline scored well on 'scope and purpose' and 'clarity and presentation'. The guidelines scored moderately on 'stakeholder involvement' in the Netherlands, but insufficiently in Germany, due mainly to the limited involvement of patients' representatives in this country. All guidelines had low scores on 'rigour of development', which was due partly to a lack of documentation and of existing evidence. 'Editorial independence' and 'applicability' had low scores in both countries as a result of how the production was organised.</p> <p>Conclusion</p> <p>Disease-oriented guidelines in social insurance medicine for the evaluation of work disability are a recent phenomenon, so far restricted to Germany and the Netherlands. The AGREE instrument is suitably applicable to assess the quality of guideline development in social insurance medicine, but some of the scoring rules need to be adapted to the context of social insurance. Existing guidelines do not meet the AGREE criteria to a sufficient level. The way patients' representatives can be involved needs further discussion. The guidelines would profit from more specific recommendations and, for providing evidence, more research is needed on the functional capacity of people with disabilities.</p

Clinical practice guidelines within the Southern African development community: a descriptive study of the quality of guideline development and concordance with best evidence for five priority diseases

<p>Abstract</p> <p>Background</p> <p>Reducing the burden of disease relies on availability of evidence-based clinical practice guidelines (CPGs). There is limited data on availability, quality and content of guidelines within the Southern African Development Community (SADC). This evaluation aims to address this gap in knowledge and provide recommendations for regional guideline development.</p> <p>Methods</p> <p>We prioritised five diseases: HIV in adults, malaria in children and adults, pre-eclampsia, diarrhoea in children and hypertension in primary care. A comprehensive electronic search to locate guidelines was conducted between June and October 2010 and augmented with email contact with SADC Ministries of Health. Independent reviewers used the AGREE II tool to score six quality domains reporting the guideline development process. Alignment of the evidence-base of the guidelines was evaluated by comparing their content with key recommendations from accepted reference guidelines, identified with a content expert, and percentage scores were calculated.</p> <p>Findings</p> <p>We identified 30 guidelines from 13 countries, publication dates ranging from 2003-2010. Overall the '<it>scope and purpose' </it>and '<it>clarity and presentation' </it>domains of the AGREE II instrument scored highest, median 58%(range 19-92) and 83%(range 17-100) respectively. '<it>Stakeholder involvement' </it>followed with median 39%(range 6-75). '<it>Applicability'</it>, '<it>rigour of development' </it>and '<it>editorial independence' </it>scored poorly, all below 25%. Alignment with evidence was variable across member states, the lowest scores occurring in older guidelines or where the guideline being evaluated was part of broader primary healthcare CPG rather than a disease-specific guideline.</p> <p>Conclusion</p> <p>This review identified quality gaps and variable alignment with best evidence in available guidelines within SADC for five priority diseases. Future guideline development processes within SADC should better adhere to global reporting norms requiring broader consultation of stakeholders and transparency of process. A regional guideline support committee could harness local capacity to support context appropriate guideline development.</p