Screening for abdominal aortic aneurysm

The official published version can be found at the link below

Global IDEA

Five members of the Global IDEA Scientific Advisory Committee respond to Dr. Moore and colleagues: Global IDEA Scientific Advisory Committee. Health and economic benefits of an accelerated program of research to combat global infectious diseases

Economic evaluation of ondansetron: Preliminary analysis using clinical trial data prior to price setting

This article has been made available through the Brunel Open Access Publishing Fund and is available from the specified link - Copyright © 1992 Macmillan Press Ltd.This study combines secondary analysis of efficacy and side-effect data from a randomised controlled trial with estimates of resource use to evaluate the likely economic effects of the new antiemetic agent ondansetron. Costs, effects and cost-effectiveness of ondansetron in the prophylaxis of acute nausea and vomiting induced by chemotherapy are assessed relative to antiemetic therapy with metoclopramide. Superior efficacy of ondansetron is quantified both in terms of significant emesis avoided and emesis management costs avoided. A simple cost analysis, with the metoclopramide dosage priced at 10 pounds, indicates that therapy with ondansetron would give equivalent net treatment costs, at a price ratio (ondansetron/metoclopramide) of 2.3 to 1. If therapeutic success is defined as the avoidance of emesis and antiemetic side-effects, then the two therapies would be equally cost-effective at a drug price ratio of 5 to 1. We conclude that, (i) economic evaluation prior to price setting is feasible and informative; (ii) such models can indicate prospective data collection priorities

Cost effectiveness of the implantable cardioverter defibrillator: a preliminary analysis

Background-An implantable cardioverter defibrillator (ICD) maybe effective in reducing the risk of sudden cardiac death. The high cost of ICD treatment, however, compared with alternatives raises the question of whether this new technology is an efficient use of scarce health care resources. Objective-To estimate the incremental cost effectiveness of the implantable cardioverter defibrillator compared with drug treatment with amiodarone in the management of patients at high risk of sudden cardiac death. Design-A cost effectiveness model was constructed from data already published and other secondary sources. Differences in patient survival were calculated from life tables for comparable ICD and amiodarone patient series. Costs were based on typical patient management protocols derived from current United Kingdom practice and interviews with physicians. Main outcome measures-Cost effectiveness of ICD treatment was computed over 20 years; all future costs and effects were discounted at 6% per year. Results-Estimated life expectancy was 111 and 6-7 years with ICD and amiodarone respectively; the discounted 20 year difference lies in the range 1P7 to 3*7 years. Discounted 20 year treatment costs were £28 400 for the ICD and £2300 for amiodarone. Cost effectiveness ofICD treatment lies in the range of £15 400 to £8200 per life-year gained. Conclusions-Cost effectiveness ofICD treatment is similar to some existing cardiac programmes funded under the NHS but uncertainty exists due to limitations of the data. Costs of ICD treatment may fall in the future as the life of the device increases and less invasive implantation methods are needed. The effectivess of ICD compared with amiodarone is currently being studied by a randomised controlled trial

Assessing the payback from health R & D: From ad hoc studies to regular monitoring

Chapter 1 : Introduction • The increasing demands for the benefits of payback from publicly funded R&D to be assessed are based partly on the need to justify or account for expenditure on R&D, and partly on the desire for information to assist resource allocation and the better management of R&D funds. The former consideration is particularly strong in relation to the R&D expenditure that comes out of the wider NHS budget. • In this report a range of categories of payback will be identified along with a variety of methods for assessing them. • The aim of the report is to make recommendations as to how the outcomes from health research might best be monitored on a regular basis. The specific context of the report is the NHS R&D Programme but many of the issues will be relevant for a wide range of funders of health R&D. • The introduction sets out not only a plan of the report but also suggests that readers familiar with the general arguments and existing literature may choose to jump to Chapter 6. Chapter 2 : Review of Existing Approaches to Assessing the Payback from Research • Existing work describes various approaches to valuing research. Some are ex ante and attempt to predict the outcomes of research being considered, others are ex post or retrospective. • The five categories of benefit or payback from health R&D that have been identified involve contributions: to knowledge; to research capacity and future research; to improved information for decision making; to the efficiency, efficacy and equity of health care services; and to the nation’s economic performance. These are shown in Table 1 of the report • The process by which R&D generates final outcomes can be modelled as a sequence. This includes primary outputs such as publications; secondary outputs in the form of policy or administrative decisions; and final outcomes which comprise the health and economic benefits. Feedback loops are also introduced and mitigate the limitations of a linear approach. • Qualitative and quantitative approaches can be used but there are immense problems with time lags and attributing outcomes, and sometimes even outputs, to specific items of research funding. • Four common methods of measuring payback can be used. Expert review, by peers or, sometimes, users is the traditional way of assessing the quality of research. Bibliometric techniques can involve not only counting publications but also using datasets such as the Science Citation Index and Wellcome’s Research Outputs Database (ROD). The various methods of economic analysis of payback are difficult to undertake given the costs and problems of acquiring relevant information and estimating benefits. Social science methods include case studies, which can provide useful information but are resource intensive, and questionnaires to researchers and potential research users. Chapter 3 : Characteristics of a Routine Monitoring System • In moving from ad hoc or research studies of payback towards a more regular monitoring it is noted that whereas there has always been a tradition of evaluation of research, in the public services in general there is now a greater emphasis on audit and performance measurement and indicators. A review of these various systems suggests we should be looking to develop a system of outcomes monitoring that incorporates performance indicators (PIs) and measurement rather than an audit system that is trying to monitor activities against predetermined targets. • Standard characteristics of performance measurement systems do not necessarily apply to research where, for example, there are non-standard outputs. Difficulties have arisen in the USA in attempting to apply the Government Performance and Results Act to research funding agencies. It is shown that because the findings of basic research, in particular, enter a knowledge pool in which people and ideas interact, it is difficult to use a PIs’ approach to track eventual outcomes. However, for some types of health research it has proved more feasible to trace the flow between research outputs and outcomes. • An outcomes monitoring system could be useful if it met the following criteria: relevant to, with as comprehensive coverage as possible of, the funders objectives; relevant to the funder’s decision making processes; encourages accurate compliance; minimises unintended consequences; and has acceptable costs. Chapter 4 : Differences Between Research Types • The range of differences between types of research can be relevant for the design of a routine monitoring system. The OECD distinguishes between basic research, applied research and experimental development. Most DH/NHS research is applied. There might be more of a tradition of publication of findings in applied research in health than in other fields. Nevertheless, the publication and incentives patterns operating in basic research mean that it would be inappropriate to use bibliometric indicators in a simple way across all fields even in health research. • Despite having some differences from health research in publication patterns and in the detailed categories of payback, the broad approach proposed in Chapter 6 could be applied to social care research. • Research that is commissioned, especially by the government, has some of the minimum conditions built into it that are associated with outcomes being generated, in particular because the funder has identified that a contribution in this area will be valuable. Chapter 5 : What Units of Research? • The term programme has various meanings including being used to describe a collection of projects on a common theme and to describe a block of funding for a research unit. • Three main streams or modes of funding can be identified: projects, which are administratively grouped into programmes including a responsive programme; institutions/centres/units; individual researchers. These 3 streams are displayed in Figure 1. It is probable that the regular data-gathering for a monitoring system would operate at the basic level of each stream or mode. • Previous work demonstrates that the full range of benefits can sometimes be applied at the level of projects, either in the responsive mode or in programmes, through the use of questionnaires to researchers. Expert and user review and user surveys have also been applied. • Institutions and centres increasingly have experience not only of traditional periodic expert review but also of producing annual reports, although there are debates about what dimensions to include in such reviews and reports. • Individuals in receipt of research development awards have completed questionnaires during and after the awards. These concentrate on the development of research capacity but can go wider. Chapter 6 : A Possible Comprehensive Outcomes Monitoring System • The proposed system is intended for DH/NHS to monitor the outcomes from its R&D in order to justify the R&D expenditure and assist with managing the portfolio. More detailed information is required for the latter purpose. • We propose a multidimensional approach be adopted to cover all the dimensions of payback and that information be gathered from three sets of sources and Table 3 shows which methods would cover which output/outcome categories. • Firstly, possibly annually, a questionnaire (possibly electronic) covering most payback categories should gather data from the basic level of each funding stream ie. from lead researchers of projects, from research institutions/centres, and from individual award holders. • Secondly, supplementary information should be gathered from external databases (including the citation indices and Wellcome’s ROD). • Thirdly, a range of approaches ie. user surveys, reviews by experts and peers, case studies including economic evaluations, and analysis of sources used in policy documents such as NICE guidelines, would be undertaken on a sample basis. They would provide not only supplementary information but, as with the external databases, would also verify the data collected directly from researchers. • These proposals can be evaluated against the criteria set out in Chapter 3: • The system is relevant to DH’s objectives of generating payback in a range of categories. • Various problems have to be overcome before the system could be fully decision relevant. Firstly it might be necessary to ask researchers to apportion the contribution made to specific outputs from various funding streams. Second, to be decision relevant the information would have to be analysed and presented in a manner consistent with funders’ decision making processes. This would involve a) showing how for each outcome and output, for example publications, data from one project or stream could be compared with those from another and b) demonstrating how different outputs and outcomes could be aggregated. • The questions of accuracy of data, minimisation of unintended consequences and the acceptability of the net costs are also addressed. Chapter 7 : Research and Monitoring • Whilst this report is primarily concerned with moving from ad hoc studies towards a routine monitoring system there are issues that need further research. • Before embarking on full implementation the feasibility needs to be tested of items such as on-line recording of data and asking researchers to attribute proportions of research outputs to separate funding agencies. • Once the system is implemented the value of some items can be better assessed, for example the additional value provided by self reporting of publications beyond that gained from relying on external databases. • The data provided by the system would provide opportunities for further payback research on, for example, links between publications and other categories of payback. • Some items such as network analysis could potentially be added to the monitoring system after further examination of them. • Finally the benefit from the monitoring system itself should be assessed.Department of Health; Wellcome Trus

The journals of importance to UK clinicians: A questionnaire survey of surgeons

Background: Peer-reviewed journals are seen as a major vehicle in the transmission of research findings to clinicians. Perspectives on the importance of individual journals vary and the use of impact factors to assess research is criticised. Other surveys of clinicians suggest a few key journals within a specialty, and sub-specialties, are widely read. Journals with high impact factors are not always widely read or perceived as important. In order to determine whether UK surgeons consider peer-reviewed journals to be important information sources and which journals they read and consider important to inform their clinical practice, we conducted a postal questionnaire survey and then compared the findings with those from a survey of US surgeons. Methods: A questionnaire survey sent to 2,660 UK surgeons asked which information sources they considered to be important and which peer-reviewed journals they read, and perceived as important, to inform their clinical practice. Comparisons were made with numbers of UK NHSfunded surgery publications, journal impact factors and other similar surveys. Results: Peer-reviewed journals were considered to be the second most important information source for UK surgeons. A mode of four journals read was found with academics reading more than non-academics. Two journals, the BMJ and the Annals of the Royal College of Surgeons of England, are prominent across all sub-specialties and others within sub-specialties. The British Journal of Surgery plays a key role within three sub-specialties. UK journals are generally preferred and readership patterns are influenced by membership journals. Some of the journals viewed by surgeons as being most important, for example the Annals of the Royal College of Surgeons of England, do not have high impact factors. Conclusion: Combining the findings from this study with comparable studies highlights the importance of national journals and of membership journals. Our study also illustrates the complexity of the link between the impact factors of journals and the importance of the journals to clinicians. This analysis potentially provides an additional basis on which to assess the role of different journals, and the published output from research

Evaluation of the NHS R & D implementation methods programme

Chapter 1: Background and introduction • Concern with research implementation was a major factor behind the creation of the NHS R&D Programme in 1991. In 1994 an Advisory Group was established to identify research priorities in this field. The Implementation Methods Programme (IMP) flowed from this and its Commissioning Group funded 36 projects. Funding for the IMP was capped before the second round of commissioning. The Commissioning Group was disbanded and eventually responsibility for the programme passed to the National Co-ordinating Centre for NHS Service Delivery and Organisation R&D (NCCSDO) which, when most projects had finished, asked the Health Economics Research Group (HERG) to conduct an evaluation. This was intended to cover: the quality of outputs; lessons to be learnt about the communication strategy and the commissioning process; and the benefits or payback from the projects. As agreed, the evaluation also addresses the questions of whether there should be a synthesis of the findings from the IMP and any further assessment of payback. Chapter 2: Methods • We adopted a wide range of quantitative and qualitative methods in the evaluation. They included: documentary analysis; interviews with key actors; questionnaires to the funded lead researchers; questionnaires to potential users; and desk analysis. Chapter 3: The outputs from the programme • As in previous assessments of research programmes, we first examined the outputs in terms of knowledge production and various items related to capacity to conduct further research. Although there was a high response rate to the questionnaire to lead researchers (30/36), missing responses mean that the data given below are incomplete. In the case of publications, however, we also made some use of data previously gathered by the programme office. • We attempted to identify publications that were in some way a specific product of IMP funding. About half (59) of the publications from the IMP projects are articles in peer reviewed journals. The journal used most frequently for publication, the BMJ, is also the one with the highest journal impact factor score of those publishing articles specifically from the programme. The recent publication datesof many articles reduces the value of citation analysis. Nevertheless, one article, Coulter et al, 1999, has already been cited on 53 occasions. Important publications, including No Magic Bullets (Oxman et al, 1995), are also associated with preliminary work undertaken for the IMP to assist priority setting. • Fifteen projects, with grants of over £1.3 million, have been awarded to IMP researchers by other funders for follow-on studies connected in some way to the IMP. We also collected details about some non-IMP researchers who are building on the IMP projects. • Research training provided in at least nine of the funded IMP projects is associated with higher/research degrees, including three MDs and four PhDs, that have been awarded or are being completed. Chapter 4: Disseminating and using the research findings • Limited thought had been given by the Implementation Methods Programme to dissemination strategies, but many of the individual researchers were active here. In response to the questionnaires, lead researchers reported making 92 presentations to academic audiences and 104 to practitioner/service groups. Some lead researchers showed that their effective dissemination led to utilisation of the findings. • The Commissioning Group gave some thought to the likely use that could be made of individual research projects, but there was limited systematic analysis of how the findings as a whole would be taken forward. Achieving impact is difficult in this complex field and less than a third of lead researchers claimed to have done so, but about half thought impact could be expected. Based mainly on reports from lead researchers, we give a brief account of how the findings from six projects are being utilised. • We sent electronic questionnaires to groups of potential users of selected projects but this produced a very low response rate. Our postal survey to Heads of Midwifery/researchers in perinatal care produced a higher response of 44%. Amongst those who did respond, there is quite a high level of knowledge about some of the programme’s projects and some level of existing and potential utilisation. We suggest, however, that in some cases there are difficulties in identifying how far the respondent’s focus is on the findings from the original research projects, and it how far it is on the impact of the IMP study that is about ways of influencing the uptake of such findings. Comments from several respondents showed strong support for the cutting edge nature of some of the research. Others, however, also indicated why findings might not be utilised by some practitioners. Several respondents advocated greater dissemination of the IMP. Chapter 5: Comparing applications with outputs • We attempted to compare the scores given to project applications with those given to projects based on their outputs. This exercise faced various problems. The final reports from all completed projects had in theory already been reviewed and given scores for their quality and relevance. In practice, not all final reports received scores. We added a refinement by giving further scores that incorporated the additional information we gathered about both publications and any uptake of the research findings. • Various limitations meant that we conducted this analysis on just 19 of the 36 projects. Nevertheless, the wide range of scores given to the outputs from projects indicates that some were much more successful than others. Our rather limited evidence suggests that there is some correlation between the scores for applications and those for outputs but it is small, which could be related to the difficulties encountered during commissioning. Chapter 6: Lessons learnt about the commissioning process • Those who established the IMP were aware that it was a different type of research field from those previously addressed within the NHS R&D Programme, but one regarded nonetheless as important. Within the NHS R&D Programme at that time a standard clinical RCT approach was strongly favoured. There was also, as ever, a need for quick results. • In developing an understanding of implementation the Advisory Group (AG) conducted cutting edge analysis, consulted widely and drew on a wide range of disciplines. Our interviewees generally took the view that the AG worked well in setting priorities and went as far as it could at the time, especially given the time constraints. • Based on our field work and analysis we identified a series of lessons that might inform future exercises. More attention was required to ensure that all relevant background disciplines were adequately taken into account in setting priorities and commissioning research. Some of these processes needed to be given more time than was available. Consultation needed to be organised in a sufficiently selective way to be of maximum benefit in such a complex area. A time-limited programme was not the most appropriate way to cover a field such as this. • In relation to the commissioning of the projects, we identified issues about the composition of commissioning groups and how people from different backgrounds (researchers, practitioners, managers and patient representatives) should best be involved. • In this new field the Commissioning Group (CG) had to work closely with applicants to develop some of the research applications. This raised issues about how, and when, this process should be handled. • Despite its own rationale, and for a variety of reasons, including the disbanding of the CG, the Implementation Methods Programme never developed an implementation or communication strategy for its own findings. • The general conclusion of those who had been involved with the IMP was that it worked as well as it could at the time, and that various important projects were commissioned. But it was only a start. Chapter 7: Should a synthesis of the findings from the programme and further payback analysis be undertaken? • From interviews and questionnaires we identified widespread, but not total, agreement that there should be some type of synthesis of the findings from the IMP. There is more debate about the form such a synthesis should take. There is some support for a more limited type of stock taking, but also wider backing for the inclusion of many different elements. These could include: a conceptual map of the field of research implementation; an exploration of how the findings from the IMP fit into the context of research implementation today; and an assessment of how far work is still needed in those areas where no projects were funded. One possible suggestion that might incorporate much of this thinking is for the establishment of a group or commission of leading researchers in the field. Their investigation could incorporate all these elements and attempt to show how the issues could be advanced. • We suggest that further work on assessing the payback from the IMP is probably not worthwhile unless it is undertaken as part of a wide-ranging synthesis. Chapter 8: Conclusions, lessons and recommendations • We conclude that the IMP was seen by many of those involved as a new and exciting field. Looking back, they were generally positive about what was started through the IMP. It commissioned a series of projects that produced some important, rigorous, and cutting edge research, at least some of which is making an impact. But this is a complex area in which traditional clinical research, health services research and the social sciences all have a role to play. A unique set of difficulties, as well as opportunities, was faced by those responsible for taking the programme forward. The intellectual challenges of constructing a programme to cover such a vast area with diverse and sometimes conflicting conceptual and methodological perspectives, were compounded by practical problems. These included the capping of the programme’s funding and the premature winding up of the Commissioning Group. As a result, this complex programme, which arguably needed better support than its more clinically orientated predecessors, did not receive it at some stages. Those involved in the programme had a considerable task – the difficulties of which were not completely appreciated at the time. They are clearer in retrospect and feed into the lessons and recommendations presented here, but it is recognised that a programme such as the SDO is already adopting some of the steps. • In relation to research commissioning and communication strategies for research programmes in general, we suggest it could be helpful if protocols were drawn up to cover certain potential difficulties. These include the remit and role of the various stakeholders represented on commissioning groups and the extent to which commissioning groups should be expected to support applicants with their proposals. Perhaps the key general lesson from this evaluation is the need for research programmes to have a proper communication strategy. This should target dissemination at relevant audiences and stress the desirability for contact to be made with potential users as early as possible in the process of devising a project. • Our other recommendations are more specifically relevant when the SDO Programme is considering an area such as implementation methods research. It would be desirable for more time to be made available for preparatory work than was allowed for the IMP and also scope provided for the programme to be able to re-visit issues and learn from early results. It is difficult to incorporate all the analysis that is required if a programme is operating in a time-limited way. • Our conclusion that research implementation is a crucial area for the NHS R&D Programme leads to the recommendation that more R&D activity is needed in this field in order to assist delivery of some key NHS agenda items. As a preliminary step, there is certainly scope for a type of stock taking of the findings from the IMP. On balance there seems also to be an argument for conducting a synthesis of work in the implementation field that goes beyond a mere collation of findings from the specific projects funded. If undertaken, it should fundamentally examine the current NHS needs for research on implementation and how they could be addressed in the light of the findings from the IMP and elsewhere. • Finally, we recommend that more attention should be given to the timing of evaluations such as this and that a phased approach should be adopted. Furthermore, researchers should be informed at the outset of their project about the likely requirements that might be placed upon them in terms of responding to requests for information by those conducting an evaluation.National Co-ordinating Centre for NHS Service Delivery and Organisation R&D (NCCSDO

Systematic review and meta-analysis of the growth and rupture rates of small abdominal aortic aneurysms: implications for surveillance intervals and their cost-effectiveness.

BACKGROUND: Small abdominal aortic aneurysms (AAAs; 3.0-5.4 cm in diameter) are usually asymptomatic and managed by regular ultrasound surveillance until they grow to a diameter threshold (commonly 5.5 cm) at which surgical intervention is considered. The choice of appropriate surveillance intervals is governed by the growth and rupture rates of small AAAs, as well as their relative cost-effectiveness. OBJECTIVES: The aim of this series of studies was to inform the evidence base for small AAA surveillance strategies. This was achieved by literature review, collation and analysis of individual patient data, a focus group and health economic modelling. DATA SOURCES: We undertook systematic literature reviews of growth rates and rupture rates of small AAAs. The databases MEDLINE, EMBASE on OvidSP, Cochrane Central Register of Controlled Trials 2009 Issue 4, ClinicalTrials.gov, and controlled-trials.com were searched from inception up until the end of 2009. We also obtained individual data on 15,475 patients from 18 surveillance studies. REVIEW METHODS: Systematic reviews of publications identified 15 studies providing small AAA growth rates, and 14 studies with small AAA rupture rates, up to December 2009 (later updated to September 2012). We developed statistical methods to analyse individual surveillance data, including the effects of patient characteristics, to inform the choice of surveillance intervals and provide inputs for health economic modelling. We updated an existing health economic model of AAA screening to address the cost-effectiveness of different surveillance intervals. RESULTS: In the literature reviews, the mean growth rate was 2.3 mm/year and the reported rupture rates varied between 0 and 1.6 ruptures per 100 person-years. Growth rates increased markedly with aneurysm diameter, but insufficient detail was available to guide surveillance intervals. Based on individual surveillance data, for each 0.5-cm increase in AAA diameter, growth rates increased by about 0.5 mm/year and rupture rates doubled. To control the risk of exceeding 5.5 cm to below 10% in men, on average a 7-year surveillance interval is sufficient for a 3.0-cm aneurysm, whereas an 8-month interval is necessary for a 5.0-cm aneurysm. To control the risk of rupture to below 1%, the corresponding estimated surveillance intervals are 9 years and 17 months. Average growth rates were higher in smokers (by 0.35 mm/year) and lower in patients with diabetes (by 0.51 mm/year). Rupture rates were almost fourfold higher in women than men, doubled in current smokers and increased with higher blood pressure. Increasing the surveillance interval from 1 to 2 years for the smallest aneurysms (3.0-4.4 cm) decreased costs and led to a positive net benefit. For the larger aneurysms (4.5-5.4 cm), increasing surveillance intervals from 3 to 6 months led to equivalent cost-effectiveness. LIMITATIONS: There were no clear reasons why the growth rates varied substantially between studies. Uniform diagnostic criteria for rupture were not available. The long-term cost-effectiveness results may be susceptible to the modelling assumptions made. CONCLUSIONS: Surveillance intervals of several years are clinically acceptable for men with AAAs in the range 3.0-4.0 cm. Intervals of around 1 year are suitable for 4.0-4.9-cm AAAs, whereas intervals of 6 months would be acceptable for 5.0-5.4-cm AAAs. These intervals are longer than those currently employed in the UK AAA screening programmes. Lengthening surveillance intervals for the smallest aneurysms was also shown to be cost-effective. Future work should focus on optimising surveillance intervals for women, studying whether or not the threshold for surgery should depend on patient characteristics, evaluating the usefulness of surveillance for those with aortic diameters of 2.5-2.9 cm, and developing interventions that may reduce the growth or rupture rates of small AAAs. FUNDING: The National Institute for Health Research Health Technology Assessment programme

Proposed methods for reviewing the outcomes of health research: the impact of funding by the UK's Arthritis Research Campaign

Background: External and internal factors are increasingly encouraging research funding bodies to demonstrate the outcomes of their research. Traditional methods of assessing research are still important, but can be merged into broader multi-dimensional categorisations of research benefits. The onus has hitherto been on public sector funding bodies, but in the UK the role of medical charities in funding research is particularly important and the Arthritis Research Campaign, the leading medical charity in its field in the UK, commissioned a study to identify the outcomes from research that it funds. This article describes the methods to be used. Methods: A case study approach will enable narratives to be told, illuminating how research funded in the early 1990s was (or was not) translated into practice. Each study will be organised using a common structure, which, with careful selection of cases, should enable cross-case analysis to illustrate the strengths of different modes and categories of research. Three main interdependent methods will be used: documentary and literature review; semi-structured interviews; and bibliometric analysis. The evaluative framework for organising the studies was previously used for assessing the benefits from health services research. Here, it has been specifically amended for a medical charity that funds a wide range of research and is concerned to develop the careers of researchers. It was further refined in three pilot studies. The framework has two main elements. First, a multi-dimensional categorisation of benefits going from the knowledge produced in peer reviewed journal articles through to the health and potential economic gain. The second element is a logic model, which, with various stages, should provide a way of organising the studies. The stock of knowledge is important: much research, especially basic, will feed into it and influence further research rather than directly lead to health gains. The cross-case analysis will look for factors associated with outcomes. Conclusions: The pilots confirmed the applicability of the methods for a full study which should assist the Arthritis Research Campaign to demonstrate the outcomes from its funding, and provide it with evidence to inform its own policies

The utilisation of health research in policy-making: Concepts, examples, and methods of assessment

Chapter 1: Introduction and Background • The importance of utilising health research in policy-making, and therefore the need to understand the mechanisms involved, is increasingly recognised. Recent reports calling for more resources to improve health in developing countries, and global pressures for accountability, draw greater attention to research-informed policy-making. • For at least twenty years there has been recognition of the multiple meanings or models of research utilisation in policy-making. It has similarly been long recognised that a range of factors is involved in the interactions between health research and policy-makers. • The emerging focus on Health Research Systems (HRS) has identified additional mechanisms through which greater utilisation of research could be achieved. Assessment of the role of health research in policy-making is best undertaken as part of a wider study that also includes the utilisation of health research by industry, medical practitioners, and the public. Chapter 2: The Nature of Policy-Making, Types of Research and Utilisation Models • Policy-making broadly interpreted includes national health policies made by government ministers and officials, policies made by local health service managers, and clinical guidelines from professional bodies. In this report, however, the main focus is on public policy-making rather than that conducted by professional bodies. The utilisation of health research in policy-making should eventually lead to desired outcomes, including health gains. Research can make a contribution in at least three phases of the policy-making process: agenda setting; policy formulation; and implementation. Descriptions of these processes, however, can over-estimate the degree of rationality in policy-making. Therefore, the analysis is informed by a review of the full range of policy-making models. These include rational and incrementalist models. • Various categories of research are likely to be used differently in health policy-making. Applied research might be more readily useable by a policy system than basic research, but health policy-makers tend to relate more willingly to natural sciences than social sciences. When research is based on the priorities of potential users, and/or is research of proven quality, this increases the possibility that it will be translated into policies. There also appears to be a greater chance of research being used in clinical policies about delivering care to patients, than in national policies on the structures of the health service. • Models of research utilisation in policy-making start with a link to rational or instrumental views of policy-making, and include descriptions of how commissioned research can help to find solutions to problems. Other models relate to an incrementalist view in which policy-making involves a series of small steps over a long period; research findings might gradually cause a shift in perceptions about an issue in a process of ‘enlightenment’. Interactive models of research utilisation stress the way in which policy-makers and researchers might develop links over a long period. Research can also be used symbolically to support decisions already taken. Chapter 3: Examples from Previous Studies • A study of health policy-making in two southern African countries illustrates how policy-making processes can be analysed. It addresses agenda setting, policy formulation and implementation. The methods used included documentary analysis and key informant interviews. • Many previous studies of research utilisation can provide lessons for future assessments. Two broad approaches can be identified. Some studies start with pieces, or programmes, of research and examine their impact. Others consider policy on a particular topic and assess the role of research in the policy-making. There are advantages and drawbacks in each approach, and overlaps between them. • To facilitate comparison, studies of research utilisation are best organised around a conceptual framework. Despite that, the influence of contextual factors in different settings makes it difficult to generalise. • The two methods used most frequently, and usually together, come from the qualitative tradition: documentary analysis and in-depth interviews. Questionnaires, bibliometric analysis, insider knowledge and historical approaches have all been applied. A few recent studies have attempted to score or scale the level of utilisation. • The examples suggest there is a greater level of utilisation and final outcomes in terms of health, health equity, and social and economic gain than is often assumed, whilst still showing much underutilisation. There is considerable variation in the degree of utilisation, both within and between studies. Chapter 4: Key Issues in the Analysis of Research Utilisation in Policy-Making • Increasing attention is focusing on the concept of interfaces between researchers and the users of research. This incorporates the idea that there are likely to be different values and interests between the two communities. • In relation to utilisation, the prioritisation debate revolves around two key aspects: whether priorities are being set that will produce research that policy-makers and others will want to use, and whether priorities are being set that will engage the interests and commitment of the research community. • Interactions across the interface between policy-makers and researchers are important in transferring research to policy-makers. This fits especially well with the interactive model of utilisation. Actions by individual researchers can be useful in generating interaction, but it is desirable to consider the role of the HRS in encouraging or facilitating interactions, networks and mechanisms at a system-wide level. The HRS could provide funding and organisational support for various items including: long-term research centres; research brokerage/translator mechanisms; the creation of official committees of policy-makers and researchers; and mechanisms for review and synthesis of research findings. • There is increased recognition of the significance of policy-makers in their role as the receptors of research. In relation to the perspective of policy-makers there is a spectrum of key questions. These range from whether relevant research is available and effectively being brought to their attention, to whether they are able to absorb it and willing to use it. The HRS has a responsibility, especially in the early parts of the spectrum, but the wider health system also has a responsibility to create appropriate institutional mechanisms and ensure there are staff willing and able to incorporate relevant research. • More attention should be given to the role of incentives, both for researchers to produce utilisable research, and for policy-makers, at the system or individual level, to use it. The assessment of utilisation becomes a key issue if rewards are to focus on relevance as well as research excellence. • An appropriate model for assessing research utilisation in policy-making combines analysis of two issues: the role of receptors and the importance of actions at the interfaces. An emphasis on the role of the receptor is necessary because ultimately it is up to the policy-maker to make the decisions. Any assessment of the success of the HRS in relation to utilisation must accept that the wider political context is beyond the control of the HRS, but consider the activities of the HRS, within its given context, to enhance the utilisation of research by increasing the permeability of the interfaces. Chapter 5: Assessment of Research Utilisation in Health Policy-Making • The reasons for assessing the utilisation of research in policy-making include: advocacy, accountability, and increased understanding. For the World Health Organization there could be a role in conducting such assessments with the aim of providing evidence of the effective use of research resources. This could support advocacy for greater resources to be made available for health research. It is important that the purposes of any assessment are taken into account in planning the methods to be used. • Previous studies demonstrated the difficulties of making generalisations about specific factors associated with high levels of utilisation. To address this in any cross-national WHO initiative involving a series of studies in a range of countries, it would be desirable to structure all the studies around a conceptual framework (such as the interfaces and receptor framework considered here) and base the studies in each country on common themes. These could include policies for the adoption of multi-drug therapy for treating leprosy, and for the equitable access to health services. • Analysis of documents and semi-structured interviews would be appropriate methods in each study assessing the role of research in policy-making on a specific policy theme. Questionnaires could also have a role. These approaches would provide triangulation of methods and data-sources and should also provide material to help identify the relative importance, in relation to the level of utilisation recorded, of the HRS mechanisms described in the previous analysis. The types and sources of research used, and reasons for their use, should also be recorded and attempts made to correlate them with the previous priority setting approaches. It is expected that each study will produce its own narrative or story of what caused utilisation in the particular context, but the data gathered could also be applied to descriptive scales of the level research utilisation. The four scales could cover the consistency of policy with research findings, and the degree of influence of research on agenda setting, policy formulation, and implementation. • The findings from the assessments in each participating country should be collated. For each policy theme or topic the analysis would compare two sets of data: the scales for level of research utilisation in each country, and the contextualised lists of the HRS activities and other mechanisms and networks thought to be important. Although the account here has focused on research impact on policy-making, the evaluations would be stronger as part of a wider analysis covering research utilisation and interactions with practitioners, industry and the public. • Given appropriate and targeted topic and country selection, this approach is likely to meet the purpose of using structured methods to provide examples of effective research utilisation. The approach should contribute towards enhanced understanding of the issues and could provide the basis of an assessment tool which, if used widely in countries, could lead to greater utilisation of health research.Research Policy and Co-operation (RPC) Department of the World Health Organization, Geneva; UK Department of Health’s Policy Research Programme; Alliance for Health Policy and Systems Research from the governments of Norway and Sweden; World Bank and International Development Research Council of Canad