The utilisation of health research in policy-making: Concepts, examples, and methods of assessment

Chapter 1: Introduction and Background • The importance of utilising health research in policy-making, and therefore the need to understand the mechanisms involved, is increasingly recognised. Recent reports calling for more resources to improve health in developing countries, and global pressures for accountability, draw greater attention to research-informed policy-making. • For at least twenty years there has been recognition of the multiple meanings or models of research utilisation in policy-making. It has similarly been long recognised that a range of factors is involved in the interactions between health research and policy-makers. • The emerging focus on Health Research Systems (HRS) has identified additional mechanisms through which greater utilisation of research could be achieved. Assessment of the role of health research in policy-making is best undertaken as part of a wider study that also includes the utilisation of health research by industry, medical practitioners, and the public. Chapter 2: The Nature of Policy-Making, Types of Research and Utilisation Models • Policy-making broadly interpreted includes national health policies made by government ministers and officials, policies made by local health service managers, and clinical guidelines from professional bodies. In this report, however, the main focus is on public policy-making rather than that conducted by professional bodies. The utilisation of health research in policy-making should eventually lead to desired outcomes, including health gains. Research can make a contribution in at least three phases of the policy-making process: agenda setting; policy formulation; and implementation. Descriptions of these processes, however, can over-estimate the degree of rationality in policy-making. Therefore, the analysis is informed by a review of the full range of policy-making models. These include rational and incrementalist models. • Various categories of research are likely to be used differently in health policy-making. Applied research might be more readily useable by a policy system than basic research, but health policy-makers tend to relate more willingly to natural sciences than social sciences. When research is based on the priorities of potential users, and/or is research of proven quality, this increases the possibility that it will be translated into policies. There also appears to be a greater chance of research being used in clinical policies about delivering care to patients, than in national policies on the structures of the health service. • Models of research utilisation in policy-making start with a link to rational or instrumental views of policy-making, and include descriptions of how commissioned research can help to find solutions to problems. Other models relate to an incrementalist view in which policy-making involves a series of small steps over a long period; research findings might gradually cause a shift in perceptions about an issue in a process of ‘enlightenment’. Interactive models of research utilisation stress the way in which policy-makers and researchers might develop links over a long period. Research can also be used symbolically to support decisions already taken. Chapter 3: Examples from Previous Studies • A study of health policy-making in two southern African countries illustrates how policy-making processes can be analysed. It addresses agenda setting, policy formulation and implementation. The methods used included documentary analysis and key informant interviews. • Many previous studies of research utilisation can provide lessons for future assessments. Two broad approaches can be identified. Some studies start with pieces, or programmes, of research and examine their impact. Others consider policy on a particular topic and assess the role of research in the policy-making. There are advantages and drawbacks in each approach, and overlaps between them. • To facilitate comparison, studies of research utilisation are best organised around a conceptual framework. Despite that, the influence of contextual factors in different settings makes it difficult to generalise. • The two methods used most frequently, and usually together, come from the qualitative tradition: documentary analysis and in-depth interviews. Questionnaires, bibliometric analysis, insider knowledge and historical approaches have all been applied. A few recent studies have attempted to score or scale the level of utilisation. • The examples suggest there is a greater level of utilisation and final outcomes in terms of health, health equity, and social and economic gain than is often assumed, whilst still showing much underutilisation. There is considerable variation in the degree of utilisation, both within and between studies. Chapter 4: Key Issues in the Analysis of Research Utilisation in Policy-Making • Increasing attention is focusing on the concept of interfaces between researchers and the users of research. This incorporates the idea that there are likely to be different values and interests between the two communities. • In relation to utilisation, the prioritisation debate revolves around two key aspects: whether priorities are being set that will produce research that policy-makers and others will want to use, and whether priorities are being set that will engage the interests and commitment of the research community. • Interactions across the interface between policy-makers and researchers are important in transferring research to policy-makers. This fits especially well with the interactive model of utilisation. Actions by individual researchers can be useful in generating interaction, but it is desirable to consider the role of the HRS in encouraging or facilitating interactions, networks and mechanisms at a system-wide level. The HRS could provide funding and organisational support for various items including: long-term research centres; research brokerage/translator mechanisms; the creation of official committees of policy-makers and researchers; and mechanisms for review and synthesis of research findings. • There is increased recognition of the significance of policy-makers in their role as the receptors of research. In relation to the perspective of policy-makers there is a spectrum of key questions. These range from whether relevant research is available and effectively being brought to their attention, to whether they are able to absorb it and willing to use it. The HRS has a responsibility, especially in the early parts of the spectrum, but the wider health system also has a responsibility to create appropriate institutional mechanisms and ensure there are staff willing and able to incorporate relevant research. • More attention should be given to the role of incentives, both for researchers to produce utilisable research, and for policy-makers, at the system or individual level, to use it. The assessment of utilisation becomes a key issue if rewards are to focus on relevance as well as research excellence. • An appropriate model for assessing research utilisation in policy-making combines analysis of two issues: the role of receptors and the importance of actions at the interfaces. An emphasis on the role of the receptor is necessary because ultimately it is up to the policy-maker to make the decisions. Any assessment of the success of the HRS in relation to utilisation must accept that the wider political context is beyond the control of the HRS, but consider the activities of the HRS, within its given context, to enhance the utilisation of research by increasing the permeability of the interfaces. Chapter 5: Assessment of Research Utilisation in Health Policy-Making • The reasons for assessing the utilisation of research in policy-making include: advocacy, accountability, and increased understanding. For the World Health Organization there could be a role in conducting such assessments with the aim of providing evidence of the effective use of research resources. This could support advocacy for greater resources to be made available for health research. It is important that the purposes of any assessment are taken into account in planning the methods to be used. • Previous studies demonstrated the difficulties of making generalisations about specific factors associated with high levels of utilisation. To address this in any cross-national WHO initiative involving a series of studies in a range of countries, it would be desirable to structure all the studies around a conceptual framework (such as the interfaces and receptor framework considered here) and base the studies in each country on common themes. These could include policies for the adoption of multi-drug therapy for treating leprosy, and for the equitable access to health services. • Analysis of documents and semi-structured interviews would be appropriate methods in each study assessing the role of research in policy-making on a specific policy theme. Questionnaires could also have a role. These approaches would provide triangulation of methods and data-sources and should also provide material to help identify the relative importance, in relation to the level of utilisation recorded, of the HRS mechanisms described in the previous analysis. The types and sources of research used, and reasons for their use, should also be recorded and attempts made to correlate them with the previous priority setting approaches. It is expected that each study will produce its own narrative or story of what caused utilisation in the particular context, but the data gathered could also be applied to descriptive scales of the level research utilisation. The four scales could cover the consistency of policy with research findings, and the degree of influence of research on agenda setting, policy formulation, and implementation. • The findings from the assessments in each participating country should be collated. For each policy theme or topic the analysis would compare two sets of data: the scales for level of research utilisation in each country, and the contextualised lists of the HRS activities and other mechanisms and networks thought to be important. Although the account here has focused on research impact on policy-making, the evaluations would be stronger as part of a wider analysis covering research utilisation and interactions with practitioners, industry and the public. • Given appropriate and targeted topic and country selection, this approach is likely to meet the purpose of using structured methods to provide examples of effective research utilisation. The approach should contribute towards enhanced understanding of the issues and could provide the basis of an assessment tool which, if used widely in countries, could lead to greater utilisation of health research.Research Policy and Co-operation (RPC) Department of the World Health Organization, Geneva; UK Department of Health’s Policy Research Programme; Alliance for Health Policy and Systems Research from the governments of Norway and Sweden; World Bank and International Development Research Council of Canad

Estimating the economic value to societies of the impact of health research: A critical review

Estimating the economic value to societies of health research is a complex but essential step in establishing and justifying appropriate levels of investment in research. The practical difficulties encountered include: identifying and valuing the relevant research inputs (when many pieces of research may contribute to a clinical advance); accurately ascribing the impact of the research; and appropriately valuing the attributed economic impact. In this review, relevant studies identified from the literature were grouped into four categories on the basis of the methods used to value the benefits of research. The first category consists of studies that value the direct cost savings that could arise from research leading either to new, less-costly treatments or to developments such as vaccines that reduce the number of patients needing treatment. The second category comprises studies that consider the value to the economy of a healthy workforce. According to this "human capital" approach, indirect cost savings arise when better health leads to the avoidance of lost production. The third category includes studies that examine gains to the economy in terms of product development, consequent employment and sales. The studies placed in the fourth category measure the intrinsic value to society of the health gain, by placing a monetary value on a life. The review did not identify any consistency of methodology, but the fourth approach has most promise as a measure of social value. Many of the studies reviewed come from industrialized nations and a proposal is made by the present reviewers for an international initiative, covering developed and developing countries, to undertake further methodological analysis and testing

Lessons from the evaluation of the UK's NHS R&D Implementation Methods Programme

Background: Concern about the effective use of research was a major factor behind the creation of the NHS R&D Programme in 1991. In 1994, an advisory group was established to identify research priorities in research implementation. The Implementation Methods Programme (IMP) flowed from this, and its commissioning group funded 36 projects. In 2000 responsibility for the programme passed to the National Co-ordinating Centre for NHS Service Delivery and Organisation R&D, which asked the Health Economics Research Group (HERG), Brunel University, to conduct an evaluation in 2002. By then most projects had been completed. This evaluation was intended to cover: the quality of outputs, lessons to be learnt about the communication strategy and the commissioning process, and the benefits from the projects. Methods: We adopted a wide range of quantitative and qualitative methods. They included: documentary analysis, interviews with key actors, questionnaires to the funded lead researchers, questionnaires to potential users, and desk analysis. Results: Quantitative assessment of outputs and dissemination revealed that the IMP funded useful research projects, some of which had considerable impact against the various categories in the HERG payback model, such as publications, further research, research training, impact on health policy, and clinical practice. Qualitative findings from interviews with advisory and commissioning group members indicated that when the IMP was established, implementation research was a relatively unexplored field. This was reflected in the understanding brought to their roles by members of the advisory and commissioning groups, in the way priorities for research were chosen and developed, and in how the research projects were commissioned. The ideological and methodological debates associated with these decisions have continued among those working in this field. The need for an effective communication strategy for the programme as a whole was particularly important. However, such a strategy was never developed, making it difficult to establish the general influence of the IMP as a programme. Conclusion: Our findings about the impact of the work funded, and the difficulties faced by those developing the IMP, have implications for the development of strategic programmes of research in general, as well as for the development of more effective research in this field

The information sources and journals consulted or read by UK paediatricians to inform their clinical practice and those which they consider important: a questionnaire survey

Background: Implementation of health research findings is important for medicine to be evidence-based. Previous studies have found variation in the information sources thought to be of greatest importance to clinicians but publication in peer-reviewed journals is the traditional route for dissemination of research findings. There is debate about whether the impact made on clinicians should be considered as part of the evaluation of research outputs. We aimed to determine first which information sources are generally most consulted by paediatricians to inform their clinical practice, and which sources they considered most important, and second, how many and which peer-reviewed journals they read. Methods: We enquired, by questionnaire survey, about the information sources and academic journals that UK medical paediatric specialists generally consulted, attended or read and considered important to their clinical practice. Results: The same three information sources – professional meetings & conferences, peerreviewed journals and medical colleagues – were, overall, the most consulted or attended and ranked the most important. No one information source was found to be of greatest importance to all groups of paediatricians. Journals were widely read by all groups, but the proportion ranking them first in importance as an information source ranged from 10% to 46%. The number of journals read varied between the groups, but Archives of Disease in Childhood and BMJ were the most read journals in all groups. Six out of the seven journals previously identified as containing best paediatric evidence are the most widely read overall by UK paediatricians, however, only the two most prominent are widely read by those based in the community. Conclusion: No one information source is dominant, therefore a variety of approaches to Continuing Professional Development and the dissemination of research findings to paediatricians should be used. Journals are an important information source. A small number of key ones can be identified and such analysis could provide valuable additional input into the evaluation of clinical research outputs

Health research improves healthcare: now we have the evidence and the chance to help the WHO spread such benefits globally

There has been a dramatic increase in the body of evidence demonstrating the benefits that come from health research. In 2014, the funding bodies for higher education in the UK conducted an assessment of research using an approach termed the Research Excellence Framework (REF). As one element of the REF, universities and medical schools in the UK submitted 1,621 case studies claiming to show the impact of their health and other life sciences research conducted over the last 20 years. The recently published results show many case studies were judged positively as providing examples of the wide range and extensive nature of the benefits from such research, including the development of new treatments and screening programmes that resulted in considerable reductions in mortality and morbidity. Analysis of specific case studies yet again illustrates the international dimension of progress in health research; however, as has also long been argued, not all populations fully share the benefits. In recognition of this, in May 2013 the World Health Assembly requested the World Health Organization (WHO) to establish a Global Observatory on Health Research and Development (R&D) as part of a strategic work-plan to promote innovation, build capacity, improve access, and mobilise resources to address diseases that disproportionately affect the world’s poorest countries. As editors of Health Research Policy and Systems (HARPS), we are delighted that our journal has been invited to help inform the establishment of the WHO Global Observatory through a Call for Papers covering a range of topics relevant to the Observatory, including topics on which HARPS has published articles over the last few months, such as approaches to assessing research results, measuring expenditure data with a focus on R&D, and landscape analyses of platforms for implementing R&D. Topics related to research capacity building may also be considered. The task of establishing a Global Observatory on Health R&D to achieve the specified objectives will not be easy; nevertheless, this Call for Papers is well timed – it comes just at the point where the evidence of the benefits from health research has been considerably strengthened

Understanding factors associated with the translation of cardiovascular research: A multinational case study approach

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.This article has been made available through the Brunel Open Access Publishing Fund.Background: Funders of health research increasingly seek to understand how best to allocate resources in order to achieve maximum value from their funding. We built an international consortium and developed a multinational case study approach to assess benefits arising from health research. We used that to facilitate analysis of factors in the production of research that might be associated with translating research findings into wider impacts, and the complexities involved. Methods: We built on the Payback Framework and expanded its application through conducting co-ordinated case studies on the payback from cardiovascular and stroke research in Australia, Canada and the United Kingdom. We selected a stratified random sample of projects from leading medical research funders. We devised a series of innovative steps to: minimize the effect of researcher bias; rate the level of impacts identified in the case studies; and interrogate case study narratives to identify factors that correlated with achieving high or low levels of impact. Results: Twenty-nine detailed case studies produced many and diverse impacts. Over the 15 to 20 years examined, basic biomedical research has a greater impact than clinical research in terms of academic impacts such as knowledge production and research capacity building. Clinical research has greater levels of wider impact on health policies, practice, and generating health gains. There was no correlation between knowledge production and wider impacts. We identified various factors associated with high impact. Interaction between researchers and practitioners and the public is associated with achieving high academic impact and translation into wider impacts, as is basic research conducted with a clinical focus. Strategic thinking by clinical researchers, in terms of thinking through pathways by which research could potentially be translated into practice, is associated with high wider impact. Finally, we identified the complexity of factors behind research translation that can arise in a single case. Conclusions: We can systematically assess research impacts and use the findings to promote translation. Research funders can justify funding research of diverse types, but they should not assume academic impacts are proxies for wider impacts. They should encourage researchers to consider pathways towards impact and engage potential research users in research processes. © 2014 Wooding et al.; licensee BioMed Central Ltd.RAND Europe and HERG, with subsequent funding from the NHFA, the HSFC and the CIHR. This research was also partially supported by the Policy Research Programme in the English Department of Health

Conducting retrospective impact analysis to inform a medical research charity’s funding strategies: The case of Asthma UK

© 2013 Hanney et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.This article has been made available through the Brunel Open Access Publishing Fund.BACKGROUND: Debate is intensifying about how to assess the full range of impacts from medical research. Complexity increases when assessing the diverse funding streams of funders such as Asthma UK, a charitable patient organisation supporting medical research to benefit people with asthma. This paper aims to describe the various impacts identified from a range of Asthma UK research, and explore how Asthma UK utilised the characteristics of successful funding approaches to inform future research strategies. METHODS: We adapted the Payback Framework, using it both in a survey and to help structure interviews, documentary analysis, and case studies. We sent surveys to 153 lead researchers of projects, plus 10 past research fellows, and also conducted 14 detailed case studies. These covered nine projects and two fellowships, in addition to the innovative case studies on the professorial chairs (funded since 1988) and the MRC-Asthma UK Centre in Allergic Mechanisms of Asthma (the ‘Centre’) which together facilitated a comprehensive analysis of the whole funding portfolio. We organised each case study to capture whatever academic and wider societal impacts (or payback) might have arisen given the diverse timescales, size of funding involved, and extent to which Asthma UK funding contributed to the impacts. RESULTS: Projects recorded an average of four peer-reviewed journal articles. Together the chairs reported over 500 papers. All streams of funding attracted follow-on funding. Each of the various categories of societal impacts arose from only a minority of individual projects and fellowships. Some of the research portfolio is influencing asthma-related clinical guidelines, and some contributing to product development. The latter includes potentially major breakthroughs in asthma therapies (in immunotherapy, and new inhaled drugs) trialled by university spin-out companies. Such research-informed guidelines and medicines can, in turn, contribute to health improvements. The role of the chairs and the pioneering collaborative Centre is shown as being particularly important. CONCLUSIONS: We systematically demonstrate that all types of Asthma UK’s research funding assessed are making impacts at different levels, but the main societal impacts from projects and fellowships come from a minority of those funded. Asthma UK used the study’s findings, especially in relation to the Centre, to inform research funding strategies to promote the achievement of impact.This study was funded by Asthma UK

The impact generated by public and charity-funded research in the UK: A systematic literature review

Objective: To identify, synthesize and critically assess the empirical evidence of the impact generated by public and charity funded health research in the United Kingdom. Methods: We conducted a systematic literature review of the empirical evidence published in English in peer-reviewed journals between 2006 and 2017. Studies meeting the inclusion criteria were selected and their findings were analysed using the Payback Framework into five main categories: knowledge, benefits to future research and research use, benefits from informing policy and product development, health and health sector benefits and broader economic benefits. We assessed the studies for risk of selection, reporting and funding bias. Results: Thirteen studies met the inclusion criteria. The majority of the studies (10 out of 13) assessed impact at multiple domains including the main 5 key themes of the Payback Framework. All of them showed a positive impact of funded research on outcomes. Of those studies, one presented low risk of bias (8%), 6 studies were classified as presenting moderate risk of bias (46%) and 6 studies presented high risk of bias (46%). Conclusions: Empirical evidence on the impact of public and charity funded research is still limited and subject to funding and selection bias. More work is needed to establish the causal effects of funded research on academic outcomes, policy, practice and the broader economy

Yes, research can inform health policy; but can we bridge the 'Do-Knowing It's Been Done' gap?

Copyright @ 2011 Hanney and Gonzalez.Provisional Abstract: This editorial introduces a new Supplement in Health Research Policy and Systems and highlighs the importance of assessing the impact of health research by examining whether we can move from 'Know-Do' to 'Do-Knowing It's Been Done'This article is made available through the Brunel Open Access publishing fund

Institutional strategies for capturing socio-economic impact of academic research

Evaluation of socio-economic impact is an emerging theme for publicly-funded academic research. Within this context the paper suggests that the concept of institutional research capital be expanded to include the capture and evaluation of socio-economic impact. Furthermore, it argues that understanding the typology of impacts and the tracking from research to impact will assist the formulation of institutional strategies for capturing socio-economic impact. A three-stage approach is proposed for capturing and planning activities to enhance the generation of high-quality impact. Stage one outlines the critical role of user engagement that facilitates the tracking of such impact. Stage two employs an analytical framework based on the criteria of ‘depth’ and ‘spread’ to evaluate impacts that have been identified. Stage three utilizes the outcomes of the framework to devise strategies, consisting of either further research (to increase depth) or more engagement (to increase spread) that will improve the generation of higher quality impact