Proposed methods for reviewing the outcomes of health research: the impact of funding by the UK's Arthritis Research Campaign

Background: External and internal factors are increasingly encouraging research funding bodies to demonstrate the outcomes of their research. Traditional methods of assessing research are still important, but can be merged into broader multi-dimensional categorisations of research benefits. The onus has hitherto been on public sector funding bodies, but in the UK the role of medical charities in funding research is particularly important and the Arthritis Research Campaign, the leading medical charity in its field in the UK, commissioned a study to identify the outcomes from research that it funds. This article describes the methods to be used. Methods: A case study approach will enable narratives to be told, illuminating how research funded in the early 1990s was (or was not) translated into practice. Each study will be organised using a common structure, which, with careful selection of cases, should enable cross-case analysis to illustrate the strengths of different modes and categories of research. Three main interdependent methods will be used: documentary and literature review; semi-structured interviews; and bibliometric analysis. The evaluative framework for organising the studies was previously used for assessing the benefits from health services research. Here, it has been specifically amended for a medical charity that funds a wide range of research and is concerned to develop the careers of researchers. It was further refined in three pilot studies. The framework has two main elements. First, a multi-dimensional categorisation of benefits going from the knowledge produced in peer reviewed journal articles through to the health and potential economic gain. The second element is a logic model, which, with various stages, should provide a way of organising the studies. The stock of knowledge is important: much research, especially basic, will feed into it and influence further research rather than directly lead to health gains. The cross-case analysis will look for factors associated with outcomes. Conclusions: The pilots confirmed the applicability of the methods for a full study which should assist the Arthritis Research Campaign to demonstrate the outcomes from its funding, and provide it with evidence to inform its own policies

An external evaluation of Phat Pak process and structure: Volume 1: Evaluation report

An evaluation was conducted by the 510 Evaluation Research Group 2007 after receiving a request from the Phat Pak Governance Group (PPGG). The evaluation focused on four areas identified in the Phat Pak service specification plan attached to it’s Ministry of Health contract (Youth participation, Communication Strategies, Decision making Processes and Skills Development). Our intention of evaluating the Phat Pak Programme has been to, through feedback from a range of participants, gauge the extent that the programme objectives (such as those highlighted in the contract are being met. More specifically, the function of the evaluation has been to help identify specific areas that are working well and those areas that need further development. It is our intention that the evaluation will provide both short and long-term direction with added insight towards increasing participation and enhancing health outcomes for youth and the wider community of the Waikato region

Project Retrosight. Understanding the returns from cardiovascular and stroke research: Methodology Report

Copyright @ 2011 RAND Europe. All rights reserved. The full text article is available via the link below.This project explores the impacts arising from cardiovascular and stroke research funded 15-20 years ago and attempts to draw out aspects of the research, researcher or environment that are associated with high or low impact. The project is a case study-based review of 29 cardiovascular and stroke research grants, funded in Australia, Canada and UK between 1989 and 1993. The case studies focused on the individual grants but considered the development of the investigators and ideas involved in the research projects from initiation to the present day. Grants were selected through a stratified random selection approach that aimed to include both high- and low-impact grants. The key messages are as follows: 1) The cases reveal that a large and diverse range of impacts arose from the 29 grants studied. 2) There are variations between the impacts derived from basic biomedical and clinical research. 3) There is no correlation between knowledge production and wider impacts 4) The majority of economic impacts identified come from a minority of projects. 5) We identified factors that appear to be associated with high and low impact. This report presents the key observations of the study and an overview of the methods involved. It has been written for funders of biomedical and health research and health services, health researchers, and policy makers in those fields. It will also be of interest to those involved in research and impact evaluation.This study was initiated with internal funding from RAND Europe and HERG, with continuing funding from the UK National Institute for Health Research, the Canadian Institutes of Health Research, the Heart and Stroke Foundation of Canada and the National Heart Foundation of Australia. The UK Stroke Association and the British Heart Foundation provided support in kind through access to their archives

The metric tide: report of the independent review of the role of metrics in research assessment and management

This report presents the findings and recommendations of the Independent Review of the Role of Metrics in Research Assessment and Management. The review was chaired by Professor James Wilsdon, supported by an independent and multidisciplinary group of experts in scientometrics, research funding, research policy, publishing, university management and administration. This review has gone beyond earlier studies to take a deeper look at potential uses and limitations of research metrics and indicators. It has explored the use of metrics across different disciplines, and assessed their potential contribution to the development of research excellence and impact. It has analysed their role in processes of research assessment, including the next cycle of the Research Excellence Framework (REF). It has considered the changing ways in which universities are using quantitative indicators in their management systems, and the growing power of league tables and rankings. And it has considered the negative or unintended effects of metrics on various aspects of research culture. The report starts by tracing the history of metrics in research management and assessment, in the UK and internationally. It looks at the applicability of metrics within different research cultures, compares the peer review system with metric-based alternatives, and considers what balance might be struck between the two. It charts the development of research management systems within institutions, and examines the effects of the growing use of quantitative indicators on different aspects of research culture, including performance management, equality, diversity, interdisciplinarity, and the ‘gaming’ of assessment systems. The review looks at how different funders are using quantitative indicators, and considers their potential role in research and innovation policy. Finally, it examines the role that metrics played in REF2014, and outlines scenarios for their contribution to future exercises

Using the Medical Research Council framework for the development and evaluation of complex interventions in a theory-based infant feeding intervention to prevent childhood obesity:The baby milk intervention and trial

Introduction. We describe our experience of using the Medical Research Council framework on complex interventions to guide the development and evaluation of an intervention to prevent obesity by modifying infant feeding behaviours. Methods. We reviewed the epidemiological evidence on early life risk factors for obesity and interventions to prevent obesity in this age group. The review suggested prevention of excess weight gain in bottle-fed babies and appropriate weaning as intervention targets; hence we undertook systematic reviews to further our understanding of these behaviours. We chose theory and behaviour change techniques that demonstrated evidence of effectiveness in altering dietary behaviours. We subsequently developed intervention materials and evaluation tools and conducted qualitative studies with mothers (intervention recipients) and healthcare professionals (intervention deliverers) to refine them. We developed a questionnaire to assess maternal attitudes and feeding practices to understand the mechanism of any intervention effects. Conclusions. In addition to informing development of our specific intervention and evaluation materials, use of the Medical Research Council framework has helped to build a generalisable evidence base for early life nutritional interventions. However, the process is resource intensive and prolonged, and this should be taken into account by public health research funders. This trial is registered with ISRTCN: 20814693 Baby Milk Trial

The shape of things to come: visions for the future of Aboriginal and Torres Strait Islander health research

This paper presents the results of a project that considered how research might best contribute to Aboriginal and Torres Strait Islander health and wellbeing in the year 2030. Executive summary In late 2012, the Lowitja Institute embarked on a project using ‘futures thinking’ to consider how research might best contribute to Aboriginal and Torres Strait Islander health and wellbeing in the year 2030. The project was motivated by a desire to ‘get ahead of the game’: to anticipate and prepare for the potential research demands of the future. In particular, there was a desire to ‘close the gap’ between the point at which important research needs are identified by policy makers or service providers, and when research findings can be delivered. To think about the research needs of the future, it was necessary to first imagine what life might be like in 2030. What might Australia be like then, and the world? And what might the lives of Aboriginal and Torres Strait Islander people be? Workshops were held around the country to consider issues and trends visible in the current landscape, and how these might play out to influence life in 2030. A range of possible scenarios emerged, clustering around two divergent futures: an inclusive, vibrant Australia in which Aboriginal and Torres Strait Islander cultures are valued and embraced as central to the Australian identity; or an Australia in which economic and/or spiritual poverty drive a rejection of diversity and increase the divide between rich and poor. Participants then grappled with the question: If this (or that) scenario occurs, then what will be needed from research? By thinking about the range of possible scenarios for life in 2030, what capabilities are required to deliver the research that will be needed to address emerging issues and create a healthy future? The inspirational and empowering answer—perhaps not surprisingly—was not simply a list of research topics. Instead, participants articulated a strong and widely shared desire for a profoundly different system of research. A vision emerged of a research system in which research and practice are closely interwoven and which would enable greater integration of health services, policy and research. Such a system would be responsive to changing research demands, but also to changing social, economic, technological and knowledge landscapes. Specific research capabilities were also identified. An urgent necessity to actively address the social determinants of health was articulated in every workshop, along with a growing sense that the health and health research sectors may need to play a facilitating role, inviting other sectors—such as education, justice, local government—to collaborate and maximise the impact of their collective efforts to bring about change. A need for more evidence and evaluation around early childhood development programs (social as well as physical development) was seen as a priority for the immediate future. The Aboriginal and Torres Strait Islander health and health research sectors have played a pioneering role in the reform of research in Australia. The strength of vision articulated by participants in this project, and the desire to see that vision become a reality, suggests the sector will succeed in its drive toward a vision of a more effective research system

The Revolution of Mobile Phone-Enabled Services for Agricultural Development (m-Agri Services) in Africa: The Challenges for Sustainability

The provision of information through mobile phone-enabled agricultural information services (m-Agri services) has the potential to revolutionise agriculture and significantly improve smallholder farmers’ livelihoods in Africa. Globally, the benefits of m-Agri services include facilitating farmers’ access to financial services and sourcing agricultural information about input use, practices, and market prices. There are very few published literature sources that focus on the potential benefits of m-Agri services in Africa and none of which explore their sustainability. This study, therefore, explores the evolution, provision, and sustainability of these m-Agri services in Africa. An overview of the current landscape of m-Agri services in Africa is provided and this illustrates how varied these services are in design, content, and quality. Key findings from the exploratory literature review reveal that services are highly likely to fail to achieve their intended purpose or be abandoned when implementers ignore the literacy, skills, culture, and demands of the target users. This study recommends that, to enhance the sustainability of m-Agri services, the implementers need to design the services with the users involved, carefully analyse, and understand the target environment, and design for scale and a long-term purpose. While privacy and security of users need to be ensured, the reuse or improvement of existing initiatives should be explored, and projects need to be data-driven and maintained as open source. Thus, the study concludes that policymakers can support the long-term benefit of m-Agri services by ensuring favourable policies for both users and implementers