Proposed methods for reviewing the outcomes of health research: the impact of funding by the UK's Arthritis Research Campaign

Background: External and internal factors are increasingly encouraging research funding bodies to demonstrate the outcomes of their research. Traditional methods of assessing research are still important, but can be merged into broader multi-dimensional categorisations of research benefits. The onus has hitherto been on public sector funding bodies, but in the UK the role of medical charities in funding research is particularly important and the Arthritis Research Campaign, the leading medical charity in its field in the UK, commissioned a study to identify the outcomes from research that it funds. This article describes the methods to be used. Methods: A case study approach will enable narratives to be told, illuminating how research funded in the early 1990s was (or was not) translated into practice. Each study will be organised using a common structure, which, with careful selection of cases, should enable cross-case analysis to illustrate the strengths of different modes and categories of research. Three main interdependent methods will be used: documentary and literature review; semi-structured interviews; and bibliometric analysis. The evaluative framework for organising the studies was previously used for assessing the benefits from health services research. Here, it has been specifically amended for a medical charity that funds a wide range of research and is concerned to develop the careers of researchers. It was further refined in three pilot studies. The framework has two main elements. First, a multi-dimensional categorisation of benefits going from the knowledge produced in peer reviewed journal articles through to the health and potential economic gain. The second element is a logic model, which, with various stages, should provide a way of organising the studies. The stock of knowledge is important: much research, especially basic, will feed into it and influence further research rather than directly lead to health gains. The cross-case analysis will look for factors associated with outcomes. Conclusions: The pilots confirmed the applicability of the methods for a full study which should assist the Arthritis Research Campaign to demonstrate the outcomes from its funding, and provide it with evidence to inform its own policies

Fulfilling Lives: Supporting people with multiple needs, Evaluation Report, Year 1

This report is prepared for the Big Lottery Fund (the Fund) by the national evaluationteam and provides emerging findings and lessons learned from the first year of thenational evaluation of the Fulfilling Lives: Supporting people with multiple needsinitiative hereafter referred to as Fulfilling Lives (multiple needs).The national evaluation has been designed to determine the degree to which the initiativeis successfully achieving its aims and how they are being achieved. The evaluation will beboth formative and summative in nature, in that, it will inform the ongoing design and delivery of Fulfilling Lives (multiple needs) and its component projects as well as assessoverall achievements and value for money to inform future decision and policy making.Within this context, the evaluation has a number of objectives:— To track and assess the achievements of the initiative and to estimate the extent to whichthese are attributable to the projects and interventions delivered.— To calculate the costs of the projects and the corresponding value of benefits to theexchequer and wider society. This will enable an assessment of value for money of theprogramme and for individual interventions.— To identify what interventions and approaches work well, for which people, families andcommunities and in which circumstances and contexts.— To assess the extent to which the Big Lottery Fund's principles are incorporated into projectdesign and delivery and to determine the degree to which these principles affect successfuldelivery and outcomes.— To explore project implementation, understand problems faced and to facilitate theidentification of solutions and lessons learned

Implementation Action Plan for organic food and farming research

The Implementation Action Plan completes TP Organics’ trilogy of key documents of the Research Vision to 2025 (Niggli et al 2008) and the Strategic Research Agenda (Schmid et al 2009). The Implementation Action Plan addresses important areas for a successful implementation of the Strategic Research Agenda. It explores the strength of Europe’s organic sector on the world stage with about one quarter of the world’s organic agricultural land in 2008 and accounting for more than half of the global organic market. The aims and objectives of organic farming reflect a broad range of societal demands on the multiple roles of agriculture and food production of not only producing commodities but also ecosystem services. These are important for Europe’s economic success, the resilience of its farms and prosperity in its rural areas. The organic sector is a leading market for quality and authenticity: values at the heart of European food culture. Innovation is important across the EU economy, and no less so within the organic sector. The Implementation Action Plan devotes its third chapter to considering how innovation can be stimulated through organic food and farming research and, crucially, translated into changes in business and agricultural practice. TP Organics argues for a broad understanding of innovation that includes technology, know-how and social/organisational innovations. Accordingly, innovation can involve different actors throughout the food sector. Many examples illustrate innovations in the organic sector includign and beyond technology. The various restrictions imposed by organic standards have driven change and turned organic farms and food businesses into creative living laboratories for smart and green innovations and the sector will continue to generate new examples. The research topics proposed by TP Organics in the Strategic Research Agenda can drive innovation in areas as wide ranging as production practices for crops, technologies for livestock, food processing, quality management, on-farm renewable energy or insights into the effects of consumption of organic products on disease and wellbeing and life style of citizens. Importantly, many approaches developed within the sector are relevant and useful beyond the specific sector. The fourth chapter addresses knowledge management in organic agriculture, focusing on the further development of participatory research methods. Participatory (or trans-disciplinary) models recognise the worth and importance of different forms of knowledge and reduced boundaries between the generators and the users of knowledge, while respecting and benefitting from transparent division of tasks. The emphasis on joint creation and exchange of knowledge makes them valuable as part of a knowledge management toolkit as they have the capacity to enhance the translation of research outcomes into practical changes and lead to real-world progress. The Implementation Action Plan argues for the wider application of participatory methods in publicly-funded research and also proposes some criteria for evaluating participatory research, such as the involvement and satisfaction of stakeholders as well as real improvements in sustainability and delivery of public goods/services. European agriculture faces specific challenges but at the same time Europe has a unique potential for the development of agro-ecology based solutions that must be supported through well focused research. TP Organics believes that the most effective approaches in agriculture and food research will be systems-based, multi- and trans-disciplinary, and that in the development of research priorities, the interconnections between biodiversity, dietary diversity, functional diversity and health must be taken into account. Chapter five of the action plan identifies six themes which could be used to organise research and innovation activities in agriculture under Europe’s 8th Framework Programme on Research Cooperation: • Eco-functional intensification – A new area of agricultural research which aims to harness beneficial activities of the ecosystem to increase productivity in agriculture. • The economics of high output / low input farming Developing reliable economic and environmental assessments of new recycling, renewable-based and efficiency-boosting technologies for agriculture. • Health care schemes for livestock Shifting from therapeutics to livestock health care schemes based on good husbandry and disease prevention. • Resilience and “sustainagility” Dealing with a more rapidly changing environment by focusing on ‘adaptive capacity’ to help build resilience of farmers, farms and production methods. • From farm diversity to food diversity and health and wellbeing of citizens Building on existing initiatives to reconnect consumers and producers, use a ‘whole food chain’ approach to improve availability of natural and authentic foods. • Creating centres of innovation in farming communities A network of centres in Europe applying and developing trans-disciplinary and participatory scientific approaches to support innovation among farmers and SMEs and improving research capacities across Europe

Advances in Teaching & Learning Day Abstracts 2005

Proceedings of the Advances in Teaching & Learning Day Regional Conference held at The University of Texas Health Science Center at Houston in 2005

User needs elicitation via analytic hierarchy process (AHP). A case study on a Computed Tomography (CT) scanner

Background: The rigorous elicitation of user needs is a crucial step for both medical device design and purchasing. However, user needs elicitation is often based on qualitative methods whose findings can be difficult to integrate into medical decision-making. This paper describes the application of AHP to elicit user needs for a new CT scanner for use in a public hospital. Methods: AHP was used to design a hierarchy of 12 needs for a new CT scanner, grouped into 4 homogenous categories, and to prepare a paper questionnaire to investigate the relative priorities of these. The questionnaire was completed by 5 senior clinicians working in a variety of clinical specialisations and departments in the same Italian public hospital. Results: Although safety and performance were considered the most important issues, user needs changed according to clinical scenario. For elective surgery, the five most important needs were: spatial resolution, processing software, radiation dose, patient monitoring, and contrast medium. For emergency, the top five most important needs were: patient monitoring, radiation dose, contrast medium control, speed run, spatial resolution. Conclusions: AHP effectively supported user need elicitation, helping to develop an analytic and intelligible framework of decision-making. User needs varied according to working scenario (elective versus emergency medicine) more than clinical specialization. This method should be considered by practitioners involved in decisions about new medical technology, whether that be during device design or before deciding whether to allocate budgets for new medical devices according to clinical functions or according to hospital department

On-line transformer condition monitoring through diagnostics and anomaly detection

This paper describes the end-to-end components of an on-line system for diagnostics and anomaly detection. The system provides condition monitoring capabilities for two in- service transmission transformers in the UK. These transformers are nearing the end of their design life, and it is hoped that intensive monitoring will enable them to stay in service for longer. The paper discusses the requirements on a system for interpreting data from the sensors installed on site, as well as describing the operation of specific diagnostic and anomaly detection techniques employed. The system is deployed on a substation computer, collecting and interpreting site data on-line

Assessing the payback from health R & D: From ad hoc studies to regular monitoring

Chapter 1 : Introduction • The increasing demands for the benefits of payback from publicly funded R&D to be assessed are based partly on the need to justify or account for expenditure on R&D, and partly on the desire for information to assist resource allocation and the better management of R&D funds. The former consideration is particularly strong in relation to the R&D expenditure that comes out of the wider NHS budget. • In this report a range of categories of payback will be identified along with a variety of methods for assessing them. • The aim of the report is to make recommendations as to how the outcomes from health research might best be monitored on a regular basis. The specific context of the report is the NHS R&D Programme but many of the issues will be relevant for a wide range of funders of health R&D. • The introduction sets out not only a plan of the report but also suggests that readers familiar with the general arguments and existing literature may choose to jump to Chapter 6. Chapter 2 : Review of Existing Approaches to Assessing the Payback from Research • Existing work describes various approaches to valuing research. Some are ex ante and attempt to predict the outcomes of research being considered, others are ex post or retrospective. • The five categories of benefit or payback from health R&D that have been identified involve contributions: to knowledge; to research capacity and future research; to improved information for decision making; to the efficiency, efficacy and equity of health care services; and to the nation’s economic performance. These are shown in Table 1 of the report • The process by which R&D generates final outcomes can be modelled as a sequence. This includes primary outputs such as publications; secondary outputs in the form of policy or administrative decisions; and final outcomes which comprise the health and economic benefits. Feedback loops are also introduced and mitigate the limitations of a linear approach. • Qualitative and quantitative approaches can be used but there are immense problems with time lags and attributing outcomes, and sometimes even outputs, to specific items of research funding. • Four common methods of measuring payback can be used. Expert review, by peers or, sometimes, users is the traditional way of assessing the quality of research. Bibliometric techniques can involve not only counting publications but also using datasets such as the Science Citation Index and Wellcome’s Research Outputs Database (ROD). The various methods of economic analysis of payback are difficult to undertake given the costs and problems of acquiring relevant information and estimating benefits. Social science methods include case studies, which can provide useful information but are resource intensive, and questionnaires to researchers and potential research users. Chapter 3 : Characteristics of a Routine Monitoring System • In moving from ad hoc or research studies of payback towards a more regular monitoring it is noted that whereas there has always been a tradition of evaluation of research, in the public services in general there is now a greater emphasis on audit and performance measurement and indicators. A review of these various systems suggests we should be looking to develop a system of outcomes monitoring that incorporates performance indicators (PIs) and measurement rather than an audit system that is trying to monitor activities against predetermined targets. • Standard characteristics of performance measurement systems do not necessarily apply to research where, for example, there are non-standard outputs. Difficulties have arisen in the USA in attempting to apply the Government Performance and Results Act to research funding agencies. It is shown that because the findings of basic research, in particular, enter a knowledge pool in which people and ideas interact, it is difficult to use a PIs’ approach to track eventual outcomes. However, for some types of health research it has proved more feasible to trace the flow between research outputs and outcomes. • An outcomes monitoring system could be useful if it met the following criteria: relevant to, with as comprehensive coverage as possible of, the funders objectives; relevant to the funder’s decision making processes; encourages accurate compliance; minimises unintended consequences; and has acceptable costs. Chapter 4 : Differences Between Research Types • The range of differences between types of research can be relevant for the design of a routine monitoring system. The OECD distinguishes between basic research, applied research and experimental development. Most DH/NHS research is applied. There might be more of a tradition of publication of findings in applied research in health than in other fields. Nevertheless, the publication and incentives patterns operating in basic research mean that it would be inappropriate to use bibliometric indicators in a simple way across all fields even in health research. • Despite having some differences from health research in publication patterns and in the detailed categories of payback, the broad approach proposed in Chapter 6 could be applied to social care research. • Research that is commissioned, especially by the government, has some of the minimum conditions built into it that are associated with outcomes being generated, in particular because the funder has identified that a contribution in this area will be valuable. Chapter 5 : What Units of Research? • The term programme has various meanings including being used to describe a collection of projects on a common theme and to describe a block of funding for a research unit. • Three main streams or modes of funding can be identified: projects, which are administratively grouped into programmes including a responsive programme; institutions/centres/units; individual researchers. These 3 streams are displayed in Figure 1. It is probable that the regular data-gathering for a monitoring system would operate at the basic level of each stream or mode. • Previous work demonstrates that the full range of benefits can sometimes be applied at the level of projects, either in the responsive mode or in programmes, through the use of questionnaires to researchers. Expert and user review and user surveys have also been applied. • Institutions and centres increasingly have experience not only of traditional periodic expert review but also of producing annual reports, although there are debates about what dimensions to include in such reviews and reports. • Individuals in receipt of research development awards have completed questionnaires during and after the awards. These concentrate on the development of research capacity but can go wider. Chapter 6 : A Possible Comprehensive Outcomes Monitoring System • The proposed system is intended for DH/NHS to monitor the outcomes from its R&D in order to justify the R&D expenditure and assist with managing the portfolio. More detailed information is required for the latter purpose. • We propose a multidimensional approach be adopted to cover all the dimensions of payback and that information be gathered from three sets of sources and Table 3 shows which methods would cover which output/outcome categories. • Firstly, possibly annually, a questionnaire (possibly electronic) covering most payback categories should gather data from the basic level of each funding stream ie. from lead researchers of projects, from research institutions/centres, and from individual award holders. • Secondly, supplementary information should be gathered from external databases (including the citation indices and Wellcome’s ROD). • Thirdly, a range of approaches ie. user surveys, reviews by experts and peers, case studies including economic evaluations, and analysis of sources used in policy documents such as NICE guidelines, would be undertaken on a sample basis. They would provide not only supplementary information but, as with the external databases, would also verify the data collected directly from researchers. • These proposals can be evaluated against the criteria set out in Chapter 3: • The system is relevant to DH’s objectives of generating payback in a range of categories. • Various problems have to be overcome before the system could be fully decision relevant. Firstly it might be necessary to ask researchers to apportion the contribution made to specific outputs from various funding streams. Second, to be decision relevant the information would have to be analysed and presented in a manner consistent with funders’ decision making processes. This would involve a) showing how for each outcome and output, for example publications, data from one project or stream could be compared with those from another and b) demonstrating how different outputs and outcomes could be aggregated. • The questions of accuracy of data, minimisation of unintended consequences and the acceptability of the net costs are also addressed. Chapter 7 : Research and Monitoring • Whilst this report is primarily concerned with moving from ad hoc studies towards a routine monitoring system there are issues that need further research. • Before embarking on full implementation the feasibility needs to be tested of items such as on-line recording of data and asking researchers to attribute proportions of research outputs to separate funding agencies. • Once the system is implemented the value of some items can be better assessed, for example the additional value provided by self reporting of publications beyond that gained from relying on external databases. • The data provided by the system would provide opportunities for further payback research on, for example, links between publications and other categories of payback. • Some items such as network analysis could potentially be added to the monitoring system after further examination of them. • Finally the benefit from the monitoring system itself should be assessed.Department of Health; Wellcome Trus