Assessing the payback from health R & D: From ad hoc studies to regular monitoring

Chapter 1 : Introduction • The increasing demands for the benefits of payback from publicly funded R&D to be assessed are based partly on the need to justify or account for expenditure on R&D, and partly on the desire for information to assist resource allocation and the better management of R&D funds. The former consideration is particularly strong in relation to the R&D expenditure that comes out of the wider NHS budget. • In this report a range of categories of payback will be identified along with a variety of methods for assessing them. • The aim of the report is to make recommendations as to how the outcomes from health research might best be monitored on a regular basis. The specific context of the report is the NHS R&D Programme but many of the issues will be relevant for a wide range of funders of health R&D. • The introduction sets out not only a plan of the report but also suggests that readers familiar with the general arguments and existing literature may choose to jump to Chapter 6. Chapter 2 : Review of Existing Approaches to Assessing the Payback from Research • Existing work describes various approaches to valuing research. Some are ex ante and attempt to predict the outcomes of research being considered, others are ex post or retrospective. • The five categories of benefit or payback from health R&D that have been identified involve contributions: to knowledge; to research capacity and future research; to improved information for decision making; to the efficiency, efficacy and equity of health care services; and to the nation’s economic performance. These are shown in Table 1 of the report • The process by which R&D generates final outcomes can be modelled as a sequence. This includes primary outputs such as publications; secondary outputs in the form of policy or administrative decisions; and final outcomes which comprise the health and economic benefits. Feedback loops are also introduced and mitigate the limitations of a linear approach. • Qualitative and quantitative approaches can be used but there are immense problems with time lags and attributing outcomes, and sometimes even outputs, to specific items of research funding. • Four common methods of measuring payback can be used. Expert review, by peers or, sometimes, users is the traditional way of assessing the quality of research. Bibliometric techniques can involve not only counting publications but also using datasets such as the Science Citation Index and Wellcome’s Research Outputs Database (ROD). The various methods of economic analysis of payback are difficult to undertake given the costs and problems of acquiring relevant information and estimating benefits. Social science methods include case studies, which can provide useful information but are resource intensive, and questionnaires to researchers and potential research users. Chapter 3 : Characteristics of a Routine Monitoring System • In moving from ad hoc or research studies of payback towards a more regular monitoring it is noted that whereas there has always been a tradition of evaluation of research, in the public services in general there is now a greater emphasis on audit and performance measurement and indicators. A review of these various systems suggests we should be looking to develop a system of outcomes monitoring that incorporates performance indicators (PIs) and measurement rather than an audit system that is trying to monitor activities against predetermined targets. • Standard characteristics of performance measurement systems do not necessarily apply to research where, for example, there are non-standard outputs. Difficulties have arisen in the USA in attempting to apply the Government Performance and Results Act to research funding agencies. It is shown that because the findings of basic research, in particular, enter a knowledge pool in which people and ideas interact, it is difficult to use a PIs’ approach to track eventual outcomes. However, for some types of health research it has proved more feasible to trace the flow between research outputs and outcomes. • An outcomes monitoring system could be useful if it met the following criteria: relevant to, with as comprehensive coverage as possible of, the funders objectives; relevant to the funder’s decision making processes; encourages accurate compliance; minimises unintended consequences; and has acceptable costs. Chapter 4 : Differences Between Research Types • The range of differences between types of research can be relevant for the design of a routine monitoring system. The OECD distinguishes between basic research, applied research and experimental development. Most DH/NHS research is applied. There might be more of a tradition of publication of findings in applied research in health than in other fields. Nevertheless, the publication and incentives patterns operating in basic research mean that it would be inappropriate to use bibliometric indicators in a simple way across all fields even in health research. • Despite having some differences from health research in publication patterns and in the detailed categories of payback, the broad approach proposed in Chapter 6 could be applied to social care research. • Research that is commissioned, especially by the government, has some of the minimum conditions built into it that are associated with outcomes being generated, in particular because the funder has identified that a contribution in this area will be valuable. Chapter 5 : What Units of Research? • The term programme has various meanings including being used to describe a collection of projects on a common theme and to describe a block of funding for a research unit. • Three main streams or modes of funding can be identified: projects, which are administratively grouped into programmes including a responsive programme; institutions/centres/units; individual researchers. These 3 streams are displayed in Figure 1. It is probable that the regular data-gathering for a monitoring system would operate at the basic level of each stream or mode. • Previous work demonstrates that the full range of benefits can sometimes be applied at the level of projects, either in the responsive mode or in programmes, through the use of questionnaires to researchers. Expert and user review and user surveys have also been applied. • Institutions and centres increasingly have experience not only of traditional periodic expert review but also of producing annual reports, although there are debates about what dimensions to include in such reviews and reports. • Individuals in receipt of research development awards have completed questionnaires during and after the awards. These concentrate on the development of research capacity but can go wider. Chapter 6 : A Possible Comprehensive Outcomes Monitoring System • The proposed system is intended for DH/NHS to monitor the outcomes from its R&D in order to justify the R&D expenditure and assist with managing the portfolio. More detailed information is required for the latter purpose. • We propose a multidimensional approach be adopted to cover all the dimensions of payback and that information be gathered from three sets of sources and Table 3 shows which methods would cover which output/outcome categories. • Firstly, possibly annually, a questionnaire (possibly electronic) covering most payback categories should gather data from the basic level of each funding stream ie. from lead researchers of projects, from research institutions/centres, and from individual award holders. • Secondly, supplementary information should be gathered from external databases (including the citation indices and Wellcome’s ROD). • Thirdly, a range of approaches ie. user surveys, reviews by experts and peers, case studies including economic evaluations, and analysis of sources used in policy documents such as NICE guidelines, would be undertaken on a sample basis. They would provide not only supplementary information but, as with the external databases, would also verify the data collected directly from researchers. • These proposals can be evaluated against the criteria set out in Chapter 3: • The system is relevant to DH’s objectives of generating payback in a range of categories. • Various problems have to be overcome before the system could be fully decision relevant. Firstly it might be necessary to ask researchers to apportion the contribution made to specific outputs from various funding streams. Second, to be decision relevant the information would have to be analysed and presented in a manner consistent with funders’ decision making processes. This would involve a) showing how for each outcome and output, for example publications, data from one project or stream could be compared with those from another and b) demonstrating how different outputs and outcomes could be aggregated. • The questions of accuracy of data, minimisation of unintended consequences and the acceptability of the net costs are also addressed. Chapter 7 : Research and Monitoring • Whilst this report is primarily concerned with moving from ad hoc studies towards a routine monitoring system there are issues that need further research. • Before embarking on full implementation the feasibility needs to be tested of items such as on-line recording of data and asking researchers to attribute proportions of research outputs to separate funding agencies. • Once the system is implemented the value of some items can be better assessed, for example the additional value provided by self reporting of publications beyond that gained from relying on external databases. • The data provided by the system would provide opportunities for further payback research on, for example, links between publications and other categories of payback. • Some items such as network analysis could potentially be added to the monitoring system after further examination of them. • Finally the benefit from the monitoring system itself should be assessed.Department of Health; Wellcome Trus

“What Gets Measured Gets Done”: Headteachers’ Responses to the English Secondary School

English secondary schools operate within a performance management system, which includes league tables reporting school performance across a number of indicators. This paper reports the results of an interview-based study, showing that head teachers care about their school’s place in the league tables, and that they believe this system affects behaviour. The effects they identify include some unintended consequences, not necessarily related to improved overall school performance, including focusing on borderline students who can boost a pivotal indicator: the number of students gaining five A*-Cs at GCSE. This behaviour reflects, in part, the dual role played by headteachers: they are both educationalists (serving the interests of all pupils); and school marketers, concerned with promoting the school to existing and prospective parents. The behaviour is also consistent with economic theory, which predicts a focus on that which is measured, potentially at the expense of that which is important, in sectors characterised by incomplete measurement, by multiple stakeholders and containing workers with diverse objectives. We conclude that, given that performance indicators do affect behaviour, it is important to minimise unintended consequences, and we suggest the use of value-added indicators of student performance.education, performance measures

Evaluating the Impact of Performance-related Pay for Teachers in England.

This paper evaluates the impact of a performance-related pay scheme for teachers in England. Using teacher level data, matched with test scores and value-added, we test whether the introduction of a payment scheme based on pupil attainment increased teacher effort. Our evaluation design controls for pupil effects, school effects and teacher effects, and adopts a difference-in-difference methodology. We find that the scheme did improve test scores and value added, on average by about half a grade per pupil. We also find heterogeneity across subjects, with maths teachers showing no improvement.Incentives, teachers pay, education reform, pupil attainment

Integration of the Control Display Navigation Unit (CDNU) Into the EA-6B Block 89A Aircraft and its Impact on Navigation Operations

This thesis was to evaluate the attributes of the recent modification and installation of the Control Display Navigation Unit (CDNU) into the EA-6B aircraft. The author conducted multiple ground and flight test events during a three year evaluation of the EA-6B Block 89A aircraft. The Block 89A modification included an embedded Global Positioning System (GPS)/ Inertial Navigation System (INS) (EGI), enhanced functionality with the recent GPS system modification, and the ability to control the navigation, weapon, and communication on one control panel. This modification was an attempt to replace a failing attitude system and also allow for additional capability. The testing performed included ILS and GPS approaches, holding, air navigation routes, low level military navigation routes, and tactical navigation. The EA-6B is currently not authorized to navigate with the GPS as the sole navigation (NAV) nor authorized to conduct GPS approaches. This evaluation revealed a need for a GPS navigation and approach authorization in the EA-6B. Funding should be started immediately to anticipate meeting the technology requirements once free flight is authorized in the US. This thesis describes the navigation modes currently used in the EA-6B aircraft. The newest 89A upgrade demonstrates great advances in navigation ability with the addition of the EGI. The CDNU as installed in the EA-6B Block 89A aircraft satisfies the FAA requirements of a flight management system (FMS). The CDNU also partially satisfies GPS certification requirements for both the FAA and DOD. The requirement necessary to certify and utilize GPS as a primary navigation source to operate in the NAS not including approaches are RAIM or RAIM equivalent. The CDNU has a function known as EHE that uses an algorithm that was shown to be accurate enough to satisfy this requirement as long as it was used in the blended mode of operation. GPS accuracy was excellent and pilot displays were easy to read and follow. The capability to execute non-precision approaches were demonstrated in the testing and with the addition of RAIM, an unalterable loadable approach, and an alert within the pilot’s primary field of view will allow GPS non-precision approach certification. RAIM capability is available with the GEM IV receivers. An unalterable approach is available with the addition of more memory in the CDNU. An alert is available by physically mounting a new warning light or by activating something on the EFIS displays

Making Sense of Blockchain in Food Supply-Chains

Blockchain is a potentially disruptive and game-changing technology that has created excitement about its potential applications. The agriculture industry in New Zealand is facing increased pressure to be able to accurately track and trace their produce in order to provide higher levels of proof to their customers. This study used a q-methodology approach to examine whether blockchain technology can be the solution to these issues and provides recommendations as to what businesses need to in order to make this a reality. The empirical research revealed four distinct groups within the industry; each with different perspectives of blockchain and its potential. Results also found that while industry experts believe blockchain implementation is inevitable and it will solve the current issues, factors such as high set-up costs and the complexity of technology may be inhibitors. Based on these findings, key recommendations on how the industry should proceed in order to overcome these factors that are preventing adoption are derived. Further research is suggested on how the challenges of food safety and security may be overcome with emerging technologies such as Blockchain, IoT and AI

Experiences of urine collection devices during suspected Urinary Tract Infections: qualitative study

Background: Up to 30% of urine samples from women with suspected urinary tract infection (UTI) are contaminated and need to be repeated, burdening health services and delaying antibiotic prescription. To prevent contamination, midstream urine (MSU) sampling, which can be difficult to achieve, is recommended. Urine collection devices (UCDs) which automatically capture MSU have been proposed as a solution. There are few studies exploring women’s experiences of using such devices. Aim: Our aim was to explore women’s experiences of urine collection and the use of UCDs during a suspected UTI. Design and Setting: An embedded qualitative study in a UK randomised controlled trial (RCT) of UCDs among women attending primary care for UTI symptoms. Method: Semi-structured, telephone interviews with 29 women who had participated in the RCT were conducted. The transcribed interviews were then thematically analysed. Results: Most of the women were dissatisfied with how they normally produced urine samples. Many were able to use the devices, found them hygienic and would use them again, even if they had initially experienced problems. Women who had not used the devices expressed interest in trying them. Potential barriers to UCD use included positioning for the sample, UTI symptoms making urine collection difficult, and waste disposal because of the single-use plastic in the UCDs. Conclusion: Most women agreed there was a need for a user- and environment-friendly device to improve urine collection. While using UCDs can be difficult for women experiencing UTI symptoms, they may be appropriate for asymptomatic sampling in other clinical populations

Helping parents know when to seek help for an acutely ill child: Evidence based co-development of a mobile phone app using complex intervention methodology

Background: Acute illness accounts for the majority of episodes of illness in children under five years of age and is the age group with the highest consultation rate in general practice in the UK. The number of children presenting to emergency care is also steadily increasing, having risen beyond pre-pandemic numbers. Such high, and increasing, rates of consultation have prompted concerns about parents’ level of knowledge and confidence in caring for their children when they are ill, and particularly when and how to seek help appropriately. Aim: The ASK SNIFF collaboration research programme identified parents’ need for accurate and accessible information to help them know when to seek help for a sick child in 2010. This paper presents the resulting programme of research which aimed to co-develop an evidence-based safety netting intervention (mobile app) to help parents know when to seek help for an acutely ill child under the age of five years in the UK. Methods: Our programme used a collaborative six step process with 147 parent and 324 health professional participants over a period of six years including: scoping existing interventions, systematic review, qualitative research, video capture, content identification and development, consensus methodology, parent and expert clinical review. Results: Our programme has produced evidence-based content for an app supported by video clips. Our collaborative approach has supported every stage of our work, ensuring that the end result reflects the experiences, perspectives and expressed needs of parents and the clinicians they consult. Conclusion: We have not found any other resource which has used this type of approach, which may explain why there is no published evaluation data demonstrating the impact of existing UK resources. Future mobile apps should be designed and developed with the service users for whom they are intended

Severe Scene Learning Impairment, but Intact Recognition Memory, after Cholinergic Depletion of Inferotemporal Cortex Followed by Fornix Transection

To examine the generality of cholinergic involvement in visual memory in primates, we trained macaque monkeys either on an object-in-place scene learning task or in delayed nonmatching-to-sample (DNMS). Each monkey received either selective cholinergic depletion of inferotemporal cortex (including the entorhinal cortex and perirhinal cortex) with injections of the immunotoxin ME20.4-saporin or saline injections as a control and was postoperatively retested. Cholinergic depletion of inferotemporal cortex was without effect on either task. Each monkey then received fornix transection because previous studies have shown that multiple disconnections of temporal cortex can produce synergistic impairments in memory. Fornix transection mildly impaired scene learning in monkeys that had received saline injections but severely impaired scene learning in monkeys that had received cholinergic lesions of inferotemporal cortex. This synergistic effect was not seen in monkeys performing DNMS. These findings confirm a synergistic interaction in a macaque monkey model of episodic memory between connections carried by the fornix and cholinergic input to the inferotemporal cortex. They support the notion that the mnemonic functions tapped by scene learning and DNMS have dissociable neural substrates. Finally, cholinergic depletion of inferotemporal cortex, in this study, appears insufficient to impair memory functions dependent on an intact inferotemporal cortex