Evaluation of the NHS R & D implementation methods programme

Chapter 1: Background and introduction • Concern with research implementation was a major factor behind the creation of the NHS R&D Programme in 1991. In 1994 an Advisory Group was established to identify research priorities in this field. The Implementation Methods Programme (IMP) flowed from this and its Commissioning Group funded 36 projects. Funding for the IMP was capped before the second round of commissioning. The Commissioning Group was disbanded and eventually responsibility for the programme passed to the National Co-ordinating Centre for NHS Service Delivery and Organisation R&D (NCCSDO) which, when most projects had finished, asked the Health Economics Research Group (HERG) to conduct an evaluation. This was intended to cover: the quality of outputs; lessons to be learnt about the communication strategy and the commissioning process; and the benefits or payback from the projects. As agreed, the evaluation also addresses the questions of whether there should be a synthesis of the findings from the IMP and any further assessment of payback. Chapter 2: Methods • We adopted a wide range of quantitative and qualitative methods in the evaluation. They included: documentary analysis; interviews with key actors; questionnaires to the funded lead researchers; questionnaires to potential users; and desk analysis. Chapter 3: The outputs from the programme • As in previous assessments of research programmes, we first examined the outputs in terms of knowledge production and various items related to capacity to conduct further research. Although there was a high response rate to the questionnaire to lead researchers (30/36), missing responses mean that the data given below are incomplete. In the case of publications, however, we also made some use of data previously gathered by the programme office. • We attempted to identify publications that were in some way a specific product of IMP funding. About half (59) of the publications from the IMP projects are articles in peer reviewed journals. The journal used most frequently for publication, the BMJ, is also the one with the highest journal impact factor score of those publishing articles specifically from the programme. The recent publication datesof many articles reduces the value of citation analysis. Nevertheless, one article, Coulter et al, 1999, has already been cited on 53 occasions. Important publications, including No Magic Bullets (Oxman et al, 1995), are also associated with preliminary work undertaken for the IMP to assist priority setting. • Fifteen projects, with grants of over £1.3 million, have been awarded to IMP researchers by other funders for follow-on studies connected in some way to the IMP. We also collected details about some non-IMP researchers who are building on the IMP projects. • Research training provided in at least nine of the funded IMP projects is associated with higher/research degrees, including three MDs and four PhDs, that have been awarded or are being completed. Chapter 4: Disseminating and using the research findings • Limited thought had been given by the Implementation Methods Programme to dissemination strategies, but many of the individual researchers were active here. In response to the questionnaires, lead researchers reported making 92 presentations to academic audiences and 104 to practitioner/service groups. Some lead researchers showed that their effective dissemination led to utilisation of the findings. • The Commissioning Group gave some thought to the likely use that could be made of individual research projects, but there was limited systematic analysis of how the findings as a whole would be taken forward. Achieving impact is difficult in this complex field and less than a third of lead researchers claimed to have done so, but about half thought impact could be expected. Based mainly on reports from lead researchers, we give a brief account of how the findings from six projects are being utilised. • We sent electronic questionnaires to groups of potential users of selected projects but this produced a very low response rate. Our postal survey to Heads of Midwifery/researchers in perinatal care produced a higher response of 44%. Amongst those who did respond, there is quite a high level of knowledge about some of the programme’s projects and some level of existing and potential utilisation. We suggest, however, that in some cases there are difficulties in identifying how far the respondent’s focus is on the findings from the original research projects, and it how far it is on the impact of the IMP study that is about ways of influencing the uptake of such findings. Comments from several respondents showed strong support for the cutting edge nature of some of the research. Others, however, also indicated why findings might not be utilised by some practitioners. Several respondents advocated greater dissemination of the IMP. Chapter 5: Comparing applications with outputs • We attempted to compare the scores given to project applications with those given to projects based on their outputs. This exercise faced various problems. The final reports from all completed projects had in theory already been reviewed and given scores for their quality and relevance. In practice, not all final reports received scores. We added a refinement by giving further scores that incorporated the additional information we gathered about both publications and any uptake of the research findings. • Various limitations meant that we conducted this analysis on just 19 of the 36 projects. Nevertheless, the wide range of scores given to the outputs from projects indicates that some were much more successful than others. Our rather limited evidence suggests that there is some correlation between the scores for applications and those for outputs but it is small, which could be related to the difficulties encountered during commissioning. Chapter 6: Lessons learnt about the commissioning process • Those who established the IMP were aware that it was a different type of research field from those previously addressed within the NHS R&D Programme, but one regarded nonetheless as important. Within the NHS R&D Programme at that time a standard clinical RCT approach was strongly favoured. There was also, as ever, a need for quick results. • In developing an understanding of implementation the Advisory Group (AG) conducted cutting edge analysis, consulted widely and drew on a wide range of disciplines. Our interviewees generally took the view that the AG worked well in setting priorities and went as far as it could at the time, especially given the time constraints. • Based on our field work and analysis we identified a series of lessons that might inform future exercises. More attention was required to ensure that all relevant background disciplines were adequately taken into account in setting priorities and commissioning research. Some of these processes needed to be given more time than was available. Consultation needed to be organised in a sufficiently selective way to be of maximum benefit in such a complex area. A time-limited programme was not the most appropriate way to cover a field such as this. • In relation to the commissioning of the projects, we identified issues about the composition of commissioning groups and how people from different backgrounds (researchers, practitioners, managers and patient representatives) should best be involved. • In this new field the Commissioning Group (CG) had to work closely with applicants to develop some of the research applications. This raised issues about how, and when, this process should be handled. • Despite its own rationale, and for a variety of reasons, including the disbanding of the CG, the Implementation Methods Programme never developed an implementation or communication strategy for its own findings. • The general conclusion of those who had been involved with the IMP was that it worked as well as it could at the time, and that various important projects were commissioned. But it was only a start. Chapter 7: Should a synthesis of the findings from the programme and further payback analysis be undertaken? • From interviews and questionnaires we identified widespread, but not total, agreement that there should be some type of synthesis of the findings from the IMP. There is more debate about the form such a synthesis should take. There is some support for a more limited type of stock taking, but also wider backing for the inclusion of many different elements. These could include: a conceptual map of the field of research implementation; an exploration of how the findings from the IMP fit into the context of research implementation today; and an assessment of how far work is still needed in those areas where no projects were funded. One possible suggestion that might incorporate much of this thinking is for the establishment of a group or commission of leading researchers in the field. Their investigation could incorporate all these elements and attempt to show how the issues could be advanced. • We suggest that further work on assessing the payback from the IMP is probably not worthwhile unless it is undertaken as part of a wide-ranging synthesis. Chapter 8: Conclusions, lessons and recommendations • We conclude that the IMP was seen by many of those involved as a new and exciting field. Looking back, they were generally positive about what was started through the IMP. It commissioned a series of projects that produced some important, rigorous, and cutting edge research, at least some of which is making an impact. But this is a complex area in which traditional clinical research, health services research and the social sciences all have a role to play. A unique set of difficulties, as well as opportunities, was faced by those responsible for taking the programme forward. The intellectual challenges of constructing a programme to cover such a vast area with diverse and sometimes conflicting conceptual and methodological perspectives, were compounded by practical problems. These included the capping of the programme’s funding and the premature winding up of the Commissioning Group. As a result, this complex programme, which arguably needed better support than its more clinically orientated predecessors, did not receive it at some stages. Those involved in the programme had a considerable task – the difficulties of which were not completely appreciated at the time. They are clearer in retrospect and feed into the lessons and recommendations presented here, but it is recognised that a programme such as the SDO is already adopting some of the steps. • In relation to research commissioning and communication strategies for research programmes in general, we suggest it could be helpful if protocols were drawn up to cover certain potential difficulties. These include the remit and role of the various stakeholders represented on commissioning groups and the extent to which commissioning groups should be expected to support applicants with their proposals. Perhaps the key general lesson from this evaluation is the need for research programmes to have a proper communication strategy. This should target dissemination at relevant audiences and stress the desirability for contact to be made with potential users as early as possible in the process of devising a project. • Our other recommendations are more specifically relevant when the SDO Programme is considering an area such as implementation methods research. It would be desirable for more time to be made available for preparatory work than was allowed for the IMP and also scope provided for the programme to be able to re-visit issues and learn from early results. It is difficult to incorporate all the analysis that is required if a programme is operating in a time-limited way. • Our conclusion that research implementation is a crucial area for the NHS R&D Programme leads to the recommendation that more R&D activity is needed in this field in order to assist delivery of some key NHS agenda items. As a preliminary step, there is certainly scope for a type of stock taking of the findings from the IMP. On balance there seems also to be an argument for conducting a synthesis of work in the implementation field that goes beyond a mere collation of findings from the specific projects funded. If undertaken, it should fundamentally examine the current NHS needs for research on implementation and how they could be addressed in the light of the findings from the IMP and elsewhere. • Finally, we recommend that more attention should be given to the timing of evaluations such as this and that a phased approach should be adopted. Furthermore, researchers should be informed at the outset of their project about the likely requirements that might be placed upon them in terms of responding to requests for information by those conducting an evaluation.National Co-ordinating Centre for NHS Service Delivery and Organisation R&D (NCCSDO

Techniques for QCD calculations by numerical integration

Calculations of observables in quantum chromodynamics are typically performed using a method that combines numerical integrations over the momenta of final state particles with analytical integrations over the momenta of virtual particles. I describe the most important steps of a method for performing all of the integrations numerically.Comment: 36 pages with 16 postscript figure

Benefits from clinicians and healthcare organisations engaging in research

In Editor’s Choice, Godlee supports and re-emphasises the positive points about National Institute for Health Research (NIHR) clinical research networks that are made in Gulland’s article.1 2 We welcome this support for research networks and for the part they can play in a more fully integrated research and healthcare system. Research engagement by clinicians and healthcare organisations is widely held to improve health services performance. However, we found the issue to be complex in our review conducted for the NIHR Health Services and Delivery Research (HS&DR) Programme in 2012-13.3 Thirty three papers were included in the analysis, and 28 were positive about improved performance, although only seven identified improved outcomes rather than improved processes. Diverse mechanisms contributed to these improvements. In a subsequent article we consider more recent evidence,4 including that UK NHS trusts active in research have lower risk adjusted mortality for acute admissions.5 Increased attention to this issue covers not only clinician participation but also organisational developments in the NIHR and NHS, such as Collaborations for Leadership in Applied Health Research and Care (CLAHRC) and Academic Health Science Networks (AHSNs).6 7 These seek to promote better integration of research and healthcare systems by strengthening research networks, developing research capacity, and ensuring that healthcare organisations (both providers and commissioners) see research as an integral component of their overall structure. Such initiatives need to be linked to further empirical analysis that considers not only the research engagement of all relevant actors but also the organisational determinants of the impact on practice of such engagement

Next-to-leading order QCD calculations with parton showers II: soft singularities

Programs that calculate observables in quantum chromodynamics at next-to-leading order typically generate events that consist of partons rather than hadrons -- and just a few partons at that. These programs would be much more useful if the few partons were turned into parton showers, which could be given to one of the Monte Carlo event generators to produce hadron showers. In a previous paper, we have seen how to generate parton showers related to the final state collinear singularities of the perturbative calculation for the example of e+ + e- --> 3 jets. This paper discusses the treatment of the soft singularities.Comment: 26 pages with 5 figures. This version is close to the version to be publishe

Halo Effect Contamination in Assessments of Web Interface Design

This paper relies on findings and theory from both the human-computer interaction and cognitive psychology literatures in order to inquire into the extent to which the halo effect contaminates web interface design assessments. As a human cognitive bias, the halo effect manifests itself when a judge's evaluations of an entity's individual characteristics are negatively or positively distorted by the judge's overall affect toward the entity being judged. These distortions and halo-induced delusions have substantial negative implications for rational decisionmaking and the ability to objectively evaluate businesses, technologies, or other humans, and should hence be a critical consideration for both managers and organizations alike. Here we inquire into the halo effect using a controlled, randomized experiment involving more than 1,200 research subjects. Subjects' preexisting affective states were activated using polarizing issues including abortion rights, immigration policy, and gun control laws. Subjects were then asked to evaluate specific interface characteristics of six different types of websites, the textual content of which either supported or contradicted their preexisting affective beliefs. Comparing subject responses to objective control evaluations revealed strong evidence of halo effect contamination in assessments of web interface design, particularly among men. In light of the results, a theoretical framework integrating elements from cognitive and evolutionary psychology is proposed to explain the origins and purpose of the halo effect

Theory in North American Information Systems Research: A Culturomic Analysis

Since its inception, North American information systems (IS) research has relied on a broad and varied collection of theories. The core of this theoretical landscape is an important determinant of the IS research community’s identity, and, as such, researchers have discussed it extensively in recent years. Nevertheless, we know few concrete facts about the composition, consistency, or evolution of this theoretical core over time. Using a set of 318 theories in conjunction with n-gram analyses, we address these issues empirically by computationally analyzing the complete text of every research paper published in three leading North American IS journals over a 24-year period. In examining these 2,215 papers and more than 3.54 billion n-gram records, we identify the theories that constitute the overall core of North American IS research and provide insights into the evolution of that core. We further identify and quantify the nature of theoretical pluralism in North American IS research and examine the evolution of the theoretical density of IS research studies over time. Finally, our results shed light on the patterns of theory co-occurrence in North American IS research studies and demonstrate how such information can facilitate increasingly imperative efforts aimed at theory consolidation and generalization

An Inquiry into Mental Models of Web Interface Design

Here we adopt a cognitive science orientation to inquire into web interface design using the lens of mental models. A mental model is an internal cognitive representation of some aspect of the real world. The human mind automatically constructs and refines these models to provide a basis from which we can reason, make decisions, and form expectations. Even without conscious awareness, cognitive theory suggests that each web user has developed a mental model of what constitutes agreeable web interface design. But what is the nature of these mental models? Further, to what extent do people possess a general, shared mental model of web interface design? We examine these questions using a novel methodological approach in conjunction with statistical analyses and thermal imaging. We find not only that people possess a shared mental model of web interface design, but also that this shared model is remarkably cohesive from person to person

General subtraction method for numerical calculation of one-loop QCD matrix elements

We present a subtraction scheme for eliminating the ultraviolet, soft, and collinear divergences in the numerical calculation of an arbitrary one-loop QCD amplitude with an arbitrary number of external legs. The subtractions consist of local counter terms in the space of the four-dimensional loop momentum. The ultraviolet subtraction terms reproduce MSbar renormalization. The key point in the method for the soft and collinear subtractions is that, although the subtraction terms are defined graph-by-graph and the matrix element is also calculated graph-by-graph, the sum over graphs of the integral of each the subtraction term can be evaluated analytically and provides the well known simple pole structure that arises from subtractions from real emission graphs, but with the opposite sign.Comment: 38 pages, 10 figures, axodraw styl

A Tree-Loop Duality Relation at Two Loops and Beyond

The duality relation between one-loop integrals and phase-space integrals, developed in a previous work, is extended to higher-order loops. The duality relation is realized by a modification of the customary +i0 prescription of the Feynman propagators, which compensates for the absence of the multiple-cut contributions that appear in the Feynman tree theorem. We rederive the duality theorem at one-loop order in a form that is more suitable for its iterative extension to higher-loop orders. We explicitly show its application to two- and three-loop scalar master integrals, and we discuss the structure of the occurring cuts and the ensuing results in detail.Comment: 20 pages. Few typos corrected, some additional comments included, Appendix B and one reference added. Final version as published in JHE