Lessons from the evaluation of the UK's NHS R&D Implementation Methods Programme

Background: Concern about the effective use of research was a major factor behind the creation of the NHS R&D Programme in 1991. In 1994, an advisory group was established to identify research priorities in research implementation. The Implementation Methods Programme (IMP) flowed from this, and its commissioning group funded 36 projects. In 2000 responsibility for the programme passed to the National Co-ordinating Centre for NHS Service Delivery and Organisation R&D, which asked the Health Economics Research Group (HERG), Brunel University, to conduct an evaluation in 2002. By then most projects had been completed. This evaluation was intended to cover: the quality of outputs, lessons to be learnt about the communication strategy and the commissioning process, and the benefits from the projects. Methods: We adopted a wide range of quantitative and qualitative methods. They included: documentary analysis, interviews with key actors, questionnaires to the funded lead researchers, questionnaires to potential users, and desk analysis. Results: Quantitative assessment of outputs and dissemination revealed that the IMP funded useful research projects, some of which had considerable impact against the various categories in the HERG payback model, such as publications, further research, research training, impact on health policy, and clinical practice. Qualitative findings from interviews with advisory and commissioning group members indicated that when the IMP was established, implementation research was a relatively unexplored field. This was reflected in the understanding brought to their roles by members of the advisory and commissioning groups, in the way priorities for research were chosen and developed, and in how the research projects were commissioned. The ideological and methodological debates associated with these decisions have continued among those working in this field. The need for an effective communication strategy for the programme as a whole was particularly important. However, such a strategy was never developed, making it difficult to establish the general influence of the IMP as a programme. Conclusion: Our findings about the impact of the work funded, and the difficulties faced by those developing the IMP, have implications for the development of strategic programmes of research in general, as well as for the development of more effective research in this field

Who needs what from a national health research system: Lessons from reforms to the English Department of Health's R&D system

This article has been made available through the Brunel Open Access Publishing Fund.Health research systems consist of diverse groups who have some role in health research, but the boundaries around such a system are not clear-cut. To explore what various stakeholders need we reviewed the literature including that on the history of English health R&D reforms, and we also applied some relevant conceptual frameworks. We first describe the needs and capabilities of the main groups of stakeholders in health research systems, and explain key features of policymaking systems within which these stakeholders operate in the UK. The five groups are policymakers (and health care managers), health professionals, patients and the general public, industry, and researchers. As individuals and as organisations they have a range of needs from the health research system, but should also develop specific capabilities in order to contribute effectively to the system and benefit from it. Second, we discuss key phases of reform in the development of the English health research system over four decades - especially that of the English Department of Health's R&D system - and identify how far legitimate demands of key stakeholder interests were addressed. Third, in drawing lessons we highlight points emerging from contemporary reports, but also attempt to identify issues through application of relevant conceptual frameworks. The main lessons are: the importance of comprehensively addressing the diverse needs of various interacting institutions and stakeholders; the desirability of developing facilitating mechanisms at interfaces between the health research system and its various stakeholders; and the importance of additional money in being able to expand the scope of the health research system whilst maintaining support for basic science. We conclude that the latest health R&D strategy in England builds on recent progress and tackles acknowledged weaknesses. The strategy goes a considerable way to identifying and more effectively meeting the needs of key groups such as medical academics, patients and industry, and has been remarkably successful in increasing the funding for health research. There are still areas that might benefit from further recognition and resourcing, but the lessons identified, and progress made by the reforms are relevant for the design and coordination of national health research systems beyond England.This article is available through the Brunel Open Access Publishing Fund

Delivering the aims of the Collaborations for Leadership in Applied Health Research and Care: understanding their strategies and contributions

Background: In 2008, the National Institute for Health Research (NIHR) in England established nine Collaborations for Leadership in Applied Health Research and Care (CLAHRCs) to develop partnerships between universities and local NHS organisations focused on improving patient outcomes through the conduct and application of applied health research. Objectives: The study explored how effectively the CLAHRCs supported the ‘translation’ of research into patient benefit, and developed ways of doing applied research that maximised its chances of being useful to the service and the capacity of the NHS to respond. It focused on three issues: (1) how the NHS influenced the CLAHRCs, and vice versa; (2) how effective multistakeholder and multidisciplinary research and implementation teams were built in the CLAHRCs; (3) how the CLAHRCs supported the use of research knowledge to change commissioning and clinical behaviour for patient benefit. Methods: The study adopted an adaptive and emergent approach and incorporated a formative evaluation. An initial phase mapped the landscape of all nine CLAHRCs and the context within which they were established, using document analysis, workshops and interviews, and a literature review. This mapping exercise identified the three research questions that were explored in phase 2 through a stakeholder survey of six CLAHRCs, in-depth case studies of two CLAHRCs, validation interviews with all nine CLAHRCs and the NIHR, and document review. Results: (1) The local remit and the requirement for matched NHS funding enhanced NHS influence on the CLAHRCs. The CLAHRCs achieved positive change among those most directly involved, but the larger issue of whether or not the CLAHRCs can influence others in and across the NHS remains unresolved. (2) The CLAHRCs succeeded in engaging different stakeholder groups, and explored what encouraged specific groups to become involved. Being responsive to people’s concerns and demonstrating ‘quick wins’ were both important. (3) There was some evidence that academics were becoming more interested in needs-driven research, and that commissioners were seeing the CLAHRCs as a useful source of support. A growing number of completed projects had demonstrated an impact on clinical practice. Conclusions: The CLAHRCs have included NHS decision-makers in research and researchers in service decision-making, and encouraged research-informed practice. All the CLAHRCs (as collaborations) adopted relationship models. However, as the complexities of the challenges they faced became clearer, it became obvious that a focus on multidisciplinary relationships was necessary, but not sufficient on its own. Attention also has to be paid to the systems within and through which these relationships operate. Recommendations for research: Future research should compare areas with an Academic Health Science Network (AHSN) and a CLAHRC with areas with just an AHSN, to understand the difference CLAHRCs make. There should be work on understanding implementation, such as the balancing of rigour and relevance in intervention studies; systemic barriers to and facilitators of implementation; and tailoring improvement interventions. There is also a need to better understand the factors that support the explicit use of research evidence across the NHS, and the processes and mechanisms that support the sustainability and scale-up of implementation projects. Research should place emphasis on examining the role of patient and public involvement in CLAHRCs and of the relation between CLAHRCs and NHS commissioners. Funding: The NIHR Health Services and Delivery Research programme

CLAHRCs in practice: combined knowledge transfer and exchange strategies, cultural change, and experimentation

Objectives: The nine NIHR CLAHRCs are collaborations between universities and local NHS organizations that seek to improve patient outcomes through the conduct and application of applied health research. The theoretical and practical context within which the CLAHRCs were set up was characterized by a considerable degree of uncertainty, and the CLAHRCs were established as a natural experiment. Methods: We adopted a formative and emergent evaluation approach. Drawing on in-depth, multi-method case studies of two CLAHRCs we explored how they pursued their remit by supporting efforts to increase the relevance and use of health research, and building relationships. Results: Both CLAHRCs: strengthened local networks and relationships; built capacity in their local academic and NHS communities to undertake and use research that meets the needs of the service; developed research and implementation methodologies; and added to understanding of the complex relation between research and implementation. There was evidence of impact of CLAHRC projects on health and social care services. Informed by the literature on implementing collaborative research initiatives, knowledge transfer and exchange and cultural change, some key lessons can be drawn. Conclusion: The CLAHRCs pursued a strategy that can be categorized as one of flexible comprehensiveness; i.e. their programmes have been flexible and responsive and they have used a range of approaches that seek to match the diverse aspects of the complex issues they face. Key features include their work on combining a range of knowledge transfer and exchange strategies, their efforts to promote cultural change, and the freedom to experiment, learn and adapt. Although the CLAHRCs do not, by themselves, have the remit or resources to bring about wholesale service improvement in health care, they do have features that would allow them to play a key role in some of the wider initiatives that encourage innovation

Developing the protocol for the evaluation of the Health Foundation's 'Engaging with Quality Initiative': an emergent approach

In 2004 a UK charity, The Health Foundation, established the 'Engaging with Quality Initiative' to explore and evaluate the benefits of engaging clinicians in quality improvement in healthcare. Eight projects run by professional bodies or specialist societies were commissioned in various areas of acute care. A developmental approach to the initiative was adopted, accompanied by a two level evaluation: eight project self-evaluations and a related external evaluation. This paper describes how the protocol for the external evaluation was developed. The challenges faced included large variation between and within the projects (in approach, scope and context, and in understanding of quality improvement), the need to support the project teams in their self-evaluations while retaining a necessary objectivity, and the difficulty of evaluating the moving target created by the developmental approach adopted in the initiative. An initial period to develop the evaluation protocol proved invaluable in helping us to explore these issues