The generation of consensus guidelines for carrying out process evaluations in rehabilitation research

Abstract Background Although in recent years there has been a strong increase in published research on theories (e.g. realist evaluation, normalization process theory) driving and guiding process evaluations of complex interventions, there is limited guidance to help rehabilitation researchers design and carry out process evaluations. This can lead to the risk of process evaluations being unsystematic. This paper reports on the development of new consensus guidelines that address the specific challenges of conducting process evaluations alongside clinical trials of rehabilitation interventions. Methods A formal consensus process was carried out based on a modified nominal group technique, which comprised two phases. Phase I was informed by the findings of a systematic review, and included a nominal group meeting with an expert panel of participants to rate and discuss the proposed statements. Phase II was an in depth semi-structured telephone interviews with expert panel participants in order to further discuss the structure and contents of the revised guidelines. Frequency of rating responses to each statement was calculated and thematic analysis was carried out on all qualitative data. Results The guidelines for carrying out process evaluations within complex intervention rehabilitation research were produced by combining findings from Phase I and Phase II. The consensus guidelines include recommendations that are grouped in seven sections. These sections are theoretical work, design and methods, context, recruitment and retention, intervention staff, delivery of the intervention and results. These sections represent different aspects or stages of the evaluation process. Conclusion The consensus guidelines here presented can play a role at assisting rehabilitation researchers at the time of designing and conducting process evaluations alongside trials of complex interventions. The guidelines break new ground in terms of concepts and theory and works towards a consensus in regards to how rehabilitation researchers should go about carrying out process evaluations and how this evaluation should be linked into the proposed trials. These guidelines may be used, adapted and tested by rehabilitation researchers depending on the research stage or study design (e.g. feasibility trial, pilot trial, etc.)

The development and theoretical application of an implementation framework for dialectical behaviour therapy:a critical literature review

Background: Dialectical behaviour therapy (DBT) is a third wave behaviour therapy combining behaviour based components with elements of mindfulness. Although DBT effectiveness has been explored, relatively little attention has been given to its implementation. Frameworks are often the basis for gathering information about implementation and can also direct how the implementation of an intervention is conducted. Using existing implementation frameworks, this critical literature review scoped the DBT implementation literature to develop and refine a bespoke DBT implementation framework. Method and results: The initial framework was developed by consolidating existing implementation frameworks and published guidance on DBT implementation. The critical literature review retrieved papers from Medline, CINAHL, PsycInfo, PubMed, and the reference lists of included papers. Framework elements were used as codes which were applied to the literature and guided the synthesis. Findings from the synthesis refined the framework.The critical literature review retrieved 60 papers but only 14 of these explicitly focused on implementation. The DBT implementation framework captured all the execution barriers and facilitators described in the literature. However, the evidence synthesis led to a more parsimonious framework as some elements (e.g., research and published guidance) were seldom discussed in DBT implementation. Conclusion: To our knowledge this is the first published review exploring DBT implementation. The literature synthesis suggests some tentative recommendations which warrant further exploration. For instance, if DBT implementation is not pre-planned, having someone in the organisation who champions DBT can be advantageous. However, as the literature is limited and has methodological limitations, further prospective studies of DBT implementation are needed

Exploring the use of Soft Systems Methodology with realist approaches:A novel way to map programme complexity and develop and refine programme theory

As the use of realist approaches gains momentum, there is a growing interest in how systems approaches can complement realist thinking. In this article, we discuss how the epistemology of Soft Systems Methodology is compatible with realist approaches. Both Soft Systems Methodology and realist approaches emphasize the necessity to engage stakeholders; through models, thedescription of contingencies and exploring the intricacies of how complex programmes really work. We outline the key elements of realist approaches and Soft Systems Methodology, and report on two novel case studies. Drawing on our own experiences, we make the case that, used in conjunction with a realist approach, Soft Systems Methodology can provide a useful tool to a) map programme complexity, and b) develop and refine stakeholders programme theories, thus increasing the transparency, reliability, validity and accuracy of the theory building and refining process in realist approaches. We highlight Soft Systems Methodology as a novel companion to realist approaches and detail the first case studies of its use

Do guidelines for treating chest disease in children use Cochrane Reviews effectively?: a systematic review

Cochrane Reviews summarise best evidence and should inform guidelines. We assessed the use of Cochrane Reviews in the UK guidelines for paediatric respiratory disease. We found 21 guidelines which made 1025 recommendations, of which 96 could be informed by a Cochrane Review. In 38/96 recommendations (40%), some or all of the relevant Cochrane Reviews were not cited. We linked recommendations to 140 Cochrane Reviews. In 37/140 (26%) cases, the guideline recommendation did not fully agree with the Cochrane Review. Guideline developers may fail to use Cochrane Reviews or may make recommendations which are not in line with best evidence

‘Mind the gaps’: the accessibility and implementation of an effective depression relapse prevention programme in UK NHS services: learning from mindfulness-based cognitive therapy through a mixed-methods study

Objectives Mindfulness-based cognitive therapy (MBCT) is an evidence-based approach for people at risk of depressive relapse to support their long-term recovery. However, despite its inclusion in guidelines, there is an ‘implementation cliff’. The study objective was to develop a better explanation of what facilitates MBCT implementation. Setting UK primary and secondary care mental health services. Design, participants and methods A national two-phase, multi-method qualitative study was conducted, which was conceptually underpinned by the Promoting Action on Research Implementation in Health Services framework. Phase I involved interviews with stakeholders from 40 service providers about current provision of MBCT. Phase II involved 10 purposively sampled case studies to obtain a more detailed understanding of MBCT implementation. Data were analysed using adapted framework analysis, refined through stakeholder consultation. Results Access to MBCT is variable across the UK services. Where available, services have adapted MBCT to fit their context by integrating it into their care pathways. Evidence was often important to implementation but took different forms: the NICE depression guideline, audits, evaluations, first person accounts, experiential taster sessions and pilots. These were used to build a platform from which to develop MBCT services. The most important aspect of facilitation was the central role of the MBCT implementers. These were generally self-designated individuals who ‘championed’ grass-roots implementation. Our explanatory framework mapped out a prototypical implementation journey, often over many years with a balance of bottom-up and top-down factors influencing the fit of MBCT into service pathways. ‘Pivot points’ in the implementation journey provided windows of either challenge or opportunity. Conclusions This is one of the largest systematic studies of the implementation of a psychological therapy. While access to MBCT across the UK is improving, it remains patchy. The resultant explanatory framework about MBCT implementation provides a heuristic that informed an implementation resource

Role of "external facilitation" in implementation of research findings: a qualitative evaluation of facilitation experiences in the Veterans Health Administration

BACKGROUND: Facilitation has been identified in the literature as a potentially key component of successful implementation. It has not, however, either been well-defined or well-studied. Significant questions remain about the operational definition of facilitation and about the relationship of facilitation to other interventions, especially to other change agent roles when used in multi-faceted implementation projects. Researchers who are part of the Quality Enhancement Research Initiative (QUERI) are actively exploring various approaches and processes, including facilitation, to enable implementation of best practices in the Veterans Health Administration health care system – the largest integrated healthcare system in the United States. This paper describes a systematic, retrospective evaluation of implementation-related facilitation experiences within QUERI, a quality improvement program developed by the US Department of Veterans Affairs. METHODS: A post-hoc evaluation was conducted through a series of semi-structured interviews to examine the concept of facilitation across several multi-site QUERI implementation studies. The interview process is based on a technique developed in the field of education, which systematically enhances learning through experience by stimulating recall and reflection regarding past complex activities. An iterative content analysis approach relative to a set of conceptually-based interview questions was used for data analysis. FINDINGS: Findings suggest that facilitation, within an implementation study initiated by a central change agency, is a deliberate and valued process of interactive problem solving and support that occurs in the context of a recognized need for improvement and a supportive interpersonal relationship. Facilitation was described primarily as a distinct role with a number of potentially crucial behaviors and activities. Data further suggest that external facilitators were likely to use or integrate other implementation interventions, while performing this problem-solving and supportive role. PRELIMINARY CONCLUSIONS: This evaluation provides evidence to suggest that facilitation could be considered a distinct implementation intervention, just as audit and feedback, educational outreach, or similar methods are considered to be discrete interventions. As such, facilitation should be well-defined and explicitly evaluated for its perceived usefulness within multi-intervention implementation projects. Additionally, researchers should better define the specific contribution of facilitation to the success of implementation in different types of projects, different types of sites, and with evidence and innovations of varying levels of strength and complexity

A realist review of interventions and strategies to promote evidence-informed healthcare: a focus on change agency

Background Change agency in its various forms is one intervention aimed at improving the effectiveness of the uptake of evidence. Facilitators, knowledge brokers and opinion leaders are examples of change agency strategies used to promote knowledge utilization. This review adopts a realist approach and addresses the following question: What change agency characteristics work, for whom do they work, in what circumstances and why?Methods The literature reviewed spanned the period 1997-2007. Change agency was operationalized as roles that are aimed at effecting successful change in individuals and organizations. A theoretical framework, developed through stakeholder consultation formed the basis for a search for relevant literature. Team members, working in sub groups, independently themed the data and developed chains of inference to form a series of hypotheses regarding change agency and the role of change agency in knowledge use.Results 24, 478 electronic references were initially returned from search strategies. Preliminary screening of the article titles reduced the list of potentially relevant papers to 196. A review of full document versions of potentially relevant papers resulted in a final list of 52 papers. The findings add to the knowledge of change agency as they raise issues pertaining to how change agents&rsquo; function, how individual change agent characteristics effect evidence-informed health care, the influence of interaction between the change agent and the setting and the overall effect of change agency on knowledge utilization. Particular issues are raised such as how accessibility of the change agent, their cultural compatibility and their attitude mediate overall effectiveness. Findings also indicate the importance of promoting reflection on practice and role modeling. The findings of this study are limited by the complexity and diversity of the change agency literature, poor indexing of literature and a lack of theory-driven approaches.Conclusion This is the first realist review of change agency. Though effectiveness evidence is weak, change agent roles are evolving, as is the literature, which requires more detailed description of interventions, outcomes measures, the context, intensity, and levels at which interventions are implemented in order to understand how change agent interventions effect evidence-informed health care.<br /

Do guidelines for treating chest disease in children use Cochrane Reviews effectively?: a systematic review

Cochrane Reviews summarise best evidence and should inform guidelines. We assessed the use of Cochrane Reviews in the UK guidelines for paediatric respiratory disease. We found 21 guidelines which made 1025 recommendations, of which 96 could be informed by a Cochrane Review. In 38/96 recommendations (40%), some or all of the relevant Cochrane Reviews were not cited. We linked recommendations to 140 Cochrane Reviews. In 37/140 (26%) cases, the guideline recommendation did not fully agree with the Cochrane Review. Guideline developers may fail to use Cochrane Reviews or may make recommendations which are not in line with best evidence

Facilitating implementation of research evidence (FIRE): A randomised controlled trial and process evaluation of two models of facilitation informed by the promoting action on research implementation in health services (PARIHS) framework

Background: The PARIHS framework proposes that successful implementation of research evidence results from the complex interplay between the evidence to be implemented, the context of implementation and the facilitation processes employed. Facilitation is defined as a role (the facilitator) and a process (facilitation strategies/methods). Empirical evidence comparing different facilitation approaches is limited; this paper reports a trial of two different types of facilitation represented in the PARIHS framework. Methods: A pragmatic cluster randomised controlled trial with embedded process evaluation was undertaken in 24 long-term nursing care settings in four European countries. In each country, sites were randomly allocated to standard dissemination of urinary incontinence guideline recommendations and one of two types of external-internal facilitation, labelled Type A and B. Type A facilitation was a less resource intensive approach, underpinned by improvement methodology; Type B was a more intensive, emancipatory model of facilitation, informed by critical social science. The primary outcome was percentage documented compliance with guideline recommendations. Process evaluation was framed by realist methodology and involved quantitative and qualitative data collection from multiple sources. Findings: Quantitative data were obtained from reviews of 2313 records. Qualitative data included over 332 hours of observations of care; 39 hours observation of facilitation activity; 471 staff interviews; 174 resident interviews; 120 next of kin/carer interviews; and 125 stakeholder interviews. There were no significant differences in the primary outcome between study arms and all study arms improved over time. Process data revealed three core mechanisms that influenced the trajectory of the facilitation intervention: alignment of the facilitation approach to the needs and expectations of the internal facilitator and colleagues; engagement of internal facilitators and staff in attitude and action; and learning over time. Data from external facilitators demonstrated that the facilitation interventions did not work as planned, issues were cumulative and maintenance of fidelity was problematic. Implications for D&I Research: Evaluating an intervention - in this case facilitation - that is fluid and dynamic within the methodology of a randomised controlled trial is complex and challenging. For future studies, we suggest a theoretical approach to fidelity, with a focus on mechanisms, as opposed to dose and intensity of the intervention

Improving quality of care through routine, successful implementation of evidence-based practice at the bedside: an organizational case study protocol using the Pettigrew and Whipp model of strategic change

BACKGROUND: Evidence-based practice (EBP) is an expected approach to improving the quality of patient care and service delivery in health care systems internationally that is yet to be realized. Given the current evidence-practice gap, numerous authors describe barriers to achieving EBP. One recurrently identified barrier is the setting or context of practice, which is likewise cited as a potential part of the solution to the gap. The purpose of this study is to identify key contextual elements and related strategic processes in organizations that find and use evidence at multiple levels, in an ongoing, integrated fashion, in contrast to those that do not. METHODS: The core theoretical framework for this multi-method explanatory case study is Pettigrew and Whipp's Content, Context, and Process model of strategic change. This framework focuses data collection on three entities: the Why of strategic change, the What of strategic change, and the How of strategic change, in this case related to implementation and normalization of EBP. The data collection plan, designed to capture relevant organizational context and related outcomes, focuses on eight interrelated factors said to characterize a receptive context. Selective, purposive sampling will provide contrasting results between two cases (departments of nursing) and three embedded units in each. Data collection methods will include quantitative tools (e.g., regarding culture) and qualitative approaches including focus groups, interviews, and documents review (e.g., regarding integration and “success”) relevant to the EBP initiative. DISCUSSION: This study should provide information regarding contextual elements and related strategic processes key to successful implementation and sustainability of EBP, specifically in terms of a pervasive pattern in an acute care hospital-based health care setting. Additionally, this study will identify key contextual elements that differentiate successful implementation and sustainability of EBP efforts, both within varying levels of a hospital-based clinical setting and across similar hospital settings interested in EBP