A realistic evaluation : the case of protocol-based care

Background 'Protocol based care' was envisioned by policy makers as a mechanism for delivering on the service improvement agenda in England. Realistic evaluation is an increasingly popular approach, but few published examples exist, particularly in implementation research. To fill this gap, within this paper we describe the application of a realistic evaluation approach to the study of protocol-based care, whilst sharing findings of relevance about standardising care through the use of protocols, guidelines, and pathways. Methods Situated between positivism and relativism, realistic evaluation is concerned with the identification of underlying causal mechanisms, how they work, and under what conditions. Fundamentally it focuses attention on finding out what works, for whom, how, and in what circumstances. Results In this research, we were interested in understanding the relationships between the type and nature of particular approaches to protocol-based care (mechanisms), within different clinical settings (context), and what impacts this resulted in (outcomes). An evidence review using the principles of realist synthesis resulted in a number of propositions, i.e., context, mechanism, and outcome threads (CMOs). These propositions were then 'tested' through multiple case studies, using multiple methods including non-participant observation, interviews, and document analysis through an iterative analysis process. The initial propositions (conjectured CMOs) only partially corresponded to the findings that emerged during analysis. From the iterative analysis process of scrutinising mechanisms, context, and outcomes we were able to draw out some theoretically generalisable features about what works, for whom, how, and what circumstances in relation to the use of standardised care approaches (refined CMOs). Conclusions As one of the first studies to apply realistic evaluation in implementation research, it was a good fit, particularly given the growing emphasis on understanding how context influences evidence-based practice. The strengths and limitations of the approach are considered, including how to operationalise it and some of the challenges. This approach provided a useful interpretive framework with which to make sense of the multiple factors that were simultaneously at play and being observed through various data sources, and for developing explanatory theory about using standardised care approaches in practice

The implausibility of ‘usual care’ in an open system: sedation and weaning practices in Paediatric Intensive Care Units (PICUs) in the United Kingdom (UK)

Background: The power of the randomised controlled trial depends upon its capacity to operate in a closed system whereby the intervention is the only causal force acting upon the experimental group and absent in the control group, permitting a valid assessment of intervention efficacy. Conversely, clinical arenas are open systems where factors relating to context, resources, interpretation and actions of individuals will affect implementation and effectiveness of interventions. Consequently, the comparator (usual care) can be difficult to define and variable in multi-centre trials. Hence outcomes cannot be understood without considering usual care and factors that may affect implementation and impact on the intervention. Methods: Using a fieldwork approach, we describe PICU context, ‘usual’ practice in sedation and weaning from mechanical ventilation, and factors affecting implementation prior to designing a trial involving a sedation and ventilation weaning intervention. We collected data from 23 UK PICUs between June and November 2014 using observation, individual and multi-disciplinary group interviews with staff. Results: Pain and sedation practices were broadly similar in terms of drug usage and assessment tools. Sedation protocols linking assessment to appropriate titration of sedatives and sedation holds were rarely used (9 % and 4 % of PICUs respectively). Ventilator weaning was primarily a medical-led process with 39 % of PICUs engaging senior nurses in the process: weaning protocols were rarely used (9 % of PICUs). Weaning methods were variably based on clinician preference. No formal criteria or use of spontaneous breathing trials were used to test weaning readiness. Seventeen PICUs (74 %) had prior engagement in multi-centre trials, but limited research nurse availability. Barriers to previous trial implementation were intervention complexity, lack of belief in the evidence and inadequate training. Facilitating factors were senior staff buy-in and dedicated research nurse provision. Conclusions: We examined and identified contextual and organisational factors that may impact on the implementation of our intervention. We found usual practice relating to sedation, analgesia and ventilator weaning broadly similar, yet distinctively different from our proposed intervention, providing assurance in our ability to evaluate intervention effects. The data will enable us to develop an implementation plan; considering these factors we can more fully understand their impact on study outcomes

Evaluating the successful implementation of evidence into practice using the PARiHS framework : theoretical and practical challenges

Background The PARiHS framework (Promoting Action on Research Implementation in Health Services) has proved to be a useful practical and conceptual heuristic for many researchers and practitioners in framing their research or knowledge translation endeavours. However, as a conceptual framework it still remains untested and therefore its contribution to the overall development and testing of theory in the field of implementation science is largely unquantified. Discussion This being the case, the paper provides an integrated summary of our conceptual and theoretical thinking so far and introduces a typology (derived from social policy analysis) used to distinguish between the terms conceptual framework, theory and model – important definitional and conceptual issues in trying to refine theoretical and methodological approaches to knowledge translation. Secondly, the paper describes the next phase of our work, in particular concentrating on the conceptual thinking and mapping that has led to the generation of the hypothesis that the PARiHS framework is best utilised as a two-stage process: as a preliminary (diagnostic and evaluative) measure of the elements and sub-elements of evidence (E) and context (C), and then using the aggregated data from these measures to determine the most appropriate facilitation method. The exact nature of the intervention is thus determined by the specific actors in the specific context at a specific time and place. In the process of refining this next phase of our work, we have had to consider the wider issues around the use of theories to inform and shape our research activity; the ongoing challenges of developing robust and sensitive measures; facilitation as an intervention for getting research into practice; and finally to note how the current debates around evidence into practice are adopting wider notions that fit innovations more generally. Summary The paper concludes by suggesting that the future direction of the work on the PARiHS framework is to develop a two-stage diagnostic and evaluative approach, where the intervention is shaped and moulded by the information gathered about the specific situation and from participating stakeholders. In order to expedite the generation of new evidence and testing of emerging theories, we suggest the formation of an international research implementation science collaborative that can systematically collect and analyse experiences of using and testing the PARiHS framework and similar conceptual and theoretical approaches. We also recommend further refinement of the definitions around conceptual framework, theory, and model, suggesting a wider discussion that embraces multiple epistemological and ontological perspectives

FIRE (facilitating implementation of research evidence) : a study protocol

Research evidence underpins best practice, but is not always used in healthcare. The Promoting Action on Research Implementation in Health Services (PARIHS) framework suggests that the nature of evidence, the context in which it is used, and whether those trying to use evidence are helped (or facilitated) affect the use of evidence. Urinary incontinence has a major effect on quality of life of older people, has a high prevalence, and is a key priority within European health and social care policy. Improving continence care has the potential to improve the quality of life for older people and reduce the costs associated with providing incontinence aids

Role of "external facilitation" in implementation of research findings: a qualitative evaluation of facilitation experiences in the Veterans Health Administration

BACKGROUND: Facilitation has been identified in the literature as a potentially key component of successful implementation. It has not, however, either been well-defined or well-studied. Significant questions remain about the operational definition of facilitation and about the relationship of facilitation to other interventions, especially to other change agent roles when used in multi-faceted implementation projects. Researchers who are part of the Quality Enhancement Research Initiative (QUERI) are actively exploring various approaches and processes, including facilitation, to enable implementation of best practices in the Veterans Health Administration health care system – the largest integrated healthcare system in the United States. This paper describes a systematic, retrospective evaluation of implementation-related facilitation experiences within QUERI, a quality improvement program developed by the US Department of Veterans Affairs. METHODS: A post-hoc evaluation was conducted through a series of semi-structured interviews to examine the concept of facilitation across several multi-site QUERI implementation studies. The interview process is based on a technique developed in the field of education, which systematically enhances learning through experience by stimulating recall and reflection regarding past complex activities. An iterative content analysis approach relative to a set of conceptually-based interview questions was used for data analysis. FINDINGS: Findings suggest that facilitation, within an implementation study initiated by a central change agency, is a deliberate and valued process of interactive problem solving and support that occurs in the context of a recognized need for improvement and a supportive interpersonal relationship. Facilitation was described primarily as a distinct role with a number of potentially crucial behaviors and activities. Data further suggest that external facilitators were likely to use or integrate other implementation interventions, while performing this problem-solving and supportive role. PRELIMINARY CONCLUSIONS: This evaluation provides evidence to suggest that facilitation could be considered a distinct implementation intervention, just as audit and feedback, educational outreach, or similar methods are considered to be discrete interventions. As such, facilitation should be well-defined and explicitly evaluated for its perceived usefulness within multi-intervention implementation projects. Additionally, researchers should better define the specific contribution of facilitation to the success of implementation in different types of projects, different types of sites, and with evidence and innovations of varying levels of strength and complexity

Realist synthesis : illustrating the method for implementation research

BackgroundRealist synthesis is an increasingly popular approach to the review and synthesis of evidence, which focuses on understanding the mechanisms by which an intervention works (or not). There are few published examples of realist synthesis. This paper therefore fills a gap by describing, in detail, the process used for a realist review and synthesis to answer the question \u27what interventions and strategies are effective in enabling evidence-informed healthcare?\u27 The strengths and challenges of conducting realist review are also considered. MethodsThe realist approach involves identifying underlying causal mechanisms and exploring how they work under what conditions. The stages of this review included: defining the scope of the review (concept mining and framework formulation); searching for and scrutinising the evidence; extracting and synthesising the evidence; and developing the narrative, including hypotheses. ResultsBased on key terms and concepts related to various interventions to promote evidenceinformed healthcare, we developed an outcome-focused theoretical framework. Questions were tailored for each of four theory/intervention areas within the theoretical framework and were used to guide development of a review and data extraction process. The search for literature within our first theory area, change agency, was executed and the screening procedure resulted in inclusion of 52 papers. Using the questions relevant to this theory area, data were extracted by one reviewer and validated by a second reviewer. Synthesis involved organisation of extracted data into evidence tables, theming and formulation of chains of inference, linking between the chains of inference, and hypothesis formulation. The narrative was developed around the hypotheses generated within the change agency theory area. ConclusionsRealist synthesis lends itself to the review of complex interventions because it accounts for context as well as outcomes in the process of systematically and transparently synthesising relevant literature. While realist synthesis demands flexible thinking and the ability to deal with complexity, the rewards include the potential for more pragmatic conclusions than alternative approaches to systematic reviewing. A separate publication will report the findings of the review. <br /

Developing the practice context to enable more effective pain management with older people: an action research approach

Background\ud \ud This paper, which draws upon an Emancipatory Action Research (EAR) approach, unearths how the complexities of context influence the realities of nursing practice. While the intention of the project was to identify and change factors in the practice context that inhibit effective person-centred pain management practices with older people (65 years or older), reflective critical engagement with the findings identified that enhancing pain management practices with older people was dependent on cultural change in the unit as a whole. \ud \ud Methods\ud \ud An EAR approach was utilised. The project was undertaken in a surgical unit that conducted complex abdominal surgery. Eighty-five percent (n = 48) of nursing staff participated in the two-year project (05/NIR02/107). Data were obtained through the use of facilitated critical reflection with nursing staff. \ud \ud Results\ud \ud Three key themes (psychological safety, leadership, oppression) and four subthemes (power, horizontal violence, distorted perceptions, autonomy) were found to influence the way in which effective nursing practice was realised. Within the theme of 'context,' effective leadership and the creation of a psychologically safe environment were key elements in the enhancement of all aspects of nursing practice. \ud \ud Conclusions\ud \ud Whilst other research has identified the importance of 'practice context' and models and frameworks are emerging to address this issue, the theme of 'psychological safety' has been given little attention in the knowledge translation/implementation literature. Within the principles of EAR, facilitated reflective sessions were found to create 'psychologically safe spaces' that supported practitioners to develop effective person-centred nursing practices in complex clinical environments

Acute effects of nicotine on visual search tasks in young adult smokers

Rationale Nicotine is known to improve performance on tests involving sustained attention and recent research suggests that nicotine may also improve performance on tests involving the strategic allocation of attention and working memory. Objectives We used measures of accuracy and response latency combined with eye-tracking techniques to examine the effects of nicotine on visual search tasks. Methods In experiment 1 smokers and non-smokers performed pop-out and serial search tasks. In experiment 2, we used a within-subject design and a more demanding search task for multiple targets. In both studies, 2-h abstinent smokers were asked to smoke one of their own cigarettes between baseline and tests. Results In experiment 1, pop-out search times were faster after nicotine, without a loss in accuracy. Similar effects were observed for serial searches, but these were significant only at a trend level. In experiment 2, nicotine facilitated a strategic change in eye movements resulting in a higher proportion of fixations on target letters. If the cigarette was smoked on the first trial (when the task was novel), nicotine additionally reduced the total number of fixations and refixations on all letters in the display. Conclusions Nicotine improves visual search performance by speeding up search time and enabling a better focus of attention on task relevant items. This appears to reflect more efficient inhibition of eye movements towards task irrelevant stimuli, and better active maintenance of task goals. When the task is novel, and therefore more difficult, nicotine lessens the need to refixate previously seen letters, suggesting an improvement in working memory

Measuring the context of care in an Australian acute care hospital: a nurse survey

BACKGROUND: This study set out to achieve three objectives: to test the application of a context assessment tool in an acute hospital in South Australia; to use the tool to compare context in wards that had undergone an evidence implementation process with control wards; and finally to test for relationships between demographic variables (in particular experience) of nurses being studied (n = 422) with the dimensions of context. METHODS: The Alberta Context Tool (ACT) was administered to all nursing staff on six control and six intervention wards. A total of 217 (62%) were returned (67% from the intervention wards and 56% from control wards). Data were analysed using Stata (v9). The effect of the intervention was analysed using nested (hierarchical) analysis of variance; relationships between nurses' experience and context was examined using canonical correlation analysis. RESULTS: Results confirmed the adaptation and fit of the ACT to one acute care setting in South Australia. There was no difference in context scores between control and intervention wards. However, the tool identified significant variation between wards in many of the dimensions of context. Though significant, the relationship between nurses' experience and context was weak, suggesting that at the level of the individual nurse, few factors are related to context. CONCLUSIONS: Variables operating at the level of the individual showed little relationship with context. However, the study indicated that some dimensions of context (e.g., leadership, culture) vary at the ward level, whereas others (e.g., structural and electronic resources) do not. The ACT also raised a number of interesting speculative hypotheses around the relationship between a measure of context and the capability and capacity of staff to influence it.Timothy J. Schultz and Alison L. Kitso