Assessing mental health clinicians’ intentions to adopt evidence-based treatments: reliability and validity testing of the evidence-based treatment intentions scale

Background: Intentions play a central role in numerous empirically supported theories of behavior and behavior change and have been identified as a potentially important antecedent to successful evidence-based treatment (EBT) implementation. Despite this, few measures of mental health clinicians’ EBT intentions exist and available measures have not been subject to thorough psychometric evaluation or testing. This paper evaluates the psychometric properties of the evidence-based treatment intentions (EBTI) scale, a new measure of mental health clinicians’ intentions to adopt EBTs. Methods: The study evaluates the reliability and validity of inferences made with the EBTI using multi-method, multi-informant criterion variables collected over 12 months from a sample of 197 mental health clinicians delivering services in 13 mental health agencies. Structural, predictive, and discriminant validity evidence is assessed. Results: Findings support the EBTI’s factor structure (χ2 = 3.96, df= 5, p = .556) and internal consistency reliability (α = .80). Predictive validity evidence was provided by robust and significant associations between EBTI scores and clinicians’ observer-reported attendance at a voluntary EBT workshop at a 1-month follow-up (OR = 1.92, p \u3c .05), self-reported EBT adoption at a 12-month follow-up (R2 = .17, p \u3c .001), and self-reported use of EBTs with clients at a 12-month follow-up (R2 = .25, p \u3c .001). Discriminant validity evidence was provided by small associations with clinicians’ concurrently measured psychological work climate perceptions of functionality (R2 = .06, p \u3c .05), engagement (R2 = .06, p \u3c .05), and stress (R2 = .00, ns). Conclusions: The EBTI is a practical and theoretically grounded measure of mental health clinicians’ EBT intentions. Scores on the EBTI provide a basis for valid inferences regarding mental health clinicians’ intentions to adopt EBTs. Discussion focuses on research and practice applications

Feasibility and Acceptability of Two Incentive-Based Implementation Strategies for Mental Health Therapists Implementing Cognitive-Behavioral Therapy: A Pilot Study to Inform a Randomized Controlled Trial

Background: Informed by our prior work indicating that therapists do not feel recognized or rewarded for implementation of evidence-based practices, we tested the feasibility and acceptability of two incentive-based implementation strategies that seek to improve therapist adherence to cognitive-behavioral therapy for youth, an evidence-based practice. Methods: This study was conducted over 6 weeks in two community mental health agencies with therapists (n = 11) and leaders (n = 4). Therapists were randomized to receive either a financial or social incentive if they achieved a predetermined criterion on adherence to cognitive-behavioral therapy. In the first intervention period (block 1; 2 weeks), therapists received the reward they were initially randomized to if they achieved criterion. In the second intervention period (block 2; 2 weeks), therapists received both rewards if they achieved criterion. Therapists recorded 41 sessions across 15 unique clients over the project period. Primary outcomes included feasibility and acceptability. Feasibility was assessed quantitatively. Fifteen semi-structured interviews were conducted with therapists and leaders to assess acceptability. Difference in therapist adherence by condition was examined as an exploratory outcome. Adherence ratings were ascertained using an established and validated observational coding system of cognitivebehavioral therapy. Results: Both implementation strategies were feasible and acceptable—however, modifications to study design for the larger trial will be necessary based on participant feedback. With respect to our exploratory analysis, we found a trend suggesting the financial reward may have had a more robust effect on therapist adherence than the social reward. Conclusions: Incentive-based implementation strategies can be feasibly administered in community mental health agencies with good acceptability, although iterative pilot work is essential. Larger, fully powered trials are needed to compare the effectiveness of implementation strategies to incentivize and enhance therapists’ adherence to evidence-based practices such as cognitive-behavioral therapy

A Repeated Cross-Sectional Study of Clinicians’ Use of Psychotherapy Techniques During 5 Years of a System-Wide Effort to Implement Evidence-Based Practices in Philadelphia

Background: Little work investigates the effect of behavioral health system efforts to increase use of evidence-based practices or how organizational characteristics moderate the effect of these efforts. The objective of this study was to investigate clinician practice change in a system encouraging implementation of evidence-based practices over 5 years and how organizational characteristics moderate this effect. We hypothesized that evidence-based techniques would increase over time, whereas use of non-evidence-based techniques would remain static. Method: Using a repeated cross-sectional design, data were collected three times from 2013 to 2017 in Philadelphia’s public behavioral health system. Clinicians from 20 behavioral health outpatient clinics serving youth were surveyed three times over 5 years (n = 340; overall response rate = 60%). All organizations and clinicians were exposed to system-level support provided by the Evidence-based Practice Innovation Center from 2013 to 2017. Additionally, approximately half of the clinicians participated in city-funded evidence-based practice training initiatives. The main outcome included clinician self-reported use of cognitive-behavioral and psychodynamic techniques measured by the Therapy Procedures Checklist-Family Revised. Results: Clinicians were 80% female and averaged 37.52 years of age (SD = 11.40); there were no significant differences in clinician characteristics across waves (all ps \u3e .05). Controlling for organizational and clinician covariates, average use of CBT techniques increased by 6% from wave 1 (M= 3.18) to wave 3 (M= 3.37, p = .021, d = .29), compared to no change in psychodynamic techniques (p = .570). Each evidence-based practice training initiative in which clinicians participated predicted a 3% increase in CBT use (p = .019) but no change in psychodynamic technique use (p = .709). In organizations with more proficient cultures at baseline, clinicians exhibited greater increases in CBT use compared to organizations with less proficient cultures (8% increase vs. 2% decrease, p = .048). Conclusions: System implementation of evidence-based practices is associated with modest changes in clinician practice; these effects are moderated by organizational characteristics. Findings identify preliminary targets to improve implementation

Sustainable Deimplementation of Continuous Pulse Oximetry Monitoring in Children Hospitalized with Bronchiolitis: Study Protocol for the Eliminating Monitor Overuse (EMO) Type III Effectiveness-Deimplementation Cluster-Randomized Trial

Background: Methods of sustaining the deimplementation of overused medical practices (i.e., practices not supported by evidence) are understudied. In pediatric hospital medicine, continuous pulse oximetry monitoring of children with the common viral respiratory illness bronchiolitis is recommended only under specific circumstances. Three national guidelines discourage its use for children who are not receiving supplemental oxygen, but guideline-discordant practice (i.e., overuse) remains prevalent. A 6-hospital pilot of educational outreach with audit and feedback resulted in immediate reductions in overuse; however, the best strategies to optimize sustainment of deimplementation success are unknown. Methods: The Eliminating Monitor Overuse (EMO) trial will compare two deimplementation strategies in a hybrid type III effectiveness-deimplementation trial. This longitudinal cluster-randomized design will be conducted in Pediatric Research in Inpatient Settings (PRIS) Network hospitals and will include baseline measurement, active deimplementation, and sustainment phases. After a baseline measurement period, 16–19 hospitals will be randomized to a deimplementation strategy that targets unlearning (educational outreach with audit and feedback), and the other 16–19 will be randomized to a strategy that targets unlearning and substitution (adding an EHR-integrated clinical pathway decision support tool). The primary outcome is the sustainment of deimplementation in bronchiolitis patients who are not receiving any supplemental oxygen, analyzed as a longitudinal difference-in-differences comparison of overuse rates across study arms. Secondary outcomes include equity of deimplementation and the fidelity to, and cost of, each deimplementation strategy. To understand how the deimplementation strategies work, we will test hypothesized mechanisms of routinization (clinicians developing new routines supporting practice change) and institutionalization (embedding of practice change into existing organizational systems). Discussion: The EMO trial will advance the science of deimplementation by providing new insights into the processes, mechanisms, costs, and likelihood of sustained practice change using rigorously designed deimplementation strategies. The trial will also advance care for a high-incidence, costly pediatric lung disease. Trial registration: ClinicalTrials.gov,NCT05132322. Registered on November 10, 2021

Organizational Culture and Climate Profiles: Relationships with Fidelity to Three Evidence-Based Practices for Autism in Elementary Schools

Background: Implementation researchers have typically studied organizational culture and climate by testing whether individual dimensions are linked to the implementation of evidence-based practices (EBPs) rather than examining how the overarching social context influences implementation. This approach may limit implementation theory and strategy development to the extent that individual dimensions of culture and climate interact, mutually reinforce or counteract one another, or exhibit non-linear relationships. This study tests whether empirically identifiable culture and climate profiles emerge in a sample of organizations and examines how these profiles relate to EBP fidelity and work attitudes that support EBP sustainment, focusing on three EBPs for youth with autism delivered in schools as an example. Methods: The study included 65 elementary schools in the U.S. that implemented three EBPs—discrete trial training, pivotal response training, and visual schedules—for youth with autism. Organizational culture and climate and work attitudes were assessed using the Organizational Social Context measure at the beginning of the school year. Observations of EBP fidelity occurred mid school-year. We used bias-adjusted stepwise latent profile modeling to (1) identify subpopulations of schools that share similar culture and climate profiles, and (2) test for mean differences across profiles in observed EBP fidelity and teacher and staff work attitudes. Results: Controlling for region, four profiles best characterized the organizational cultures and climates of schools. Teachers and staff in schools with a comprehensive profile (high proficiency culture, positive climate) exhibited higher fidelity to two of three EBPs (d’s = .95 to 1.64) and reported superior work attitudes (d’s = .71 to 1.93) than teachers and staff in all other schools. Teachers and staff in supportive schools (low rigidity culture, positive climate) had better work attitudes, but not better fidelity, than those in schools with indifferent (low culture/climate, elevated stress) and constrained (high rigidity and resistance, high stress) profiles. Conclusions: Organizational culture and climate profiles are a strong predictor of EBP fidelity and work attitudes that support EBP sustainment, highlighting the importance of an organization’s overarching social context when developing implementation theory and strategies. Strategies that foster a comprehensive profile may improve EBP implementation

Navigating the Storm: How Proficient Organizational Culture Promotes Clinician Retention in the Shift to Evidence-Based Practice

Objective Clinician turnover is a major concern as mental health systems and organizations invest substantial resources in the implementation of evidence-based practice (EBP). In this study, we identify malleable factors associated with reduced clinician turnover during a systemwide EBP implementation initiative. Specifically, we examine how proficient organizational culture (i.e., norms and behavioral expectations that clinicians prioritize improvement in client well-being and exhibit competence in up-to-date treatment practices), EBP implementation climate (i.e., perceptions that the organization’s policies, procedures, and practices support EBP use), and change in these organizational characteristics relate to clinician turnover during a system-wide EBP transformation. Method Data were collected from 236 clinicians in 19 mental health clinics across 3 years of a system- wide EBP implementation initiative in the City of Philadelphia. Clinicians reported on proficient organizational culture and EBP implementation climate at baseline (T1) and twoyear follow-up (T2). Administrators reported on clinician turnover at three-year follow-up (T3). Hypotheses were tested via multilevel mediation analyses incorporating mixed effects logistic regression models. Results Controlling for organization size, clinician job satisfaction, attitudes towards EBP, job tenure, and age, higher levels of proficient organizational culture and improvement in proficient culture from baseline to two-year follow-up predicted reduced clinician turnover in the year following; these effects were mediated by EBP implementation climate and by improvement in EBP implementation climate, respectively. Conclusions Organizations with more proficient cultures have more supportive EBP implementation climates that predict reduced clinician turnover during system-wide EBP implementation initiatives. Strategies that target these antecedents in mental health service organizations may contribute to reduced clinician turnover

A Preliminary Evaluation: Demographic and Clinical Profiles and Changes in Functioning in Children Receiving Psychosocial Rehabilitation

The present study is the first to examine the demographic and clinical profiles at intake of children with emotional disturbances who received Child Psychosocial Rehabilitation (CPSR), a relatively new treatment for children suffering with emotional disturbance(ED). Fifty-three children ranging in age from 4 to 18 years received CPSR from a for-profit outpatient child and adolescent mental health clinic located in southwestern Idaho for a minimum of six months. The children\u27s demographic and clinical profiles were examined. In addition, the relationship between the relative change in psychological, emotional, and behavioral functioning as measured by CAFAS (Hodges, 1989, 1994) and PECFAS (Hodges, 1994) scores and the children\u27s age, gender, ethnicity, current living arrangement, type of mental health diagnosis, and severity of impairment at intake were evaluated. Significant improvement in the children\u27s overall functioning was found after six months of treatment. No significant between-group differences were found for gender, age, ethnicity, living situation, diagnosis, number of diagnoses, and overall impairment at intake

Testing a Theory of Strategic Implementation Leadership, Implementation Climate, and Clinicians’ Use of Evidence-Based Practice: A 5-Year Panel Analysis

Background: Implementation theory suggests that first-level leaders, sometimes referred to as middle managers, can increase clinicians’ use of evidence-based practice (EBP) in healthcare settings by enacting specific leadership behaviors (i.e., proactive, knowledgeable, supportive, perseverant with regard to implementation) that develop an EBP implementation climate within the organization; however, longitudinal and quasi-experimental studies are needed to test this hypothesis. Methods: Using data collected at three waves over a 5-year period from a panel of 30 outpatient children’s mental health clinics employing 496 clinicians, we conducted a quasi-experimental difference-in-differences study to test whether within-organization change in implementation leadership predicted within-organization change in EBP implementation climate, and whether change in EBP implementation climate predicted within-organization change in clinicians’ use of EBP. At each wave, clinicians reported on their first-level leaders’ implementation leadership, their organization’s EBP implementation climate, and their use of both EBP and non-EBP psychotherapy techniques for childhood psychiatric disorders. Hypotheses were tested using econometric two-way fixed effects regression models at the organization level which controlled for all stable organizational characteristics, population trends in the outcomes over time, and time-varying covariates. Results: Organizations that improved from low to high levels of implementation leadership experienced significantly greater increases in their level of EBP implementation climate (d = .92, p = .017) and within-organization increases in implementation leadership accounted for 11% of the variance in improvement in EBP implementation climate beyond all other covariates. In turn, organizations that improved from low to high levels of EBP implementation climate experienced significantly greater increases in their clinicians’ average EBP use (d = .55, p = .007) and within-organization improvement in EBP implementation climate accounted for 14% of the variance in increased clinician EBP use. Mediation analyses indicated that improvement in implementation leadership had a significant indirect effect on clinicians’ EBP use via improvement in EBP implementation climate (d = .26, 95% CI [.02 to .59]). Conclusions: When first-level leaders increase their frequency of implementation leadership behaviors, organizational EBP implementation climate improves, which in turn contributes to increased EBP use by clinicians. Trials are needed to test strategies that target this implementation leadership–EBP implementation climate mechanism

Required Sample Size to Detect Mediation in 3-Level Implementation Studies

Background: Statistical tests of mediation are important for advancing implementation science; however, little research has examined the sample sizes needed to detect mediation in 3-level designs (e.g., organization, provider, patient) that are common in implementation research. Using a generalizable Monte Carlo simulation method, this paper examines the sample sizes required to detect mediation in 3-level designs under a range of conditions plausible for implementation studies. Method: Statistical power was estimated for 17,496 3-level mediation designs in which the independent variable (X) resided at the highest cluster level (e.g., organization), the mediator (M) resided at the intermediate nested level (e.g., provider), and the outcome (Y) resided at the lowest nested level (e.g., patient). Designs varied by sample size per level, intraclass correlation coefficients of M and Y, effect sizes of the two paths constituting the indirect (mediation) effect (i.e., X→M and M→Y), and size of the direct effect. Power estimates were generated for all designs using two statistical models—conventional linear multilevel modeling of manifest variables (MVM) and multilevel structural equation modeling (MSEM)—for both 1- and 2-sided hypothesis tests. Results: For 2-sided tests, statistical power to detect mediation was sufficient (≥0.8) in only 463 designs (2.6%) estimated using MVM and 228 designs (1.3%) estimated using MSEM; the minimum number of highest-level units needed to achieve adequate power was 40; the minimum total sample size was 900 observations. For 1-sided tests, 808 designs (4.6%) estimated using MVM and 369 designs (2.1%) estimated using MSEM had adequate power; the minimum number of highest-level units was 20; the minimum total sample was 600. At least one large effect size for either the X→M or M→Y path was necessary to achieve adequate power across all conditions. Conclusions: While our analysis has important limitations, results suggest many of the 3-level mediation designs that can realistically be conducted in implementation research lack statistical power to detect mediation of highest-level independent variables unless effect sizes are large and 40 or more highest-level units are enrolled. We suggest strategies to increase statistical power for multilevel mediation designs and innovations to improve the feasibility of mediation tests in implementation research