The Integration of the Program Evaluation Standards into an Evaluation Toolkit for a Transformative Model of Care for Mental Health Service Delivery

Background: Stepped Care 2.0 (SC2.0) is a transformative model of mental health service delivery. This model was created by Stepped Care Solutions (SCS), a not-for-profit consultancy that collaborates with governments, public service organizations, and other institutions that wish to redesign their mental health and addictions systems of care. The SC2.0 model is based on 10 foundational principles and 9 core components that can be flexibly adapted to an organization’s or community’s needs. The model supports groups to reorganize and deliver mental health care in an evidence-informed, person-centric way. SCS partnered with evaluators from the Centre for Health Evaluation and Outcome Sciences (CHÉOS) to create a toolkit that provides evaluation guidance. The toolkit includes a theory of change, guidance on selecting evaluation questions and designs, and an evaluation matrix including suggested process and outcome metrics, all of which can be tailored to each unique implementation of the SC2.0 model. The objective of this resource is to support organizations and communities to conduct high-quality evaluations for the purpose of continuous improvement (a core component of the model of care) and to assess the model’s impact. Purpose: The purpose of this paper is to discuss the integration of the program evaluation standards (PES) into an evaluation toolkit for SC2.0. Setting: In this paper, we describe the toolkit development, focusing on how the PES were embedded in the process and tools. We explore how the integration of the PES into the toolkit supports evaluators to enhance the quality of their evaluation planning, execution, and meta-evaluation. Intervention: Not applicable Research Design: Not applicable Data Collection and Analysis: Not applicable Findings: In this paper, we describe the toolkit development, focusing on how the PES were embedded in the process and tools. We explore how the integration of the PES into the toolkit supports evaluators to enhance the quality of their evaluation planning, execution, and meta-evaluation. Keywords: program evaluation standards; evaluation; mental healt

The Implementation Playbook: Study protocol for the development and feasibility evaluation of a digital tool for effective implementation of evidence-based innovations

BACKGROUND: Evidence-based innovations can improve health outcomes, but only if successfully implemented. Implementation can be complex, highly susceptible to failure, costly and resource intensive. Internationally, there is an urgent need to improve the implementation of effective innovations. Successful implementation is best guided by implementation science, but organizations lack implementation know-how and have difficulty applying it. Implementation support is typically shared in static, non-interactive, overly academic guides and is rarely evaluated. In-person implementation facilitation is often soft-funded, costly, and scarce. This study seeks to improve effective implementation by (1) developing a first-in-kind digital tool to guide pragmatic, empirically based and self-directed implementation planning in real-time; and (2) exploring the tool\u27s feasibility in six health organizations implementing different innovations. METHODS: Ideation emerged from a paper-based resource, The Implementation Game©, and a revision called The Implementation Roadmap©; both integrate core implementation components from evidence, models and frameworks to guide structured, explicit, and pragmatic planning. Prior funding also generated user personas and high-level product requirements. This study will design, develop, and evaluate the feasibility of a digital tool called The Implementation Playbook©. In Phase 1, user-centred design and usability testing will inform tool content, visual interface, and functions to produce a minimum viable product. Phase 2 will explore the Playbook\u27s feasibility in six purposefully selected health organizations sampled for maximum variation. Organizations will use the Playbook for up to 24 months to implement an innovation of their choosing. Mixed methods will gather: (i) field notes from implementation team check-in meetings; (ii) interviews with implementation teams about their experience using the tool; (iii) user free-form content entered into the tool as teams work through implementation planning; (iv) Organizational Readiness for Implementing Change questionnaire; (v) System Usability Scale; and (vi) tool metrics on how users progressed through activities and the time required to do so. DISCUSSION: Effective implementation of evidence-based innovations is essential for optimal health. We seek to develop a prototype digital tool and demonstrate its feasibility and usefulness across organizations implementing different innovations. This technology could fill a significant need globally, be highly scalable, and potentially valid for diverse organizations implementing various innovations

Early Psychosis Intervention-Spreading Evidence-based Treatment (EPI-SET) : Protocol for an effectiveness-implementation study of a structured model of care for psychosis in youth and emerging adults

Introduction While early psychosis intervention (EPI) has proliferated in recent years amid evidence of its effectiveness, programmes often struggle to deliver consistent, recovery-based care. NAVIGATE is a manualised model of EPI with demonstrated effectiveness consisting of four components: individualised medication management, individual resiliency training, supported employment and education and family education. We aim to implement NAVIGATE in geographically diverse EPI programmes in Ontario, Canada, evaluating implementation and its effect on fidelity to the EPI model, as well as individual-level outcomes (patient/family member-reported and interviewer-rated), system-level outcomes (captured in provincial administrative databases) and engagement of participants with lived experience. Methods and analysis This is a multisite, non-randomised pragmatic hybrid effectiveness-implementation type III mixed methods study coordinated at the Centre for Addiction and Mental Health (CAMH) in Toronto. Implementation is supported by the Provincial System Support Program, a CAMH-based programme with provincial offices across Ontario, and Extension of Community Healthcare Outcomes Ontario Mental Health at CAMH and the University of Toronto. The primary outcome is fidelity to the EPI model as measured using the First Episode Psychosis Services-Fidelity Scale. Four hundred participants in the EPI programmes will be recruited and followed using both individual-level assessments and health administrative data for 2 years following NAVIGATE initiation. People with lived experience will be engaged in all aspects of the project, including through youth and family advisory committees. Ethics and dissemination Research ethics board approval has been obtained from CAMH and institutions overseeing the local EPI programmes. Study findings will be reported in scientific journal articles and shared with key stakeholders including youth, family members, programme staff and policymakers. Trial registration number NCT03919760; Pre-results

The impact of cultural variables and multicultural competence: a model of early therapy process

The prevalence in psychotherapy research of studies conducted with European-American clients has created a need for research on the impact of ethnicity and culture on the process of psychotherapy. This study tested a model examining the impact of client acculturation, individualism-collectivism and match in worldview between client and counsellor on the development of the therapeutic alliance and premature termination. Client openness was hypothesized to mediate the relationship between the cultural variables and the working alliance. Therapist multicultural competence was tested as both a mediator and a moderator of the impact of the independent variables on client openness and the working alliance. Forty one dyads (41 clients, 15 counsellors) were followed during the early phase of therapy. Counsellors and clients completed measures of perceived counsellor multicultural competence and Individualism-Collectivism and clients also reported their current levels of acculturation. After each early session, clients reported their perceptions of openness and alliance. Counsellors indicated if clients terminated prematurely. The data supported a smaller model than originally proposed in which counsellor multicultural competence related to client openness, which in turn affected the strength of the working alliance. Multicultural competence also moderated the impact on openness and alliance for clients with a stronger heritage identity and on the alliance for less individualistic clients. In these findings, client rather than counsellor ratings predicted psychotherapy process; these two perspectives of competence were not correlated. Results suggest that multicultural competence has an important impact on openness and the working alliance with all diverse clients, more specifically with clients closer to their heritage identity and less individualistic values. The findings also indicate that client cultural variables (acculturation, individualism-collectivism andLa prépondérance des études en psychothérapie sont menées avec les clients de la culture majoritaire. Ceci a généré un besoin de saisir l'impact de la culture sur le processus thérapeutique. Cette étude a évalué un modèle qui explique l'impact des éléments tels que l'intégration, les valeurs individualistes-collectivistes et de l'accord ou désaccord des valeurs entre client et thérapeute sur le développement du rapport thérapeutique et de la fin prématurée. Le modèle suggère que le dégré de franchise du client est lié aux variables culturelles et est important au développement du rapport thérapeutique. La compétence multiculturelle du thérapeute a été évaluée en tant que médiateur et modérateur. Quarante-un clients et quinze conseillers ont été suivis durant la première phase de thérapie. Les thérapeutes et clients ont complété des questionnaires évaluant leur perspective de la compétence multiculturelle des thérapeutes ainsi que leurs propres valeurs individualistes-collectivistes. Les clients ont aussi évalué leur dégré d'intégration. Après les premiers entretiens, les clients ont évalué leur dégré de franchise et l'alliance thérapeutique. Les conseillers ont indiqué si les clients ont mis fin prématurément à la thérapie. Les données ont soutenu un modèle où la compétence multiculturelle du thérapeute a prédit la franchise du client, et par conséquent le niveau de l'alliance thérapeutique. La compétence multiculturelle a aussi tempéré l'impact de la franchise du client et de l'alliance thérapeutique pour les clients avec une plus forte identité culturelle et sur l'alliance pour les clients moins individualistes. La perception du client, plutôt que celui du thérapeute a prédit le processus de psychothérapie. De plus, ces deux perspectives de compétence n'avaient pas de corrélation. Les résultats suggèrent que la compétence multiculturelle a un impact important sur la franchis

Guiding Principles for Implementing Stepped Care in Mental Health: Aligning on the Bigger Picture

Stepped care models have the potential to provide a much needed framework for the development of comprehensive mental health systems. However, definitions of stepped care lack consistency, resulting in varying interpretations reflected in its implementation. To help foster greater alignment in research and practice, we propose a set of principles for stepped care which can provide guidance on how to bridge multiple mental health services together, reduce fragmentation, and respond to the full breadth of mental health needs along a continuum of care in diverse settings. We hope that articulating these principles will spur mental health stakeholders to translate them into actionable standards

Service providers endorse integrated services model for youth with mental health and substance use challenges: findings from a discrete choice experiment

Abstract Background Given high rates of mental health and substance challenges among youth and substantial system access barriers, system innovation is required. Integrated youth services (IYS) models aim to transform youth mental health and substance use services by creating integrative, collaborative models of care in youth-friendly settings. This study examines service provider perspectives on the key service components to include in IYS models. Method A discrete choice experiment modeled service provider preferences for the service components of IYSs. The sample includes 388 service provider/agency leader participants (age 18+) from youth-serving organizations in Ontario. Importance scores and utility values were calculated for 12 attributes represented by four levels each. Latent class analysis identified subgroups of participants with different preferences. Results The majority of participants were direct service providers working in larger organizations in the mental health and/or substance use sectors in large urban centers. Participants strongly endorsed service models that provide rapid access to the widest variety of culturally sensitive service options, with supplementary e-health services, in youth-focused community settings with evening and weekend hours. They prefer caregiver involvement in youth services and treatment decisions and support youth and family engagement. Latent class analyses reveal three segments of service providers: a Youth-Focused Service Accessibility segment representing 62.1% (241/388) of participants, a Service Options segment representing 27.6% (107/388) of participants, and a Caregiver Integration segment representing 10.3% (40/388) of participants. Within these segments, the degree of prioritization of the various service components differ; however, the overall endorsement of the service components remains largely consistent across classes for most attributes. The segments did not differ based on demographic or agency characteristics. Conclusions The core characteristics of IYS settings for youth with mental health and substance use challenges, i.e., rapid access to a wide range of youth-oriented services, are strong priorities of service providers and youth-serving agency leaders. These findings confirm that youth-oriented service providers endorse the importance and relevance of IYS models as a whole; strong service provider buy-in to the model is expected to facilitate development, implementation and scaling of IYS models. Hearing stakeholder perspectives, including those of service providers, youth, and caregivers, is essential to developing, effectively implementing, and scaling effective youth services

Implementing One-at-a-Time Therapy in community addiction and mental health centres: a retrospective exploration of the implementation process and initial outcomes

Abstract Background The Department of Health of the Government of New Brunswick and Regional Health Authorities elected to implement Stepped Care 2.0 (SC2.0) in 2021, and began with One-at-a-Time (OAAT) therapy in Community Addiction and Mental Health Centres (CAMHCs) to facilitate rapid access to addiction and mental healthcare. This study: 1) explicated the process of implementing OAAT therapy as it aligned to evidence-based implementation frameworks and strategies; 2) assessed readiness for change among providers during the implementation; and 3) evaluated initial client and system outcomes. Methods The process of implementing OAAT therapy within CAMHCs was documented and retrospectively aligned with the Active Implementation Frameworks-Stages of Implementation, Consolidated Framework for Implementation Research, and incorporated strategies endorsed by the Expert Recommendations for Implementing Change. Providers working in CAMHCs completed online asynchronous courses in OAAT therapy and SC2.0, and were recruited to participate in research on perceptions of organizational readiness. Initial outcomes of the implementation were evaluated through client satisfaction surveys administered in CAMHCs and system performance indicators. Results Aligning with implementation stages, key strategies included: 1) continuously monitoring readiness and soliciting stakeholder feedback for iterative improvement; 2) building a representative implementation team with engaged leaders; 3) creating a comprehensive implementation plan on staff training, communication, and system changes; and 4) supporting sustainability. Providers who participated in research (N = 170, ~ 50% response rate) agreed that their organization was ready for implementation, and that OAAT therapy delivered within a SC2.0 framework was acceptable, appropriate, and feasible. More than 3,600 OAAT therapy sessions were delivered during the initial implementation stage, and waitlists were reduced by 64.1%. The majority of clients who completed surveys (N = 1240, ~ 35% response rate) reported that their OAAT therapy session was helpful, with a minority reporting that additional intervention was needed. Conclusions Thoughtful planning and execution, aligned with evidence-based implementation frameworks and strategies, played an important role in this provincial change initiative. Implementation steps outlined can help inform others looking to enact large-scale change