The Sustainability of New Programs and Innovations: A Review of the Empirical Literature and Recommendations for Future Research

Background: The introduction of evidence-based programs and practices into healthcare settings has been the subject of an increasing amount of research in recent years. While a number of studies have examined initial implementation efforts, less research has been conducted to determine what happens beyond that point. There is increasing recognition that the extent to which new programs are sustained is influenced by many different factors and that more needs to be known about just what these factors are and how they interact. To understand the current state of the research literature on sustainability, our team took stock of what is currently known in this area and identified areas in which further research would be particularly helpful. This paper reviews the methods that have been used, the types of outcomes that have been measured and reported, findings from studies that reported long-term implementation outcomes, and factors that have been identified as potential influences on the sustained use of new practices, programs, or interventions. We conclude with recommendations and considerations for future research. Methods: Two coders identified 125 studies on sustainability that met eligibility criteria. An initial coding scheme was developed based on constructs identified in previous literature on implementation. Additional codes were generated deductively. Related constructs among factors were identified by consensus and collapsed under the general categories. Studies that described the extent to which programs or innovations were sustained were also categorized and summarized. Results: Although “sustainability” was the term most commonly used in the literature to refer to what happened after initial implementation, not all the studies that were reviewed actually presented working definitions of the term. Most study designs were retrospective and naturalistic. Approximately half of the studies relied on self-reports to assess sustainability or elements that influence sustainability. Approximately half employed quantitative methodologies, and the remainder employed qualitative or mixed methodologies. Few studies that investigated sustainability outcomes employed rigorous methods of evaluation (e.g., objective evaluation, judgement of implementation quality or fidelity). Among those that did, a small number reported full sustainment or high fidelity. Very little research has examined the extent, nature, or impact of adaptations to the interventions or programs once implemented. Influences on sustainability included organizational context, capacity, processes, and factors related to the new program or practice themselves. Conclusions: Clearer definitions and research that is guided by the conceptual literature on sustainability are critical to the development of the research in the area. Further efforts to characterize the phenomenon and the factors that influence it will enhance the quality of future research. Careful consideration must also be given to interactions among influences at multiple levels, as well as issues such as fidelity, modification, and changes in implementation over time. While prospective and experimental designs are needed, there is also an important role for qualitative research in efforts to understand the phenomenon, refine hypotheses, and develop strategies to promote sustainment

The Transdiagnostic Intervention for Sleep and Circadian Dysfunction (TranS-C) for serious mental illness in community mental health part 3: study protocol to evaluate sustainment in a hybrid type 2 effectiveness-implementation cluster-randomized trial.

BACKGROUND: Although research on the implementation of evidence-based psychological treatments (EBPTs) has advanced rapidly, research on the sustainment of implemented EBPTs remains limited. This is concerning, given that EBPT activities and benefits regularly decline post-implementation. To advance research on sustainment, the present protocol focuses on the third and final phase-the Sustainment Phase-of a hybrid type 2 cluster-randomized controlled trial investigating the implementation and sustainment of the Transdiagnostic Intervention for Sleep and Circadian Dysfunction (TranS-C) for patients with serious mental illness and sleep and circadian problems in community mental health centers (CMHCs). Prior to the first two phases of the trial-the Implementation Phase and Train-the-Trainer Phase-TranS-C was adapted to fit the CMHC context. Then, 10 CMHCs were cluster-randomized to implement Standard or Adapted TranS-C via facilitation and train-the-trainer. The primary goal of the Sustainment Phase is to investigate whether adapting TranS-C to fit the CMHC context predicts improved sustainment outcomes. METHODS: Data collection for the Sustainment Phase will commence at least three months after implementation efforts in partnering CMHCs have ended and may continue for up to one year. CMHC providers will be recruited to complete surveys (N = 154) and a semi-structured interview (N = 40) on sustainment outcomes and mechanisms. Aim 1 is to report the sustainment outcomes of TranS-C. Aim 2 is to evaluate whether manipulating EBPT fit to context (i.e., Standard versus Adapted TranS-C) predicts sustainment outcomes. Aim 3 is to test whether provider perceptions of fit mediate the relation between treatment condition (i.e., Standard versus Adapted TranS-C) and sustainment outcomes. Mixed methods will be used to analyze the data. DISCUSSION: The present study seeks to advance our understanding of sustainment predictors, mechanisms, and outcomes by investigating (a) whether the implementation strategy of adapting an EBPT (i.e., TranS-C) to the CMHC context predicts improved sustainment outcomes and (b) whether this relation is mediated by improved provider perceptions of treatment fit. Together, the findings may help inform more precise implementation efforts that contribute to lasting change. TRIAL REGISTRATION: ClinicalTrials.gov identifier: NCT05956678 . Registered on July 21, 2023

A randomized controlled dismantling trial of post-workshop consultation strategies to increase effectiveness and fidelity to an evidence-based psychotherapy for Posttraumatic stress disorder

Abstract: Background: Posttraumatic Stress Disorder (PTSD) is a serious mental health condition with substantial costs to individuals and society. Among military veterans, the lifetime prevalence of PTSD has been estimated to be as high as 20%. Numerous research studies have demonstrated that short-term cognitive-behavioral psychotherapies, such as Cognitive Processing Therapy (CPT), lead to substantial and sustained improvements in PTSD symptoms. Despite known benefits, only a minority of clinicians provide these therapies. Transferring this research knowledge into clinical settings remains one of the largest hurdles to improving the health of veterans with PTSD. Attending a workshop alone is insufficient to promote adequate knowledge transfer and sustained skill; however, relatively little research has been conducted to identify effective post-training support strategies. Methods: The current study investigates whether clinicians receiving post-workshop support (six-month duration) will deliver CPT with greater fidelity (i.e., psychotherapy adherence and competence) and have improved patient outcomes compared with clinicians receiving no formal post-workshop support. The study conditions are: technology-enhanced group tele-consultation; standard group tele-consultation; and fidelity assessment with no consultation. The primary outcome is independent assessment (via audio-recordings) of the clinicians’ adherence and competence in delivering CPT. The secondary outcome is observed changes in patient symptoms during and following treatment as a function of clinician fidelity. Post-consultation interviews with clinicians will help identify facilitators and barriers to psychotherapy skill acquisition. The study results will inform how best to implement and transfer evidence-based psychotherapy (e.g., CPT) to clinical settings to attain comparable outcomes to those observed in research settings. Discussion: Findings will deepen our understanding of how much and what type of support is needed following a workshop to help clinicians become proficient in delivering a new protocol. Several influences on clinician learning and patient outcomes will be discussed. An evidence-based model of clinical consultation will be developed, with the ultimate goal of informing policy and influencing best practice in clinical consultation. Trial registration: ClinicalTrials.gov: NCT0186176

Evaluation of measures of sustainability and sustainability determinants for use in community, public health, and clinical settings: a systematic review

Background Sustainability is concerned with the long-term delivery and subsequent benefits of evidence-based interventions. To further this field, we require a strong understanding and thus measurement of sustainability and what impacts sustainability (i.e., sustainability determinants). This systematic review aimed to evaluate the quality and empirical application of measures of sustainability and sustainability determinants for use in clinical, public health, and community settings. Methods Seven electronic databases, reference lists of relevant reviews, online repositories of implementation measures, and the grey literature were searched. Publications were included if they reported on the development, psychometric evaluation, or empirical use of a multi-item, quantitative measure of sustainability, or sustainability determinants. Eligibility was not restricted by language or date. Eligibility screening and data extraction were conducted independently by two members of the research team. Content coverage of each measure was assessed by mapping measure items to relevant constructs of sustainability and sustainability determinants. The pragmatic and psychometric properties of included measures was assessed using the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). The empirical use of each measure was descriptively analyzed. Results A total of 32,782 articles were screened from the database search, of which 37 were eligible. An additional 186 publications were identified from the grey literature search. The 223 included articles represented 28 individual measures, of which two assessed sustainability as an outcome, 25 covered sustainability determinants and one explicitly assessed both. The psychometric and pragmatic quality was variable, with PAPERS scores ranging from 14 to 35, out of a possible 56 points. The Provider Report of Sustainment Scale had the highest PAPERS score and measured sustainability as an outcome. The School-wide Universal Behaviour Sustainability Index-School Teams had the highest PAPERS score (score=29) of the measure of sustainability determinants. Conclusions This review can be used to guide selection of the most psychometrically robust, pragmatic, and relevant measure of sustainability and sustainability determinants. It also highlights that future research is needed to improve the psychometric and pragmatic quality of current measures in this field. Trial registration This review was prospectively registered with Research Registry (reviewregistry1097), March 2021

Protocol for the ROSE sustainment (ROSES) study, a sequential multiple assignment randomized trial to determine the minimum necessary intervention to maintain a postpartum depression prevention program in prenatal clinics serving low-income women

Background: More research on sustainment of interventions is needed, especially return on investment (ROI) studies to determine cost-benefit trade-offs for effort required to sustain and how much is gained when effective programs are sustained. The ROSE sustainment (ROSES) study uses a sequential multiple assignment randomized (SMART) design to evaluate the effectiveness and cost-effectiveness of a stepwise approach to sustainment of the ROSE postpartum depression prevention program in 90 outpatient clinics providing prenatal care to pregnant women on public assistance. Postpartum depression (PPD) is common and can have lasting consequences. Outpatient clinics offering prenatal care are an opportune place to provide PPD prevention because most women visit while pregnant. The ROSE (Reach Out, Stay Strong, Essentials for mothers of newborns) program is a group educational intervention to prevent PPD, delivered during pregnancy. ROSE has been found to reduce cases of PPD in community prenatal settings serving low-income pregnant women. Methods: All 90 prenatal clinics will receive enhanced implementation as usual (EIAU; initial training + tools for sustainment). At the first time at which a clinic is determined to be at risk for failure to sustain (i.e., at 3, 6, 9, 12, and 15 months), that clinic will be randomized to receive either (1) no additional implementation support (i.e., EIAU only), or (2) low-intensity coaching and feedback (LICF). If clinics receiving LICF are still at risk at subsequent assessments, they will be randomized to either (1) EIAU + LICF only, or (2) high-intensity coaching and feedback (HICF). Additional follow-up interviews will occur at 18, 24, and 30 months, but no implementation intervention will occur after 18 months. Outcomes include (1) percent sustainment of core program elements at each time point, (2) health impact (PPD rates over time at each clinic) and reach, and (3) ROI (costs and cost-effectiveness) of each sustainment step. Hypothesized mechanisms include sustainment of capacity to deliver core elements and engagement/ownership. Discussion: This study is the first randomized trial evaluating the ROI of a stepped approach to sustainment, a critical unanswered question in implementation science. It will also advance knowledge of implementation mechanisms and clinical care for an at-risk population

Proceedings of the 3rd Biennial Conference of the Society for Implementation Research Collaboration (SIRC) 2015: advancing efficient methodologies through community partnerships and team science : Seattle, WA, USA. 24-26 September 2015.

Introduction to the 3rd Biennial Conference of the Society for Implementation Research Collaboration: advancing efficient methodologies through team science and community partnerships Cara Lewis, Doyanne Darnell, Suzanne Kerns, Maria Monroe-DeVita, Sara J. Landes, Aaron R. Lyon, Cameo Stanick, Shannon Dorsey, Jill Locke, Brigid Marriott, Ajeng Puspitasari, Caitlin Dorsey, Karin Hendricks, Andria Pierson, Phil Fizur, Katherine A. Comtois A1: A behavioral economic perspective on adoption, implementation, and sustainment of evidence-based interventions Lawrence A. Palinkas A2: Towards making scale up of evidence-based practices in child welfare systems more efficient and affordable Patricia Chamberlain A3: Mixed method examination of strategic leadership for evidence-based practice implementation Gregory A. Aarons, Amy E. Green, Mark. G. Ehrhart, Elise M. Trott, Cathleen E. Willging A4: Implementing practice change in Federally Qualified Health Centers: Learning from leaders’ experiences Maria E. Fernandez, Nicholas H. Woolf, Shuting (Lily) Liang, Natalia I. Heredia, Michelle Kegler, Betsy Risendal, Andrea Dwyer, Vicki Young, Dayna Campbell, Michelle Carvalho, Yvonne Kellar-Guenther A3: Mixed method examination of strategic leadership for evidence-based practice implementation Gregory A. Aarons, Amy E. Green, Mark. G. Ehrhart, Elise M. Trott, Cathleen E. Willging A4: Implementing practice change in Federally Qualified Health Centers: Learning from leaders’ experiences Maria E. Fernandez, Nicholas H. Woolf, Shuting (Lily) Liang, Natalia I. Heredia, Michelle Kegler, Betsy Risendal, Andrea Dwyer, Vicki Young, Dayna Campbell, Michelle Carvalho, Yvonne Kellar-Guenther A5: Efficient synthesis: Using qualitative comparative analysis and the Consolidated Framework for Implementation Research across diverse studies Laura J. Damschroder, Julie C. Lowery A6: Establishing a veterans engagement group to empower patients and inform Veterans Affairs (VA) health services research Sarah S. Ono, Kathleen F. Carlson, Erika K. Cottrell, Maya E. O’Neil, Travis L. Lovejoy A7: Building patient-practitioner partnerships in community oncology settings to implement behavioral interventions for anxious and depressed cancer survivors Joanna J. Arch, Jill L. Mitchell A8: Tailoring a Cognitive Behavioral Therapy implementation protocol using mixed methods, conjoint analysis, and implementation teams Cara C. Lewis, Brigid R. Marriott, Kelli Scott A9: Wraparound Structured Assessment and Review (WrapSTAR): An efficient, yet comprehensive approach to Wraparound implementation evaluation Jennifer Schurer Coldiron, Eric J. Bruns, Alyssa N. Hook A10: Improving the efficiency of standardized patient assessment of clinician fidelity: A comparison of automated actor-based and manual clinician-based ratings Benjamin C. Graham, Katelin Jordan A11: Measuring fidelity on the cheap Rochelle F. Hanson, Angela Moreland, Benjamin E. Saunders, Heidi S. Resnick A12: Leveraging routine clinical materials to assess fidelity to an evidence-based psychotherapy Shannon Wiltsey Stirman, Cassidy A. Gutner, Jennifer Gamarra, Dawne Vogt, Michael Suvak, Jennifer Schuster Wachen, Katherine Dondanville, Jeffrey S. Yarvis, Jim Mintz, Alan L. Peterson, Elisa V. Borah, Brett T. Litz, Alma Molino, Stacey Young McCaughanPatricia A. Resick A13: The video vignette survey: An efficient process for gathering diverse community opinions to inform an intervention Nancy Pandhi, Nora Jacobson, Neftali Serrano, Armando Hernandez, Elizabeth Zeidler- Schreiter, Natalie Wietfeldt, Zaher Karp A14: Using integrated administrative data to evaluate implementation of a behavioral health and trauma screening for children and youth in foster care Michael D. Pullmann, Barbara Lucenko, Bridget Pavelle, Jacqueline A. Uomoto, Andrea Negrete, Molly Cevasco, Suzanne E. U. Kerns A15: Intermediary organizations as a vehicle to promote efficiency and speed of implementation Robert P. Franks, Christopher Bory A16: Applying the Consolidated Framework for Implementation Research constructs directly to qualitative data: The power of implementation science in action Edward J. Miech, Teresa M. Damush A17: Efficient and effective scaling-up, screening, brief interventions, and referrals to treatment (SBIRT) training: a snowball implementation model Jason Satterfield, Derek Satre, Maria Wamsley, Patrick Yuan, Patricia O’Sullivan A18: Matching models of implementation to system needs and capacities: addressing the human factor Helen Best, Susan Velasquez A19: Agency characteristics that facilitate efficient and successful implementation efforts Miya Barnett, Lauren Brookman-Frazee, Jennifer Regan, Nicole Stadnick, Alison Hamilton, Anna Lau A20: Rapid assessment process: Application to the Prevention and Early Intervention transformation in Los Angeles County Jennifer Regan, Alison Hamilton, Nicole Stadnick, Miya Barnett, Anna Lau, Lauren Brookman-Frazee A21: The development of the Evidence-Based Practice-Concordant Care Assessment: An assessment tool to examine treatment strategies across practices Nicole Stadnick, Anna Lau, Miya Barnett, Jennifer Regan, Scott Roesch, Lauren Brookman-Frazee A22: Refining a compilation of discrete implementation strategies and determining their importance and feasibility Byron J. Powell, Thomas J. Waltz, Matthew J. Chinman, Laura Damschroder, Jeffrey L. Smith, Monica M. Matthieu, Enola K. Proctor, JoAnn E. Kirchner A23: Structuring complex recommendations: Methods and general findings Thomas J. Waltz, Byron J. Powell, Matthew J. Chinman, Laura J. Damschroder, Jeffrey L. Smith, Monica J. Matthieu, Enola K. Proctor, JoAnn E. Kirchner A24: Implementing prolonged exposure for post-traumatic stress disorder in the Department of Veterans Affairs: Expert recommendations from the Expert Recommendations for Implementing Change (ERIC) project Monica M. Matthieu, Craig S. Rosen, Thomas J. Waltz, Byron J. Powell, Matthew J. Chinman, Laura J. Damschroder, Jeffrey L. Smith, Enola K. Proctor, JoAnn E. Kirchner A25: When readiness is a luxury: Co-designing a risk assessment and quality assurance process with violence prevention frontline workers in Seattle, WA Sarah C. Walker, Asia S. Bishop, Mariko Lockhart A26: Implementation potential of structured recidivism risk assessments with justice- involved veterans: Qualitative perspectives from providers Allison L. Rodriguez, Luisa Manfredi, Andrea Nevedal, Joel Rosenthal, Daniel M. Blonigen A27: Developing empirically informed readiness measures for providers and agencies for the Family Check-Up using a mixed methods approach Anne M. Mauricio, Thomas D. Dishion, Jenna Rudo-Stern, Justin D. Smith A28: Pebbles, rocks, and boulders: The implementation of a school-based social engagement intervention for children with autism Jill Locke, Courtney Benjamin Wolk, Colleen Harker, Anne Olsen, Travis Shingledecker, Frances Barg, David Mandell, Rinad S. Beidas A29: Problem Solving Teletherapy (PST.Net): A stakeholder analysis examining the feasibility and acceptability of teletherapy in community based aging services Marissa C. Hansen, Maria P. Aranda, Isabel Torres-Vigil A30: A case of collaborative intervention design eventuating in behavior therapy sustainment and diffusion Bryan Hartzler A31: Implementation of suicide risk prevention in an integrated delivery system: Mental health specialty services Bradley Steinfeld, Tory Gildred, Zandrea Harlin, Fredric Shephard A32: Implementation team, checklist, evaluation, and feedback (ICED): A step-by-step approach to Dialectical Behavior Therapy program implementation Matthew S. Ditty, Andrea Doyle, John A. Bickel III, Katharine Cristaudo A33: The challenges in implementing muliple evidence-based practices in a community mental health setting Dan Fox, Sonia Combs A34: Using electronic health record technology to promote and support evidence-based practice assessment and treatment intervention David H. Lischner A35: Are existing frameworks adequate for measuring implementation outcomes? Results from a new simulation methodology Richard A. Van Dorn, Stephen J. Tueller, Jesse M. Hinde, Georgia T. Karuntzos A36: Taking global local: Evaluating training of Washington State clinicians in a modularized cogntive behavioral therapy approach designed for low-resource settings Maria Monroe-DeVita, Roselyn Peterson, Doyanne Darnell, Lucy Berliner, Shannon Dorsey, Laura K. Murray A37: Attitudes toward evidence-based practices across therapeutic orientations Yevgeny Botanov, Beverly Kikuta, Tianying Chen, Marivi Navarro-Haro, Anthony DuBose, Kathryn E. Korslund, Marsha M. Linehan A38: Predicting the use of an evidence-based intervention for autism in birth-to-three programs Colleen M. Harker, Elizabeth A. Karp, Sarah R. Edmunds, Lisa V. Ibañez, Wendy L. Stone A39: Supervision practices and improved fidelity across evidence-based practices: A literature review Mimi Choy-Brown A40: Beyond symptom tracking: clinician perceptions of a hybrid measurement feedback system for monitoring treatment fidelity and client progress Jack H. Andrews, Benjamin D. Johnides, Estee M. Hausman, Kristin M. Hawley A41: A guideline decision support tool: From creation to implementation Beth Prusaczyk, Alex Ramsey, Ana Baumann, Graham Colditz, Enola K. Proctor A42: Dabblers, bedazzlers, or total makeovers: Clinician modification of a common elements cognitive behavioral therapy approach Rosemary D. Meza, Shannon Dorsey, Shannon Wiltsey-Stirman, Georganna Sedlar, Leah Lucid A43: Characterization of context and its role in implementation: The impact of structure, infrastructure, and metastructure Caitlin Dorsey, Brigid Marriott, Nelson Zounlome, Cara Lewis A44: Effects of consultation method on implementation of cognitive processing therapy for post-traumatic stress disorder Cassidy A. Gutner, Candice M. Monson, Norman Shields, Marta Mastlej, Meredith SH Landy, Jeanine Lane, Shannon Wiltsey Stirman A45: Cross-validation of the Implementation Leadership Scale factor structure in child welfare service organizations Natalie K. Finn, Elisa M. Torres, Mark. G. Ehrhart, Gregory A. Aarons A46: Sustainability of integrated smoking cessation care in Veterans Affairs posttraumatic stress disorder clinics: A qualitative analysis of focus group data from learning collaborative participants Carol A. Malte, Aline Lott, Andrew J. Saxon A47: Key characteristics of effective mental health trainers: The creation of the Measure of Effective Attributes of Trainers (MEAT) Meredith Boyd, Kelli Scott, Cara C. Lewis A48: Coaching to improve teacher implementation of evidence-based practices (EBPs) Jennifer D. Pierce A49: Factors influencing the implementation of peer-led health promotion programs targeting seniors: A literature review Agathe Lorthios-Guilledroit, Lucie Richard, Johanne Filiatrault A50: Developing treatment fidelity rating systems for psychotherapy research: Recommendations and lessons learned Kevin Hallgren, Shirley Crotwell, Rosa Muñoz, Becky Gius, Benjamin Ladd, Barbara McCrady, Elizabeth Epstein A51: Rapid translation of alcohol prevention science John D. Clapp, Danielle E. Ruderman A52: Factors implicated in successful implementation: evidence to inform improved implementation from high and low-income countries Melanie Barwick, Raluca Barac, Stanley Zlotkin, Laila Salim, Marnie Davidson A53: Tracking implementation strategies prospectively: A practical approach Alicia C. Bunger, Byron J. Powell, Hillary A. Robertson A54: Trained but not implementing: the need for effective implementation planning tools Christopher Botsko A55: Evidence, context, and facilitation variables related to implementation of Dialectical Behavior Therapy: Qualitative results from a mixed methods inquiry in the Department of Veterans Affairs Sara J. Landes, Brandy N. Smith, Allison L. Rodriguez, Lindsay R. Trent, Monica M. Matthieu A56: Learning from implementation as usual in children’s mental health Byron J. Powell, Enola K. Proctor A57: Rates and predictors of implementation after Dialectical Behavior Therapy Intensive Training Melanie S. Harned, Marivi Navarro-Haro, Kathryn E. Korslund, Tianying Chen, Anthony DuBose, André Ivanoff, Marsha M. Linehan A58: Socio-contextual determinants of research evidence use in public-youth systems of care Antonio R. Garcia, Minseop Kim, Lawrence A. Palinkas, Lonnie Snowden, John Landsverk A59: Community resource mapping to integrate evidence-based depression treatment in primary care in Brazil: A pilot project Annika C. Sweetland, Maria Jose Fernandes, Edilson Santos, Cristiane Duarte, Afrânio Kritski, Noa Krawczyk, Caitlin Nelligan, Milton L. Wainberg A60: The use of concept mapping to efficiently identify determinants of implementation in the National Institute of Health--President’s Emergent Plan for AIDS Relief Prevention of Mother to Child HIV Transmission Implementation Science Alliance Gregory A. Aarons, David H. Sommerfeld, Benjamin Chi, Echezona Ezeanolue, Rachel Sturke, Lydia Kline, Laura Guay, George Siberry A61: Longitudinal remote consultation for implementing collaborative care for depression Ian M. Bennett, Rinad Beidas, Rachel Gold, Johnny Mao, Diane Powers, Mindy Vredevoogd, Jurgen Unutzer A62: Integrating a peer coach model to support program implementation and ensure long- term sustainability of the Incredible Years in community-based settings Jennifer Schroeder, Lane Volpe, Julie Steffen A63: Efficient sustainability: Existing community based supervisors as evidence-based treatment supports Shannon Dorsey, Michael D Pullmann, Suzanne E. U. Kerns, Nathaniel Jungbluth, Lucy Berliner, Kelly Thompson, Eliza Segell A64: Establishment of a national practice-based implementation network to accelerate adoption of evidence-based and best practices Pearl McGee-Vincent, Nancy Liu, Robyn Walser, Jennifer Runnals, R. Keith Shaw, Sara J. Landes, Craig Rosen, Janet Schmidt, Patrick Calhoun A65: Facilitation as a mechanism of implementation in a practice-based implementation network: Improving care in a Department of Veterans Affairs post-traumatic stress disorder outpatient clinic Ruth L. Varkovitzky, Sara J. Landes A66: The ACT SMART Toolkit: An implementation strategy for community-based organizations providing services to children with autism spectrum disorder Amy Drahota, Jonathan I. Martinez, Brigitte Brikho, Rosemary Meza, Aubyn C. Stahmer, Gregory A. Aarons A67: Supporting Policy In Health with Research: An intervention trial (SPIRIT) - protocol and early findings Anna Williamson A68: From evidence based practice initiatives to infrastructure: Lessons learned from a public behavioral health system’s efforts to promote evidence based practices Ronnie M. Rubin, Byron J. Powell, Matthew O. Hurford, Shawna L. Weaver, Rinad S. Beidas, David S. Mandell, Arthur C. Evans A69: Applying the policy ecology model to Philadelphia’s behavioral health transformation efforts Byron J. Powell, Rinad S. Beidas, Ronnie M. Rubin, Rebecca E. Stewart, Courtney Benjamin Wolk, Samantha L. Matlin, Shawna Weaver, Matthew O. Hurford, Arthur C. Evans, Trevor R. Hadley, David S. Mandell A70: A model for providing methodological expertise to advance dissemination and implementation of health discoveries in Clinical and Translational Science Award institutions Donald R. Gerke, Beth Prusaczyk, Ana Baumann, Ericka M. Lewis, Enola K. Proctor A71: Establishing a research agenda for the Triple P Implementation Framework Jenna McWilliam, Jacquie Brown, Michelle Tucker A72: Cheap and fast, but what is “best?”: Examining implementation outcomes across sites in a state-wide scaled-up evidence-based walking program, Walk With Ease Kathleen P Conte A73: Measurement feedback systems in mental health: Initial review of capabilities and characteristics Aaron R. Lyon, Meredith Boyd, Abigail Melvin, Cara C. Lewis, Freda Liu, Nathaniel Jungbluth A74: A qualitative investigation of case managers’ attitudes toward implementation of a measurement feedback system in a public mental health system for youth Amelia Kotte, Kaitlin A. Hill, Albert C. Mah, Priya A. Korathu-Larson, Janelle R. Au, Sonia Izmirian, Scott Keir, Brad J. Nakamura, Charmaine K. Higa-McMillan A75: Multiple pathways to sustainability: Using Qualitative Comparative Analysis to uncover the necessary and sufficient conditions for successful community-based implementation Brittany Rhoades Cooper, Angie Funaiole, Eleanor Dizon A76: Prescribers’ perspectives on opioids and benzodiazepines and medication alerts to reduce co-prescribing of these medications Eric J. Hawkins, Carol A. Malte, Hildi J. Hagedorn, Douglas Berger, Anissa Frank, Aline Lott, Carol E. Achtmeyer, Anthony J. Mariano, Andrew J. Saxon A77: Adaptation of Coordinated Anxiety Learning and Management for comorbid anxiety and substance use disorders: Delivery of evidence-based treatment for anxiety in addictions treatment centers Kate Wolitzky-Taylor, Richard Rawson, Richard Ries, Peter Roy-Byrne, Michelle Craske A78: Opportunities and challenges of measuring program implementation with online surveys Dena Simmons, Catalina Torrente, Lori Nathanson, Grace Carroll A79: Observational assessment of fidelity to a family-centered prevention program: Effectiveness and efficiency Justin D. Smith, Kimbree Brown, Karina Ramos, Nicole Thornton, Thomas J. Dishion, Elizabeth A. Stormshak, Daniel S. Shaw, Melvin N. Wilson A80: Strategies and challenges in housing first fidelity: A multistate qualitative analysis Mimi Choy-Brown, Emmy Tiderington, Bikki Tran Smith, Deborah K. Padgett A81: Procurement and contracting as an implementation strategy: Getting To Outcomes® contracting Ronnie M. Rubin, Marilyn L. Ray, Abraham Wandersman, Andrea Lamont, Gordon Hannah, Kassandra A. Alia, Matthew O. Hurford, Arthur C. Evans A82: Web-based feedback to aid successful implementation: The interactive Stages of Implementation Completion (SIC)TM tool Lisa Saldana, Holle Schaper, Mark Campbell, Patricia Chamberlain A83: Efficient methodologies for monitoring fidelity in routine implementation: Lessons from the Allentown Social Emotional Learning Initiative Valerie B. Shapiro, B.K. Elizabeth Kim, Jennifer L. Fleming, Paul A. LeBuffe A84: The Society for Implementation Research Collaboration (SIRC) implementation development workshop: Results from a new methodology for enhancing implementation science proposals Sara J. Landes, Cara C. Lewis, Allison L. Rodriguez, Brigid R. Marriott, Katherine Anne Comtois A85: An update on the Society for Implementation Research Collaboration (SIRC) Instrument Review Projec