Learning Evaluation: Blending Quality Improvement and Implementation Research Methods to Study Healthcare Innovations

Background: In healthcare change interventions, on-the-ground learning about the implementation process is often lost because of a primary focus on outcome improvements. This paper describes the Learning Evaluation, a methodological approach that blends quality improvement and implementation research methods to study healthcare innovations. Methods: Learning Evaluation is an approach to multi-organization assessment. Qualitative and quantitative data are collected to conduct real-time assessment of implementation processes while also assessing changes in context, facilitating quality improvement using run charts and audit and feedback, and generating transportable lessons. Five principles are the foundation of this approach: (1) gather data to describe changes made by healthcare organizations and how changes are implemented; (2) collect process and outcome data relevant to healthcare organizations and to the research team; (3) assess multi-level contextual factors that affect implementation, process, outcome, and transportability; (4) assist healthcare organizations in using data for continuous quality improvement; and (5) operationalize common measurement strategies to generate transportable results. Results: Learning Evaluation principles are applied across organizations by the following: (1) establishing a detailed understanding of the baseline implementation plan; (2) identifying target populations and tracking relevant process measures; (3) collecting and analyzing real-time quantitative and qualitative data on important contextual factors; (4) synthesizing data and emerging findings and sharing with stakeholders on an ongoing basis; and (5) harmonizing and fostering learning from process and outcome data. Application to a multi-site program focused on primary care and behavioral health integration shows the feasibility and utility of Learning Evaluation for generating real-time insights into evolving implementation processes. Conclusions: Learning Evaluation generates systematic and rigorous cross-organizational findings about implementing healthcare innovations while also enhancing organizational capacity and accelerating translation of findings by facilitating continuous learning within individual sites. Researchers evaluating change initiatives and healthcare organizations implementing improvement initiatives may benefit from a Learning Evaluation approach

A national evaluation of a dissemination and implementation initiative to enhance primary care practice capacity and improve cardiovascular disease care: the ESCALATES study protocol.

BACKGROUND: The Agency for Healthcare Research and Quality (AHRQ) launched the EvidenceNOW Initiative to rapidly disseminate and implement evidence-based cardiovascular disease (CVD) preventive care in smaller primary care practices. AHRQ funded eight grantees (seven regional Cooperatives and one independent national evaluation) to participate in EvidenceNOW. The national evaluation examines quality improvement efforts and outcomes for more than 1500 small primary care practices (restricted to those with fewer than ten physicians per clinic). Examples of external support include practice facilitation, expert consultation, performance feedback, and educational materials and activities. This paper describes the study protocol for the EvidenceNOW national evaluation, which is called Evaluating System Change to Advance Learning and Take Evidence to Scale (ESCALATES). METHODS: This prospective observational study will examine the portfolio of EvidenceNOW Cooperatives using both qualitative and quantitative data. Qualitative data include: online implementation diaries, observation and interviews at Cooperatives and practices, and systematic assessment of context from the perspective of Cooperative team members. Quantitative data include: practice-level performance on clinical quality measures (aspirin prescribing, blood pressure and cholesterol control, and smoking cessation; ABCS) collected by Cooperatives from electronic health records (EHRs); practice and practice member surveys to assess practice capacity and other organizational and structural characteristics; and systematic tracking of intervention delivery. Quantitative, qualitative, and mixed methods analyses will be conducted to examine how Cooperatives organize to provide external support to practices, to compare effectiveness of the dissemination and implementation approaches they implement, and to examine how regional variations and other organization and contextual factors influence implementation and effectiveness. DISCUSSION: ESCALATES is a national evaluation of an ambitious large-scale dissemination and implementation effort focused on transforming smaller primary care practices. Insights will help to inform the design of national health care practice extension systems aimed at supporting practice transformation efforts in the USA. CLINICAL TRIAL REGISTRATION: NCT02560428 (09/21/15)

Understanding Care Integration from the Ground Up: Five Organizing Constructs that Shape Integrated Practices.

PURPOSE: To provide empirical evidence on key organizing constructs shaping practical, real-world integration of behavior health and primary care to comprehensively address patients\u27 medical, emotional, and behavioral health needs. METHODS: In a comparative case study using an immersion-crystallization approach, a multidisciplinary team analyzed data from observations of practice operations, interviews, and surveys of practice members, and implementation diaries. Practices were drawn from 2 studies of practices attempting to integrate behavioral health and primary care: Advancing Care Together, a demonstration project of 11 practices located in Colorado, and the Integration Workforce Study, a study of 8 practices across the United States. RESULTS: We identified 5 key organizing constructs influencing integration of primary care and behavioral health: 1) Integration REACH (the extent to which the integration program was delivered to the identified target population), 2) establishment of continuum of care pathways addressing the location of care across the range of patient\u27s severity of illness, 3) approach to patient transitions: referrals or warm handoffs, 4) location of the integration workforce, and 5) participants\u27 mental model for integration. These constructs intertwine within an organization\u27s historic and social context to produce locally adapted approaches to integrating care. Contextual factors, particularly practice type, influenced whether specialty mental health and substance use services were colocated within an organization. CONCLUSION: Interaction among 5 organizing constructs and practice context produces diverse expressions of integrated care. These constructs provide a framework for understanding how primary care and behavioral health services can be integrated in routine practice

Preparing the Workforce for Behavioral Health and Primary Care Integration.

PURPOSE: To identify how organizations prepare clinicians to work together to integrate behavioral health and primary care. METHODS: Observational cross-case comparison study of 19 U.S. practices, 11 participating in Advancing Care Together, and 8 from the Integration Workforce Study. Practices varied in size, ownership, geographic location, and experience delivering integrated care. Multidisciplinary teams collected data (field notes from direct practice observations, semistructured interviews, and online diaries as reported by practice leaders) and then analyzed the data using a grounded theory approach. RESULTS: Organizations had difficulty finding clinicians possessing the skills and experience necessary for working in an integrated practice. Practices newer to integration underestimated the time and resources needed to train and organizationally socialize (onboard) new clinicians. Through trial and error, practices learned that clinicians needed relevant training to work effectively as integrated care teams. Training efforts exclusively targeting behavioral health clinicians (BHCs) and new employees were incomplete if primary care clinicians (PCCs) and others in the practice also lacked experience working with BHCs and delivering integrated care. Organizations\u27 methods for addressing employees\u27 need for additional preparation included hiring a consultant to provide training, sending employees to external training programs, hosting residency or practicum training programs, or creating their own internal training program. Onboarding new employees through the development of training manuals; extensive shadowing processes; and protecting time for ongoing education, mentoring, and support opportunities for new and established clinicians and staff were featured in these internal training programs. CONCLUSION: Insufficient training capacity and practical experience opportunities continue to be major barriers to supplying the workforce needed for effective behavioral health and primary care integration. Until the training capacity grows to meet the demand, practices must put forth considerable effort and resources to train their own employees

Designing Clinical Space for the Delivery of Integrated Behavioral Health and Primary Care.

PURPOSE: This study sought to describe features of the physical space in which practices integrating primary care and behavioral health care work and to identify the arrangements that enable integration of care. METHODS: We conducted an observational study of 19 diverse practices located across the United States. Practice-level data included field notes from 2-4-day site visits, transcripts from semistructured interviews with clinicians and clinical staff, online implementation diary posts, and facility photographs. A multidisciplinary team used a 4-stage, systematic approach to analyze data and identify how physical layout enabled the work of integrated care teams. RESULTS: Two dominant spatial layouts emerged across practices: type-1 layouts were characterized by having primary care clinicians (PCCs) and behavioral health clinicians (BHCs) located in separate work areas, and type-2 layouts had BHCs and PCCs sharing work space. We describe these layouts and the influence they have on situational awareness, interprofessional bumpability, and opportunities for on-the-fly communication. We observed BHCs and PCCs engaging in more face-to-face methods for coordinating integrated care for patients in type 2 layouts (41.5% of observed encounters vs 11.7%; P \u3c .05). We show that practices needed to strike a balance between professional proximity and private work areas to accomplish job tasks. Private workspace was needed for focused work, to see patients, and for consults between clinicians and clinical staff. We describe the ways practices modified and built new space and provide 2 recommended layouts for practices integrating care based on study findings. CONCLUSION: Physical layout and positioning of professionals\u27 workspace is an important consideration in practices implementing integrated care. Clinicians, researchers, and health-care administrators are encouraged to consider the role of professional proximity and private working space when creating new facilities or redesigning existing space to foster delivery of integrated behavioral health and primary care