Evaluability Assessment: Clarifying Organizational Support and Data Availability

Evaluability assessment (EA) emerged in the 1970s as a way to ensure a program was ready for summative evaluation.  The primary purpose was assessing the presence of measureable program objectives (Trevisan, 2007), yet evaluators conducting EA encountered difficulty with unclear, ambiguous methods (Smith, 2005).  To address this concern, the purpose of this study was to clarify two aspects of EA, organizational support and data availability.  In practice, organizational stakeholders must support the evaluation project to ensure it is pursued to completion.  In addition, the availability of operational data facilitates analysis of the evaluand effect.  This qualitative study consisted of interviews with evaluators, organizational stakeholders, and technology personnel followed by thematic analysis.  The findings indicate the importance of specific organizational and data related considerations that affect evaluability. The researchers recommend considerations that elaborate upon the existing EA framework.  The recommended evaluability considerations assist evaluators in identifying ill-advised evaluations and enhancing the likelihood of success in ongoing studies

The Role of Leadership and Culture in Creating Meaningful Assessment: A Mixed Methods Case Study

With increased demands for institutional accountability and improved student learning, involvement in assessment has become a fundamental role of higher education faculty (Rhodes, 2010). However, faculty members and administrators often question whether assessment efforts do indeed improve student learning (Hutchings, 2010). This mixed methods case study of a faculty inquiry project explored how factors linked to organizational context (Kezar, 2013) are related to commitment to assessment and to use of assessment data by faculty members. Results indicated key best practices, such as developing faculty leaders and communities of practice to exchange ideas. The study provides insights for institutional administrators and faculty members seeking to develop a culture of assessment

Understanding why child welfare clinic attendance and growth of children in the nutrition surveillance programme is below target: lessons learnt from a mixed methods study in Ghana

Background: Growth monitoring and promotion (GMP) programmes promote not only child health but serve as a service delivery strategy to enhance coverage for other crucial nutrition-specific interventions. This study compared community-based and facility-based GMP programme with respect to attendance rates, children’s nutritional status, caregivers’ satisfaction with services received and perceptions of service providers and users on factors influencing utilization. Methods: Explanatory sequential mixed methods study conducted in Ga West municipality, Ghana. It comprised 12-month secondary data analysis using growth monitoring registers of 220 infants aged 0–3 months enrolled in two community-based (CB = 104) and two facility-based (FB = 116) child welfare clinics; cross-sectional survey (exit interview) of 232 caregiver-child pairs accessing CB (n = 104) and FB services (n = 116); and in-depth interviews with 10 health workers and 15 mothers. Quantitative data were analyzed through Fisher’s exact, unpaired t-tests, and logistic regression at 95% confidence interval (CI) using SPSS version 20. Qualitative data were analyzed by thematic content analysis using ATLAS.ti 7.0. Results: Mean annual attendance to both programmes was similar with an average of six visits per year. Only 13.6% of caregiver-child pairs attained more than nine visits in the 12-months period. At least 60% of children in both programs had improved weight-for-age z-scores (WAZ) scores during participation. Predictors for improved WAZ were being underweight at baseline (AOR:11.1, 95%CI:4.0–31.0), annual attendance of at least six visits (AOR:2.2, 95%CI:1.1–4.1) and meeting the Ghana Health Service target of nine visits (AOR:4.65, 95%CI:1.4–15.1). Compared to 31.5% CB users, significant proportion of FB caregivers (57.4%) were visited at home. Half were dissatisfied with services received (CB:55.6% vs. FB:62.0%, p = 0.437) citing long waiting times, negative staff attitude and extortions of money. Regarding perceptions on factors hindering service utilization, emerged themes included extremes of maternal age, high parity, postpartum socio-cultural beliefs and practices, financial commitments, undue delays, unprofessional staff behaviours, high premium on vaccination and general misconceptions about the programme. Conclusion: The association of increased attendance with improved growth reaffirms the need to strengthen primary healthcare systems to improve service delivery; sensitize caregivers on contribution of growth monitoring and promotion to early child development; and increase contacts through home visits

The Pandemic Leadership Model: A Study of Medical Student Values During COVID-19

Background: Leadership training in medical school continues to grow. Little information exists to guide leadership program development. Concurrently, the COVID-19 pandemic provides a real-world crucible of leadership, allowing insight into qualities and characteristics medical students value. We aim to determine what students value in leadership during a pandemic and the implicit framework students use. Methods: We conducted a cross-sectional, qualitative study using a five-item novel survey instrument developed by a consensus group of experts from family medicine, leadership development, medical education, and survey research to elicit student perceptions of effective and ineffective leadership qualities and examples during the COVID-19 pandemic at the University of Michigan Medical School. We used thematic analysis to identify overarching themes to build a model of leadership integrated with existing theory. Results: 162 students participated across all years of medical school. We identified themes of Communication, Other-Orientation, Personal Characteristics, Decisive Action, and Use of Information. These five themes were then built into the model of Pandemic Leadership within the context of complexity leadership theory and collective leadership theory. This model represents qualities and characteristics students value in good leaders during a crisis. Conclusion: This study is unique in its focus on student perceptions of leadership qualities during a real-world laboratory for leadership. We hope that this information, along with the pandemic leadership model, can serve as the first step toward relevant leadership training programs in medical education. Leadership training programs in medical education would likely benefit from grounding in the student values identified by this study

Medical Students\u27 Experiences and Outcomes Using a Virtual Human Simulation to Improve Communication Skills: Mixed Methods Study

Background: Attending to the wide range of communication behaviors that convey empathy is an important but often underemphasized concept to reduce errors in care, improve patient satisfaction, and improve cancer patient outcomes. A virtual human (VH)–based simulation, MPathic-VR, was developed to train health care providers in empathic communication with patients and in interprofessional settings and evaluated through a randomized controlled trial. Objective: This mixed methods study aimed to investigate the differential effects of a VH-based simulation developed to train health care providers in empathic patient-provider and interprofessional communication. Methods: We employed a mixed methods intervention design, involving a comparison of 2 quantitative measures—MPathic-VR–calculated scores and the objective structured clinical exam (OSCE) scores—with qualitative reflections by medical students about their experiences. This paper is a secondary, focused analysis of intervention arm data from the larger trial. Students at 3 medical schools in the United States (n=206) received simulation to improve empathic communication skills. We conducted analysis of variance, thematic text analysis, and merging mixed methods analysis. Results: OSCE scores were significantly improved for learners in the intervention group (mean 0.806, SD 0.201) compared with the control group (mean 0.752, SD 0.198; F1,414=6.09; P=.01). Qualitative analysis revealed 3 major positive themes for the MPathic-VR group learners: gaining useful communication skills, learning awareness of nonverbal skills in addition to verbal skills, and feeling motivated to learn more about communication. Finally, the results of the mixed methods analysis indicated that most of the variation between high, middle, and lower performers was noted about nonverbal behaviors. Medium and high OSCE scorers most often commented on the importance of nonverbal communication. Themes of motivation to learn about communication were only present in middle and high scorers. Conclusions: VHs are a promising strategy for improving empathic communication in health care. Higher performers seemed most engaged to learn, particularly nonverbal skills

Enhancing Nonverbal Communication Through Virtual Human Technology: Protocol for a Mixed Methods Study

Background: Communication is a critical component of the patient-provider relationship; however, limited research exists on the role of nonverbal communication. Virtual human training is an informatics-based educational strategy that offers various benefits in communication skill training directed at providers. Recent informatics-based interventions aimed at improving communication have mainly focused on verbal communication, yet research is needed to better understand how virtual humans can improve verbal and nonverbal communication and further elucidate the patient-provider dyad. Objective: The purpose of this study is to enhance a conceptual model that incorporates technology to examine verbal and nonverbal components of communication and develop a nonverbal assessment that will be included in the virtual simulation for further testing. Methods: This study will consist of a multistage mixed methods design, including convergent and exploratory sequential components. A convergent mixed methods study will be conducted to examine the mediating effects of nonverbal communication. Quantitative (eg, MPathic game scores, Kinect nonverbal data, objective structured clinical examination communication score, and Roter Interaction Analysis System and Facial Action Coding System coding of video) and qualitative data (eg, video recordings of MPathic–virtual reality [VR] interventions and student reflections) will be collected simultaneously. Data will be merged to determine the most crucial components of nonverbal behavior in human-computer interaction. An exploratory sequential design will proceed, consisting of a grounded theory qualitative phase. Using theoretical, purposeful sampling, interviews will be conducted with oncology providers probing intentional nonverbal behaviors. The qualitative findings will aid the development of a nonverbal communication model that will be included in a virtual human. The subsequent quantitative strand will incorporate and validate a new automated nonverbal communication behavior assessment into the virtual human simulation, MPathic-VR, by assessing interrater reliability, code interactions, and dyadic data analysis by comparing Kinect responses (system recorded) to manually scored records for specific nonverbal behaviors. Data will be integrated using building integration to develop the automated nonverbal communication behavior assessment and conduct a quality check of these nonverbal features. Results: Secondary data from the MPathic-VR randomized controlled trial data set (210 medical students and 840 video recordings of interactions) were analyzed in the first part of this study. Results showed differential experiences by performance in the intervention group. Following the analysis of the convergent design, participants consisting of medical providers (n=30) will be recruited for the qualitative phase of the subsequent exploratory sequential design. We plan to complete data collection by July 2023 to analyze and integrate these findings. Conclusions: The results from this study contribute to the improvement of patient-provider communication, both verbal and nonverbal, including the dissemination of health information and health outcomes for patients. Further, this research aims to transfer to various topical areas, including medication safety, informed consent processes, patient instructions, and treatment adherence between patients and providers

Enhancing Nonverbal Communication Through Virtual Human Technology: Protocol for a Mixed Methods Study

Background: Communication is a critical component of the patient-provider relationship; however, limited research exists on the role of nonverbal communication. Virtual human training is an informatics-based educational strategy that offers various benefits in communication skill training directed at providers. Recent informatics-based interventions aimed at improving communication have mainly focused on verbal communication, yet research is needed to better understand how virtual humans can improve verbal and nonverbal communication and further elucidate the patient-provider dyad. Objective: The purpose of this study is to enhance a conceptual model that incorporates technology to examine verbal and nonverbal components of communication and develop a nonverbal assessment that will be included in the virtual simulation for further testing. Methods: This study will consist of a multistage mixed methods design, including convergent and exploratory sequential components. A convergent mixed methods study will be conducted to examine the mediating effects of nonverbal communication. Quantitative (eg, MPathic game scores, Kinect nonverbal data, objective structured clinical examination communication score, and Roter Interaction Analysis System and Facial Action Coding System coding of video) and qualitative data (eg, video recordings of MPathic–virtual reality [VR] interventions and student reflections) will be collected simultaneously. Data will be merged to determine the most crucial components of nonverbal behavior in human-computer interaction. An exploratory sequential design will proceed, consisting of a grounded theory qualitative phase. Using theoretical, purposeful sampling, interviews will be conducted with oncology providers probing intentional nonverbal behaviors. The qualitative findings will aid the development of a nonverbal communication model that will be included in a virtual human. The subsequent quantitative strand will incorporate and validate a new automated nonverbal communication behavior assessment into the virtual human simulation, MPathic-VR, by assessing interrater reliability, code interactions, and dyadic data analysis by comparing Kinect responses (system recorded) to manually scored records for specific nonverbal behaviors. Data will be integrated using building integration to develop the automated nonverbal communication behavior assessment and conduct a quality check of these nonverbal features. Results: Secondary data from the MPathic-VR randomized controlled trial data set (210 medical students and 840 video recordings of interactions) were analyzed in the first part of this study. Results showed differential experiences by performance in the intervention group. Following the analysis of the convergent design, participants consisting of medical providers (n=30) will be recruited for the qualitative phase of the subsequent exploratory sequential design. We plan to complete data collection by July 2023 to analyze and integrate these findings. Conclusions: The results from this study contribute to the improvement of patient-provider communication, both verbal and nonverbal, including the dissemination of health information and health outcomes for patients. Further, this research aims to transfer to various topical areas, including medication safety, informed consent processes, patient instructions, and treatment adherence between patients and providers. International Registered Report Identifier (IRRID): DERR1-10.2196/4660

Evaluability Assessment: Clarifying Organizational Support and Data Availability

Evaluability assessment (EA) emerged in the 1970s as a way to ensure a program was ready for summative evaluation.  The primary purpose was assessing the presence of measureable program objectives (Trevisan, 2007), yet evaluators conducting EA encountered difficulty with unclear, ambiguous methods (Smith, 2005).  To address this concern, the purpose of this study was to clarify two aspects of EA, organizational support and data availability.  In practice, organizational stakeholders must support the evaluation project to ensure it is pursued to completion.  In addition, the availability of operational data facilitates analysis of the evaluand effect.  This qualitative study consisted of interviews with evaluators, organizational stakeholders, and technology personnel followed by thematic analysis.  The findings indicate the importance of specific organizational and data related considerations that affect evaluability. The researchers recommend considerations that elaborate upon the existing EA framework.  The recommended evaluability considerations assist evaluators in identifying ill-advised evaluations and enhancing the likelihood of success in ongoing studies

Descripciones de las prácticas de muestreo en cinco enfoques de investigación cualitativa en educación y ciencias de la salud

Obwohl vereinzelte Vorschläge zu Samplegrößen vorliegen, steht Literatur hierzu nach wie vor eher selten zur Verfügung. Um die Ergebnisse aus qualitativer Forschung transparenter und verlässlicher zu machen, sind jedoch praktische Vorschläge für die Konzeption von Studien und für die Budgetierung von Ressourcen erforderlich. Der vorliegende Artikel untersucht deshalb qualitative Samplegrößen und -praktiken innerhalb der Erziehungs- und Gesundheitswissenschaften für unterschiedliche Designs: Fallstudien, Ethnografie, Grounded-Theory-Methodologie, narrative Forschungsansätze und Phänomenologie. Einbezogen wurden die 51 am häufigsten zitierten Artikel aus diesen Bereichen entlang vorgängiger Kategorien und wesentlicher Charakteristika, die sich im Laufe der Studie ergaben. Insgesamt fand sich eine durchschnittliche Samplegröße von 87 Fällen, in weniger als der Hälfte der Artikel wurde die Samplingstrategie explizit benannt. Im Beitrag werden die Ergebnisse für jeden Ansatz beschrieben und Empfehlungen gegeben, die Forschenden, Gutachter/innen, Fördereinrichtungen und Studierenden helfen sollen, qualitative Samplingpraktiken besser einordnen und bewerten zu können.URN: http://nbn-resolving.de/urn:nbn:de:0114-fqs1502256Although recommendations exist for determining qualitative sample sizes, the literature appears to contain few instances of research on the topic. Practical guidance is needed for determining sample sizes to conduct rigorous qualitative research, to develop proposals, and to budget resources. The purpose of this article is to describe qualitative sample size and sampling practices within published studies in education and the health sciences by research design: case study, ethnography, grounded theory methodology, narrative inquiry, and phenomenology. I analyzed the 51 most highly cited studies using predetermined content categories and noteworthy sampling characteristics that emerged. In brief, the findings revealed a mean sample size of 87. Less than half of the studies identified a sampling strategy. I include a description of findings by approach and recommendations for sampling to assist methodologists, reviewers, program officers, graduate students, and other qualitative researchers in understanding qualitative sampling practices in recent studies.URN: http://nbn-resolving.de/urn:nbn:de:0114-fqs1502256Aunque existen recomendaciones para la determinación de tamaños de la muestra cualitativa, la literatura parece contener pocos casos de investigación sobre el tema. Es necesaria una guía práctica para la determinación de tamaños de muestra para realizar rigurosa investigación cualitativa, para desarrollar propuestas y para presupuestar recursos. El propósito de este artículo es describir prácticas de muestreo y de tamaño de la muestra cualitativa en los estudios publicados en educación y ciencias de la salud por el diseño de investigación: estudio de caso, etnografía, metodología de teoría fundamentada, investigación narrativa y fenomenología. He analizado los 51 estudios más altamente citados usando las categorías de contenido predeterminadas y las características más notables de muestreo que surgieron. En resumen, los resultados revelaron un tamaño de muestra media de 87. Menos de la mitad de los estudios identificaron una estrategia de muestreo. Incluyo una descripción de los resultados por enfoque y recomendaciones para el muestreo a fin de ayudar a metodólogos, revisores, responsables de programa, estudiantes de posgrado y otros investigadores cualitativos para lograr comprensión de las prácticas de muestreo cualitativo en estudios recientes.URN: http://nbn-resolving.de/urn:nbn:de:0114-fqs150225