Equity “On the Sideline”: A Mixed Methods Study of New England Evaluation Practice in 2020

Background: Centering equity in evaluations is increasingly recognized as an important professional responsibility of evaluators. While some theoretical and practical guidance exists, the evaluation field has limited empirical research on equity within evaluation practice. Purpose: This paper explores whether and how evaluators address inequities and advance equity throughout evaluation phases drawing on select findings from a larger study. Setting: The study focuses on American Evaluation Association-affiliated evaluators in the New England region of the United States who work in a variety of areas (e.g., health, education).   Intervention: Not applicable Research Design: The study uses a complementarity, sequential mixed methods design comprised of a researcher-developed online questionnaire administered to a census and snowball sample of practicing evaluators (n=82) and individual, semi-structured interviews with a subset of this sample selected to maximize variation (n=21). Quantitative data were analyzed using descriptive statistics (i.e., means and standard deviations, frequencies). Qualitative data were analyzed using a collaborative process of deductive and inductive coding followed by thematic analysis. Findings: Eight overarching findings suggest that despite evaluators’ attempts to center equity, it remains largely “on the sideline.” This is due to evaluators’ need to work against some conventional professional and methodological norms, within contractual and contextual constraints, and with limited professional preparation. &nbsp

Equity “On the Sideline”

Background: Centering equity in evaluations is increasingly recognized as an important professional responsibility of evaluators. While some theoretical and practical guidance exists, the evaluation field has limited empirical research on equity within evaluation practice. Purpose: This paper explores whether and how evaluators address inequities and advance equity throughout evaluation phases drawing on select findings from a larger study. Setting: The study focuses on American Evaluation Association-affiliated evaluators in the New England region of the United States who work in a variety of areas (e.g., health, education).   Intervention: Not applicable Research Design: The study uses a complementarity, sequential mixed methods design comprised of a researcher-developed online questionnaire administered to a census and snowball sample of practicing evaluators (n=82) and individual, semi-structured interviews with a subset of this sample selected to maximize variation (n=21). Quantitative data were analyzed using descriptive statistics (i.e., means and standard deviations, frequencies). Qualitative data were analyzed using a collaborative process of deductive and inductive coding followed by thematic analysis. Findings: Eight overarching findings suggest that despite evaluators’ attempts to center equity, it remains largely “on the sideline.” This is due to evaluators’ need to work against some conventional professional and methodological norms, within contractual and contextual constraints, and with limited professional preparation.

Educational Accountability: Investigating the Collective Capacity of a Middle School

181 p.Thesis (Ph.D.)--University of Illinois at Urbana-Champaign, 2008.The main findings of this instrumental mixed methods case study were that, in this context, external accountability led to increased attention to test scores, curriculum, and instruction. External accountability influenced the school's system of internal accountability by fostering better alignment between instruction and state curriculum standards. External accountability also influenced the school's collective capacity, particularly with respect to leadership and teachers' ability to provide more academic support to students and a more nurturing environment. At the same time, teachers had varied student expectations that deterred them from achieving even higher levels of collective instructional capacity.U of I OnlyRestricted to the U of I community idenfinitely during batch ingest of legacy ETD

Values engagement in evaluation: Ideas, illustrations, and implications. American Journal of Evaluation. Advance online publication. doi: 10.1177/1098214011422592

Abstract Values-engagement in evaluation involves both describing stakeholder values and prescribing certain values. Describing stakeholder values is common practice in responsive evaluation traditions. Prescribing or advocating particular values is only explicitly part of democratic, culturally responsive, critical, and other openly ideological traditions in evaluation, but we argue that it is implicit in all evaluation approaches and practices. In this article, we discuss various conceptualizations of values-engagement in evaluation. We further present a specific form of values-engaged evaluation that is committed to descriptive and prescriptive valuing, with an emphasis on its prescriptive advancement of the values of inclusion and equity. Examples from field experience illustrate these two countenances and underscore the multiple challenges invoked by intentional engagement with the values dimensions of evaluation. The examples come from evaluations of science, technology, engineering, and mathematics (STEM) educational programs

Sowing the Seeds: Sociocultural Resistance In the Psychological Sciences

Objective: This article problematizes the use of resilience as a psychological and developmental indication of well-being. We base our argument on the possibility that resilience theories internalize responsibility for survival within the individual, and that survival is dependent on the ability to assimilate to injustice. Resistance, on the other hand, represents acts of intentional, active, and often collective survival which can expose and oppose social injustice. Method: Bringing together transdisciplinary scholarship on resistance, we propose a conceptual framework of sociocultural resistance. This framework seeks to forward studies of health that acknowledge the complexity of relationships, culture, and power constitutive of the human condition. Results: We provide examples of sociocultural resistance in the psychological and developmental sciences and suggest the use of diverse theory and methods in the study of resistance. Conclusions: Resistance research is a timely, necessary, and critical turning point in the social sciences with the potential to change unjust systems and promote a nuanced view of health

Strengthening Capacity and Equity in New England Evaluation

This report presents results from the Strengthening Capacity and Equity in New England Evaluation (SCENE) study. The SCENE study was conducted between August 2020 and August 2021 by a research group at Boston College with funding from the Barr Foundation.The purpose was three-fold: (1) to characterize evaluation providers, evaluation practices, and evaluators working in the New England area; (2) to explore whether and how evaluators address inequities and advance equity; and (3) to identify ways to strengthen capacity and equity among evaluators within the region. By providing an initial assessment to inform future capacity building and learning initiatives, we aim to inform regional evaluation practitioners and commissioners interested in advancing equity in evaluation. Findings are particularly relevant to local affiliates of the American Evaluation Association, including: Greater Boston Evaluators Network, Vermont Evaluation Network, Connecticut Area Evaluators' Network, Maine Evaluation Society. We also sought to address gaps in the evaluation literature regarding what equity and equity-focused evaluation mean and look like in practice. We plan to submit study results for publication/dissemination in academic journals

Exploring Changes in Two Types of Self-Efficacy Following Participation in a Chronic Disease Self-Management Program

Chronic conditions and falls are related issues faced by many aging adults. Stanford’s Chronic Disease Self-Management Program (CDSMP) added brief fall-related content to the standardized 6-week workshop; however, no research had examined changes in Fall-related self-efficacy (SE) in response to CDSMP participation. This study explored relationships and changes in SE using the Self-Efficacy to Manage Chronic Disease Scale (SEMCD Scale) and the Fall Efficacy Scale (FallE Scale) in participants who successfully completed CDSMP workshops within a Southern state over a 10-month period. SE scale data were compared at baseline and post-intervention for 36 adults (mean age 74.5, SD ± 9.64). Principal component analysis (PCA), using oblimin rotation was completed at baseline and post-intervention for the individual scales and then for analysis combining both scales as a single scale. Each scale loaded under a single component for the PCA at both baseline and post-intervention. When both scales were entered as single meta-scale, the meta-scale split along two factors with no double loading. SEMCD and FallE Scale scores were significantly correlated at baseline and post-intervention, at least p<.05. A significant proportion of participants improved their scores on the FallE Scale post-intervention (p=.038). The magnitude of the change was also significant only for the FallE Scale (p=.043). The SEMCD Scale scores did not change significantly. Study findings from the exploratory PCA and significant correlations indicated that the SEMCD Scale and the FallE Scale measured two distinct but related types of SE. Though the scale scores were correlated at baseline and post-intervention, only the FallE Scale scores significantly differed post-intervention. Given this relationship and CDSMP’s recent addition of a10-minute fall prevention segment, further exploration of CDSMP’s possible influence on Fall-related SE would provide useful understanding for health promotion in aging adults

The complexity of DLB: U.S.‐based Dementia with Lewy Body Consortium

Background The clinical syndrome of dementia with Lewy bodies (DLB) is well defined with good positive predictive value for the presence of Lewy body pathology at autopsy. However, the presence of Lewy body pathology in dementia is not always linked to the clinical syndrome of DLB, particularly early in the clinical course (e.g., prodromal and mild cognitive impairment), and co‐pathologies are quite frequent. This clinical and pathologic complexity leads to many challenges including early recognition and diagnosis, clinical management, and development of symptomatic and disease modifying therapies. To address these issues, a number of consortia have been established to longitudinally study DLB and related Lewy body diseases, including a US based Dementia with Lewy Bodies Consortium (DLBC). Method The U.S.‐based DLBC is a multi‐centered, NIH funded, project to longitudinally and systematically study subjects with DLB and mild cognitive impairment/high likelihood DLB (MCI‐DLB). The primary goal of the DLBC is to generate biospecimens linked to clinical, imaging, and neuropathologic characterization in a longitudinally followed cohort across the participating DLBC sites. Biofluids, including blood and cerebrospinal fluid (CSF), and data generated by the DLBC will be ultimately be available to investigators through the NINDS‐funded Parkinson’s Disease Biomarker Program. Result To date, one hundred and thirteen subjects have been enrolled in the DLBC. Preliminary results from the first ninety‐two subjects note a male predominance in the sample (82 of 92, 89%). At baseline visit, mean age was 69.4 (SD 7.1 years), baseline UPDRS‐MDS Part III was 28.9 (15.7 SD) and MoCA 20.4 (SD 5.4). Neuropathologic data are limited, but suggest that an elevated CSF p‐tau may assist in clinically differentiating DLB with or without high levels of Alzheimer’s disease pathology. Imaging results including volumetric MRI, amyloid imaging and DaTscan are pending. Conclusion The development of DLB cohorts, such as the US based DLBC, is essential to understanding the complexity and pathobiologic significance of the clinical syndrome, biomarker characteristics and imaging findings in patients with underlying Lewy body disease. These results continue to emphasize the importance of further development of biomarkers in DLB and related neurodegenerative disorders