Equity “On the Sideline”: A Mixed Methods Study of New England Evaluation Practice in 2020

Background: Centering equity in evaluations is increasingly recognized as an important professional responsibility of evaluators. While some theoretical and practical guidance exists, the evaluation field has limited empirical research on equity within evaluation practice. Purpose: This paper explores whether and how evaluators address inequities and advance equity throughout evaluation phases drawing on select findings from a larger study. Setting: The study focuses on American Evaluation Association-affiliated evaluators in the New England region of the United States who work in a variety of areas (e.g., health, education).   Intervention: Not applicable Research Design: The study uses a complementarity, sequential mixed methods design comprised of a researcher-developed online questionnaire administered to a census and snowball sample of practicing evaluators (n=82) and individual, semi-structured interviews with a subset of this sample selected to maximize variation (n=21). Quantitative data were analyzed using descriptive statistics (i.e., means and standard deviations, frequencies). Qualitative data were analyzed using a collaborative process of deductive and inductive coding followed by thematic analysis. Findings: Eight overarching findings suggest that despite evaluators’ attempts to center equity, it remains largely “on the sideline.” This is due to evaluators’ need to work against some conventional professional and methodological norms, within contractual and contextual constraints, and with limited professional preparation. &nbsp

Equity “On the Sideline”

Background: Centering equity in evaluations is increasingly recognized as an important professional responsibility of evaluators. While some theoretical and practical guidance exists, the evaluation field has limited empirical research on equity within evaluation practice. Purpose: This paper explores whether and how evaluators address inequities and advance equity throughout evaluation phases drawing on select findings from a larger study. Setting: The study focuses on American Evaluation Association-affiliated evaluators in the New England region of the United States who work in a variety of areas (e.g., health, education).   Intervention: Not applicable Research Design: The study uses a complementarity, sequential mixed methods design comprised of a researcher-developed online questionnaire administered to a census and snowball sample of practicing evaluators (n=82) and individual, semi-structured interviews with a subset of this sample selected to maximize variation (n=21). Quantitative data were analyzed using descriptive statistics (i.e., means and standard deviations, frequencies). Qualitative data were analyzed using a collaborative process of deductive and inductive coding followed by thematic analysis. Findings: Eight overarching findings suggest that despite evaluators’ attempts to center equity, it remains largely “on the sideline.” This is due to evaluators’ need to work against some conventional professional and methodological norms, within contractual and contextual constraints, and with limited professional preparation.

PLAYING TOWARD MULTIPLICITY: DISORIENTING INTRA-ACTIONS WITH MATERIALS & LEARNING IN AN ESCAPE ROOM

In this paper, we follow the co-constitutive material-participant relationships that propel action in escape room play, particularly how they open and close paths for learning. We focus on how learning is organized in one escape room game, The Author’s Enigma, as intertwined with conversations on play and learning to consider the relational values and ideologies that appear through more-than-human encounters. We contribute a critical (new) materialist draw toward material-participant intra-actions to notice the production of narrowed messages, as well as openings that lead toward multiplicity. Thinking-with-theory; we take up Ahmed’s (2006) concepts of orientation and disorientation to consider objects’ arrivals and their not-yet-presence. We also move with Barad’s (2003) concept of intra-action, perceiving more-than-human actors as ‘matter-in-the-process-of-becoming’. Following a conch shell as a vibrant material in the ecology of escape, we trace participants and the shell through moments of intra-action across escape play. We detail the communicative forms produced through material intra-actions as both open and closed, while producing resonant logics with players in the room. We discuss implications for: 1) game-based approaches to education; 2) disruption to assimilative lenses for sensing and supporting learning; and 3) valuing particular relational arrangements with materials

Strengthening Capacity and Equity in New England Evaluation

This report presents results from the Strengthening Capacity and Equity in New England Evaluation (SCENE) study. The SCENE study was conducted between August 2020 and August 2021 by a research group at Boston College with funding from the Barr Foundation.The purpose was three-fold: (1) to characterize evaluation providers, evaluation practices, and evaluators working in the New England area; (2) to explore whether and how evaluators address inequities and advance equity; and (3) to identify ways to strengthen capacity and equity among evaluators within the region. By providing an initial assessment to inform future capacity building and learning initiatives, we aim to inform regional evaluation practitioners and commissioners interested in advancing equity in evaluation. Findings are particularly relevant to local affiliates of the American Evaluation Association, including: Greater Boston Evaluators Network, Vermont Evaluation Network, Connecticut Area Evaluators' Network, Maine Evaluation Society. We also sought to address gaps in the evaluation literature regarding what equity and equity-focused evaluation mean and look like in practice. We plan to submit study results for publication/dissemination in academic journals