Communicating About Evaluation: A Conceptual Model and Case Example

Background: Despite consensus within the evaluation community about what is distinctive about evaluation, confusion among stakeholders and other professions abounds. The evaluation literature describes how those in the social sciences continue to view evaluation as applied social science and part of what they already know how to do, with the implication that no additional training beyond the traditional social sciences is needed. Given the lack of broader understanding of the specialized role of evaluation, the field struggles with how best to communicate about evaluation to stakeholders and other professions. Purpose: This paper addresses the need to clearly communicate what is distinctive about evaluation to stakeholders and other professions by offering a conceptual tool that can be used in dialogue with others. Specifically, we adapt a personnel evaluation framework to map out what is distinctive about what evaluators know and can do. We then compare this map with the knowledge and skill needed in a related profession (i.e., assessment) in order to reveal how the professions differ.  Setting: Not applicable. Intervention: Not applicable. Research Design: Not applicable. Data Collection and Analysis: Not applicable. Findings: We argue that using a conceptual tool such as the one presented in this paper with comparative case examples would clarify for outsiders the distinct work of evaluators. Additionally, we explain how this conceptual tool is flexible and could be extended by evaluation practitioners in a myriad of ways. Keywords: evaluation knowledge; evaluation skill; profession; professionalizatio

Mind Body Health Interventions in Preschoolers

There are myriad of mind body health (MBH) interventions that are effective for the preschool population. Supports may include, yet are not limited to, journal expression, yoga, music therapy, mindfulness, video self-modeling, and muscle relaxation. These particular interventions have resulted in positive changes for preschoolers with anxiety, depression, stress, and attention, as well as various physical conditions. Benefits of these MBH interventions include being effective with respect to teacher time, economically feasible, and are well-received by parents, teachers, and preschoolers

Parametric Tests for Two Population Means under Normal and Non-Normal Distributions

A simulation study was conducted to explore the performance of the independent means t-test, Satterthwaite’s approximate t-test, and the conditional t-test under various conditions. This study analyzed Type I error control and statistical power of these testing approaches and provided guidance on the proper selection among them

Youth Engagement in Electoral Activities: A Collaborative Evaluation of a Civic Education Project

Youth civic engagement is recognized as an essential component necessary for the preservation of democratic practices; however, inadequate levels of civic participation persist among young people. Past research has shown that young people are more likely to participate in politics when they are informed. We present survey data collected from middle and high school students during a collaborative evaluation of a civic education project. A total of 4492 students participated in the pre-survey, and 3180 students participated in the post-survey. The aim of this part of the evaluation was to explore students’ civic engagement before and after the infusion of civic education into the curriculum. We discuss reasons hindering students’ intent to vote, their satisfaction level with the 2008 presidential election, and how students planned to become involved in the 2012 presidential election. The results of the post-survey show an increase in students registered to vote and students’ intent to register to vote after the infusion of civic education into the curriculum. The findings suggest that civic education seems to positively influence students’ political behavior

Effectiveness of Propensity Score Methods in a Multilevel Framework: A Monte Carlo Study

Propensity score analysis has been used to minimize the selection bias in observational studies to identify causal relationships. A propensity score is an estimate of an individual\u27s probability of being placed in a treatment group given a set of covariates. Propensity score analysis aims to use the estimate to create balanced groups, akin to a randomized experiment. This study used Monte Carlo methods to examine the appropriateness of using propensity score methods to achieve balance between groups on observed covariates and reproduce treatment effect estimates in multilevel studies. Specifically, this study examined the extent to which four different propensity score estimation models and three different propensity score conditioning methods produced balanced samples and reproduced the treatment effects with clustered data. One single-level logistic model and three multilevel models were investigated. Conditioning methods included: (a) covariance adjustment, (b) matching, and (c) stratification. Design factors investigated included: (a) level-1sample size, (b) level-2 sample size, (c) level-1 covariate relationship to treatment, (d) level-2 covariate relationship to treatment, (e) level-1 covariate relationship to outcome, (f) level-2 covariate relationship to outcome, and (g) population effect size. The results of this study suggest the degree to which propensity score analyses are able to create balanced groups and reproduce treatment effect estimates with clustered data is largely dependent upon the propensity score estimation model and conditioning method selected. Overall, the single-level logistic and random intercepts models fared slightly better than the more complex multilevel models while covariance adjustment and matching methods tended to be more stable in terms of balancing groups than stratification. Additionally, the results indicate propensity score analysis should not be conducted with small samples. Finally, this study did not identify an estimation model or conditioning method that was consistently able to create adequately balanced groups and reproduce treatment effect estimates

Communicating About Evaluation: A Conceptual Model and Case Example

Background: Despite consensus within the evaluation community about what is distinctive about evaluation, confusion among stakeholders and other professions abounds. The evaluation literature describes how those in the social sciences continue to view evaluation as applied social science and part of what they already know how to do, with the implication that no additional training beyond the traditional social sciences is needed. Given the lack of broader understanding of the specialized role of evaluation, the field struggles with how best to communicate about evaluation to stakeholders and other professions. Purpose: This paper addresses the need to clearly communicate what is distinctive about evaluation to stakeholders and other professions by offering a conceptual tool that can be used in dialogue with others. Specifically, we adapt a personnel evaluation framework to map out what is distinctive about what evaluators know and can do. We then compare this map with the knowledge and skill needed in a related profession (i.e., assessment) in order to reveal how the professions differ.  Setting: Not applicable. Intervention: Not applicable. Research Design: Not applicable. Data Collection and Analysis: Not applicable. Findings: We argue that using a conceptual tool such as the one presented in this paper with comparative case examples would clarify for outsiders the distinct work of evaluators. Additionally, we explain how this conceptual tool is flexible and could be extended by evaluation practitioners in a myriad of ways. Keywords: evaluation knowledge; evaluation skill; profession; professionalizatio

Battling Inertia in Educational Leadership: CRT Praxis for Race Conscious Dialogue

The purpose of this article is to illustrate how institutional racism is mediated by faculty negotiating power and privilege in the selection of Black (African American) women into an educational leadership preparation program. Critical race theory (CRT) praxis is used to analyze the faculty dynamics in the candidate selection process situated in a race neutral institutional culture. This reflective case study of an educational leadership department draws on qualitative data such as field notes from faculty conversations, experiential knowledge, memos, and quantitative data describing the disproportionate rejection of Black women applying to an educational leadership program in the US. Efforts to confront a race neutral process prompted by the higher rejection rate of Black women in comparison to their white counterparts prompted some faculty to engage in race conscious discourse. Faculty in departments of educational leadership who provoke race conscious dialogue on how they are implicated in institutional racism will likely face risks they will need to (em)brace for the battle against inertia

Modeling Social Justice Educational Leadership: Self-Assessment for Equity (SAFE)

In this chapter we present a model of self-assessment for equity (SAFE) to guide faculty in leadership preparation programs concerned with equity and justice. The conceptual framework for this model derives from a review of the literature on social justice in educational leadership preparation and addresses the gap in the literature on the evaluation of educational leadership programs. We discuss how we conceptualize and implement SAFE in the context of the mission statement guiding the educational leadership program in which we work

Youth Engagement in Electoral Activities: A Collaborative Evaluation of a Civic Education Project

Youth civic engagement is recognized as an essential component necessary for the preservation of democratic practices; however, inadequate levels of civic participation persist among young people. Past research has shown that young people are more likely to participate in politics when they are informed. We present survey data collected from middle and high school students during a collaborative evaluation of a civic education project. A total of 4492 students participated in the pre-survey, and 3180 students participated in the post-survey. The aim of this part of the evaluation was to explore students’ civic engagement before and after the infusion of civic education into the curriculum. We discuss reasons hindering students’ intent to vote, their satisfaction level with the 2008 presidential election, and how students planned to become involved in the 2012 presidential election. The results of the post-survey show an increase in students registered to vote and students’ intent to register to vote after the infusion of civic education into the curriculum. The findings suggest that civic education seems to positively influence students’ political behavior

How do Propensity Score Methods Measure up In the Presence of Measurement Error?

Considering that the absence of measurement error in research is a rare phenomenon and its effects can be dramatic, we examine the impact of measurement error on propensity score (PS) analysis used to minimize selection bias in behavioral and social nonexperimental studies. A Monte Carlo study was conducted to explore the effects of measurement error on treatment effect and balance estimates in PS analysis across seven different PS conditioning methods. In general, the results indicate that even low levels of measurement error in the covariates lead to substantial statistical bias in estimates of treatment effects and concomitant reduction in confidence interval coverage across all methods of conditioning on the PS