Steps Toward Evaluation as Decluttering: Learnings from Hawaiian Epistemology

This paper discusses one of the more contemporary challenges in development and in global health--lots of good ideas from well-meaning insiders and outsiders that end up cluttering both the physical and mental spaces of what can be loosely termed as “attempts” at development. Given the place-based nature of indigenous thought, we turn to Hawaiian epistemology at looking to insights for clarity on how one can negotiate interactions to declutter place and also confuse identity.  We believe that evaluation as a field can help in bringing greater recognition of the need for models of development and learning that respect the importance of de-cluttering.  Implications for a decolonized approach to evaluation are discussed

Building Spaces for Dialogues to Rethink Evaluator Competencies: Lessons from the Webinars Organized by the Evaluation Centre for Complex Health Interventions

Background:  There is a need to rethink evaluator competencies given the harsh and paralyzing realities of COVID.  The pandemic was a time where there was a need to balance diverse perspectives given the limited scientific evidence that existed when faced with a genuinely unprecedented time. In the Fall of 2021 (September to October), the Evaluation Centre for Complex Health Interventions in partnership with the Asia Pacific Evaluation Association organized a three-part webinar series in response to the multiple issues that surfaced during COVID-19, and specifically, the implications of the pandemic for rethinking evaluator competencies and evaluator training. The presenters were from multiple countries including India, Canada, USA, UK, and South Africa. Purpose: The presenters pushed for more responsive evaluation approaches to address inequities and sustainability and for a decolonized approach to knowledge building.  The webinar raised a number of themes that have potential implications for future discussions on evaluator competencies including:  enhancing evaluation contributions to the Sustainable Development Goals (SDGs), the need to rethink evaluation criteria,  the need to embrace and address varieties of uncertainties,  focus on diversity and heterogeneity;  understanding the role of contexts in complex programs and policies;  the need to reconceptualize sustainability;  being more explicit about inequities and vulnerabilities; and the need to pay attention to systems and system dynamics. Setting:  The webinars were organized by the Evaluation Centre and the Asia Pacific Evaluation Association on a Zoom platform. Intervention: Not applicable. Research Design: Not applicable. Data Collection and Analysis: Not applicable. Findings: Not applicable

Ten steps to making evaluation matter

This paper proposes ten steps to make evaluations matter. The ten steps are a combination of the usual recommended practice such as developing program theory and implementing rigorous evaluation designs with a stronger focus on more unconventional steps including developing learning frameworks, exploring pathways of evaluation influence, and assessing spread and sustainability. Consideration of these steps can lead to a focused dialogue between program planners and evaluators and can result in more rigorously planned programs. The ten steps can also help in developing and implementing evaluation designs that have greater potential for policy and programmatic influence. The paper argues that there is a need to go beyond a formulaic approach to program evaluation design that often does not address the complexity of the programs. The complexity of the program will need to inform the design of the evaluation. The ten steps that are described in this paper are heavily informed by a Realist approach to evaluation. The Realist approach attempts to understand what is it about a program that makes it work.Evaluation design Program theory Pathways of influence Learning frameworks Design Learning Spread

The potential of an online educational platform to contribute to achieving sustainable development goals: a mixed-methods evaluation of the Peoples-uni online platform

Abstract Background This paper reports on an online platform, People’s Open Access Education Initiative (Peoples-uni), as a means of enhancing access to master’s level public health education for health professionals. Peoples-uni seeks to improve population health in low- and middle-income countries by building public health capacity through e-learning at very low cost. We report here an evaluation of the Peoples-uni programme, conducted within the context of Sustainable Development Goal 4, which seeks to “ensure inclusive and quality education for all and promote lifelong learning” by 2030. The evaluation seeks to address the following three questions: (1) Did Peoples-uni meet its intended goals? (2) What were the different types of impacts that students experienced? (3) What suggestions for future changes in Peoples-uni did students recommend? Methods A mixed methods evaluation consisted of two parts, namely an online survey and a telephone interview. A total of 119 master’s level graduates were invited to participate; responses were obtained from 71 of those invited, giving a response rate of 60%. Respondents were spread across 31 countries. Interviews were conducted with 18 respondents. Results There was strong evidence that Peoples-uni had achieved its stated goals. Potential impacts on students included knowledge to enhance practice and appreciation of context, enhanced research capacity through knowledge of public health, critical thinking and evidence-based programming, and empowerment of students about the potential of education as a means of improving their lives. Accreditation through future partnerships with local universities was recommended by students. Conclusions Peoples-uni has been able to deliver a credible public health master’s level educational programme, with positive impacts on the students who graduated. Challenges are to find a way to accredit the programme to ensure its sustainability and to see how to take full advantage of the current, and future, graduates to turn this from an education programme into a capacity-building programme with real impact

Incorporating gender, equity, and human rights into the action planning process: moving from rhetoric to action

Background: Mainstreaming of gender, equity, and human rights (GER) is an important focus of the World Health Organization (WHO) and other UN organizations. This paper explores the role of action plans in mainstreaming GER. This paper is informed by a theory-driven evaluation lens. Design: A theory of change framework explored the following seven dimensions of how action plans can implement mainstreaming of GER: awareness of the foundations of GER; understanding of context; planning to impact GER; implementation for GER; monitoring, evaluation, and learning; planning for sustainability; agenda setting and buy-in. The seven dimensions were used to analyze the action plans. Reviewers also explored innovations within each of the action plans for the seven dimensions. Results: GER mainstreaming is more prominent in the foundation, background, and planning components of the plan but becomes less so along the theory of change including implementation; monitoring and evaluation; sustainability; and agenda setting and buy-in. Conclusions: Our analysis demonstrates that much more can be done to incorporate GER considerations into the action planning process. Nine specific recommendations are identified for WHO and other organizations. A theory-driven approach as described in the paper is potentially helpful for developing clarity by which action plans can help with mainstreaming GER considerations

Heterogeneities in utilization of antenatal care in Uttar Pradesh, India: the need to contextualize interventions to individual contexts

Background: This paper explores the heterogeneities in antenatal care (ANC) utilization in India’s most populated state, Uttar Pradesh. Taking an intersectionality lens, multiple individual- and district-level factors are used to identify segments of any antenatal care usage in Uttar Pradesh Objective: This paper seeks to understand the multilevel contexts of ANC utilization. The planning and programming challenge is that such knowledge of contextual specificity is rarely known upfront at the initial stages of planning or implementing an intervention. Exploratory data analysis might be needed to identify such contextual specificity. Methods: Tree-structured regression methods are used to identify segments and interactions between factors. The results from the tree-structured regression were complemented with multilevel models that controlled for the clustering of individuals within districts. Results: Heterogeneities in utilization of any ANC were observed. The multiple segments of ANC utilization that were developed went from a low utilization of 23.7% for respondents who were not literate and did not have home ownership to a high of 82.4% for respondents who were literate and at the highest level of wealth. Key variables that helped define the segments of ANC utilization include: woman’s literacy, ownership of home, wealth index, and district-level sex ratio. Based on the multilevel model of any ANC utilization, cross-level interactions also were obtained between sex ratio and ownership of home as well as between sex ratio and literacy. Increases in sex ratio increased the influence of ownership of home on any ANC, while increases in sex ratio reduced the impact of woman’s literacy on receiving any ANC. Conclusion: We argue that a focus on heterogeneous segments of utilization can help build knowledge of the mechanisms that underlie inequities in maternal health utilization. Such knowledge of heterogeneity needs to be incorporated in contextualizing interventions to meet a variety of recipients’ needs