Challenges in impact evaluation of development interventions: opportunities and limitations for randomized experiments

In recent years debates on as well as funding of impact evaluations of development interventions have flourished. Unfortunately, controversy regarding the promotion and application of randomized experiments (RE) has led to a sense of polarization in the development policy and evaluation community. As some proponents claim epistemological supremacy of REs (with respect to attribution) the counter reaction among others has been rejection. Needless to say, such extreme positions are counterproductive to reaching a goal that is commonly endorsed: to learn more about what works and why in development. This paper discusses the prospects and limitations of REs from the perspective of three categories of challenges in impact evaluation: delimitation and scope, attribution versus explanation, and implementation challenges. The implicit lesson is twofold. First of all, the question ‘to randomize or not to randomize’ is overrated in the current debate. Limitations in scope, applicability as well as implementation will necessarily restrict the use of REs in development impact evaluation. There is a risk that the current popularity of REs in certain research and policy circles might lead to a backlash as too high expectations of REs may quicken its demise. More importantly, given the nature and scope of the challenges discussed in the paper, more energy should be devoted to developing and testing ‘rigorous’ mixed method approaches within a framework of theory-driven evaluation.

Dealing with Stakeholder Values in the Evaluation of Development Programs. A Methodological Framework for Mid Term Evaluation

In mid term program evaluations evaluators are often confronted with the double task of retrospectively judging the program's merit and worth while at the same time advising decision makers concerning future adjustments in courses of action. In such cases, it can be argued that it is particularly important that evaluators take into account the divergent views and needs of different stakeholder groups. In principle, program theory evaluation can constitute a sound basis for dealing with the double objective of retrospective judgment and proactive program improvement. However, as argued in the paper, current approaches in program theory evaluation may not be sufficiently equipped to systematically deal with divergent stakeholder values. Taking into account lessons from the literature on stakeholder values in evaluation, an alternative methodological framework is presented. The framework combines program theory evaluation with elements of multicriteria decision aid. An example is used to illustrate the framework.

Impact Evaluations and Development: Nonie Guidance on Impact Evaluation

In international development, impact evaluation is principally concerned with final results of interventions (programs, projects, policy measures, reforms) on the welfare of communities, households, and individuals, including taxpayers and voters. Impact evaluation is one tool within the larger toolkit of monitoring and evaluation (including broad program evaluations, process evaluations, ex ante studies, etc.).The Network of Networks for Impact Evaluation (NONIE) was established in 2006 to foster more and better impact evaluations by its membership -- the evaluation networks of bilateral and multilateral organizations focusing on development issues, as well as networks of developing country evaluators. NONIE's member networks conduct a broad set of evaluations, examining issues such as project and strategy performance, institutional development, and aid effectiveness. By sharing methodological approaches and promoting learning by doing on impact evaluations, NONIE aims to promote the use of this more specific approach by its members within their larger portfolio of evaluations. This document, by Frans Leeuw and Jos Vaessen, has been developed to support this focus.For development practitioners, impact evaluations play a keyrole in the drive for better evidence on results and development effectiveness. They are particularly well suited to answer important questions about whether development interventions do or do not work, whether they make a difference, and how cost-effective they are. Consequently, they can help ensure that scarce resources are allocated where they can have the most developmental impact

Methodological Challenges in Impact Evaluation: The Case of the Global Environment Facility (GEF)

In this paper, we explore some of the methodological challenges that evaluators face in assessing the impacts of complex intervention strategies. We illustrate these challenges, using the specific example of an impact evaluation of one of the six focal areas of the Global Environment Facility; its biodiversity program. The paper discusses how theory-based evaluation can provide a basis for meeting some of the challenges presented.

Measuring Outcome and Impact of Small Development Projects: Lessons from the Evaluation of a Training Project on Low External Input Agriculture in Guatemala

Impact studies at the level of the beneficiary are not very common in the case of small-scale development projects. In this paper we will present the case of an evaluation study on a training project in Low External Input Agriculture in Guatemala. The evaluation design is based on a simple quasiexperimental design and complemented by qualitative methods of data collection. We want to illustrate that the type of mixed method evaluation used in the case can constitute a useful alternative to study the outcome and impact of small-scale development interventions given their specific constraints of money, expertise and time.

Poverty, institutions and interventions: a framework for an institutional analysis of poverty and local anti-poverty interventions

At a time when technological innovations are making our world increasingly smaller and our production systems are becoming increasingly more efficient, the benefits of economic growth and development as a whole have not been able to reach all of society. Indeed, many poor countries, characterised by their disadvantageous position in the global society and continuously plagued by weak governments, internal strife and natural disasters have missed out on many of the benefits of growth and development. Within countries that do gain advantage from the various developments of globalisation, significant groups continue to be excluded from the benefits of this new-found prosperity. It is quite significant that a generalised conclusion such as this is still a reality at the turn of the century, despite decades of national and international effort to promote development and combat poverty.