Towards Systemic Evaluation

Problems of conventional evaluation models can be understood as an impoverished ‘conversation’ between realities (of non-linearity, indeterminate attributes, and ever-changing context), and models of evaluating such realities. Meanwhile, ideas of systems thinking and complexity science—grouped here under the acronym STCS—struggle to gain currency in the big ‘E’ world of institutionalized evaluation. Four evaluation practitioners familiar with evaluation tools associated with STCS offer perspectives on issues regarding mainstream uptake of STCS in the big ‘E’ world. The perspectives collectively suggest three features of practicing systemic evaluation: (i) developing value in conversing between bounded values (evaluations) and unbounded reality (evaluand), with humility; (ii) developing response-ability with evaluand stakeholders based on reflexivity, with empathy; and (iii) developing adaptive rather than mere contingent use(fulness) of STCS ‘tools’ as part of evaluation praxis, with inevitable fallibility and an orientation towards bricolage (adaptive use). The features hint towards systemic evaluation as core to a reconfigured notion of developmental evaluation

Purity or pragmatism? : Reflecting on the use of systematic review methodology in development

Systematic review methodology pioneered in health care has been increasingly applied to development questions of importance in lower- and middle-income countries. This paper reports one such review on the topic of microfinance in sub-Saharan Africa and reflects on the number of pragmatic methodological compromises made when applying the method to a new field. These compromises relate to multidisciplinary teamwork, application of regional filters, drawing on evidence from additional study types and exploring mechanisms for change through the development and testing of a causal pathway. The paper concludes that a pragmatic rigorous approach to systematically reviewing evidence of effectiveness is needed for international development

Pragmatic meta analytic studies: learning the lessons from naturalistic evaluations of multiple cases

This paper explores the concept of pragmatic meta‐analytic studies in eLearning. Much educational technology literature focuses on developers and teachers describing and reflecting on their experiences. Few connections are made between these experiential ‘stories’. The data set is fragmented and offers few generalisable lessons. The field needs guidelines about what can be learnt from such single‐case reports. The pragmatic meta‐analytic studies described in this paper have two common aspects: (1) the cases are related in some way, and (2) the data are authentic, that is, the evaluations have followed a naturalistic approach. We suggest that examining a number of such cases is best done by a mixed‐methods approach with an emphasis on qualitative strategies. In the paper, we overview 63 eLearning cases. Three main meta‐analytic strategies were used: (1) meta‐analysis of the perception of usefulness across all cases, (2) meta‐analysis of recorded benefits and challenges across all cases, and (3) meta‐analysis of smaller groups of cases where the learning design and/or use of technology are similar. This study indicated that in Hong Kong the basic and non‐interactive eLearning strategies are often valued by students, while their perceptions of interactive strategies that are potentially more beneficial fluctuate. One possible explanation relates to the level of risk that teachers and students are willing to take in venturing into more innovative teaching and learning strategies

Gold standard or fool's gold: the pursuit of certainty in experimental criminology

This article assesses some of the claims made for experimental research in the field of rehabilitation of offenders. It suggests that both policy officials and evaluators have tended to over-invest financially and intellectually in a technocratic model of reducing reoffending that emphasizes programmes for offenders, and to under-invest in models that see the process as a complex ‘people changing’ skill. It argues that the complexity of this process renders it hard to evaluate using experimental methods of evaluation such as randomized controlled trials (RCTs). RCTs provide strong internal validity, but in complex settings offer weak external validity, making it hard to generalize from the experimental setting to other settings. The article suggests that the proper role for evaluative research in this field should be seen as building and testing middle-level theories about how best to change offenders’ behaviour

Making Out-of-School Time Matter: Evidence for an Action Agenda

Presents findings from a review of literature that identifies and addresses the level of demand for OST services, the effectiveness of the offerings, quality in OST programs, how to encourage participation, and how to build further community capacity

Participatory plant breeding and the evolution of landraces: A case study in the organic farms of the Collserola Natural Park

Although consumers appreciate the genetic diversity among tomato landraces, traditional varieties have been displaced from commercial agriculture. Their cultivation through organic farming in natural parks can contribute to their resurgence. With this aim, we developed a participatory plant breeding (PPB) program in Collserola Natural Park (Barcelona, Spain) to promote the conservation of the Mando landrace and to obtain new varieties adapted to local organic farmsPostprint (published version

Impact assessments in finance and private sector development : what have we learned and what should we learn ?

Until recently rigorous impact evaluations have been rare in the area of finance and private sector development. One reason for this is the perception that many policies and projects in this area lend themselves less to formal evaluations. However, a vanguard of new impact evaluations on areas as diverse as fostering microenterprise growth, microfinance, rainfall insurance, and regulatory reform demonstrates that in many circumstances serious evaluation is possible. The purpose of this paper is to synthesize and distil the policy and implementation lessons emerging from these studies, use them to demonstrate the feasibility of impact evaluations in a broader array of topics, and thereby help prompt new impact evaluations for projects going forward.Access to Finance,,Debt Markets,Banks&Banking Reform,Microfinance