Learning about Measuring Advocacy and Policy Change: Are Baselines always Feasible and Desirable?

This paper captures some recent challenges that emerged from establishing a baseline for an empowerment and accountability fund. It is widely accepted that producing a baseline is logical and largely uncontested – with the recent increased investment in baselines being largely something to be welcomed. This paper is therefore not a challenge to convention, but rather a note of caution: where adaptive programming is necessary, and there are multiple pathways to success, then the ‘baseline-endline’ survey tradition has its limitations. This is particularly so for interventions which seek to alter complex political-economic dynamics, such as between citizens and those in power

Evaluation and Impact Investing: A Review of Methodologies to Assess Social Impact

This report highlights the paradox within impact investing: the prioritisation of ‘social impact’ without prioritising ‘impact evidence’. The growth of metrics, ratings and certification-based approaches has sought to address this gap but this only goes so far, and there is a need for a more evaluative approach to assessing impact. Drawing on the field of development evaluation, the report sets out five criteria for a ‘more evaluative’ way of assessing impact (impact, differential impact, plausible causality, aggregation, and accountability). The report then reviews a subset of more than 100 resources against these criteria and concludes that while there are some promising methods, each has different strengths and limitations in providing a more robust assessment of impact. As such, the report warns that trade-offs need to be carefully considered, for instance between different methods that provide greater standardisation versus those that provide greater specificity – and the cost/benefit trade-offs for investors in using each approach. Furthermore, just one method is unlikely to be sufficient by itself and there is a need for more guidance, innovation and learning for investors on methodological choices and how best to combine and complement different approaches for assessing impact in a cost-effective manner. Without such innovations it will become harder for impact investors to differentiate themselves from the more orthodox investment industry.UK Department for International Developmen

Learning about Theories of Change for the Monitoring and Evaluation of Research Uptake

This paper captures lessons from recent experiences on using ‘theories of change’ amongst organisations involved in the research–policy interface. The literature in this area highlights much of the complexity inherent in the policymaking process, as well as the challenges around finding meaningful ways to measure research uptake. As a tool, ‘theories of change’ offers much, but the paper argues that the very complexity and dynamism of the research-to-policy process means that any theory of change will be inadequate in this context. Therefore, rather than overcomplicating a static depiction of change at the start (to be evaluated at the end), incentives need to be in place to regularly collect evidence around the theory, test it periodically, and then reflect and reconsider its relevance and assumptions

Gauging Demand for Evidence and Accountability in Impact Investing by using Twitter Social Network Analysis: A Methodology

This report explains the methodology used to analyse the demand for evidence and accountability within the impact investment market. Social network analysis (SNA), qualitative interviews and ego network analysis were used to explore actors within this market (or system), drawing on data from Twitter to consider connections, influence and conversations about social impact. To date, all data collection for the Twitter SNA has been completed from 10 May 2016 to 18 July 2016, and a Policy Briefing has been produced from the initial findings (O’Flynn and Barnett 2016). Qualitative interviews and ego network analysis as part of the second half of the research are ongoing. A longer report regarding the main analysis and findings is also planned.UK Department for International Developmen

Assessing contrasting strategies for ensuring ethical practice within evaluation: institutional review boards and professionalisation

This paper explores the application of ethics in two contrasting approaches to evaluation: one that views evaluation as essentially a research project, and the other that sees evaluation as an extension of project management. We argue that the growth in so-called rigorous impact evaluation, characterised by practitioners as evaluation using experimental or quasi-experimental methods, has seen evaluation treated increasingly as a subset of research. This has entailed greater use of ethical committees, and specifically institutional review boards (IRBs), as many academics promoting the use of experimental methods are based in the USA. Elsewhere, evaluation is treated more as a management activity, with professionalisation initiatives such as membership standards and ethical guidance often used in the place of formal review. In this paper, we question whether the simultaneous growth in usage of IRBs and professionalisation addresses the ethical issues faced by evaluators