Case selection for robust generalisation in impact evaluation:lessons from QuIP impact evaluation studies

What wider lessons can be drawn from a single impact evaluation study? This article examines how case study and source selection contribute to useful generalisation. Practical suggestions for making these decisions are drawn from a set of qualitative impact studies. Generalising about impact is a deliberative process of building, testing and refining useful theories about how change happens. To serve this goal, purposive selection can support more credible generalisation than random selection by systematically and transparently drawing upon prior knowledge of variation in actions, contexts, and outcomes to test theory against diverse, deviant and anomalous cases

The Samagra anti-poverty programme in Madhya Pradesh: integrating household data, overcoming silo-problems and leaving nobody behind.

AbstractIf it is ‘to leave nobody behind’ in pursuit of the Sustainable Development Goals (SDGs), then a state requires a comprehensive and continuously updated database of individuals and households within its jurisdiction. Consolidation of this data can also assist in overcoming silo-based fragmentation in state delivery of anti-poverty programmes and services. We explore the feasibility of establishing such a database through an extended case study of Samagra (meaning ‘all comprehensive’) in the Indian state of Madhya Pradesh. Samagra was developed to facilitate integrated monitoring and management of all major government-to-people (G2P) cash transfers in the state, and (more radically) to support a paradigm shift from a demand-led to an entitlement-based approach to benefit delivery. Samagra is a unified population monitoring system based on continuous recording of household level demographic events, linked both to operational records of participation in diverse G2P programmes and to financial accounts provided by designated financial institutions operating within a five-kilometre radius of each household. The paper offers a practitioner’s account of Samagra’s introduction, including implementation challenges, policy outcomes and issues meriting further research and discussion.<br/

Friend or Patron? Social Relations across the National NGO-Donor Divide in Ghana

The article examines the institutions governing relations between grant using national NGOs and grant giving international donors in three regions of Ghana (Upper West, Northern and Greater Accra Region). Formal procedural rules and professional norms can be viewed as necessary to minimise opportunities for informal patronage, rent-seeking and corruption made possible by the unequal access to resources. Qualitative research confirmed that friendships originating in kinship and ethnicity, school links and past collaboration can also weaken NGO sustainability. But it also highlighted the positive role informal networks, connections, personal contacts, friendship and face-to-face contact play in enhancing collaboration between donors and national NGOs, building trust and strengthening lines of accountability - with non-adherence to shared norms resulting in sanctions and reputation loss. These findings echo Eyben (2010) in affirming the positive role of informal relations, and highlighting how they can complement formal rules and professional norms governing NGO-donor relations rather than undermining them. Compared a narrow emphasis on clientelism, the research throws a more positive light on the role of informal institutions and provides a more nuanced conceptual foundation for assessing ‘formalisation’ as a normative strategy. Donors concerned with supporting civil society need to be wary of trying to do so remotely and in ways that reduce opportunities for closer interaction and investment in trustful relationships

Approaches to evidence synthesis in international development : a research agenda

Abstract: This paper discusses the spectrum of synthesis methods available to generate, explore and text theory, their value to the field of international development and innovations required to make better use of the primary research available. It argues for clearer distinctions between syntheses produced as public goods, and those tailored to specific circumstances; and strengthening knowledge systems through greater use of maps to navigate existing and missing evidence, harmonised outcomes and measures, and advances in automation technologies. Improved methods and guidance are required for synthesising formative research and investigating contextual factors. Engaging stakeholders and working across academic disciplines support the production of policy‐relevant syntheses and inspire methods development

Financial inclusion, vulnerability, andmental models: From physical accessto effective use of financial servicesin a low income area of Mexico City

This paper addresses the issue of how to improve access to regulated financial services,hereafter referred to as financial inclusion. More widespread use of regulated financial servicescan enhance domestic resource mobilisation and improve allocative efficiency in the use ofcapital (World Bank 2007). However, the focus here is on financial inclusion as an instrument forpoverty reduction. This section considers why financial inclusion is of particular policy interestboth internationally and in Mexico. It then briefly reviews alternative theoretical approaches toanalysing financial inclusion and resulting welfare outcomes, particularly for relatively poor andvulnerable people.Section two presents a case study from a low income area of Mexico City. This illustrates howchoice of relevant theory for thinking about financial inclusion can be informed by empiricalresearch. We first use quantitative data to analyse the extent to which use of financial servicesvaries with education, employment, asset ownership and other indicators likely to affect personspecifictransaction costs. We then draw on complementary qualitative data to analyse otherinfluences on access to and use of financial services. Section three concludes that effective useof financial services is not only determined by individuals’ economic characteristics andexogenous transaction costs, but also by more complex cognitive and social processes. Thishighlights the dangers of relying too heavily on a narrowly economistic framework for analysis offinancial exclusion. More specifically, we argue for more research into how the diverse andchanging mental models of poor people influence their use and non-use of financial services

Attributing Development Impact: The Qualitative Impact Protocol Case Book

Substantiating cause and effect is one of the great conundrums for those aiming to have a social impact, be they an NGO, social impact investment fund, or multinational corporation. All face the same quandary: how do you know whether, or how, you contributed to an observed social change? A wide range of impact evaluation methodologies exist to address this need, ranging from informal feedback loops to highly elaborate surveys. But generating useful and credible information in a timely and cost-effective way remains an elusive goal, particularly for organizations working in complex, rapidly evolving and diverse contexts.Attributing Development Impact brings together responses to this challenge using an innovative impact evaluation approach called the Qualitative Impact Protocol (QuIP). This is a transparent, flexible and relatively simple set of guidelines for collecting, analysing and sharing feedback from intended beneficiaries about significant drivers of change in their lives. Innovative features include the use of ‘blindfolded’ interviewing to mitigate pro-project bias, and the application of a flexible coding system to make analysis and reporting faster and more transparent.The QuIP has now been used in 18 countries (including Ethiopia, India, Malawi, Mexico, Tanzania, Uganda and UK) with activities to promote food security, rural livelihoods, factory working conditions, medical training, community empowerment and microcredit for house improvement. This book includes comprehensive ‘how to’ QuIP guidelines and practical insights based on case studies from these countries into how to address the numerous methodological challenges thrown up by impact evaluation.Essential reading for evaluation specialists within NGOs, governments and donor agencies; social impact investors; community development practitioners; and researchers and students interested in evaluation methodologies