French version available in IDRC Digital Library: Constations en bref - examens externes des programmes : Gouvernance, sécurité et justice (GSJ) et Croissance pour tous (CPT)External Program Reviews at IDRC are a two-part process. They begin with the program reflecting on its own achievements in terms of three issues: strategy and evolution, outcomes, and the main lessons learned from experience. Following this, an independent and external panel judges the appropriateness of program implementation, the quality of research, and the relevance and significance of program outcomes – all with a view to identifying key issues for management consideration. In combination, the internal and external review facilitate instrumental learning and accountability

IDRC. Policy and Evaluation Division

International Development Research Centre: IDRC Digital Library

                                                 1 EVALUATING IDRC RESULTS External Program Reviews   FINDINGS BRIEF – External Program Reviews  Governance, Security and Justice (GSJ) & Supporting Inclusive Growth (SIG)     This findings brief provides a high level overview of the External Reviews of GSJ and SIG programs. Addressing these two separate evaluations in turn, it begins with contextual information on the programs, it summarizes the findings of their external reviews, and outlines key issues for Board consideration.  External Reviews are conducted by experienced, esteemed, and fully independent subject matter experts. The GSJ review (pp. 1-4) was prepared by Paige Arthur, Anna Paterson, and Necla Tschirgi. The SIG review (pp. 5-7) was conducted by Osvaldo Feinstein, Kunal Sen, and Lyn Squire.  External Reviews address four issues: 1 - To what extent are program outcomes relevant and significant? 2 - Overall, was the quality of research supported by the program acceptable? 3 - How did the program perform in implementing its prospectus? 4 - What are the key issues for IDRC’s Board of Governors and senior management?  For a research organization, there are implicit challenges in the simultaneous evaluation of program implementation and results. To measure results, time must pass for research to be conducted and disseminated for public consumption and uptake. To measure process, recent programming actions must be assessed in order to derive timely and actionable findings. To ensure that results (issues 1 and 2 above) and process (issue 3) are measured, IDRC External Reviews consider the spectrum of a program’s activities undertaken on a 5 year cycle, but there may be tradeoffs in the emphasis of older vs newer projects.  To reach evidence-informed answers to the above questions, evaluators use multiple data collection and analysis methods. Approaches to ensure validity of the findings are fully described in the ‘methodology’ sections of the evaluation reports, and are quality assessed.  As a point of accountability and a way of sharing our learning, External Review reports (and the programs’ self-assessment reports) will be made available to the general public on the IDRC’s website. External Program Reviews at IDRC are a two-part process. They begin with the program reflecting on its own achievements in terms of three issues: strategy and evolution, outcomes, and the main lessons learned from experience. Following this, an independent and external panel judges the appropriateness of program implementation, the quality of research, and the relevance and significance of program outcomes – all with a view to identifying key issues for management consideration. In combination, the internal and external review facilitate instrumental learning and accountability. 2  Overview of the Governance, Security and Justice (GSJ) Program  The goal of GSJ is to support the creation of policy-relevant knowledge on the conditions for increasing the legitimacy and accountability of public authorities in the areas of governance, security, and justice. The GSJ prospectus outlined that the program was expected to contribute to local and global debates on governance through strategic research investments in fragile and conflict affected states, as well as countries in transition.  The program was rolled out in April 2011, and encapsulates four former IDRC program areas: Peace, Conflict and Development; Women’s Rights and Citizenship; ICT4D; and the Middle East Governance Fund. From its inception to the beginning of this evaluation period (September 2014), the GSJ program allocated a total of CAD 43.7 million. The average value of projects increased from $435,552 in 2011–2012 to $544,264 in 2013–2014. A snapshot of the GSJ program is available in the GSJ dashboard (see GSJ Final Prospectus Report). The Implementation Plan presented to the Board at the same time as the evaluation addresses the planned future direction.   Summary of GSJ External Review Findings  GSJ outcomes and results:  The evaluation examined knowledge generation, capacity building and policy influence outcomes. A description of the evolution and course corrections taken by GSJ, over the period of review, is addressed in the ‘implementation and strategy’ section below. With respect to knowledge generation, the evaluation found three clusters of significant and relevant research findings that represent the beginnings of a critical mass on a particular thematic issue, usually regionally defined. However, the individual project contributions stop short of what one would call “program-level” contributions to knowledge generation, as they have not as yet coalesced sufficiently around common research questions. In short, GSJ’s outcomes so far mainly relate to parallel, rather than coordinated, research efforts. The evaluation found that GSJ’s main outcomes with respect to capacity building were driven through approaches to building individual skills and knowledge development, rather than institutional capacity development. This was a positive area of GSJ success. It is noted, however, that individual approaches are more relevant to countries and/or regions with already high institutional research capacity than they are to those with lower institutional capacity.  Programming should be mindful of this caveat. The evaluation found that GSJ’s influence on policy and practice is relevant and moderately significant. Primarily at the local/national level, the outcome of a GSJ-led prioritization of research relevant to local contexts. 3  Quality of GSJ supported research:  Overall the quality of GSJ-supported research was deemed to be acceptable or good, with the exception of the dimension of assessing the potentially negative consequences of research. However, the quality of research varied considerably within and between projects and regions.  Specifically, GSJ research measured well on issues of relevance and originality, with a particular strength in mixed methods designs bringing together scholars from different disciplines in new ways. They also scored well for their inclusiveness of vulnerable populations and engagement with local knowledge, reflecting GSJ’s focus on working with Southern partners including at the grass-roots level. Gender responsiveness was variable for different reasons. The technical merit of research outputs also varied, reflecting differences in research capacity across regions. Many research outputs failed to elucidate the methods and data on which findings were based, let alone to acknowledge any methodological limitations in the majority of research sampled. The evaluation suggests that GSJ has played a leading role in IDRC in pushing grantees to think about ethics; however, the evaluation did not find enough evidence to assess whether and to what extent ethics protocols had been used, including in research with human respondents.  This was a noted concern.  As a particular example of good practice in research use, the evaluation highlighted three baseline studies supported by GSJ. The panel judged that the three studies achieved exemplary uptake. These were studies for which there was clear demand from partner donors, and were considered particularly impactful by these users.  GSJ implementation and strategy:  An overarching finding related to implementation of the program is that the GSJ prospectus was extremely broad in scope and, as a result, the GSJ team had to undertake continuous adaptations and innovations to implement its program. The evaluation notes the GSJ portfolio was thinly spanned across the wide breadth of the prospectus and thereby made it difficult to identify a clear research agenda contributing to a common knowledge base. This is a concerning finding primarily related to implementation, but one that stretches into issues of GSJ results and GSJ research quality.  That said, the evaluation applauds GSJ’s ongoing efforts to bring greater focus and clarity to the prospectus through various mechanisms. These include launching several competitive calls around a narrower set of research topics; supporting region-specific, clustered thematic projects based on regional strategies; and collaborating with selected donors in order to increase available resources for expanded programming. 4  What are the key issues for IDRC’s Board of Governors and senior management?  Overall, GSJ has supported research in a broad range of themes and regions. The projects under the GSJ umbrella have generated new knowledge, with a particular strength in inter- disciplinary research, and they have achieved selected policy relevance especially at the local contextual level, but also in some international policy forums. The program is forging an identity out of this broad inheritance, and it is thinking about how to strengthen its thematic focus and maximize the impact of its programmatic resources, including the use of new funding modalities. Within this process there is room to consider the issues below that have emerged from the review.   Often the desired scale of impact is considerably larger than the scale of funding. Partnerships that leverage funds from other donors offer a solution to the problem.  The GSJ team struggled with multiple tensions between objectives that were only partly a result of GSJ’s broad prospectus scope. These tensions cannot simply be solved at the program level and require greater attention at the level of corporate strategy. These include tensions between the concurrent goals of knowledge generation, capacity building and policy influence, which may not all be achievable and realistic within the same project.  The scope and boundaries of the GSJ field and programming desires are broad. Going beyond IDRC’s stand-alone, grants plus model, innovative programming modalities recently deployed by GSJ should be actively encouraged for greater coherence and impact.  Greater attention should be paid to gathering adequate monitoring data to identify and manage performance issues, to assess the cost of research and capacity building in different contexts and, crucially, to facilitate learning across broad programme portfolios.  Establishing cohesive program focus is an issue that requires greater attention at very start of the programming period. This is the defining difference between research project success and IDRC program success.  Where research projects aim to build the capacity of southern researchers, research outputs should be encouraged to meet minimum standards of research quality. Standardizing such basic requirements could considerably improve the quality of research outputs across different types of research. 5  Overview of the Supporting Inclusive Growth Program (SIG)  The Supporting Inclusive Growth (SIG) Program was initiated in April 2011. The future direction of the program is outlined in the Implementation Plan, being concurrently presented to the Board. The purpose of the SIG program is to support research that provides a better understanding of how policy can promote growth and inclusion simultaneously. The core of the SIG program clusters around two foci – enterprise development and employment opportunities – with gender being a cross-cutting theme. The program builds on some of the results and lines of activity of three previous IDRC programs – Globalization, Growth and Poverty (GGP), Women’s Rights and Citizenship (WRC) and Information and Communications Technologies for Development (ICT4D). During the period April 2011 - July 2014, SIG supported 120 projects in 50 countries, which varied significantly in size, ranging from small amounts of seed funding to core grants up to CAD 3 million for the Partnership for Economic Policy. A simple visualization of the SIG program is accessible in the SIG dashboard (see SIG Final Prospectus Report).   Summary of SIG External Review Findings  SIG outcomes and results:  The evaluation assessed outcomes in the three key areas for IDRC programs – knowledge generation, capacity building, and policy influence. A description of the evolution and course corrections of the program, over the period of review, is addressed under ‘implementation and strategy’ below.  Regarding contribution to knowledge, it found mixed performance. Much of SIG-supported research was of high quality (especially for uptake and use) and was validated as such by international experts. However, the vast majority of SIG research projects is intended for national or regional development agents (including policy-makers) and this implies a trade- off. In simple terms, this focus implies that little of the contextually important knowledge created is likely to become well known in larger international development circles. The identification of this trend is not a negative result, but worth noting. Furthermore, the program may have improved its overall contribution to knowledge creation if a more cohesive framework to selecting projects was used.  Almost all projects were implemented by Southern researchers so the scope for capacity building was immense. The panel was struck, however, by the limited extent of formal capacity building efforts in the sample of projects examined by the panel – the results on capacity building were judged low in almost half of the projects sampled for review.  The panel rated the program’s efforts to influence policy as very good.  SIG presence was instrumental in realizing a powerful and sustained policy push. In the panel’s estimation, no 6  other institution can match this record. The key to SIG’s success in this respect has been the persistent, start-to-finish interaction with policy-makers. The only caveat, but an important one, is that no systematic approach to quality control of research outputs prepared for policy-makers emerged from the review of project documentation, and follow-up questions with Southern researchers qualified that findings were not routinely reviewed for accuracy and quality of analysis prior to delivery to policy-makers.   Quality of SIG supported research:  The dimensions of research quality examined by the panel were rated as acceptable or good, with the exception of gender responsiveness, considered less than acceptable on average. Thus, the methodological standards in the design and execution of the research, the extent of engagement with local knowledge, and the originality of the research were all considered acceptable, while relevance and efforts to reach policy-makers were judged to be good and, indeed, a strength of the program. An additional review of value-for-money indicated a good result for the SIG program. This analysis is detailed in Appendix J & K of the full report.  An interesting and important result to emerge from the analysis is the strong positive correlation between methodological standards and efforts to reach policy-makers. This suggests that successful projects had both a rigorous methodology and a strong policy focus while unsuccessful ones were weak methodologically and inadequately positioned to reach policy- makers.   SIG implementation and strategy:  The overarching question posed by the SIG program is ‘what policies can promote growth and inclusion simultaneously’. The actual research, however, focuses on the two underlying themes of enterprise development and decent jobs with gender as a cross cutting theme. The panel considers implementation with respect to the two research foci to be acceptable but notes a gap between them and the overarching issue of ‘inclusive growth’.  With that noted, the panel also observed a progressive improvement in the coherence of the program as carry-over projects from previous programs gave way to ones brought in solely at the current team’s discretion. This is especially true of the cross-cutting theme of gender (an issue of capacity and research quality) – absent from many of the earlier projects but very evident in later ones like the large-scale Growth and Women’s Economic Empowerment Program partnership.  Overall, reviewers noted positive trends in efforts by the SIG program to course correct and improve over the prospectus period. 7  What are the key issues for IDRC’s Board of Governors and senior management?  This evaluation brought to light several issues, the most noteworthy being the critical importance of involving experienced, topic-knowledgeable researchers (Northern or Southern) in IDRC research to ensure maximum contribution to knowledge without compromising capacity building and policy influence. This is a general point. Where such expertise was absent, the resulting analysis was disappointingly limited.  Other issues emerging from the evaluation include:   Engaging with partners should be managed in a way that maintains IDRC focus on Southern researchers and on IDRC’s agenda. While working with partners such as DFID and Hewlett Foundation is a welcome feature of SIG’s evolution in recent years, there are possible trade-offs as many of the Southern research institutions with which IDRC has built long-term relationships may not receive funding if partnerships result in more of SIG funds being channeled through global competitive calls with a greater share of resources going to Northern-led projects.   Policy inputs require careful review before delivery to policy-makers. This could be addressed by allocating resources specifically for this purpose.   Identifying opportunities to link programing to major external development activities such as World Development Reports or Human Development Reports and doing so with sufficient lead time would increase visibility and effectiveness of IDRC programs.   A strong conceptual framework is critical for a tight research program focus and an eventual prominent impact. 

Evaluating IDRC results : external program reviews

https://idl-bnc-idrc.dspacedirect.org/bitstream/10625/54450/1/IDL-54450.pdf

Evaluating IDRC results : external program reviews

Abstract

Similar works

Full text

Available Versions

International Development Research Centre: IDRC Digital Library