The core of the NCDP Program clusters around four foci: tobacco control, alcohol control, healthy diets and physical activity. The evaluation examined capacity building, policy influence and knowledge generation outcomes. The Governance for Equity in Health Systems (GEHS) program aims to strengthen individual researchers, research teams and institutions in low- and middle-income countries by supporting grantees to examine health system challenges towards increasing equity, improving governance and promoting systems integration. With regard to capacity strengthening, the evaluation found multiple lines of evidence suggesting that the GEHS program has built capacities or influenced capacity development at national and regional levels

IDRC. Policy and Evaluation Division

International Development Research Centre: IDRC Digital Library

1  FINDINGS BRIEF – External Program Reviews  Non-Communicable Disease Prevention (NCDP) & Governance for Equity in Health Systems (GEHS) Programs    This findings brief provides a high level overview of the External Reviews of the NCDP and GEH programs. Addressing these two separate evaluations in turn, it begins with contextual information on the programs, summarizes the findings of their external reviews, and outlines key issues for Board consideration.   External Reviews are conducted by experienced, recognized and fully independent subject matter experts. The NCDP review (pp. 2-4) was prepared by Stella Bialous, Rachel Nugent, and Sylviane Ratte. The GEHS review (pp. 5-7) was conducted by Joseph Decosas, Sarah McFarlane and John Frank.   External Reviews address four issues:  1 - To what extent are program outcomes relevant and significant?  2 - Overall, was the quality of research supported by the program acceptable?  3 - How did the program perform in implementing its prospectus?  4 - What are the key issues for IDRC’s Board of Governors and senior management?   For a research organization, there are implicit challenges in the simultaneous evaluation of program implementation and results. To measure results, time must pass for research to be conducted and disseminated for public consumption and uptake. To measure process, recent programming actions must be assessed in order to derive timely and actionable findings. To ensure that results (issues 1 and 2 above) and process (issue 3) are assessed, IDRC External Reviews consider the spectrum of a program’s activities undertaken on a 5 year cycle, but there may be trade-offs in the emphasis of older vs. newer projects.   To reach evidence-informed answers to the above questions, evaluators use multiple data collection and analysis methods. Approaches to ensure validity of the findings are fully described in the ‘methodology’ sections of the evaluation reports, and are quality assessed by IDRC’s Policy and Evaluation Division. Program Reviews at IDRC are a two-part process.  They begin with the program reflecting on its own achievements in terms of 3 issues: strategy and evolution, outcomes, and the main lessons learned from experience. Following this, an independent and external panel judges the appropriateness of program implementation, the quality of research, and the relevance and significance of program outcomes – all with a view to identifying key issues for management consideration.  In combination, the internal and external review facilitate instrumental learning and accountability.   EVALUATING IDRC RESULTS External Program Reviews 2  As a point of accountability and a way of sharing our learning, External Review reports (and the programs’ self-assessment reports) will be made available to the general public on the IDRC’s website. Overview of the Non-Communicable Disease Prevention (NCDP) Program  The goal of NCDP was to support locally-led research designed to inform the adoption and effective implementation of policies and programs that are low cost but can have a high impact on reducing the common non-communicable disease (NCD) risk factors and improve overall population health in low- and middle-income countries (LMICs).   NCDP began in June 2011 and built on a strong base of evidence and experience from IDRC’s Research for International Tobacco Control (RITC) program.  The Program portfolio for 2011-2014 included 74 projects (research projects, research support projects, and awards projects) with a combined value of CA$21.7 million. The core of the NCDP Program clusters around four foci: Tobacco control, alcohol control, healthy diets and physical activity.  In 2014 NCDP merged with the Ecosystems and Human Health program. The External Review panel restricted its assessment to 3.5 years of NCDP programming alone, given that the ECOHEALTH program was externally reviewed by a separate panel at the end of 2014.   Summary of NCDP External Review Findings  NCDP Outcomes and results:  The evaluation examined capacity building, policy influence and knowledge generation outcomes. A description of the program’s evolution and course corrections taken by NCDP over the period under review is addressed in the “implementation and strategy” section below.  With respect to capacity building, the evaluation found that the Program achieved a high level of performance in increasing capacity to conduct high-quality research, with examples of capacity building well-calibrated to development and/or research priorities and challenges in specific project operating contexts.  In the area of policy influence, multiple sources of evidence consulted suggested that the Program is a leader in promoting high-quality, pioneering research in NCDP risk factors with a focus on policy change as an outcome. This is an impressive achievement in a relatively young field in Lower and Middle Income Countries (LMICs). The evaluators noted that while large projects demonstrated that scaling can produce positive policy outcomes, there is also value in also supporting smaller research projects, particularly in areas where new knowledge is being developed.   The knowledge generation goal was attained at a moderate to high level across projects reviewed. The Program was particularly successful in generating knowledge in the fiscal policy research thematic. The evaluators noted, however, that IDRC is often not credited for their knowledge generation contributions.  This is due in part to the fact that project findings may be published in formats that serve to influence policy within project or country contexts, but do not 3  necessarily allow for dissemination of results to wider audiences.  Quality of NCDP Supported Research:  Overall the NCDP supported research assessed was deemed to be of good to very good quality.  The Program was productive with examples of influential research that involved local researchers and appeared in international peer reviewed journals. With a few exceptions, the evaluation noted that the Program was able to ensure that projects included strategies to position research for use by different audiences.  Since NCDP program themes are embedded in an emerging research area, projects should incorporate concurrent research capacity strengthening activities; the Program showed a strong record of capacity strengthening in the design and conduct of NCDP’s research projects.   NCDP Implementation and Strategy:  Overall, NCDP performed very well in an environment that faced a number of political and data challenges and in the midst of internal reorganization. Funding allocation was coherent and consistent with the Prospectus. The NCDP pioneered a new research focus on Non-Communicable Disease risk factors, filling an evidence gap and complementing other funders’ contributions.   Lessons from the former tobacco control program were well integrated and transition to NCDP was mostly smooth and efficient. Choices were made to strengthen NCDP’s position by narrowing the Program’s focus (physical activity focus was dropped). The rationale for these choices was coherent, efficient and appropriate. There is evidence of efforts to develop partnerships internally and externally.  These efforts need further strengthening with support from different parts of IDRC, if the program seeks to achieve greater visibility.   Key Issues for IDRC’s Board of Governors and Senior Management  The issues raised by the evaluation pertain mainly to capitalizing on the contributions the NCDP Program has made to NCD prevention in LMICs and how IDRC can expand on those accomplishments as part of its strategic objectives in the 2015-2020 period.   Capitalize on partnership opportunities: NCDP’s investments have been instrumental in galvanizing the attention of other funders who now have a better understanding of the importance of NCD prevention. This offers future opportunities to retain its niche in this field, strengthening and expanding partnerships with interested agencies and funders.   Build Leaders:  Having constructed a network of researchers who are now capable of producing high quality, policy relevant research, IDRC is well-positioned to continue to build research leadership in the NCDP field.  This could be achieved by increasing support to strategic networks working in NCD prevention and by enhancing connections with other IDRC programs such as the Think Tank Initiative and the former EcoHealth Program, moving NCDP work to a new level and in exciting new directions. 4   Communicate, Synthesize and Share Results: Overall IDRC has a strong reputation and is uniquely positioned in an area where there are limited funders. Greater synthesis, communication and sharing of existing and emerging results carries with it the potential to help advance and energize the global NCD prevention network of researchers, funders, and policy workers.  Well-documented lessons and solutions would also widen opportunities for country-level impacts and would bolster partnership discussions with new funders using evidence-informed arguments.   5  Overview of the Governance for Equity in Health Systems (GEHS) Program  The IDRC Governance for Equity in Health Systems (GEHS) program has the goal to strengthen individual researchers, research teams and institutions in low- and middle-income countries (LMICs) to build research capacities, contribute to appropriate and innovative methodologies and generate a body of credible knowledge to influence policies and practices at all levels of the health system from local to global. The program does this by supporting its grantees to examine health system challenges and work towards increasing equity, improving governance and promoting systems integration.   The 2011-2016 Program builds on two previous phases of work which began in 2002.  This work started with the recognition that strengthening health systems was a new field, especially in LMICs with a limited number of capable institutions and researchers. IDRC was and continues to be viewed as one of the most important donors in this area.  During its third programing period, the GEHS program integrated projects and staff from the former Information and Communications Technologies for Development program area and from the Global Health Research Initiative. At the time of this External Review, GEHS had supported 65 research projects with a total budget of $40.8 million.  Summary of GEHS External Review Findings  GEHS Outcomes and results:  The 2011-2016 Prospectus identified four outcome areas focusing on the goal of improved health in LMICs through strengthened equitable health systems: capacity strengthening of LMIC researchers and institutions; development of research methodologies; knowledge generation; and influence of policies, practices, agendas and funding priorities. A description of the program’s evolution and course corrections taken by GEHS over the period under review is addressed in the “implementation and strategy” section below.  The evaluation verified the achievements described by GEHS as being generally accurate for projects reviewed and noted that the Program demonstrated a positive influence on how health systems research (HSR) is undertaken in LMICs. The panel confirmed that the projects produced outcomes that were relevant to the strengthening of health systems, particularly at the national and regional levels and that this work has generated or will likely generate significant contributions to health systems development.  With regard to capacity strengthening, the evaluation found multiple lines of evidence suggesting that the GEHS program has built capacities or influenced capacity development at national and regional levels. The African Doctoral Dissertation Research Fellowships project was cited as an excellent example of a model that builds capacity of individuals and their institutions.  To achieve a critical mass, evaluators cited two projects with long term support that have exceled. The evaluation noted, however, that GEHS could better understand its capacity development results and could gain greater visibility by more systematically tracking information on the publications and career paths of the individuals it supports. 6   With regard to methodological development, the panel highlighted several examples of GEHS achievements in bringing together researchers from different disciplines to generate knowledge that is robust and replicable. The evaluation documents a number of examples of scientifically sound and consistently used research methods. The collaboration with the World Health Organization’s  Alliance for Health Policy and Systems Research which resulted in an influential journal supplement and training package was noted as having great promise for impacting the course of health systems research.  Regarding contribution to knowledge, the panel noted that GEHS has made important contributions in building knowledge to influence local health policies and practices and that at a global level, the projects reviewed are expanding the reach and deepening the health systems research knowledge base.     GEHS performed well in supporting research that is positioned to influence policies and practices with examples of policy papers, training manuals and other materials that enhanced the receptivity of findings among different audiences. A key good practice involved engaging key audiences early in the research process. The panel also noted that the program has started to develop a critical mass of researchers who are networking across countries and regions to build and sustain capacity and who are undertaking research that is influencing national, regional and global practices, policies and agendas.   Quality of GEHS Supported Research:  Overall, the panel rated the quality of GEHS supported research as generally good, and often very good and demonstrated high levels of integrity and adherence to research ethics standards. There were limited examples of research that was of less than acceptable quality. In these cases, the knowledge products were noted as early outputs of projects with less experienced investigators that had potential to improve with ongoing technical and capacity building support.  While at least half of the sampled projects were implemented in countries with considerable political volatility and weak governance, the evaluation found that this did not affect research quality. IDRC’s approach of close interaction with and targeted support to grantees was the main mitigation factor allowing for early detection of potential risks and rapid action.  Among the quality parameters examined, gender responsiveness was the weakest – meaning that in some cases the research was less strong in analyzing and reporting on the gender dimensions of the research. The evaluation noted that gender responsiveness continues to require further strengthening in Health Systems Research in general.  GEHS Implementation and Strategy:  The evaluation found that GEHS made well-balanced investments between 2011 and 2015 across its intended outcome areas. Capacity building received 39 percent of funding which was 7  considered appropriate given the status of Health Systems Research (HSR) as a relatively new and emergent field in LMICs. The three areas of knowledge generation, methodological development and influencing policy and practice each received roughly 20 percent of investments.  The quality and the scope of activities supported by GEHS during this period were fully in line with the expectations raised in its Prospectus, and in the view of the panel, actually exceeded expectations, given changes faced during the program period. The evaluation judged the spread of project topics, settings, research partners (old vs. new), size and cost to be appropriate with the GEHS team working hard to develop and support projects in under-funded and challenging areas. The Program demonstrated a high level of adaptability, by identifying and effectively mitigating risks, adapting to contextual changes at IDRC and in the program countries, and by recognizing and engaging in emergent new opportunities such as the increased focus of the Canadian Government on the support of maternal, neonatal and child health, and the emergent international focus on Universal Health Coverage. The panel noted positively GEHS’ learning-based approach to project development with regular reflective practice and collective team discussions as a strength.  Key Issues for IDRC’s Board of Governors and Senior Management  The overall judgement of the panel is that the GEHS program performed at a high level, with strong fidelity to the 2011-2016 Prospectus. With the support of GEHS, individuals, institutions and networks in LMICs have grown to occupy strong positions to influence local, national, regional and global policy agendas.  The panel raised four issues for consideration:   Build on synergies and unique leadership:  Through GEHS, IDRC has played a critical role among funders of health research for development and the Centre is viewed as a leader in developing and supporting research capacity with a governance and equity lens in LMICs.  IDRC should seek ways to maintain its unique leadership in this field.    Communicate and publicize results: The program is not as visible globally as it might be. IDRC should further publicize and disseminate its work in Health Systems Research and ensure representation on relevant advisory and implementation bodies.  Opportunities for doing this exist within the IMCHA portfolio, given that strengthened health systems are a key prerequisite for progress in reproductive, maternal, neonatal and child health.  Review partnership good practices to widen impact: To build on GEHS’ achievements in growing cadre of health systems researchers in LMICs, IDRC should consider long-term partnerships with other funders for the implementation of a regional institution-based approach in West Africa. Such partnerships would allow for greater support to research mentorship programs, networks, and other capacity strengthening initiatives.   Strengthen monitoring and evaluation of results: While it is difficult to attribute changes in health outcomes to investments in research or research capacity building, IDRC needs to get better at defining and monitoring results beyond the immediate outputs of projects.  

Evaluating IDRC results

https://idl-bnc-idrc.dspacedirect.org/bitstream/10625/55257/1/IDL-55257.pdf

Evaluating IDRC results

Abstract

Similar works

Full text

Available Versions

International Development Research Centre: IDRC Digital Library