Quarry, Wendy

Ramirez, Ricardo

International Development Research Centre: IDRC Digital Library

In_Focus Evaluation: A Brief    Wendy Quarry & Ricardo Ramirez New Economy Development Group 30 April, 2007     Introduction In late 2006, the IDRC Evaluation Unit in partnership with the Communications Division contracted the New Economy Development Group (NEDG) to evaluate the In_Focus collection. The NEDG team (Wendy Quarry and Ricardo Ramirez), both professionals in Communication for Development, was asked to assess both the formative and summative aspects of the materials and to focus attention on three key productions: Seeds that Give; Eco-Health and Fixing Health Systems (TEHIP).   In_Focus Evaluation Objectives The evaluation is intended to be used by both Communications and Programs and Partnership Branch (PPB) staff to help them learn what has worked well in the formative aspects of the production and what could be improved. The intended uses of the summative part of this evaluation will be to determine the results and influence of the projects and the extent to which they are achieving their objectives (or achieving objectives that were unintended). The Evaluation Unit is also interested in the approach and findings as this type of communication evaluation adds a new dimension to the evaluation work of the Centre    The objectives of the evaluation are: 1. Assess the extent to which the projects are meeting their aims and objectives; 2. Document the results of the projects (reaches and outcomes) and analyze their influence; 3. Provide reflections on the strengths and weaknesses and the process and outputs of the In_Focus projects and the communication and dissemination of the material in relation to the subject matter content and context of each field.  Methodology Faced with the complexity of the research topic, we (the team) proposed a phased approach to the research process.  Roughly, the overall phases consisted of: a preparation phase (to get a preliminary overview of the In_Focus process and potential impact); the preparation of a detailed workplan (to guide the evaluation process in partnership with the Steering Committee); development of a methodological approach to data collection (for both the formative and summative aspects of the evaluation), data collection, analysis and report writing. Each phase was reviewed by the Steering Committee prior to moving forward from one to the next.  In_Focus Evaluation BRIEF  2  The data collection tools for the formative aspect of the evaluation included: individual or small group semi-structured interviews and focus groups. We paid attention to what was unique and what was common across the different projects. We explored the proposed uses of the different media products and the extent to which they were targeted to specific audiences. To do this we focused on IDRC staff (both at Headquarters and at Regional offices) and on IDRC partners (and met with 2 ghost writers).   The formative aspect also included a review of how similar organizations bridge the gap between research and policy though an on-line forum facilitated by the Pelican Initiative of the European Centre for Development Policy Management (ECDPM) in the Netherlands. Our on-line session focused on the question: “Research and Communication: Bridging the Research-Policy Gap?” and ran for six weeks from February 9th to March 23rd. 2007.    The data collection tools for the summative aspect of the evaluation included: individual interviews, and focus groups discussions. These were complemented by an on-line survey that was available from 9 Feb to 6 April in English, French and Spanish. The evaluators used email lists provided by the program division as no single database was available. A total of 105 surveys were received.  Finally, the summative aspect included a review of how similar organizations track the impact of their dissemination efforts of comparable products. This was done through a second on-line forum facilitated by the Communication Initiative from March 25th. to April 21st., 2007. Our session focused on the area of planning, producing, disseminating and evaluating the impact of advocacy or communication materials.  One field trip was made to both Syria (ICARDA in Aleppo) and Cairo to allow us to meet IDRC Partners and others who had had access to the In_Focus collection. This travel opened up the opportunity for face-to-face interviews with some of the potential audience for the material and widened the exploration of the In_Focus potential and use.  It was unfortunate that we could not manage more face to face interviews with “users” of the IDRC collection.  A trip to Tanzania was cancelled due to lack of availability of people familiar with the series and able to find the time to meet with the researcher. A trip to Mexico and Honduras was cancelled for the same reason.  Issues for Consideration From the beginning, it was very clear that the collection resonated strongly with different audiences. Style and format received positive response; size, length, writing style, and an attractive look are well appreciated by IDRC staff members, by others researchers and academics. A particular strength has been the collection‟s service to IDRC knowledge management - in terms of its attractive packaging of years of substantive research. A pleasant surprise has been the popularity of the materials as training resources for universities. In_Focus Evaluation BRIEF  3  The collection, however, misses the mark almost entirely for the initial objective of bringing research results to the attention of policymakers. It is not the type of product that can alone bring research results to policy. Apart from providing materials that are too long for this particular audience, the collection is not put together with the benefit of in-depth communication strategies (audience research) to figure out how to reach this audience, and how to provide just-in time-briefs to support linkages when policy windows open.    This is due, in part we believe, to the relative isolation of the work of the Communications Division from knowledge widely held within the building and among staff in regional offices and field placements.  While the collection has created an important precedent in reducing the silos between the Communications Division and the Programs -an achievement that deserves credit- there is scope for more work to reduce a noticeable in-house isolation.    Some Program officers have sound developmental communication instincts that are not adequately recognized nor harnessed into communication strategies; the Evaluation Unit‟s recent writings on how research gets into policy, offer practical insights that have not been utilized; and the organization's relevant Communication for Development experience has not been put to work internally.  Altogether, the combination of this internal knowledge could have helped bring the In_Focus collection closer to its original objective of bringing research to the attention of policy.   These findings led us to recommend the following:   Build on the success of a shared inter-departmental project to develop a more holistic approach to In_Focus productions  We suggest that a small working group be set up consisting of one to two people from: The Evaluation Unit; Communications Division and Program Initiatives to brainstorm on best methods for bringing research to policy. This could either be a working group over several weeks or a one-day initiative with facilitation.      Replace the Knowledge Pyramid with an alternative organizing framework for future In_Focus projects based on a dialogue between research and policy  The Knowledge Pyramid is a successful strategy for organizing and presenting knowledge. It also allows different audiences to access differing levels of material. It is however supply driven and as such has a prevailing one-way flow of information from the different source (anywhere in the pyramid) to the user. This sets the stage for a broadcast approach to materials dissemination that has no in-built cycle of monitoring and evaluation. It therefore follows that a second challenge is in the lack of or limited audience research and pre-testing phases. If In_Focus Evaluation BRIEF  4 the need to get research results into a policy agenda remains a key In_Focus objective then a more responsive and two-way mode of presenting (and acknowledging) information would be more appropriate.   Clarify objectives of each Project from the beginning  The original In_Focus objective of bringing research into the policy agenda clearly evolved into a multitude of differing objectives. This evolution was not, however made explicit nor did the particular objectives appear to be clarified at the beginning of each In_Focus production. Since clearly stating the objective of any communication initiative is an important first step in the communication planning process, we suggest that each In_Focus production start with a clear definition of the objectives for that particular project. In this way the different audiences can be clarified with research into their information needs. It may follow that the In_Focus format (book, CD and case studies) may not be the best way to meet each particular set of objectives.    Set clear and transparent selection criteria and charter  The present In_Focus selection criteria and Charter (contract) is inconclusive and open to change. It would be useful to develop a clear and transparent selection process accessible to all staff -regional and headquarters- within the IDRC. We found that regional staff were not aware of the intent behind the In_Focus collection nor of the selection process. Regional staff may be well placed to assess the degree of interest for a new production.    Develop full communication strategies for each project  We found that the communication strategies that were developed for the In_Focus Collection were, in essence, distribution strategies that failed to clearly identify and research specific audiences (they were also put in place after the project was well underway). A communication strategy based on the project objectives, key audiences and research into the particular audience knowledge and information needs (plus best channels for communication) is sound communication practice. It sets the stage for identifying a variety of media and methods appropriate to the set of objectives and the targeted audience for each objective.     Differentiate Ottawa Vs in-country production  IDRC-Ottawa could produce those materials that respond to knowledge management objectives (eg In_Focus projects that include an organizational archival objective), as well as materials intended for Canadian users (eg fact sheets for the timely drafting of policy briefs; websites to profile organizational achievements, training guides for teachers and faculty). At the same time, IDRC In_Focus Evaluation BRIEF  5 could focus efforts in select countries where partners are best positioned to produce and disseminate materials for their priority users.   We suggest at least three parallel production activities: materials that are directed at Canadian audiences should be produced in Ottawa; while projects should be able to do that same in-country. Common products with a common design remain appropriate when there are shared audiences with common needs (e.g. teachers). We feel books are the most likely to respond to two objectives: internal knowledge management, combined with dissemination to a broad practitioner-academic population. On the other hand, a close consultation with teachers may lead to proposals for a second generation of publications that are specifically developed for training programs.  Each case will be unique, and where the overlap between Ottawa and in-country production is too small, there may be room for separate communication efforts to be developed outside the In_Focus collection.   Budget for all resources from the start  We recommend that prior to the signing of the Charter, a focus group be put in place. The group would involve Program and Communications staff (veterans of the Collection process) and new staff (both the writer and his/her manager) interested in working on the potential project. The findings of the Research – Policy studies for the Centre could be integrated into an M&E component from the start.  The purpose of the discussion would be to offer a full opportunity for those wanting to be involved in a new project initiative to discuss the work load and expectations with people who had experienced the process. A budget would be put in place to include an estimate of person days and a financial investment   Develop a distribution tracking system  We were greatly hampered in assessing the summative aspect of In_Focus by the lack of any system to track distribution. To circumvent this problem we suggest instituting what we have dubbed, the One Third rule.   Allocate 1/3 of the books printed for general distribution without tracking who gets them. Our findings show that readers like to have these materials for general reference. In addition the materials do position the organization in the development research field. This 1/3 would continue to be given out at conferences and launches, and by staff and regional offices as courtesy „business cards‟.  Allocate 1/3 that will be tracked in a general manner: buyers‟ In_Focus Evaluation BRIEF  6 contact information will be kept.  Last, allocate 1/3 to be distributed through regional offices or specific projects in a targeted way, and with strict rules regarding the tracking of user information. The 1/3; 1/3; 1/3 proportion leaves room for variations, but what is important is the explicit setting of distribution decisions from the start. This applies both to Ottawa-based production and to decentralized, country-based production.    In addition we recommend a clear policy spelling out which materials are sold or given away. In general, users will value a product that has a value attached to it, though this does not necessarily mean it is the user who pays for it.   Access multi-organizational initiatives on research/policy issues  Finally we conclude that it is not always necessary to go at it alone. There are other organizations globally addressing the same research/policy issues as a community of practice; this allows lessons to be exchanged and innovation through interaction. In addition to building up cross usage of in-house knowledge, teams engaged within IDRC could benefit from joining multi-organizational initiatives such as the Research in Use Program that currently hosts several UK-based organizations (http://www.researchintouse.com/index.php?section=1    

In_focus evaluation : a brief

https://idl-bnc-idrc.dspacedirect.org/bitstream/10625/45805/1/132275.pdf

In_focus evaluation : a brief

Abstract

Similar works

Full text

Available Versions

International Development Research Centre: IDRC Digital Library