Article thumbnail

Meta-evaluations in government and government institutions: A case study example from the Australian Centre for International Agricultural Research

By T.P. Madzivhandila, Garry R. Griffith, Euan M. Fleming and A.E. Nesamvuni

Abstract

In this paper we draw on impact assessment work of the Australian Centre for International Agricultural Research (ACIAR) to present an example of meta-evaluation – an evaluation of evaluations – in an agricultural research, development and extension setting. We explore quality issues relating to evaluation studies in the context of government institutions. Program evaluation standards (PES) are divided into categories of utility, feasibility, propriety and accuracy to provide a framework for the meta-evaluation. The PES are presented as a universal measure of evaluation study quality. The intent of using them here is to judge the adequacy of PES as a universal quality measure or meta-evaluation base and to extract useful insights from ACIAR program evaluation activities when developing a meta-evaluation model for the Limpopo Department of Agriculture (LDA). Our meta-evaluation is undertaken of 63 impact assessment reports. First, the literature guiding the conduct of a meta-evaluation is reviewed. Second, an assessment (the meta-evaluation) of the evaluation studies is carried out for 19 sampled reports from a population of relevant reports fitting the dimension of the analysis, and results are presented and discussed. Also, lessons learned are presented, using the framework provided by the meta-evaluation criteria. Third, taking into account the lessons learned, implications are drawn for a proposed systematic meta-evaluation of the LDA. Finally, we conclude that all the PES cannot be equally emphasized in a meta-evaluation model. At ACIAR, 70% of the standards were at least partially addressed. Therefore, we succeeded in using the PES in judging the ACIAR evaluation quality. As such, they can be an important base when developing an evaluation model but should be applied in a contextualized manner.Meta-evaluation, Evaluation Quality, Program Evaluation Standards, Evaluation Model, Australian Centre for International Agricultural Research, Limpopo Department of Agriculture (South Africa), International Development,

OAI identifier:
Downloaded from http://purl.umn.edu/59098

To submit an update or takedown request for this paper, please submit an Update/Correction/Removal Request.

Suggested articles

Citations

  1. (2001a), 'Development of AES standards',
  2. (1991). A
  3. (1992). A case study of evaluation auditing as metaevaluation',
  4. (1984). A rating scale for assessing the quality of responsive/illuminative evaluations',
  5. (2008). A review and impact assessment of ACIAR's fruit fly research partnerships 10984-2007', Impact assessment study report: IAS 56 (Canberra: Australian Centre for International Agricultural Research).
  6. (2009). ACIAR Database for Impact Assessments (ADIA): an outline of the database structure and a guide to its operation', ACIAR impact assesment series:
  7. (2008). Analysis of ACIAR's returns on investment: Appropriateness, efficiency and effectiveness', ACIAR impact assesment series:
  8. (2005). Applying the Joint Committee’s
  9. (2005). Assessing international evaluations: An example from USAID’s gemocracy and Governance program',
  10. (2004). Assessment of rodent control projects in Vietnam: Adoption and impact', Impact assessment study report: IAS 24 (Canberra: Australian Centre for International Agricultural Research).
  11. (2009). Australian Centre for Intermational Agricultural Research 'Impact Assessment Programme', measuring impact accessed 26
  12. (1998). Australiuan tree species selection in China', Impact assessment study report:
  13. Beyond Ethics: Why We Need Evaluation Standards',
  14. (2001). Breeding and feeding of pigs in Australia and Vietnam', Impact assessment study report:
  15. (1999). Building capacity for organizational learning through evaluative inquiry',
  16. (2008). Concurrent Meta-Evaluation: A Critique ',
  17. (1998). Controll of new castle desease in village chickens', Impact assessment study report: IAS 01 (Canberra: Australian Centre for International Agricultural Research).
  18. (2004). Data- and literature-based reflections on Western European evaluation standards and practices', New Directions for Evaluations,
  19. (1997). Defining and describing the paradigm issues in mixed-methods evaluation',
  20. (2003). Descriptive Metaevaluation: Case Study of an Interdisciplinary Curriculum', Evaluation and the Health Professions,
  21. Establishment of a protected area in Vanuatu', Impact assesment study report:
  22. Eucalypt tree improvement in China', Impact assessment study report:
  23. (1997). Evaluating evaluation: Where do we go from here?',
  24. (2007). Evaluating Measures Taken Against Right-Wing Extremism',
  25. (1999). Evaluation and Performance Management in Government',
  26. (1988). Evaluation for decisions: Is anybody there? Does anybody care?',
  27. (2001). Evaluation in organizations: A systematic approach to enhancing learning, performance, and change
  28. (2003). Evaluation of gender mainstreaming: Ideas from a meta-evaluation study',
  29. (1999). Evaluation research and the performance management movement: From estrangement to useful integration?',
  30. (2007). Evaluation theory, models, and applications (San Francisco:
  31. (2002). Evaluation utilization as argumentation',
  32. (2004). Evaluation: A systematic approach (London:
  33. (1999). Evaluative inquiry for learning organization (Thousand Oaks:
  34. (1991). Foundations of program evaluation: Theories of practice (California:
  35. (1998). Framing participatory evaluation',
  36. (2008). Guidelines for assessing the impacts of ACIAR’s research activities', ACIAR impact assesment series:
  37. (2001). How modern democracies are shaping evaluation and the emerging challenges for evaluation',
  38. (2008). Impact of increasing efficiency and productivity of ruminants in India by the use of protectednutrient technology', Impact assessment study report: IAS 53 (Canberra: Australian Centre for International Agricultural Research).
  39. (1999). Improved drying of high moisture seeds', Impact assessment study report: IAS 14 (Canberra: Australian Centre for International Agricultural Research).
  40. (2003). Improved methods in diagnosis, epidemiology, and information management of footand-mouth deseases in South-East Asia', Impact assessment study report: IAS 21 (Canberra: Australian Centre for International Agricultural Research).
  41. (1999). Inclusive evaluation: Implications of transformative theory for evaluation',
  42. (2003). Influential evaluations',
  43. (2000). Learning dimensions of evaluation use',
  44. M (2004a), 'Acacia hybrids in Vietnam', Impact assessment study report:
  45. (2005). Management of fruit flies in Pacific', Impact assessment study report:
  46. (2001). Managing for results: Roles for evaluators in a new management era',
  47. (2004). Meta-analysis, sytematic reviews, and research syntheses',
  48. (2009). Meta-evaluation as a means of examining evaluation influence',
  49. (2000). Meta-evaluation as a tool for learning: A case study of the European structural fund evaluations in Finland',
  50. (2000). Meta-evaluation as a tool for learning:A case study of the European Structural Fund, evaluations in Finland',
  51. (2009). Meta-evaluation in practice: Selection and application of criteria',
  52. (1978). Meta-evaluation research',
  53. (1983). Meta-evaluation',
  54. (2004). Meta-evaluations: Discovering what works better in welfare provisions',
  55. (2007). Minimising impacts of fungal disease of eucalypts in South East Asia', Impact assessment study report: IAS 49 (Canberra: Australian Centre for International Agricultural Research).
  56. (2007). Mite pest of honey bees in the Asia Pacific region', Impact assessment study report:
  57. (2007). Mixed methods in social inquiry
  58. (1990). Multiple stakeholders, several policy interests: To whom are we ultimately ethically responsible?',
  59. (2004). National Evaluation Standards for Australia and New Zealand: Many questions but few Answers', New Directions for Evaluations,
  60. (1998). Pigeanpea improvement', Impact assessment study report: IAS 06 (Canberra: Australian Centre for International Agricultural Research).
  61. (2007). Policy framework for the Government- Wide Monitoring and Evaluation System',
  62. (1993). Politics and evaluation: A reprise with mellower over tones',
  63. (1998). Post harverst R&D concerning tropical fruits', Impact assesment study report:
  64. (1997). pragmatic basis for mixed methods designs',
  65. (2004). Program evaluation: Altrenative approaches and practical guidelines
  66. (1994). Programme evaluation standards (Thousand Oaks:
  67. (2006). Programme evaluation: Forms and appraches
  68. (2005). Proposal and implementation plan for the Government-Wide Monitoring and Evaluation System',
  69. (2005). Quality, context, and use: Issues in achieving the goals of metaevaluation',
  70. (1997). Realistic evaluation (Los Angeles:
  71. (2000). Reconceptualizing evaluation use: An intergrated theory of influence',
  72. (2009). Research design: Qualitative, quantitative, and mixed methods approaches (Thousand Oaks:
  73. (2005). Review of the retuns to ACIAR's bilateral investments', Impact Assessment Series no. 35 (Canberra: Australian Centre for International Agricultural Research).
  74. (2004). Setting standards and providing guidelines: The means toward what End?',
  75. (2004). Setting standards and providing guidelines',
  76. (1988). Stakeholder participation and utilization in programme evaluation',
  77. (2000). Standards for evaluation: On the way to guiding principles in German evaluation',
  78. (2009). that works for developing countries and Australia', Home accessed 26
  79. (1999). The interface between evaluation and public policy',
  80. (2001). The learning dimension of evaluation use',
  81. (1995). The logic of evaluation and evaluation practice',
  82. (2001). The metaevaluation imperative',
  83. (2007). The practice of social research
  84. (2001). The practice of social research (Cape Town:
  85. (2000). The program evaluation standards in international settings',
  86. (2007). The role of agriculture in development: Implications for Sub-Saharan Africa',
  87. (2005). The role of agriculture in development', International policy brief (Wageningen:
  88. (2005). The Role of Agriculture in Pro-Poor Growth in Sub-Saharan Africa',
  89. (1996). The Theory behind Practical Evaluation',
  90. (2003). The user-oriented evaluator’s role in formulating a program theory: Using a theory-driven Approach',
  91. (1998). Theories of evaluation practice',
  92. (1997). Thoughts for a new evaluation society',
  93. (2008). Transformative research and evaluation
  94. (2008). Two stage grain drying in the Phillipienes', Impact Assessment study report: IAS 59 (Canberra: Australian Centre for International Agricultural Research).
  95. (2001). Unfinished business: Causes and values',
  96. (2004). Unpacking the participatory process',
  97. (1986). Using evaluation to improve government performance',
  98. (1997). Utilization-focused evaluation (London: Thousand Oaks).
  99. (2006). Water management in public irrigation schemes in Vietnam', Impacy assesment studies report: IAS 43 (Canberra: Australian Centre for International Agricultural Research).