Abschlussbericht für DFG Projekt "A 'gold standard' of institutional assessment? Operationalizing and explaining political biases in large numbers of international organization evaluation reports"

Abstract

This research project examined the role of political influences in evaluation processes within international organizations (IOs). Evaluations are widely used as tools for accountability and learning, but concerns have been raised about their neutrality and independence. While previous research has largely relied on perception-based evidence, this study systematically analysed the content of evaluation reports to assess whether political biases shape evaluation findings and recommendations. At the core of the project was a quantitative content analysis of 1,082 evaluation reports. Based on novel conceptualizations of evaluation biases and using a state-of-the-art fine-tuned BERT language model, nearly one million sentences from these reports were classified as positive, neutral, or negative. Additionally, the recommendations given in a sample of 240 evaluation reports were manually coded regarding the type and depth of recommendations given. Findings show on the one hand that evaluation findings—the assessments of IO performance—do not exhibit systematic biases based on whether evaluation units (in terms of their budget, staffing and agenda) are controlled by IO administrations or member states. Evaluation recommendations on the other hand, do reflect stakeholder influence. Reports from IO administration-controlled evaluation units contained broader, less specific recommendations that tended to favour increasing organizational resources while avoiding proposals for additional oversight. In contrast, member state-controlled evaluations were more targeted and focused on strengthening accountability mechanisms. These patterns suggest that political considerations influence how evaluation results are translated into policy recommendations. Another key finding concerns the role of the commissioning entity. Evaluations commissioned by decentralized operational units, which are closely involved in project/program implementation, tended to be systematically more positive than those conducted by independent central evaluation units. This suggests that decentralized evaluations may be subject to direct or indirect pressure to present findings in a more favourable light.All data and the language model were published. Substantive findings were presented at conferences and published in a book with Oxford University Press and in leading journals of political science (with peer review). Beyond academic contributions, the project was characterized by intensive exchange with practitioners. Regular consultations were held with evaluation professionals from the UN system, development agencies, and international organizations. Key findings were presented to the UN Evaluation Group and shared in various practitioner networks to ensure that the research remained relevant for those working directly with evaluation processes