Abschlussbericht für DFG Projekt "A 'gold standard' of institutional assessment? Operationalizing and explaining political biases in large numbers of international organization evaluation reports"

This research project examined the role of political influences in evaluation processes within international organizations (IOs). Evaluations are widely used as tools for accountability and learning, but concerns have been raised about their neutrality and independence. While previous research has largely relied on perception-based evidence, this study systematically analysed the content of evaluation reports to assess whether political biases shape evaluation findings and recommendations. At the core of the project was a quantitative content analysis of 1,082 evaluation reports. Based on novel conceptualizations of evaluation biases and using a state-of-the-art fine-tuned BERT language model, nearly one million sentences from these reports were classified as positive, neutral, or negative. Additionally, the recommendations given in a sample of 240 evaluation reports were manually coded regarding the type and depth of recommendations given. Findings show on the one hand that evaluation findings—the assessments of IO performance—do not exhibit systematic biases based on whether evaluation units (in terms of their budget, staffing and agenda) are controlled by IO administrations or member states. Evaluation recommendations on the other hand, do reflect stakeholder influence. Reports from IO administration-controlled evaluation units contained broader, less specific recommendations that tended to favour increasing organizational resources while avoiding proposals for additional oversight. In contrast, member state-controlled evaluations were more targeted and focused on strengthening accountability mechanisms. These patterns suggest that political considerations influence how evaluation results are translated into policy recommendations. Another key finding concerns the role of the commissioning entity. Evaluations commissioned by decentralized operational units, which are closely involved in project/program implementation, tended to be systematically more positive than those conducted by independent central evaluation units. This suggests that decentralized evaluations may be subject to direct or indirect pressure to present findings in a more favourable light.All data and the language model were published. Substantive findings were presented at conferences and published in a book with Oxford University Press and in leading journals of political science (with peer review). Beyond academic contributions, the project was characterized by intensive exchange with practitioners. Regular consultations were held with evaluation professionals from the UN system, development agencies, and international organizations. Key findings were presented to the UN Evaluation Group and shared in various practitioner networks to ensure that the research remained relevant for those working directly with evaluation processes

Von Nachrichten zu Netzwerken. Textbasierte Methoden zur Messung interorganisationaler Beziehungen

Die Erhebung interorganisationaler Netzwerkdaten ist komplex und zeitintensiv. Bisher beziehen sich automatisierte Erhebungen vornehmlich auf rechtliche oder finanzielle Beziehungen (beispielsweise Eigentum). Darüber hinaus sind auch strategische Beziehungen von hoher Relevanz, die aber selten unter eine Berichtspflicht fallen. Obwohl deren Aufnahme und Beendigung häufig öffentlich in den Medien kommuniziert werden, bleiben diese Daten bei der Konstruktion interorganisationaler Netzwerke meist unberücksichtigt. Basierend auf Schwenkler und Zheng (2020) geht dieser Beitrag der Frage nach, inwiefern Zeitungsartikel für die Datenerhebung im Kontext interorganisationaler Netzwerke genutzt wer-den können. Dabei präsentiert er eine beispielhafte Erhebung über die API-Schnittstelle der London Stock Exchange Group (LSEG), die die Beziehungen zwischen Unternehmen anhand der Häufigkeit einer entsprechenden Berichterstattung im jeweiligen Jahr erfasst. Der aktuelle Stand des Projekts deutet bereits auf methodische Herausforderungen hin. Diese bestehen vor allem in der Messung der Stärke von Beziehungen, der zeitlichen Bestimmung von Beziehungen sowie ihrer thematischen Klassifikation. Der vorliegende Beitrag soll unter anderem diskutieren, inwiefern Large Language Models (LLM) eine mögliche Lösung für diese Herausforderungen darstellen. Damit fokussiert er sich vor allem darauf, skalierbare Erhebungstechniken für die Netzwerkforschung zu testen, um bestehende Ansätze in diesem Feld weiterzuentwickeln