Linking Evaluation Theory and Practice: Exploring Ray Rist’s Enduring Legacy

Background: Over the course of his career, Ray C. Rist made substantial contributions to advancing evaluation theory and practice globally. This includes leading the International Research Group for Policy and Program Evaluation (INTEVAL) for nearly forty years, overseeing the publication of approximately forty books in the Comparative Policy Evaluation series. Purpose: This special edition explores Ray Rist's influence and legacy over his career, and specifically through INTEVAL, examining how his achievements have impacted the domains of evaluation, audit, and learning. The aim is to identify enduring lessons for evaluators, auditors and policy makers regarding some of the issues addressed by Ray and INTEVAL. Setting: The special edition encompasses the influence and legacy of Ray throughout his career spanning academia, government, international institutions, and voluntary service. Intervention: Not applicable. Research Design: Not applicable. Data Collection and Analysis: Contributors analyzed Ray's publication record, citation impacts, leadership approach, and collaborative methodologies. They also examine the influence of some of INTEVAL’s publications under the leadership of Ray and their influence on evaluative thinking and practice. The special edition draws on historical analysis, case studies, and theoretical frameworks to examine Ray's contributions to evaluation theory and practice, with particular focus on his collaborative work within INTEVAL and other organizations. Findings: Ray Rist's nearly 200 publications have been cited almost 13,000 times, with significant impact both in his early academic work and later evaluation publications. His leadership style created a productive voluntary network of evaluation professionals that has sustained itself for four decades. Key contributions include pioneering work on policy instruments, results-based monitoring and evaluation systems, evaluation in accountability structures, and global evaluation capacity development. This special edition documents how Ray's collaborative approach advanced evaluation theory while maintaining strong connections to practice across diverse global contexts

Implementation Fidelity: The Disconnect Between Theory and Practice

The current paper combines a conceptual analysis of major reviews of implementation fidelity studies (e.g., Dane & Schneider, 1998; Mowbray et al., 2003; O’Donnell, 2008) with reflections on fidelity measurement practice in field evaluations.  It claims that practice is impoverished by the failure to recognize the existence of competing conceptualizations of fidelity, rooted in different theoretical perspectives. Different evaluation contexts may be better matched to one or the other of these perspectives. Confusion about how fidelity should be defined in a given funding program or evaluation prevents evaluators from instituting a maximally useful fidelity measurement program.   Difficulties inherent to creating high-quality fidelity measures contribute to the problem. The causes and consequences of this disconnect between fidelity theory and fidelity practice are discussed herein. Preliminary suggestions for solutions are advanced

From Perpetuation to Disruption of Disadvantages: Learning from a Young Ray Rist and Implications for the Future of the Field of Evaluation

Background: Given the amplification of vulnerabilities created by the COVID-19 pandemic, the need for evaluation to more intentionally explore the intended and unintended consequences of interventions in contemporary society has increased. In this paper, we analyze one of Ray Rist's earliest papers, "Student Social Class and Teacher Expectations: The Self-Fulfilling Prophecy in Ghetto Education" to explore whether it has lessons for contemporary evaluators  Purpose: This paper seeks to learn lessons for evaluation (in identifying themes important to addressing inequities) from a seminal paper published more than 50 years ago. We explore lessons that research study conducted in a classroom in the USA might have for contemporary evaluators. Setting: Not applicable. Intervention: Not applicable. Research Design: Not applicable. Data Collection and Analysis: This paper analyzes key themes from Rist's seminal paper and leverages our experience as evaluators working in multiple settings to assess the implication of his paper to contemporary evaluations. The analysis is informed by a realist lens that recognizes the importance of contexts and mechanisms in the generation of outcomes. Findings: The following four implications of Rist's ideas are discussed: (1) the need to pay close attention to the architecture of interventions; (ii) explore how the implementation of some interventions can lead to exclusion of individuals; (iii) evaluation as a field needs to move from verdicts to explanations; (iv) evaluators need to pay attention to the dynamics of mutually reinforcing processes that exacerbate disadvantages over time

From Ongoing Streams of Evidence to Final Synthesis

Background: Protecting the global environment requires complex program and project designs intended to produce multiple benefits in the various environmental domains (biodiversity, land degradation, water, climate change, and pollution) while benefiting the people, especially local communities living in the areas of intervention. In this context, learning is a key requirement to which independent evaluation must contribute. This necessitates bringing together streams of evidence from multiple perspectives and sources. Purpose: The article brings out lessons learned in designing and conducting comprehensive evaluations of large-scale programs and projects funded by the Global Environment Facility (GEF) using streams of evidence. It also describes the utility and utilization of such evaluations for making decisions about future funding. Setting; GEF is the oldest public financier for the global environment and serves as a financial mechanism to five multilateral environmental agreements in the areas of biodiversity, climate change, land degradation, and pollutants. Intervention: Comprehensive evaluations of the GEF. Research Design: Not applicable. Data Collection and Analysis: Not applicable. Findings: The article draws upon the sixth and seventh comprehensive evaluations of the GEF, which were key inputs to the Fund’s replenishment negotiations. The article outlines the transition from standalone studies to continuous evaluation streams, which provided real-time insights to improve decision-making in the GEF. These evaluations assessed the GEF’s continued relevance and ability to effectively support the recipient countries in protecting the global environment. The evaluations covered project performance, governance, integration in programming, and transformational impact, as well as methodologies and addressing emerging challenges like financial constraints and the pandemic’s impact. The findings of these evaluations directly influenced GEF’s strategic direction and policy, leading to program refinements. The comprehensive evaluation experience demonstrates to the power of bringing together evidence from multiple studies into a stream that can provide both timely information on specific aspects and a comprehensive diagnostic of an organisation’s performance, strengths and weaknesses

Democracy, Accountability, and Evaluation

Background: The focus of the article concerns the final step in the accountability chain in democratic societies, the one between the people and their elected representatives. The importance of this relationship has meant that questions regarding accountability and independent scrutiny have been important in the democratic discussions for more than 2300 years. Based on a discussion of this relationship, the article asks the question of what role evaluation can have in strengthening this relationship. Purpose: Trust and mistrust must be balanced in a democratic society. An important prerequisite for this balance is that the citizens know that those to whom they have entrusted power are under constant and independent scrutiny. Setting: The expansion of evaluation has meant that the amount of information, which potentially can be useful in the final step of the accountability chain, has increased. This development suggests that it is today easier than in previous times to hold the elected accountable. However, surveys and indexes show a different picture. More “old” democracies are described as flawed democracies. And it is not difficult to note developments in many democratic nations which seem to reflect increased mistrust in the elected. We can also find indications of a lack of trust in evaluative information delivered by government offices, audit, inspections and research bodies. This seems most obvious in extreme situations, crises, breakdowns, suspicions of abuse of power and corruption or what is seen as just shocking incompetence, where routine oversight does not give trustworthy answers to the questions raised. The article points out that such situations seem to demand something extraordinary, ad hoc accountability mechanisms. Intervention: N/a Research design: N/a Data collection and analysis: Systematic qualitative analysis. Particular attention is paid to the publications of Inteval in the area of accountability, and the role of investigating commissions in Sweden. Findings: Since the 1960s accountability has been an important part of the evaluation discourse and has also been seen as an important purpose for evaluation. However, the article points out that the debate about accountability reveals tensions within the evaluation field. The conclusion, given the background of lack of trust in many democracies, is that it is important that the evaluation community, more than earlier, emphasizes the importance of accountability and discuss how the evaluation practice can contribute to enhanced accountability.&nbsp

Evaluation, Inteval, and Two National Audit Offices (U.S. GAO & Netherlands Court of Audit): Ray Rist’s Contributions and Leadership in the Early Years of INTEVAL

Background: This paper describes Ray Rist’s intellectual, research-focused work in the early years of Inteval (largely between the mid-1980s till around mid-1990s). It sets out what the position and roles of the (then) USA General Accounting Office (GAO) and the Netherlands Court of Audit (NCA) were with respect to evaluation. Purpose: Differences and similarities between the GAO and NCA experience are described, with the GAO as a ‘first wave’ organization in the evaluation profession and the NCA as a ‘second wave’ organization, as it learned from the USA experiences. The labelling of first and second wave developments in evaluation was formulated by Derlien (1989). There is a particular focus on the role of leadership in shaping practice. Setting: During the period of the mid-1980’s till mid-1990’s’, Rist and the author of this article were both working at respectively GAO and NCA in the field of policy evaluation/program evaluation and methodology. The setting was such that they worked together on a number of occasions and products, amongst others dealing with how (performance) audits and evaluation developed; the introduction (in the NCA) of government-wide (comparative) evaluations of tools of government like subsidies, information campaigns and inspections (covering all ministries), how evaluation was related to the learning capability of governments and how Inteval as a group of evaluators, auditors and other social scientists developed. Intervention: Not applicable Research design: Not applicable Data collection and analysis: Not applicable Findings: A number of lessons are set out about the role of leadership in evaluation

Capacity Development in Evaluation: The Role of the International Program for Development Evaluation Training (IPDET)

Background: Over the last decades evaluation has truly become a global field of practice. Of particular note has been the significant progress in the institutionalization of evaluation in low- and middle-income countries. Ray Rist, working at the Independent Evaluation Group (IPDET) of the World Bank, has been an influential champion of evaluation capacity development, among other things through establishing and leading the International Program for Development Evaluation Training (IPDET), a flagship global training program in evaluation. Purpose: The article focuses on the role of IPDET in developing evaluation capacity. It highlights IPDET's contributions to the field, particularly in low- and middle-income countries, and discusses factors contributing to its success and future challenges. Setting: IPDET was initially hosted by Carlton University from 2001 to 2016, offering a comprehensive training program on evaluation. In 2017, the University of Bern and the University of Saarbrücken took over, introducing a new program concept. The program has adapted to changes, including moving online during the COVID-19 pandemic and integrating into the Global Evaluation Initiative (GEI) network. Findings: IPDET's success is attributed to its strong brand, external financial support, quality control, and community spirit. The program has trained thousands of participants globally and continues to adapt to evolving needs. Key factors for future success include differentiating itself from competition from other training programs, adapting to technological advances, and maintaining relevance in the evolving field of evaluation. The article emphasizes the importance of IPDET's holistic approach to training and its integration into the GEI network to enhance its impact

Ecology and Evolutionary Biology’s Unique Contribution to Evaluation

Background: Evaluation recognizes the need to consider three constructs –  program change over time, the consequences of program action over time, and relationships between programs with their environments. Our methods for studying these constructs are home grown, i.e. they have developed almost exclusively within our field. These constructs, however, have a long and deep history in the fields of ecology and evolutionary biology (EEB). Thus it makes sense to consider how EEB might contribute to the models. methodologies, and data analysis strategies that evaluation applies to program change, outcomes, and program/environment interactions.  Further, in recent years evaluation has been paying ever greater attention to how complex system behavior affects programs and their outcomes. Much in the fields of EEB can be seen as a subset of complexity. Purpose: This article has two purposes: 1) to convince evaluators that EEB can empower their efforts to evaluate change over time in programs, outcomes, and program/environment effects, and 2) to spur the growth of a group of evaluators with an interest in further exploring EEB’s contribution to our field Setting: Not applicable. Intervention: Not applicable. Research Design: Not applicable. Data Collection and Analysis: Not applicable. Findings: Not applicable

Conclusions on the Way Forward for Evaluation in Difficult Times

Background: The evaluation profession faces unprecedented challenges in an era of global uncertainty, with shifting political climates, rising populism, and complex societal problems requiring innovative evaluation approaches. Moments like these are opportune times to harvest learnings from the past to inform current and future evaluation practice. Purpose: This concluding chapter reflects on Ray Rist's career and the broader implications of INTEVAL's work for the future of evaluation in uncertain times, highlighting key characteristics that can guide evaluators navigating contemporary challenges. Setting: The article examines Ray Rist's influence across academic, governmental, and international institutions, including his work with INTEVAL, the General Accounting Office, the World Bank, IPDET, and IDEAS. Intervention: Not applicable. Research Design: Not applicable. Data Collection and Analysis: Not applicable. Findings: Four key characteristics define Ray Rist's approach. These characteristics guided his approach to INTEVAL over the past 40 years and remain relevant for evaluators today: (1) commitment to continuous professional development and lifelong learning; (2) dedication to addressing injustice through evaluation and audit; (3) focus on institutionalizing evaluation capacity globally, particularly in the Global South; and (4) emphasis on open collaboration and inclusive discussion. These characteristics have enabled INTEVAL to remain relevant despite changing times and can guide evaluators facing current global challenges including climate change, migration, and increasing populism

INTEVAL as a Positively Charged Social Network

Background: In this article Bastøe and Haslie seeks to employ theoretical insights from two connected bodies of literature to understand the unique characteristics of The International Evaluation Research Group (Inteval). One perspective draws on insights from social network theories about how some networks are supportive and innovative while others are not. The second perspective draws on organizational learning and Community of Practice theories which illuminate how informal networks facilitate learning, innovation, and knowledge sharing. Purpose: This article aims to explore Inteval as a positively charged social network, employing theories on social networks and communities of practice to understand its impact. Setting: Inteval is a multidisciplinary, multicultural, and multinational group of evaluation experts who convene annually to discuss innovative topics in evaluation, contributing to book publications in the Comparative Policy Evaluation Series. Intervention: N/A Research Design: The article uses a theoretical approach, drawing on social network and community of practice theories, supported by empirical observations and experiences within the Inteval network. Data Collection and Analysis: The analysis is based on the author's extensive engagement with Inteval, supplemented by literature on social networks and communities of practice. Findings: Inteval exemplifies the theoretical insights concerning features of a positively charged social network. Ray Rist's central role and personal connections have been pivotal in maintaining the network. Theories on organizational learning emphasize the importance of mutual engagement, joint enterprise, and shared repertoire, which are evident in Inteval's practices. The density and reciprocity within the network, evidenced by frequent and meaningful interactions, contribute to the group's longevity and effectiveness. The non-hierarchic structure further supports knowledge sharing and lowers barriers to participation. These theoretical insights underscore the significance of trust, belonging, and a supportive network culture in sustaining Inteval’s success and offer valuable lessons for other communities of practice