Systems Mapping

This open access book explores a range of new and older systems mapping methods focused on representing causal relationships in systems. In a practical manner, it describes the methods and considers the differences between them; describes how to use them yourself; describes how to choose between and combine them; considers the role of data, evidence, and stakeholder opinion; and describes how they can be useful in a range of policy and research settings. This book provides a key starting point and general-purpose resource for understanding complex adaptive systems in practical, actionable, and participatory ways. The book successfully meets the growing need in a range of social, environmental, and policy challenges for a richer more nuanced, yet actionable and participatory understanding of the world. The authors provide a clear framework to alleviate any confusion about the use of appropriate terms and methods, enhance the appreciation of the value they can bring, and clearly explain the differences between approaches and the resulting outputs of mapping processes and analysis

Building a system-based theory of change using participatory systems mapping

Theory of Change diagrams are commonly used within evaluation. Due to their popularity and flexibility, Theories of Change can vary greatly, from the nuanced and nested, through to simplified and linear. We present a methodology for building genuinely holistic, complexity-appropriate, system-based Theory of Change diagrams, using Participatory Systems Mapping as a starting point. Participatory System Maps provide a general-purpose resource that can be used in many ways; however, knowing how to turn their complex view of a system into something actionable for evaluation purposes is difficult. The methodology outlined in this article gives this starting point and plots a path through from systems mapping to a Theory of Change evaluators can use. It allows evaluators to develop practical Theories of Change that take into account feedbacks, wider context and potential negative or unexpected outcomes. We use the example of the energy trilemma map presented elsewhere in this special issue to demonstrate

Participatory systems mapping for complex energy policy evaluation

The use of complexity science in evaluation has received growing attention over the last 20 years. We present the use of a novel complexity-appropriate method – Participatory Systems Mapping – in two real-world evaluation contexts and consider how this method can be applied more widely in evaluation. Participatory Systems Mapping involves the production of a causal map of a system by a diverse set of stakeholders. The map, once refined and validated, can be analysed and used in a variety of ways in an evaluation or in evaluation planning. The analysis approach combines network analysis with subjective information from stakeholders. We suggest Participatory Systems Mapping shows great potential to offer value to evaluators due to the unique insights it offers, the relative ease of its use, and its complementarity with existing evaluation approaches and methods

How does the commissioning process hinder the uptake of complexity-appropriate evaluation?

This paper investigates the role of evaluation commissioning in hindering the take-up of complexity-appropriate evaluation methods, using findings from interviews with 19 UK evaluation commissioners and contractors. We find, against a backdrop of a need to ‘do more with less’ and frustration with some traditional approaches, the commissioning process is perceived to hinder adoption of complexity-appropriate methods because of its inherent lack of time and flexibility, and assessment processes which struggle to compare methods fairly. Participants suggested a range of ways forward, including more scoping and dialogue in commissioning processes, more accommodation of uncertainty, fostering of demand from policy users, more robust business cases, and more radical overhauls of the commissioning process. Findings also emphasised the need to understand how the commissioning process interacts with the wider policy making environment and evidence culture, and how this manifests itself in different attitudes to risk in commissioning from different actors

Systems Mapping

This open access book explores a range of new and older systems mapping methods focused on representing causal relationships in systems. In a practical manner, it describes the methods and considers the differences between them; describes how to use them yourself; describes how to choose between and combine them; considers the role of data, evidence, and stakeholder opinion; and describes how they can be useful in a range of policy and research settings. This book provides a key starting point and general-purpose resource for understanding complex adaptive systems in practical, actionable, and participatory ways. The book successfully meets the growing need in a range of social, environmental, and policy challenges for a richer more nuanced, yet actionable and participatory understanding of the world. The authors provide a clear framework to alleviate any confusion about the use of appropriate terms and methods, enhance the appreciation of the value they can bring, and clearly explain the differences between approaches and the resulting outputs of mapping processes and analysis

Cased-based modelling and scenario simulation for ex-post evaluation

Despite 20 years of increasing acceptance, implementing complexity-appropriate methods for ex-post evaluation remains a challenge: instead of focusing on complex interventions, methods need to help evaluators better explore how policies (no matter how simple) take place in real-world, open, dynamic systems where many intertwined factors about the cases being targeted affect outcomes in numerous ways. To assist in this advance, we developed case-based scenario simulation, a new visually intuitive evaluation tool grounded in a data-driven, case-based, computational modelling approach, which evaluators can use to explore counterfactuals, status-quo trends, and what-if scenarios for some potential set of real or imagined interventions. To demonstrate the value and versatility of case-based scenario simulation we explore four published evaluations that differ in design (cross sectional, longitudinal, and experimental) and purpose (learning or accountability), and present a prospective view of how case-based scenario simulation could support and enhance evaluators’ efforts in these complex contexts

Reflections On the Use of Complexity-Appropriate Computational Modeling for Public Policy Evaluation in the UK

In the UK, calls for the application of insights from the study of complex adaptive systems to public policy evaluation are beginning to be taken seriously in government. Policymakers and analysts are accepting the fallibility of overly simplistic, definitive, or linear analysis, or are finding traditional forms of analysis and evidence less appropriate or feasible. Through our work in CECAN (the Centre for the Evaluation of Complexity Across the Nexus), we reflect on our experiences and the practical challenges of using complexity-appropriate computational modeling with policy analysts and evaluators in UK central government. As an example, we discuss our work with the COMPLEX-IT toolkit, which uses a selection of case-based computational modeling approaches. We end by suggesting ways forward for applied complexity scientists, and policy evaluators and analysts to make more effective use of these methods

Participatory complexity in tourism policy: Understanding sustainability programmes with participatory systems mapping

Linear logic models are insufficient to understand how interventions work in complex areas such as sustainable tourism. We present Participatory Systems Mapping (PSM), a novel method to develop shared understandings and collective management of complex policy issues among stakeholders. We use PSM with stakeholders in Barcelona to support the design of an upcoming evaluation of an existing sustainability programme. Discussion during workshops, and analysis of the PSM map produced, suggest sharing best practices and improving peer-to-peer learning are pivotal to improving sustainability. We show how a complex systems approach, implemented via PSM, can provide a more holistic understanding of the contexts and interactions of tourism policy. We offer learning and guidance on how the method can be used by others

How do policy evaluators understand complexity?

There is a well-documented interest in how insights from the study of complexity can be applied to policy evaluation. However, important questions remain as to how complexity is understood and used by policy evaluators. We present findings from semi-structured interviews with 30 UK policy evaluators working in food, energy, water and environment policy domains. We explore how they understand, use and approach complexity, and consider the implications for evaluation research and practice. Findings reveal understandings of complexity arising from contextual factors, scale-related issues and perceptions of unpredictability. The evidence indicates terminological and analogical use of complexity and its concepts by policy evaluators, but limited evidence of its literal use. Priorities for the future include framing complexity more pragmatically and as an opportunity not a cost. Communicating this up the policy hierarchy is the key to progressing complexity-appropriate evaluation – this can be enabled by strengthening links between policy evaluation and academic communities