Hummelbrunner, Richard

Reynolds, Martin

English

Open Research Online (Open)

Open Research OnlineThe Open University’s repository of research publicationsand other research outputsSystems thinking, learning and values in evaluationOtherHow to cite:Hummelbrunner, R. and Reynolds, M. (2013). Systems thinking, learning and values in evaluation. EvaluationConnections: The European Evaluation Society Newsletter, June 2013, pp.9-10.For guidance on citations see FAQs.c© 2013 Richard Hummelbrunner and Martin ReynoldsVersion: Accepted ManuscriptLink(s) to article on publisher’s website:http://www.europeanevaluation.org/sites/default/files/ees newsletter/ees-newsletter-2013-06-june.pdfCopyright and Moral Rights for the articles on this site are retained by the individual authors and/or other copyrightowners. For more information on Open Research Online’s data policy on reuse of materials please consult the policiespage.oro.open.ac.uk1   Systems Thinking, Learning and Values in Evaluation Richard Hummelbrunner and Martin Reynolds   Applying the three core systems concepts – interrelationships, perspectives and boundaries (see Bob Williams’ article in this issue) - for the evaluation of a “situation” has implications for the type of learning that it generates. It  also helps to make explicit its value base. This article proposes a conceptual framework that  connects the three systems concepts with learning and values.1   First we examine the implications for learning. To do so we use a model that has been widely used in the Organizational Development literature. It is based on the work of Gregory Bateson (1972) as well as Chris Argyris and Donald Schön (1978) and addresses the purpose and extent of learning. It distinguishes between three types of learning:   − Single loop learning (Learning to adapt): results in a change of strategy or tactics  without questioning the underlying goals or assumptions. It helps to control individual behaviour within existing decision making protocols;  provides short-term solutions to implementation problems and deals with symptoms more than root causes . The core question is ‘Are we doing things right?` − Double loop learning (Learning to change): by reflecting on goals and assumptions, one probes the generative mechanisms of problems, their underlying causes and their consequences. This leads to adjustments in strategy and to better mid- and long-term course corrections in response to contextual changes. The core question is ‘Are we doing the right things?’ − Triple loop learning (Learning to learn): by reflecting on the learning mechanisms, existing rules are challenged and possibly changed in ways that affect knowledge acquisition and behaviour, i.e. by identifying different patterns of recognising and handling problems or coping more effectively with contextual changes. The core question is ‘What makes this the right thing to do?  Although each of these levels addresses different questions, the progression from single to double and triple loop learning can be expected to lead to deeper and more sustainable learning.  Based on this model, we associate each of the three systems concepts with a specific loop of learning. Figure 1 below illustrates these connections for the generic case of evaluating the effects of an intervention:                                                 1 The ideas in this article were originally presented by the authors in a panel session at the workshop ‘Impact, Learning and Innovation’ at the Institute of Development Studies, Brighton UK (March 26-27 2013).  2    Fig. 1: Types of Learning and Systems Concepts  − Single loop learning: The focus is on interrelationships, primarily between the intervention and its effects, but also within them (e.g. between the actions of an intervention or the various effects produced). In case of divergence from original plans, adaptive recommendations are made; for example, modifying a strategy or activities in order to better achieve stated aims and objectives. Significantly, the purpose of the intervention is not questioned. − Double loop learning: Assumptions underpinning an intervention can only be reflected if multiple perspectives are taken into account. When acknowledging that a situation can be framed in different ways, this also questions the purpose and goals of an intervention.  − Triple loop learning: Here the focus is on the boundaries inevitably made with any intervention and its evaluation. Reflecting on boundary judgements is very helpful (and needed) for critically reflecting on the rules and relations of power that affect behaviour and cognition patterns (Flood and Romm, 1996) . This notably involves looking at the power relations that determine the boundaries of an intervention and its evaluation, including the role of evaluation commissioners and evaluators themselves.   The key role of the evaluator is in assigning value. Each of the learning loops can be associated with a different set of values:  − Single loop learning is based on instrumental values embedded in  an intervention. These underpin the intervention logic and can be derived from the respective documents, either Perspectives BoundariesSingle loop learningDouble loop learningTriple loop learningInterrelationshipsAre we doing things right? Are we doing the right things?What makes this the right thing to do?Questions purpose, goalsPower for determining purpose and goalsIntervention Effects3  explicitly or (probably more often) only implicitly. Utility is perhaps the best example of such value. Instrumental values inform evaluative measures regarding issues of ‘efficacy’ (does it work?) and ‘efficiency’ (how well does it work using available resources?).  − Double loop learning is based on the intrinsic value underpinning the various framings of an intervention and/or the wider situation being evaluated. They can include personal,organizational or social values. Intrinsic values inform evaluative measures regarding issues of ‘relevance’ (why is it important that the intervention works and works well?) and ‘effectiveness’ (are the right things getting done?)2  − Triple loop learning is based on critical value; that is, value in reflecting on the rules and customs that govern dominant behaviour and cognition patterns in a particular context. Critical values inform evaluative measures regarding issues of equity and emancipation (what and who determines the importance of some measures of success over others?)  Figure 2 is an attempt to integrate all of these associations into a single framework.     Fig. 2: A systems - based framework for rigor in evaluation                                                2 The term ‘effectiveness’ can have different meanings in different contexts of use; sometimes used interchangeably with efficacy.  In this article we make a clear distinction between evaluative criteria of efficacy (“getting things done“) and efficiency (“getting things done right”) – both of which constitute single loop learning -  and effectiveness (“getting the right things done”). Effectiveness in this latter sense invokes double loop learning.  InterventionPerspectivesBoundariesSingle loop learningDouble loop learningTriple loop learningInterrelationshipsInstrumental values Critical valuesIntrinsicvaluesSystemic practice  Sustainable learningEffects4   The framework can be used to interrogate  the coherence among the various components of  an evaluation assignment. For example, is the type of learning envisaged in line with the evaluation`s value base? Can the value-base be modified or expanded if a deeper level of learning is envisaged or needed? Which of the systems concepts might be more appropriately applied in making value judgments in an evaluation?   The progression from single to triple loop learning is expected to lead to more sustainable learning. Similarly, the progression in focusing from Interrelationships to Perspectives and Boundaries indicates the extent and depth of systemic practice.  Both sustainable learning and systemic practice in evaluation can be enhanced by applying progressively wider measures of value. This does not imply that the ‘upper’ level should and can always be reached. Often only one specific level might be feasible or can be appropriately attained given the actual circumstance and conditions of an evaluation.  But the framework is helpful for reflecting on the constraints and limitations of an evaluation, as well as pointing at hidden opportunities that might otherwise be missed.  This framework applies systems thinking to the evaluation process by proposing three sets of typologies, their respective boundaries as well as some suggested correspondence between them. We believe that reflections based on such a framework can add rigour to evaluation practice. The value base of an evaluation can be made more explicit and congruent with the evaluation mandate. Addressing the appropriate value base (in coherence with the envisaged learning type) can enhance the relevance, validity and credibility of evaluations.  We are aware that this framework is still tentative and provisional, and that some connections and their implications invite further exploration. Additional associations are possible and could be integrated in this framework. For instance, the OECD’s Development Assistance Committee (DAC) evaluation criteria (relevance, , effectiveness, efficiency,  sustainability, impact) could be grouped and more appropriately aligned along the same three levels (i.e., efficacy/efficiency;  relevance; and sustainability/ impact). The framework may also be applied to inform assessments of governance and/or performance by distinguishing between different levels of authority (power over, power with, and power to) and agency (personal, organizational, social).   References  Bateson, G. (1972). "The logical categories of learning and communication." Steps to an Ecology of Mind: 279-308.  Argyris, C. and D. Schön (1978). Organizational Learning. London, Addison-Wesley.  Flood, R. L. and N. Romm (1996). Diversity Management: Triple Loop Learning. Chichester, John Wiley & Sons.  

Systems thinking, learning and values in evaluation

http://oro.open.ac.uk/37718/1/Connections%20613_Final%20PrePub.pdf

Systems thinking, learning and values in evaluation

Abstract

Similar works

Full text

Available Versions

Open Research Online (Open)