PowerPoint presentationThe presentation provides leading questions towards understanding the role of evaluation in health care and in health policy and how these might apply and have impact on health care in China. Policy resistance is the tendency for interventions to be defeated by the system’s response to the intervention itself. The presentation reminds its audience that interventions are embedded in social systems and shaped by this context. Evaluation can be conceived of as an intervention with short- and long-term goals. One of the intermediate goals is to influence policy. The long-term goal is to improve individual lives

Carden, Fred

Sridharan, Sanjeev

International Development Research Centre: IDRC Digital Library

Fred Carden Sanjeev Sridharan AN INTRODUTION TO EVALUATION: The role of Potential Use in Developing Evaluations  Day 1   Monday 9:10 to 9:40 AM Goals of the Workshop   Goals  To build evaluation capacity in the health sector in China;   Build evaluation capacity for Health Systems Reform in China;    Help inform important decisions facing the implementation of the health system reform efforts   Help with the evaluation of the Clinical Pathways Program;   Build evaluation as a field   How can this Workshop make a difference?  ..this “workshop” itself is an intervention...it is part of a larger intervention   Question for reflection: How can the workshop and the mentoring program make a difference to evaluation culture  in the health sector in China? Why is this workshop relevant for the multiple participants  Different Stakeholders   Workshop Participants  China Health Economics Institute  International Development Research Centre  Li Ka Shing Knowledge Institute   But...all of this only matters if it  IMPROVES HEALTH OUTCOMES OF INDIVIDUALS Types of Evaluation Formative Developmental Summative   Different Models of Evaluation   Evaluation as Learning/Evaluation as Accountability  Evaluation as Testing/Evaluation as Navigation   SOME INITIAL QUESTIONS  What are we trying to learn?   What are multiple types of learning?  How do we know if the system if being reformed?  Whose health outcomes need to improve as a result of the health systems reform?  Did  that really happen?  What is evaluation? An incomplete definition    Evaluation is defined both as a means of assessing performance and to identify alternative ways to deliver   “evaluation is the systematic collection and analysis of evidence on the outcomes of programs to make judgments about their relevance, performance and alternative ways to deliver them or to achieve the same results.”   Why is this an incomplete definition? The Knowledge Translational Problem for Health Systems Evaluation  Do we know enough about the system before we intervene?    The more pragmatic question: How can evaluations help  understand what needs to be done to impact health outcomes?   The Social World as a Complex, Dynamic System  Understanding the System BEFORE we Intervene  Understanding the System WHILE we Intervene   “Solutions” Can Also Create New Problems Meadows DH, Richardson J, Bruckmann G. Groping in the dark: the first decade of global modelling. New York, NY: Wiley, 1982. Merton RK. The unanticipated consequences of purposive social action. American Sociological Review 1936;1936:894-904. Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971;73(3):53-68. Policy resistance is the tendency for interventions to be delayed, diluted, or defeated by the response of the system to the intervention itself. -- Meadows, Richardson, Bruckman System-as-Cause Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971;73(3):53-68. Meadows DH. Leverage points: places to intervene in a system. Sustainability Institute, 1999.  Available at <http://www.sustainabilityinstitute.org/pubs/Leverage_Points.pdf>. Richardson GP. Feedback thought in social science and systems theory. Philadelphia, PA: University of Pennsylvania Press, 1991. Sterman JD. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin McGraw-Hill, 2000. Possible Solutions: 1. Focus on specific user-driven questions; 2. Build knowledge utilization into the knowledge generation process; 3. Focus on evaluation capacities (not just evaluation method); 4. Build dynamic relationship between the implementation and the evaluation; 5. Build trust in the evaluation  The challenge for this workshop: How can we make evaluations more useful?   1:30 to 2:10 pm The Purpose of Evaluation   Purpose of evaluation    (Mark, Henry and Julnes, 2000)  Assessing merit and worth  Causal questions, Experimental trials, observational studies  Programme and organizational improvement  Formative evaluation  Oversight and compliance  Knowledge development  Neglected purpose of many evaluations So why are evaluations so often not very useful? UN Office of the Internal Oversight Services, 2008  A Critique of Results-Based Management (2008).  “Results-based management at the United Nations has been an administrative chore of little value to accountability and decision-making.” The UN Critique of performance management and evaluation  Lack of strategic direction and cross-organizational performance incentives   Problems of attribution and trivializing innovation   Trivializing outcomes   The practice of  evaluation lacks rigor   A lack of purpose The UN Critique (2)  Lack of clarity on the consequences of good and poor performance   Lack of clarity on the capacity needed to build a results-based management system   Technical solutions are not a substitute for substantive clarity  Discussion  How can we avoid the mistakes discussed in the UN report?   What steps do we need to take to ensure that our evaluations are useful? 2:10 to 2:50 pm The role of evaluation in Health System Reform efforts How is evaluations of system reform efforts different from evaluations of narrow programs? On programs and systems   “The problem is one of complexity. The health interventions in question are not singular schemes or finite treatments but concern the design, implementation, management and regulation of entire services. These services have a multiplicity of goals, many of them relating to the fulfillment of long-term ambitions.” Pawson et al. (2004) 5 minutes discussion on why the Health Systems Reform in China is an example of a complex intervention The nature of system reform (complex interventions) A not very complex intervention:  Primary Prevention Have a Heart Paisley The Nature of Systems (Sterman, 2006)  Constantly changing;  Governed by feedback;  Non-linear, History-dependent;  Adaptive and evolving;  Characterized by trade-offs;  Policy resistance: “The result is policy resistance, the tendency for interventions to be defeated by the system’s response to the intervention itself.”  Timeline of impact is unclear  Features of complex interventions (Pawson et al., 2004)  The intervention is a theory or theories  The intervention involves the actions of people.  The intervention consists of a chain of steps  These chains of steps or processes are often not linear, and involve negotiation and feedback at each stage.  Interventions are embedded in social systems and how they work is shaped by this context.  Interventions are prone to modification as they are implemented.   Interventions are open systems and change through learning as stakeholders come to understand them.      Different models of organizing knowledge of complex systems The Big challenge: Changing complex systems can take time.   When can we expect changes?  Experience at the U.S. Centers for Disease Control  “Popular conceptions about how certain phenomena change over time may often fail to account for real-world sources of inertia and delay and may suggest that things can change more rapidly than is actually possible.”   “Findings from our study indicate that the Health People 2010 objective for reducing diagnosed diabetes prevalence by 38% will not be achieved — not because of ineffective or underfunded health protection efforts but because the objective itself is unattainable. Moreover, if current investments in diabetes screening and disease management continue to succeed in diagnosing a greater number of people and in enabling people to live longer with the disease, then diagnosed prevalence will move still farther away from the HP 2010 target.”  Question for discussion   What kinds of questions would you like the evaluation you are conducting to answer?   How will you know what success looks like? When will you know it? What are some early markers of success of your intervention?   What are some aspects of the implementation of the system reform implementation you most unsure about? How can evaluations help?    2:50 pm to 3:20 pm? How can evaluations help influence policies and programs? The Big Questions  What are pathways of influence by which evaluations impacts policies?    Conceptualizing evaluation as an intervention with short and long term goals. One of the intermediate goals is to influence policy. The long term goal is to improve individual lives  ? Policy Influence Social Betterment  Evaluation as Intervention  Individual Evaluation Project  Multiple  evaluations projects  ? Policy Influence Social Betterment  Evaluation as Intervention  Individual Evaluation Projects  Multiple evaluation projects  Monitoring and Evaluation  Evaluative Thinking Monitoring and Evaluation  Evaluative Thinking Monitoring and Evaluation  Evaluative Thinking Monitoring and Evaluation  Evaluative Thinking Monitoring and Evaluation System Evaluation  Quality Pathways of Policy Influence Carden (2009) Conceptualizing  research and evaluation as complex systems Evaluation as Intervention                       Overall Context                     Mechanism: So, what can you do?            Outcomes             Stability of Decision Making Institutions Policy Absorption Capacity Nature of Governance Opportunities for countries in transition Economic crisis and pressures on government   Relationship Building   Communication   Networks of Influence   Institutional Mechanisms Policy Influence “Better” Decisions Social Betterment                               Clear Government Demand  Government interest but leadership absent Interest but no capacity  Interesting research but research uninteresting for Government Government disinterest or hostility Evaluation Quality Evaluation Networks Action Research New tools Process Outcomes             Policy Capacity Broader Policy Horizons Policy Regimes   Government Context Capacity, Horizons and Regimes  Capacity: “Research can strengthen the institutional framework supporting policymaking by enhancing the policy community’s own collective ability to assess and communicate innovative ideas, and by cultivating new talents for analytical and applying research advice”  Horizon: “Researchers can improve the intellectual framework surrounding policymaking by introducing new ideas to the policy agenda, by ensuring that information comes to policymakers in a form and language that they can grasp and use, and by fostering helpful dialogue between researchers and decision makers”  Decision Regimes: “The quality of a policy can be determined as much by the procedures of deliberation and decision as by its content.” Discuss  Explore implications of the above model for the Evaluation of the Health System Reform?   What are some ways in which the evaluations can be useful for decision making?  Model 2: Pathways of Evaluation Influence Mel Mark and Gary Henry, Evaluation, 2003 Examples of individual mechanisms   Salience.  Salience refers to the importance of an issue as judged by an individual.    Elaboration: How much an individual actually thinks about an issue.   Examples of interpersonal mechanisms  Justification:  refers to the use of research findings to support a previously held position.    Persuasion:  Persuasion refers generally to the attempts of one party to change the attitudes of another, usually through direct communication.   Examples of collective mechanisms   Policy-oriented learning.  Policy-oriented learning refers to “relatively enduring alterations of thought or behavioral intentions that result from experience and/or new information and that are concerned with the attainment or revision of policy objectives”    Policy change refers here either to major or to minor, more incremental modifications of policies (Sabatier & Jenkins-Smith, 1993) within the environment where the evaluation occurred.   3:35 to 4:15 pm  Evaluation Approaches Some examples of evaluation approaches  Theory based approaches (Realist Evaluation)   Empowerment/Participatory approach   Utilization based approaches   Economic approaches   Exemplars from the literature   Strengths and limitations of each the above evaluations   The Experimenting Society  A brief history:  Donald Campbell and the Experimenting Society   The role of ‘experiments’ in social decision making   The rise of impact evaluation   Experiments and Quasi-experiments Challenging Evaluation:   An Introduction to Outcome Mapping   www.idrc.ca/evaluation   www.outcomemapping.ca What is outcome mapping?  A framework or way of thinking that allows researchers to plot human behaviour and actions and assess their contribution to the aims of research projects and programs Looking at the bigger picture  Outcome mapping allows a researcher to draw a picture that includes more than just the science A flexible, multiple-use tool  Outcome mapping can be used to  Plan changes  Record performance  Form the basis of an evaluation plan View the map for what it is  The map is NOT the territory so be prepared for surprises along the way  Keep your eyes wide open  Being attentive along the journey…   is as important as the destination  Change in  Behaviour Change in State Outcomes = Behaviour Change Changes in state are usually outside the scope of a project Changes in behaviour, action and relationships are the focus CONTRIBUTION ATTRIBUTION Influence not Control INFLUENCE CONTROL « Those Who Dream Make a Difference! » Aim high  Link dreams to actions Sphere of Influence Your responsibility is here Know what it is Focus on these relationships Challenging Evaluation:   An Introduction to Outcome Mapping   www.idrc.ca/evaluation   www.outcomemapping.ca Discussion  Do any of these approaches have relevance for the evaluation of clinical pathways and the health systems reform ?  Will any of these approaches help you frame your evaluation questions differently?   4:15 to 5:05 pm Questions to consider for developing an evaluation plan 1. Understanding the Intervention   What are the different components of a program? Are the components stable? What are the implications of the stability of program components to evaluation design?   Relevance of the concept for  Health System Reform and the Clinical Pathways Program 2. Program Theory  What is the program theory? Why and how is the program likely to work? Under what contexts is the program likely to work?    Discuss relevance of ideas of program theory for Health Systems Reform and the Clinical Pathways  3. Linking evidence to the Program Theory  Does the evidence base support the program theory linkages?     Discussion:  How was evidence used in developing the clinical pathways?    Role of evidence in health systems reform 4. Understanding timelines of impact  What is the timeline of impact? How long will it take for the intervention to impact key outcomes? What are the key outcomes?   Discuss relevance of this concept for Clinical Pathways and Health Systems Reform 5. Learning Frameworks  What kinds of learning will the evaluation provide?    Discuss  What do we hope to learn from the evaluations of Clinical pathways and Health Systems Reform?      6. Utilization/Influence  What are the pathways of influence of the evaluation - how will the evaluation influence the decisions involved in the present program as well as future programs and policies? How will the evaluation be utilized? 7. Measures, Methods and Design  What does success look like?  What measures are being used to measure the key outcomes?    How will we know if  the program is successful? What design is being implemented  to answer this question?   What methods will answer the key  evaluation questions ?      Discuss:  Relevance to the evaluation of Health Systems Reform and Clinical Pathways 8. Learning about Program  How will the evaluations be designed to learn about the program theory of the intervention?     9. Spread and 10. Sustainability  What kinds of learning will be “spread” through the evaluation? Relationship of the evaluation to issues of generalizibility and scaling-up   How will the evaluation help with decisions to sustain the intervention? Why sustainability of the intervention is an important concept for evaluations? Discussion  Discussions on useful/influential evaluations   Discuss features of evaluations that were useful. Why were the evaluations useful? How did the evaluations help with decision making?   Discussion on structure of an evaluation plan 

Introdution to evaluation : the role of potential use in developing evaluations; day 1

https://idl-bnc-idrc.dspacedirect.org/bitstream/10625/48297/1/IDL-48297.pdf

Introdution to evaluation : the role of potential use in developing evaluations; day 1

Abstract

Similar works

Full text

Available Versions

International Development Research Centre: IDRC Digital Library