The politicisation of evaluation: constructing and contesting EU policy performance

Although systematic policy evaluation has been conducted for decades and has been growing strongly within the European Union (EU) institutions and in the member states, it remains largely underexplored in political science literatures. Extant work in political science and public policy typically focuses on elements such as agenda setting, policy shaping, decision making, or implementation rather than evaluation. Although individual pieces of research on evaluation in the EU have started to emerge, most often regarding policy “effectiveness” (one criterion among many in evaluation), a more structured approach is currently missing. This special issue aims to address this gap in political science by focusing on four key focal points: evaluation institutions (including rules and cultures), evaluation actors and interests (including competencies, power, roles and tasks), evaluation design (including research methods and theories, and their impact on policy design and legislation), and finally, evaluation purpose and use (including the relationships between discourse and scientific evidence, political attitudes and strategic use). The special issue considers how each of these elements contributes to an evolving governance system in the EU, where evaluation is playing an increasingly important role in decision making

Assessing the impact of area-based initiatives in deprived neighborhoods: The example of S. João de Deus in Porto, Portugal

Though there have been many area-based initiatives to regenerate rundown areas in numerous cities around the world, many of them involving the demolition of stigmatized housing estates, far fewer attempts have been made to assess the effects of these initiatives upon the fortunes of displaced households and those who remain in these areas. By presenting the results of an empirical in-depth case study on the effects of an area-based initiative targeted at one of the most deprived neighborhoods in Porto, this article raises several epistemological concerns related to the goals, ideological assumptions, and social and spatial effects of these initiatives. Among other interrelated issues, the article discusses the impact of conflicting ideologies upon processes of radical strategy shift and of social and territorial marginalization and appeals to the need for more pluralistic approaches to evaluation.I gratefully acknowledge the financial support of the Portuguese Fundação para a Ciência e Tecnologia for research grant SFRH/BPD/75863/2011, POPH/FSE

A realist evaluation of the management of a well- performing regional hospital in Ghana

<p>Abstract</p> <p>Background</p> <p>Realist evaluation offers an interesting approach to evaluation of interventions in complex settings, but has been little applied in health care. We report on a realist case study of a well performing hospital in Ghana and show how such a realist evaluation design can help to overcome the limited external validity of a traditional case study.</p> <p>Methods</p> <p>We developed a realist evaluation framework for hypothesis formulation, data collection, data analysis and synthesis of the findings. Focusing on the role of human resource management in hospital performance, we formulated our hypothesis around the high commitment management concept. Mixed methods were used in data collection, including individual and group interviews, observations and document reviews.</p> <p>Results</p> <p>We found that the human resource management approach (the actual intervention) included induction of new staff, training and personal development, good communication and information sharing, and decentralised decision-making. We identified 3 additional practices: ensuring optimal physical working conditions, access to top managers and managers' involvement on the work floor. Teamwork, recognition and trust emerged as key elements of the organisational climate. Interviewees reported high levels of organisational commitment. The analysis unearthed perceived organisational support and reciprocity as underlying mechanisms that link the management practices with commitment.</p> <p>Methodologically, we found that realist evaluation can be fruitfully used to develop detailed case studies that analyse how management interventions work and in which conditions. Analysing the links between intervention, mechanism and outcome increases the explaining power, while identification of essential context elements improves the usefulness of the findings for decision-makers in other settings (external validity). We also identified a number of practical difficulties and priorities for further methodological development.</p> <p>Conclusion</p> <p>This case suggests that a well-balanced HRM bundle can stimulate organisational commitment of health workers. Such practices can be implemented even with narrow decision spaces. Realist evaluation provides an appropriate approach to increase the usefulness of case studies to managers and policymakers.</p

BMC Public Health

BACKGROUND: Population health intervention research raises major conceptual and methodological issues. These require us to clarify what an intervention is and how best to address it. This paper aims to clarify the concepts of intervention and context and to propose a way to consider their interactions in evaluation studies, especially by addressing the mechanisms and using the theory-driven evaluation methodology. MAIN TEXT: This article synthesizes the notions of intervention and context. It suggests that we consider an "interventional system", defined as a set of interrelated human and non-human contextual agents within spatial and temporal boundaries generating mechanistic configurations - mechanisms - which are prerequisites for change in health. The evaluation focal point is no longer the interventional ingredients taken separately from the context, but rather mechanisms that punctuate the process of change. It encourages a move towards theorization in evaluation designs, in order to analyze the interventional system more effectively. More particularly, it promotes theory-driven evaluation, either alone or combined with experimental designs. CONCLUSION: Considering the intervention system, hybridizing paradigms in a process of theorization within evaluation designs, including different scientific disciplines, practitioners and intervention beneficiaries, may allow researchers a better understanding of what is being investigated and enable them to design the most appropriate methods and modalities for characterizing the interventional system. Evaluation methodologies should therefore be repositioned in relation to one another with regard to a new definition of "evidence", repositioning practitioners' expertise, qualitative paradigms and experimental questions in order to address the intervention system more profoundly

BMC Public Health

BACKGROUND: Connected health devices and applications (referred to hereafter as "SDApps" - Smart devices and applications) are being portrayed as a new way for prevention, with the promise of accessibility, effectiveness and personalization. Many effectiveness evaluations (experimental designs) with strong internal validity exist. While effectiveness does appear to vary, the mechanisms used by these devices have not yet been thoroughly investigated. This article seeks to unpack this black box, and describes the process of elaboration of an intervention theory for healthy eating and physical activity SDApps. It includes a set of requirements relative to their impact on social health inequalities. METHODS: To build this theory, we drew on theory-driven approaches and in particular on the theory of change (ToC) method. To this end, we developed a cumulative and iterative process combining scientific data from the literature with knowledge from experts (researchers and practitioners) and from patients or users. It was a 3-step process, as follows: 1 - identifying the evidence base; 2 - developing the theory through design intervention and creating realistic expectations, including in our case specific work on social health inequalities (SHIs); 3 - modeling process and outcome. RESULTS: We produced an evidence-based theory according to the ToC model, based on scientific evidence and knowledge from experts and users. It sets out a causal pathway leveraging 11 key mechanisms - theoretical domains - with which 50 behavior change techniques can be used towards 3 ultimate goals: Capacity, Opportunity, Motivation - Behavior (COM-B). Furthermore, the theory specifically integrates requirements relative to the impact on SHIs. CONCLUSIONS: This theory is an aid to SDAapp design and evaluation and it can be used to consider the question of the possible impact of SDApps on the increase in inequalities. Firstly, it enables developers to adopt a more overarching and thorough approach to supporting behavior change, and secondly it encourages comprehensive and contributive evaluations of existing SDApps. Lastly, it allows health inequalities to be fully considered

Advancing the application of systems thinking in health:analysing the contextual and social network factors influencing the use of sustainability indicators in a health system - a comparative study in Nepal and Somaliland

BACKGROUND: Health systems strengthening is becoming a key component of development agendas for low-income countries worldwide. Systems thinking emphasizes the role of diverse stakeholders in designing solutions to system problems, including sustainability. The objective of this paper is to compare the definition and use of sustainability indicators developed through the Sustainability Analysis Process in two rehabilitation sectors, one in Nepal and one in Somaliland, and analyse the contextual factors (including the characteristics of system stakeholder networks) influencing the use of sustainability data. METHODS: Using the Sustainability Analysis Process, participants collectively clarified the boundaries of their respective systems, defined sustainability, and identified sustainability indicators. Baseline indicator data was gathered, where possible, and then researched again 2 years later. As part of the exercise, system stakeholder networks were mapped at baseline and at the 2-year follow-up. We compared stakeholder networks and interrelationships with baseline and 2-year progress toward self-defined sustainability goals. Using in-depth interviews and observations, additional contextual factors affecting the use of sustainability data were identified. RESULTS: Differences in the selection of sustainability indicators selected by local stakeholders from Nepal and Somaliland reflected differences in the governance and structure of the present rehabilitation system. At 2 years, differences in the structure of social networks were more marked. In Nepal, the system stakeholder network had become more dense and decentralized. Financial support by an international organization facilitated advancement toward self-identified sustainability goals. In Somaliland, the small, centralised stakeholder network suffered a critical rupture between the system's two main information brokers due to competing priorities and withdrawal of international support to one of these. Progress toward self-defined sustainability was nil. CONCLUSIONS: The structure of the rehabilitation system stakeholder network characteristics in Nepal and Somaliland evolved over time and helped understand the changing nature of relationships between actors and their capacity to work as a system rather than a sum of actors. Creating consensus on a common vision of sustainability requires additional system-level interventions such as identification of and support to stakeholders who promote systems thinking above individual interests