Puzzling about policy paradigms : precision and progress

No abstrac

Reassessing the concept of policy paradigm : aligning ontology and methodology in policy studies

The concept of policy paradigm, which was developed by Peter Hall (1993), has given rise to an abundance of literature on ideas and policy dynamics. Despite its theoretical value, however, the original concept was insufficiently specified and has been used in problematic ways by many policy scholars, thereby undermining the validity of their descriptive and causal inferences. The main argument defended in this article is that scholars studying policy paradigms should devote as much attention to the ideas of policy makers as to the policies they adopt, while distinguishing between the two constructs as much as possible. This article reviews these issues and put forward seven guidelines that aim to better circumscribe the concept of policy paradigms and improve research on ideas and policy dynamics

Ideas and welfare reform in Saskatchewan : entitlement, workfare or activation?

Plusieurs provinces canadiennes ont mis en œuvre des réformes substantielles de leur régime d’aide sociale dans les années 1990. Or, nos connaissances à propos des idées qui sous-tendent ces réformes sont étonnamment limitées. En particulier, peu d’études empiriques ont directement examiné les idées des acteurs de politique. Cet article présente une étude de cas rétrospective d’une initiative saskatchewanaise majeure, soit la réforme Building Independence (BI), et examine sa congruence avec trois paradigmes d’aide sociale. Les données proviennent d’entrevues avec des acteurs de politique. Cette étude conclut que les idées sous-jacentes à BI sont étroitement alignées avec le paradigme de l’activation, mais présente certaines similitudes avec les paradigmes du droit social (entitlement) et du workfare. Ce résultat est ensuite discuté à la lumière de trois enjeux : 1) les autochtones, 2) la perspective des prestataires d’aide sociale, et 3) l’idéologie de la Troisième voie. Des pistes de recherche sur les idées et le changement de politique viennent conclure cet article.Many provinces have enacted substantial reforms of their social assistance regime in the 1990s. However, we know surprisingly little about the ideas that underlie welfare reform in Canada. In particular, few empirical studies have directly examined the ideas of policy actors. This article presents a retrospective case study of a major policy initiative, namely Saskatchewan’s Building Independence (BI), and examines its alignment with three paradigms of social assistance. Data come primarily from interviews with policy actors. This study concludes that the policy ideas informing BI align closely with the activation paradigm, but also share some similarities with the entitlement and workfare paradigms. The significance of this finding is then discussed in light of three issues: 1) Aboriginal people, 2) the perspective of social assistance clients, and 3) Third Way ideology. To conclude this article, a research agenda on ideas and policy change is proposed

Program awareness, administrative burden, and non-take-up of Québec’s supplement to the work premium

Program take-up is a necessary condition for program effectiveness. Yet, non-take-up is a significant challenge for many social programs, including Québec’s Supplement to the Work Premium (SWP), a refundable tax credit targeted toward long-term welfare clients. Based on interviews with 21 public actors and 46 program participants and nonparticipants, this study explains how low program awareness, the low value of the benefit and the significant administrative burden borne by potential and actual participants contribute to the non-take-up of the SWP in this sample. Moreover, four policy implications and recommendations that can inform the design, implementation, and evaluation of social programs, are derived from this study

Taking explanation seriously in political science

The concept of explanation has attracted considerable attention in the social sciences and, in particular, in political science. However, scholars are not always familiar with what explaining political phenomena means, let alone with what it entails for developing sound causal arguments. This essay introduces Craig Parsons’ fourfold typology of explanation (institutional, ideational, structural, and psychological) before assessing its value for the causal analysis of political behaviour and processes. As argued, despite its limitations, his typology clearly maps the explanations in political science while helping scholars to combine them more rigorously, when needed. This is why Parsons typology has the potential to move political scientists to the ‘next level’, as far as explanation is concerned

L'examen de qualité des évaluations fédérales : une méta-évaluation réussie?

La qualité des évaluations représente un enjeu fondamental au sein du mouvement de la gestion et des politiques fondées sur des données probantes. Le Secrétariat du Conseil du Trésor du Canada (SCT) a ainsi réalisé en 2004 un examen de la qualité des évaluations fédérales d’une envergure considérable. Au regard de sa pertinence, de sa méthodologie, et de la crédibilité de ses conclusions, cette méta-évaluation est-elle réussie? Cet article offre une réponse à cette question en présentant une évaluation de l’examen du SCT. Malgré des lacunes importantes au niveau de la théorie et des critères de qualité, du devis de recherche, du processus de codage, et de l’analyse des données, cette méta-évaluation présente des conclusions qui sont dans l’ensemble pertinentes et d’un niveau de crédibilité acceptable. Les enseignements que l’on peut tirer de cet examen sont proposés à titre de recommandations à l’intention des évaluateurs et des responsables des services d’évaluation désirant réaliser un examen similaire mais qui ne souffrirait pas des mêmes lacunes.Evaluation quality is a fundamental issue within the evidence- based management and policy movement. The Treasury Board Secretariat of Canada (TBS) conducted a wide-ranging review of the quality of federal evaluations in 2004. In terms of its relevance and methodology and the credibility of its conclusions, is the metaevaluation a success? This article answers the question by presenting an evaluation of the TBS quality review. Despite serious shortcomings with respect to the quality theory and criteria, design, coding process, and data analysis, the meta-evaluation conclusions are relevant and credible overall. Lessons learned from this quality review are proposed as recommendations for evaluators and officials responsible for evaluation units who wish to conduct a similar review that does not suffer from the same shortcomings

La participation à l'évaluation : du concept à la mesure

La popularité croissante des approches participatives représente une tendance lourde dans le champ de l’évaluation des politiques. La prolifération des définitions et des termes utilisés pour désigner la participation génère cependant beaucoup de confusion chez les chercheurs et praticiens du domaine. Il n’existe en outre aucun instrument de mesure adéquat de la participation, ce qui freine l’avancement des connaissances. Trois questions de recherche structurent cette thèse : 1) Qu’est-ce que la participation à l’évaluation?; 2) Comment traduire ce concept en un instrument de mesure opérationnel?; et 3) Est-ce que cet instrument mesure la participation de manière fidèle et valide? Une conceptualisation cohérente de l’évaluation participative s’inscrivant dans la foulée des travaux de Cousins et Whitmore (1998) est d’abord proposée. Cette conceptualisation, fondée sur la logique des conditions nécessaires et suffisantes, est opérationnalisée en un instrument de mesure de la participation. L’instrument (Participatory Evaluation Measurement Instrument – PEMI) fait ensuite l’objet d’une validation empirique à partir d’un échantillon de 40 cas d’évaluation tirés de la littérature et d’un sondage auprès de leurs auteurs. Trois éléments sont appréciés quantitativement : 1) la fidélité intercodeur; 2) la convergence des scores des codeurs et des auteurs sur le PEMI; et 3) la convergence des scores des auteurs sur le PEMI et un instrument de mesure alternatif. De manière générale, cette étude suggère que le PEMI génère des scores dont la fidélité et la validité sont d’un niveau acceptable. Troisièmement, une étude de validation du PEMI combinant méthodes qualitatives et quantitatives est présentée. Le recours aux méthodes mixtes a généré un cycle inattendu – mais bénéfique – de révision de l’instrument et de validation quantitative supplémentaire. Les résultats de validation suggèrent que la version révisée du PEMI, désormais fondée sur une structure conceptuelle hybride, est plus en phase avec l’opinion des répondants quant au niveau de participation des cas d’évaluation. La valeur ajoutée des méthodes mixtes à des fins de validation est également discutée. Une réflexion sur le potentiel scientifique de l’instrument de mesure, en particulier dans le cadre de recherches empiriques sur la relation entre participation et utilisation de l’évaluation, vient conclure cette thèse.The growing popularity of participatory approaches represents an important trend in the field of program evaluation. The proliferation of definitions and terms used to designate stakeholder participation, however, generates a lot of confusion among researchers and practitioners. Moreover, the dearth of adequate instruments to measure participation hinders knowledge accumulation. This dissertation is structured around three research questions: 1) What is stakeholder participation in evaluation? 2) How is this concept translated into an operational measurement instrument? and 3) Does this instrument allow for the reliable and valid measurement of stakeholder participation? A systematic and coherent conceptualization of participatory evaluation is first proposed based on the work of Cousins and Whitmore (1998). This conceptualization, which is based on the logic of necessary and sufficient conditions, is operationalized in a measurement instrument. The instrument (Participatory Evaluation Measurement Instrument – PEMI) is then empirically validated using a sample of 40 evaluation cases from the literature and a survey of their authors. Three elements are quantitatively assessed: 1) intercoder reliability; 2) convergence between coders’ and authors’ scores on the PEMI; and 3) convergence between authors’ scores on the PEMI and an alternative measurement instrument. Considered globally, this study suggests that the PEMI can generate reliable and valid scores. Finally, a validation study combining qualitative and quantitative methods is presented. The use of mixed methods has generated an unexpected but most welcome cycle of instrument revision and further quantitative validation. The validation results suggest that the revised version of the PEMI, now based on a hybrid conceptual structure, is more in line with our respondents’ opinions with respect to the level of stakeholder participation in their particular evaluation case. The added value of mixed methods for validation purposes is also discussed using counterfactual reasoning. Reflections on the scientific and practical potential of the measurement instrument, on the relationship between stakeholder participation and evaluation use in particular, conclude this dissertation

Measuring stakeholder participation in evaluation : an empirical validation of the Participatory Evaluation Measurement Instrument (PEMI)

Background. Stakeholder participation is an important trend in the field of program evaluation. Although a few measurement instruments have been proposed, they either have not been empirically validated or do not cover the full content of the concept. Objectives. This study consists of a first empirical validation of a measurement instrument that fully covers the content of participation, namely the Participatory Evaluation Measurement Instrument (PEMI). It specifically examines 1) the intercoder reliability of scores derived by two research assistants on published evaluation cases; 2) the convergence between the scores of coders and those of key respondents (i.e., authors); and 3) the convergence between the authors’ scores on the PEMI and the Evaluation Involvement Scale (EIS). Sample. A purposive sample of 40 cases drawn from the evaluation literature was used to assess reliability. One author per case in this sample was then invited to participate in a survey; 25 fully usable questionnaires were received. Measures. Stakeholder participation was measured on nominal and ordinal scales. Cohen’s kappa, the intraclass correlation coefficient and Spearman’s rho were used to assess reliability and convergence. Results. Reliability results ranged from fair to excellent. Convergence between coders’ and authors’ scores ranged from poor to good. Scores derived from the PEMI and the EIS were moderately associated. Conclusions. Evidence from this study is strong in the case of intercoder reliability and ranges from weak to strong in the case of convergent validation. Globally, this suggests that the PEMI can produce scores that are both reliable and valid

Using systematic review methods within a Ph.D. dissertation in political science : challenges and lessons learned from practice

Systematic review and synthesis methods have gained wide acceptance within the social sciences and, as a result, many postgraduate students now consider using them for their thesis or dissertation research. However, students are rarely aware of all the concrete implications that their decision entails. This reflective narrative reports the experience of a political science student who began to conduct a systematic review as part of his PhD dissertation but who did not complete it. The aim of this article is to identify challenges and lessons learned from this experience and to formulate recommendations for postgraduate students who wish to make an informed choice with respect to the use of these methods