The power of PISA – limitations and possibilities for educational research

On 6th December 2016, the Programme for International Student Assessment (PISA) releases its report on the achievements of 15-year-olds from 72 countries and economies around the world. This triennial international survey aims to evaluate education systems across 72 contexts by testing skills in Mathematics, Science and Reading Literacy. This is the sixth cycle of PISA and the OECD suggests countries and economies now have the capability to compare the results over time to ‘assess the impact of education policy decisions’ . Compared to other education studies, the media coverage of PISA must be described as massive (Meyer and Benavot, 2013, Baird et al., 2016) and, as with previous years, it is expected that PISA will attract considerable discussion among policy makers, educators and researchers (Wiseman, 2014). It is therefore timely to present a thematic issue of Assessment in Education, where we publish four articles that have analysed previous datasets from the PISA studies each commenting upon the challenges, limitations and potential future assessment research on the PISA data. The articles touch upon issues regarding sampling, language, item difficulty and demands, as well as the secondary analyses of students’ reported experiences of formative assessment in the classroom. One important message from the authors in this thematic Special Issue is the need for a more complex discussion around the use and misuse of PISA data, and the importance of pointing to the limitations of how the results are presented to policy makers and the public. In an area where the media produces narratives on schools and education systems based upon rankings in PISA, researchers in the field of large-scale assessment studies have a particularly important role in stepping up and advising on how to interpret and understand these studies, while warning against potential misuse. </p

On the supranational spell of PISA in policy

BACKGROUND: PISA results appear to have a large impact upon government policy. The phenomenon is growing, with more countries taking part in PISA testing and politicians pointing to PISA results as reasons for their reforms. PURPOSE: The aims of this research were to depict the policy reactions to PISA across a number of jurisdictions, to see whether they exhibited similar patterns and whether the same reforms were evident. SOURCES OF EVIDENCE: We investigated policy and media reactions to the 2009 and 2012 PISA results in six cases: Canada, China (Shanghai), England, France, Norway and Switzerland. Cases were selected to contrast high-performing jurisdictions (Canada, China) with average performers (England, France, Norway and Switzerland). Countries that had already been well reported on in the literature were excluded (Finland, Germany). DESIGN AND METHODS: Policy documents, media reports and academic articles in English, French, Mandarin and Norwegian relating to each of the cases were critically evaluated. RESULTS: A policy reaction of ‘scandalisation’ was evident in four of the six cases; a technique used to motivate change. Five of the six cases showed ‘standards-based reforms’ and two had reforms in line with the ‘ideal-governance’ model. However, these are categorisations: the actual reforms had significant differences across countries. There are chronological problems with the notion that PISA results were causal with regard to policy in some instances. Countries with similar PISA results responded with different policies, reflecting their differing cultural and historical education system trajectories. CONCLUSIONS: The connection between PISA results and policy is not always obvious. The supranational spell of PISA in policy is in the way that PISA results are used as a magic wand in political rhetoric, as though they conjure particular policy choices. This serves as a distraction from the ideological basis for reforms. The same PISA results could motivate a range of different policy solutions

Never Mind the Gap: Formative Assessment Confronted with Dewey’s and Gadamer’s Concept of Experience

The notion of “closing the learning gap” is widely used in the conceptualisation of formative assessment. It builds on an unarticulated assumption that students' learning can and should be controlled towards predefined outcomes. This article discusses this control assumption in the light of the concept of the American philosopher John Dewey and the German philosopher Hans‐Georg Gadamer. Their conceptualisation challenges the idea of learning as a linear and controllable process that results in stable and predictable outcomes. Using the concept of experience, we argue that learning follows a continuous circular movement where previous experiences condition future interpretations and that every experience changes the subject. This process of change is both unpredictable and diverse and requires that attention is paid to the uniqueness of each situation and to students as subjects. Following the discussion, we propose a model for considering the extensiveness and rigidity of formative assessment practices and that authors pay attention to whether they conceptualise formative assessment in a way that promotes student and teacher “gap closing” and control.publishedVersio

‘[It] isn’t designed to be assessed how we assess’:Rethinking assessment for qualification in the context of the implementation of the Curriculum for Wales

This paper reports teacher and learner perspectives on how assessment and reform influences pedagogical practices and behaviours. The research was conducted in a context of policy reform, at a time when Wales’ revised General Certificate of Secondary Education (GCSE) specifications had been implemented, and learners were preparing for their assessments; but, also during the period of debate on the development of Wales’ new curriculum, which has taken a distinct and contrasting position on assessment to the assumptions underlying the reform of Welsh GCSEs implemented from 2015. These data, therefore, offer unique insights into the affordances and limitations of two sharply contrasting systems at a time of considerable change, offering reflections on the current curriculum and its attendant assessment practices, and also a prospective analysis of how the principles embedded in the new curriculum could challenge these existing assumptions and conventions. Findings suggest that teachers and learners currently inhabit an assessment‐driven system, which encourages performative practices in pedagogy and is governed by external accountability; and that these practices are at odds with the principles of assessment articulated in Successful Futures. Consequently, teachers in this study expressed uncertainty about how assessment for certification purposes at GCSE could be compatible with the principles of the Curriculum for Wales

Assessment in the service of learning: challenges and opportunities or Plus ça Change, Plus c’est la même Chose

This paper begins with a brief overview of literature indicating that, although there have been significant advances in the field’s capacity to conduct both formative and summative assessments over the past decades, those advances have not been matched by comparable impact. The bulk of the paper is devoted to a series of examples from the Mathematics Assessment Project that illustrate issues of methods, and the unrealized potential for advances

The power of PISA – limitations and possibilities for educational research

On 6th December 2016, the Programme for International Student Assessment (PISA) releases its report on the achievements of 15-year-olds from 72 countries and economies around the world. This triennial international survey aims to evaluate education systems across 72 contexts by testing skills in Mathematics, Science and Reading Literacy. This is the sixth cycle of PISA and the OECD suggests countries and economies now have the capability to compare the results over time to ‘assess the impact of education policy decisions’ . Compared to other education studies, the media coverage of PISA must be described as massive (Meyer and Benavot, 2013, Baird et al., 2016) and, as with previous years, it is expected that PISA will attract considerable discussion among policy makers, educators and researchers (Wiseman, 2014). It is therefore timely to present a thematic issue of Assessment in Education, where we publish four articles that have analysed previous datasets from the PISA studies each commenting upon the challenges, limitations and potential future assessment research on the PISA data. The articles touch upon issues regarding sampling, language, item difficulty and demands, as well as the secondary analyses of students’ reported experiences of formative assessment in the classroom. One important message from the authors in this thematic Special Issue is the need for a more complex discussion around the use and misuse of PISA data, and the importance of pointing to the limitations of how the results are presented to policy makers and the public. In an area where the media produces narratives on schools and education systems based upon rankings in PISA, researchers in the field of large-scale assessment studies have a particularly important role in stepping up and advising on how to interpret and understand these studies, while warning against potential misuse. </p

The power of PISA – limitations and possibilities for educational research

On 6th December 2016, the Programme for International Student Assessment (PISA) releases its report on the achievements of 15-year-olds from 72 countries and economies around the world. This triennial international survey aims to evaluate education systems across 72 contexts by testing skills in Mathematics, Science and Reading Literacy. This is the sixth cycle of PISA and the OECD suggests countries and economies now have the capability to compare the results over time to ‘assess the impact of education policy decisions’ . Compared to other education studies, the media coverage of PISA must be described as massive (Meyer and Benavot, 2013, Baird et al., 2016) and, as with previous years, it is expected that PISA will attract considerable discussion among policy makers, educators and researchers (Wiseman, 2014). It is therefore timely to present a thematic issue of Assessment in Education, where we publish four articles that have analysed previous datasets from the PISA studies each commenting upon the challenges, limitations and potential future assessment research on the PISA data. The articles touch upon issues regarding sampling, language, item difficulty and demands, as well as the secondary analyses of students’ reported experiences of formative assessment in the classroom. One important message from the authors in this thematic Special Issue is the need for a more complex discussion around the use and misuse of PISA data, and the importance of pointing to the limitations of how the results are presented to policy makers and the public. In an area where the media produces narratives on schools and education systems based upon rankings in PISA, researchers in the field of large-scale assessment studies have a particularly important role in stepping up and advising on how to interpret and understand these studies, while warning against potential misuse. </p