Innovation, evaluation design and typologies of professional learning

Background: Current policy discourses emphasise the importance of evidence in education, including evidencing the impact of teacher professional learning on student outcomes. Randomised controlled trial (RCT) designs are promoted to measure ‘impact’. Recent debates about this reflect longer standing methodological disputes. Advocates of comparative approaches contend that these are uniquely capable of establishing causality. However, others dispute this and consider their application in education as often being flawed. Whilst acknowledging the importance of these debates, our concern is how RCTs and similar evaluation designs are specifically used to evaluate innovations in which professional learning is important. Purpose: Arguably, professional learning is often under-theorised within experimental and quasi-experimental designs. The purpose of this paper is to address this by encouraging developers of innovations and evaluators to consider a proposed typology of professional learning and other important relevant methodological issues. This is so that developers of innovations that involve professional learning are better able to theorise their endeavours and to support more appropriate design of RCTs and other forms of evaluation of innovations. Sources of evidence: Theoretical and methodological literature from diverse fields is drawn on, namely: descriptions of RCT implementation and process evaluation designs; research on effective professional development; and theoretical models of professional learning. Insights and theories from this literature are used to develop and illustrate the typology and to identify methodological concerns and potential ways to address these. Main argument: In trials of those innovations that involve professional learning, there is both assessment of the extent to which professional learning occurs and also of whether resulting changes in practice improve outcomes. A novel typology of three different ways that professional learning may occur in innovations is proposed. This is related to the centrality (or not) of professional learning to the innovation’s success and related to the form and purpose of the professional learning involved. The three analytical categories described are pedagogical professional learning, technical professional learning and curriculum professional learning. Based on this typology, features of professional learning that are likely to lead to impact on student outcomes are discussed

The Statistics of Pupil Segregation in England

In the context of the 2020 exam 'debacle', this paper examines two statistical 'icebergs' that measure aspects of between-school and within-school variance in the English education system; participation in private education and the ‘the school effect’. These are 'icebergs' because they both display a superficial appearance of being relatively small but both have hidden, socially divisive, depths. The historical obsession with pupil segregation in England is discussed with reference to the disgraced psychologist Cyril Burt and more recent policy makers enthrallment by eugenicist beliefs. In terms of private schools, between 40 and 50% of pupils from families with greatest economic and cultural capital are shown to be privately educated compared with an overall average of 7%. In terms of the school effect, a majority of variance in pupil attainment is shown to reside at the structural level of school and (more importantly) classroom levels. In other words, the English education system is socially engineered so that grades are more determined by structural location than the efforts or agency of pupils or teachers. The analyses lead into a discussion of between-school and within-school pupil segregation. Finally, the paper calls for a new post-Covid ‘Spirit’ to de-engineer the system to unleash the talent, ability and potential of pupils, students and teachers

ScratchMaths: evaluation report and executive summary

Since 2014, computing has been part of the primary curriculum. ‘Scratch’ is frequently used by schools, and the EEF funded this trial to test whether the platform could be used to improve pupils’ computational thinking skills, and whether this in turn could have a positive impact on Key Stage 2 maths attainment. Good computational thinking skills mean pupils can use problem solving methods that involve expressing problems and their solutions in ways that a computer could execute – for example, recognising patterns. Previous research has shown that pupils with better computational thinking skills do better in maths. The study found a positive impact on computational thinking skills at the end of Year 5 – particularly for pupils who have ever been eligible for free school meals. However, there was no evidence of an impact on Key Stage 2 maths attainment when pupils were tested at the end of Year 6. Many of the schools in the trial did not fully implement ScratchMaths, particularly in Year 6, where teachers expressed concerns about the pressure of Key Stage 2 SATs. But there was no evidence that schools which did implement the programme had better maths results. Schools may be interested in ScratchMaths as an affordable way to cover aspects of the primary computing curriculum in maths lessons without any adverse effect on core maths outcomes. This trial, however, did not provide evidence that ScratchMaths is an effective way to improve maths outcomes

An onion of white advantage? (in UK HE degree attainment)

After a couple of decades examining the statistics of structural educational inequality in the English education system from early years to university access, I first examined attainment differences amongst UK graduates across ethnic, socioeconomic and gender groups in 2015. The statistical practice that I observed both shocked and confused me. Shocked to see how out-of-step HE practices were with all educational levels leading up to HE. Confused that HE was a ‘system exemplar’ for poor statistical practice (the educational ‘capstone’ status of the University/Academy had made me assume/expect different/better). The confusion has since receded, but shock remains and has helped motivate a fascination in the quantitative methodology of UK HE equality analyses. This paper focuses specifically on HE equality analyses relating to student ethnicity. In becoming acclimatised to HE ethnicity equality analyses, the more you look, the more problems you will see. In this paper, I present the statistical problems as a series ‘onion layers’ all of which could be placed under two broad methodological themes: measurement and sample representation. In developing a Critical Race Theory of statistics (QuantCrit), Gillborn et al. (2018) highlighted how statistical data and analyses can be deployed to support or obstruct progress in racial equity. This paper draws on the principles of QuantCrit to consider the substantive implications of the layers of statistical obfuscation found in UK HE ethnicity equality analyses. One thing is clear, current practice commonly results in analyses that understate the extent of educational advantage enjoyed by white graduates in UK HE. Gillborn, D., Warmington, P. & Demack, S. (2018) QuantCrit: education, policy, ‘Big Data’ and principles for a critical race theory of statistics, Race Ethnicity and Education, 21:2, 158-179, DOI: 10.1080/13613324.2017.137741

Oracy curriculum, culture and assessment toolkit

This report evaluates a developmental project designed by School 21 and the University of Cambridge to improve Year 7 students' oracy skills. The project involved developing an Oracy Skills Framework, which sets out the physical, linguistic, cognitive, and social-emotional oracy skills required by students for education and life. Other components which were informed by this framework are: • a dedicated Year 7 oracy curriculum comprising weekly oracy lessons; • oracy in every lesson; • building a whole school oracy culture; and • an Oracy Assessment Toolkit. These components were piloted and further developed with Year 7 students within School 21 from September 2013 to July 2014. During the final stages of the project the components were brought together to create an 'Oracy Curriculum, Culture and Assessment Toolkit' that can be adopted by other schools and a website was created, Voice 21 (http://voice21.org/), containing guidance and resources for schools using the Toolkit. This report focuses on an evaluation of: 1. The approaches and materials which formed the Oracy Curriculum, Culture and Assessment Toolkit, including an indicative impact finding on the impact on Year 7 pupils in School 21. 2. What further development of the Oracy Curriculum, Culture and Assessment Toolkit is needed and would enable a more robust evaluation of its impact

Personal, social, health and economic (PSHE) education: A mapping study of the prevalent models of delivery and their effectiveness

In October 2008, then Schools Minister Ji, Knight announced that Personal, Social, Health and Economic (PSHE) education would become compulsory (for Key Stages 1-4). Following this, In November 2009, Sheffield Hallam University was contracted by DCSF (now DfE) to conduct a mapping exercise of PSHE education in primary and secondary schools in England. This resulted from a recommendation in the Macdonald Review, whcih identified the need for research to establish and report on the prevalent models of delivery for PSHE education and their effectiveness in improving outcomes for children and young people (Macdonald, 2009:8)

An Evaluation of the Further Mathematics Support Programme : Research Report

The four key focus areas for the 2014-16 evaluation were to consider: • capacity and capability building • reach to schools/colleges, teachers and students • effectiveness of the programme (quality and impact) • sustainability beyond the end of the programme The evaluation also sought to assess the viability and value of quantitative modelling of Further Mathematics and FMSP activity, including identifying any relevant issues in drawing together disparate databases, to provide a quantitative baseline for future evaluation and to develop tools for future evaluation, including assessing the value of focus group interviews with students as a data collection method. The evaluation affirms previous findings about the quality and value of FMSP activity and the positive regard of stakeholders for the programme. It supports a continuation of a varied programme offer. The analysis highlights the important role that the FMSP has and can play in supporting Further Mathematics culture particularly in contexts and centres in challenging circumstances. The evaluation confirms the impact that the FMSP has had on widening participation in Further Mathematics, but also indicates that access to Further Mathematics - and so to both the intrinsic benefits of this as well as access to further opportunities - continues to be more available to students who are socially and economically advantaged