A case against school effectiveness

This paper considers the model of school effectiveness (SE) currently dominant in research, policy and practice in England (although the concerns it raises are international). It shows, principally through consideration of initial and propagated error, that SE results cannot be relied upon. By considering the residual difference between the predicted and obtained score for all pupils in any phase of education, SE calculations leave the results to be disproportionately made up of relative error terms. Adding contextual information confuses but does not help this situation. Having shown and illustrated the sensitivity of SE to this propagation of initial errors, and therefore why it is unworkable, the paper considers some of the reasons why SE has become dominant, outlines the damage this dominant model causes, and begins to shape alternative ways of considering what schools do

Measuring is more than assigning numbers

'Measurement is fundamental to research-related activities in social science (hence this Handbook). In my own field of education research, perhaps the most discussed element of education lies in test scores. Examination results are measurements, the number of students attaining a particular standard in a test is a measurement; indeed the standard of a test is a measurement. The allocation of places at school, college or university, student:teacher ratios, funding plans, school timetables, staff workloads, adult participation rates, and the stratification of educational outcomes by sex, social class, ethnicity or geography for example, are all based on measurements. Good and careful work has been done in all of these areas (Nuttall 1987). However, the concept of measurement itself remains under-examined, and is often treated in an uncritical way. In saying this I mean more than the usual lament about qualitative:quantitative schism or the supposed reluctance of social scientists to engage with numeric analysis (Gorard et al. 2004a). I mean that even where numeric analysis is being conducted, the emphasis is on collecting, collating, analysing, and reporting the kinds of data generated by measurement, with the process of measurement and the rigor of the measurement instrument being somewhat taken for granted by many commentators. Issues that are traditionally considered by social scientists include levels of measurement, reliability, validity, and the creation of complex indices (as illustrated in some of the chapters contained in this volume). But these matters are too often dealt with primarily as technical matters – such as how to assess reliability or which statistical test to use with which combination of levels of measurement. The process of quantification itself is just assumed'

School experience as a potential determinant of post-compulsory participation

This paper considers the views of young people aged 14 to 16 about their future education, training and occupation. It is based on a study of around 3,000 year 11 pupils in 45 educational settings in England during 2007/08, supplemented by documentary analysis, official statistics, and interviews and surveys with staff and parents. Pupil-reported plans to continue in formal education and their aspirations for professional occupations are heavily stratified by individual and family background, including prior attainment. This is as expected. But once this variation has been accounted for, in a logistic regression model, there is both a small school mix ‘effect’ and a much larger school experience effect. The patterns in the pupil stories suggest that there are some simple levers available to policy-makers and to practitioners for the improvement of young peoples’ plans to participate

Research Design, as Independent of Methods

Objectives •Readers of this chapter should be in a better position to: •Understand the process of research design •Place their own and others work within a full cycle or programme of ongoing research •Understand why good research almost always involves a mixture of evidence •Defend themselves from those who want numbers and text to be enemies rather than allies •Argue that good research is more ethical for society than poor researc

Who is missing from higher education?

This paper discusses the difficulties of establishing a clear count of UK higher education students in terms of the categories used for widening participation, such as occupational background or ethnicity. Using some of best and most complete data available, such as the annual figures from the Higher Education Statistics Agency, the paper then establishes that there is little evidence of a simple consistent pattern of under-representation within these categories, except perhaps for men, and students of white ethnicity. However, once prior qualifications are taken into account, there is no evidence that potential students are unfairly and disproportionately denied access to HE in terms of occupation, ethnicity, sex or disability. This has important implications for what we mean by widening participation in HE, and how we might achieve it

Equity - and its relationship to citizenship education

'In the second half of the seminar, Professor Gorard will draw out the policy implications of some EU-funded research on pupils’ views on equity in schools and on the segregation of school intakes across different European countries. In particular, he will argue that in order to assess whether school systems are equitable we need to pay attention to the lived experience of pupils in schools, and that concerns about equality and quality in education need not be in tension with one another. Professor Gorard’s argument is that the use of school improvement models has led to an undue emphasis on the most visible indicators of schooling - examination results – which may marginalise other purposes and benefits of schooling. Social, ethnic and economic segregation between schools matters, but not primarily for the sake of test results because, for pupils, schools are their life and not merely a preparation for it. In general the lessons from international studies are that the mix of pupils between schools whether in terms of occupational class, income or sex has little impact on attainment. So policy makers can feel free to use criteria other than effectiveness for deciding on the pattern of intakes to schools including criteria related to equity

Doing data analysis

'Research is about more than empirical evidence, but evidence is at the heart of finding out more about the social and education world. One way of marshalling evidence on a topic, or to answer a research question, is to use the findings of others as published in the literature. This use of evidence at third-hand is common – in the notorious literature review for a PhD, for example. I say ‘third-hand’ because the analyst does not have access to the primary evidence, nor are they re-presenting an analysis of the data. They are presenting a summary of what a previous author presented about an analysis of data. Done well, with a clear focus, such a review of literature can be useful, at least in establishing what others think, how a topic is usually researched, and why the topic might be important to research further. Some of the inherent weaknesses of using the accounts of others might be overcome by ensuring that all of the relevant literature was used, even accounts of unsuccessful studies and evidence from unpublished studies, and then conducting a full meta-analysis of the results (I recommend using a Bayesian approach, see appendix to Gorard et al. 2004, which allows the relatively simple combination of different kinds of evidence). But such systematic reviews of evidence are rare, very difficult to do properly, and both expensive and time-consuming. And anyway this second approach does not overcome the chief drawbacks of the literature which are that we have no direct access to the evidence of others, and often face a very partial view of the assumptions made and the analyses conducted.

Commentary on the evaluation of teacher effectiveness through student test scores

The Tennessee Value-added Assessment System is claimed to be able to estimate the impact of teachers on their students’ progress. This has led to further claims, such as that teacher quality is paramount in improving student progress. However, TVAAS and similar schemes should not be relied upon. Explanations of TVASS in the public domain are incomplete and poorly presented. TVAAS is not a ‘test’ of anything, and other analysts have attributed the same student progress residuals as used in TVAAS to school, classroom, district, leadership, social and economic factors. The analysis appears to be circular – effective teachers are defined by progress of students so students making progress have necessarily effective teachers. The analysis anyway cannot be sustained with the kinds of data available. The estimated level of missing data, and of measurement and representational error in the data that is present, suggest that the estimated residuals for each student are composed largely of error terms. Sanders and colleagues make elementary statistical errors, such as using significance tests with population data

Accessing and understanding research in education

Introductory comments This chapter describes the range of evidence that exists relevant to educational studies, and discusses how to locate, read, and use such evidence. The evidence exists in publicly available datasets as well as in the writings of others. The chapter introduces some generic methods of assessing the quality and usefulness of evidence, including a key quality control principle. The chapter ends by outlining some ideas for future research