Canning, John

English

Question 22 (Q22), ‘Overall, I am satisfied with the quality of the course' stands alone in the National Student Survey (NSS). University league tables include it more than any other question. however, despite being the most commonly used measure from the NSS the questions has received very little scrutiny

University of Brighton Research Portal

1  Half a million unsatisfied graduates? Increasing scrutiny of National Student Survey’s ‘overall’ question.  John Canning Senior Lecturer in Learning and Teaching in Higher Education Centre for Learning and Teaching University of Brighton j.canning@brighton.ac.uk     Question 22 (Q22), ‘Overall, I am satisfied with the quality of the course’ stands alone in the National Student Survey (NSS). University league tables include it more than any other question. It covers a multitude of sins. It is the litmus test of a course. However good or bad the assessment, the facilities, the course organisation, this overall question can condemn or redeem a course. Like checkmate in chess, a boxer’s knockout punch, a judge’s verdict it does not matter who tried hardest, made the best moves or made the best arguments. In many respects it is the only judgment that counts.   In 2014 86% of students agreed or strongly agreed with Q22, a figure colourfully highlighted on the HEFCE (2014a) website.  Does good in part mean good overall? Marsh and Cheng (2008) recognised that Q22 was under-researched, especially in terms of its relationship with other 21 questions.  Q22 has such a taken for granted status that it has escaped individual scrutiny. Overall satisfaction is the headline figure and university leaders set targets for their own institutions.  This is not to suggest that the other 21 questions are ignored though. It is reasonable to assume extended effort, better coaching and developing better arguments to improve scores on Questions 1-21 will lead to a greater chance of positive verdict on Question 22.   Correlations between overall satisfaction with most of the other 21 items are fairly high (around 0.7) though correlation does not necessarily equal causation. Even the direction of possible causation is unclear—is a student satisfied overall when the other 21 components (plus other unknown issues not addressed) are in place, or does a satisfied student take the view that all is well with the rest of the experience?    Yorke (2009) notes all the questions in NSS are positive leading the possibility of acquiescence bias (or ‘yeah-saying’). If some if the preceding questions were negative would the overall satisfaction be 86%, greater than 86% or less than 86%? If the NSS included questions such ‘The teaching on my course was boring’, ‘Staff failed to give me feedback on my work’, or the ‘The equipment and resources required for my course were outdated’ then we may see a propensity for lower overall satisfaction—I am entering the realm of pure speculation here but the order of questions might be important. Would overall satisfaction be higher or lower if Q22 came first or the questions were generated to each student at random? Although Yorke found no statistically significant differences between responses when he made some of the existing questions negative, he suspects that the highly politicised ‘real world’ environment of the NSS may make for different results than the controlled environments of his experiments.   2  Question 22 does not come under scrutiny in Callender et al’s (2014) recent review of the NSS which recommends rephrasing and replacing certain questions in time for the 2017 survey. The ‘overall quality’ question is rated most highly by potential students (Renfrew et al 2010), but the assumption that this question is unproblematic could be usefully explored further. To my knowledge there has not been any study of the placing and ordering of the NSS questions.  Sir Humphry Appleby, the erudite and pompous Cabinet secretary in 1980s comedy ‘Yes Prime Minister’, shows his protégé Bernard Woolly how opinion polls can be manipulated. A majority view, either yes or no, on any question can be obtained by the biasing the questions which proceed it, then only reporting on the final question (Jay and Lynn 1986).  The importance placed on Q22 in league tables and university publicity should lead us to scrutinise the ‘overall’ more, not less. How do respondents really interpret this question? As academics we often assume students demarcate clear boundaries between the learning and teaching experience and ‘externals’ such as the quality of university accommodation, sports facilities or social life.  We may assume there is a strict boundary between the course and the experience of university outside the course, but this does not mean the students think about this in the same way.  These outside factors can enable or inhibit the students’ ability to benefit fully from the course. How can a student be satisfied with overall quality of the course if he/she is unable to attend classes due to outside problems, some of which may be within the university’s sphere of influence?     Alarmingly big numbers  Then we have the psychology of percentages and big numbers. Is 86% a large or small percentage? At first glance it appears large. A vast majority of students are satisfied –more than eight out of 10 or four out of five in marketing speak. No doubt many organisations would be delighted with that level of satisfaction.    However, let us forget the percentages for a moment and look at actual numbers. In 2014 53,252 students did not agree or strongly agree with the Q22 statement.  53,252 is a big number, almost enough to fill the repurposed London Olympic stadium.  Extend that figure over the ten year period of the NSS and suddenly, the population of unsatisfied and ambivalent graduates is about the same as Leeds. As the NSS is undertaken in the final year of the course these unsatisfied students are mostly successful students. At the very least they have made it through the first, second or third year of assessment (depending on course length). These are not our drop-outs and failures.   The unsatisfied graduate should give us more cause for retrospection than at present. Students are graduating, not only £50,000 plus in debt, but in possession of a degree certificate from our institution and three or four years of an unsatisfactory experience on their CV.  If we are dissatisfied with our car insurance company or latest vacuum cleaner most of us would go no further than avoiding that company or manufacturer in the future and perhaps warning our friends. Three years’ work, £50,000 of debt and an identity as a graduate of a particular institution and course cannot be resolved by doing better research next time, or accepting a bad experience as part of life’s rich tapestry. For students who made it to the final year then left without a qualification the outlook will be even worse.  The unsatisfied graduate may eventually consider that their experience was satisfactory after all. They may be successful and realise the value of their course was greater than expected. 3  Conversely they may attribute personal success as having been achieved despite their university studies.  Percentages of 80 or 90 plus must not be allowed to distract us from the troublesome truth that tens of thousands of student express negative views on the NSS each year.  Looking at the actual numbers troubles me in a way the percentages do not. Do we accept that there will always be students for whom university is a less than satisfactory experience and concentrate our efforts on trying to make these numbers as few as possible? Or do these vast majority positive experiences feed our complacency?  In conclusion  A student in a prior project reported ‘No-one knows what the question means, but everyone can recognise that the responses will be used to make the university look good’ (Canning et al, 2011: 5). That 16% of students still can’t agree or strongly agree with Question 22 is troubling as damaging the reputation of one’s alma mater appears irrational. Q22 offers an insight into a truth about the student learning experience, albeit ‘Through a glass, darkly’. With more than 50,000 students unable to agree with this overall question each year we need more thought and research into understanding this question.    References Callender, C,, Ramsden, P., and Griggs, J. (2014) Review of the National Student Survey. Bristol: HEFCE. http://tinyurl.com/kmmgogu  Canning, J, et al (2011) Understanding the National Student Survey: Investigations in Languages, Linguistics and Area Studies. Southampton: LLAS Centre for Languages, Linguistics and Area Studies http://tinyurl.com/noqe5h9  HEFCE (2014a) National Student Survey results hit 10-year high http://tinyurl.com/lcfjmd4  HEFCE (2014b) UK Review of the Provision of Information about Higher Education : National Student Survey Results and Trends Analysis 2005-2013. Bristol: HEFCE. http://tinyurl.com/pmoajvf  Jay, Antony, and Jonathan Lynn. ‘The Ministerial Broadcast’. Yes Prime Minister. BBC. http://tinyurl.com/l5ahs5w Marsh, H. W. and Cheng, J. (2008). National Student Survey of Teaching in UK Universities: Dimensionality, Multilevel Structure, and Differentiation at the Level of University and Discipline: Preliminary Results. York: Higher Education Academy. Renfrew K et al (2010) Understanding the Information Needs of Users of Public Information  About Higher Education. Manchester: Oakleigh http://tinyurl.com/olvxvrg        Yorke, M. ‘‘Student Experience’ Surveys: Some Methodological Considerations and an Empirical Investigation’. Assessment & Evaluation in Higher Education 34: 721–39.     

Half a million unsatisfied graduates? Increasing scrutiny of National Student Survey's ‘overall' question

Kingston University Research Repository

Half a million unsatisfied graduates? Increasing scrutiny of the National Student Survey's 'overall' question

https://cris.brighton.ac.uk/ws/files/368877/jcanning%20Educational%20Developments%20submission.pdf

Half a million unsatisfied graduates? Increasing scrutiny of National Student Survey's ‘overall' question

Abstract

Similar works

Full text

Available Versions

University of Brighton Research Portal

Kingston University Research Repository