Due to a clerical error identical chemistry exam papers were set two years running. In the second year that the paper was used it was distributed as a ‘past paper’ for use as a revision aid, and lecturers worked through all the questions during classes. The students were also provided with model answers. Despite this, the cohort of students that had seen and reviewed the questions (n=50) performed no better than the previous year’s students (n=68) who had no prior knowledge of the questions. After the mistake was discovered the students were given a short survey to assess their reactions to the paper. Most thought the practice paper had helped them revise, furthermore they did not notice that they had already seen the exam paper. The students’ results and reactions shed doubt on the value of working through exam questions in lessons

Lorch, M

Prior, T J

English

University of Leicester Open Journals

Issue 8                                                                                            49 Communication  T J Prior and M Lorch* Department of Chemistry University of Hull Hull  HU6 7RX *m.lorch@hull.ac.uk         It is commonplace for students to request past exam papers as revision aids. ….the number of requests, anecdotal evidence and studies suggests that students think this type of revision exercise is extremely useful     A clerical error provides a unique opportunity to study the effect of previously seen exam questions on exam results   Abstract Due to a clerical error identical chemistry exam papers were set two years running. In the second year that the paper was used it was distributed as a ‘past paper’ for use as a revision aid, and lecturers worked through all the questions during classes. The students were also provided with model answers. Despite this, the cohort of students that had seen and reviewed the questions (n=50) performed no better than the previous year’s students (n=68) who had no prior knowledge of the questions. After the mistake was discovered the students were given a short survey to assess their reactions to the paper. Most thought the practice paper had helped them revise, furthermore they did not notice that they had already seen the exam paper. The students’ results and reactions shed doubt on the value of working through exam questions in lessons.  Introduction It is commonplace for students to request past exam papers as revision aids. They also frequently ask teachers and lecturers to work though these past questions within lessons and/or provide model answers. Judging from the number of requests, anecdotal evidence and studies suggests that students think this type of revision exercise is extremely useful1.   Since exams have long been used as a means of assessment it is not surprising that there has been much research into the usefulness of practicing exam questions2. The general consensus is that students who repeatedly practice questions that are similar to those appearing in the final exam achieve better results than those who do not practice1,2d,3. This hardly seems surprising; after all we all know the old adage ‘practice makes perfect’. But it is heartening to know that research backs up what our common sense tells us to be true. And so we tend to grant our students’ wishes and work through exam papers with them.   However, in general these studies follow look at the effect of formative assessments using carefully constructed situations where students are allowed to practice multiple questions. In practice this scenario does not always occur and more often we simply work through exam questions in front of the students. There is evidence to suggest that this strategy of, essentially, giving students past papers with model answers is not helpful; students mistake understanding how the teacher arrived at the correct answer for understanding how to arrive at it2c. Instead it is much more useful to allow students to practice the exam questions and then review the answers in lessons2c.   We have been presented with a unique opportunity, arising from a clerical error, to further assess the effect of reviewing exams questions on the achievements of students.          A clerical error provides a unique opportunity to study the effect of previously seen exam questions 50                                                                              New Directions Communication           Methodology An exam paper was set for foundation year chemistry students. It used short answers and problem solving to test knowledge, application and understanding.  The past paper had been used the previous year, was in the public domain and freely available to students months before the exam. The paper was also distributed in class and all lecturers worked through the past paper in lectures, thus providing students with model answers. The exam for the course was sat approximately four weeks after the end of the lecture course.  One month after the students had completed the exam they were asked to complete a survey to judge their perceptions of the exam questions and how useful they perceived the revision lectures to be. The survey consisted of the four questions/statements with a 5-point answer scale plus a free answer section (Appendix 1).   The students’ exam results were compared to the previous year’s cohort (who completed the same course with the same style of revision lectures but without having seen their final exam paper). Results in other assessments where used to compare the abilities of the two year groups.  Lectures were well attended with 86% of the cohort regularly present.  Furthermore, 82% of the students accessed the virtual learning environment web pages where the past papers and model answers were hosted. Thus, the vast majority of students were directly provided with or accessed the past papers and model answers, via either lectures or e-learning resources.  Exam results The cohort that sat a previously unseen exam paper achieved a mean mark of 59.3% with a standard deviation (SD) of 20.7 (n=68). In the following year, when the students had already seen their final exam paper, the mean mark was 56.2% with a SD of 23.7% (n=50). The distribution of the marks is similar for the two cohorts (Figure 1). In other forms of assessment on the same chemistry module the mean marks were 80.92% and 81.6% respectively.   Survey results The survey was distributed to 30 students out of the cohort of 50, all those that received the survey returned it. The students that responded were almost unanimous in their belief that reviewing past papers and access to model answers helps them to revise (Figure 2). Only one student thought that going over past papers was not useful and none thought that access to model answers was of no use. There was similar agreement in the comparison of difficulty of the paper they sat with the paper they practised; 94% reported that they were about the same difficulty. On the subject the similarity of the practice and real exam papers, all recognised a similarity, but only one student noticed that the papers were identical. Interestingly one student reported in the free answer section “One question was identical, the others were very similar”.  Discussion The common sense expectation was that the students who had already seen and practiced their final exam paper would fare significantly better than those that, the year before, sat the paper blind. The students’ results do not bear this out. Instead the similarity between the two cohorts’ marks is striking and if anything those that had already been exposed to the paper performed slightly worse. Judging from other assessments in the module, which took the form of laboratory work and open book short answer questions, the two cohorts were of similar ability. Therefore the exam results could not be explained by one cohort being of a lower ability and then having their mark boosted by prior exposure to the exam paper.   Furthermore, the students’ performance in the exam runs contrary to their perceptions of the exam paper. They were almost unanimous in their belief that reviewing the past papers and model answers helped them prepare for the exam, but this is not born out by their marks. Even more striking is the fact that all but one of the students failed to notice that they had already seen the exam paper. This fact alone calls into question the usefulness of reviewing past papers with students. Together with previous studies2c our observations suggests that for the majority of students, simple delivery of material and the revision by presentation of model answers is not effective.  Figure 1: Student exam performance. The cohort that had seen the paper prior to sitting the exam achieved a mean mark of 56.2% (black) compared to 59.3% for the cohort who had not seen the exam paper (grey). A clerical error provides a unique opportunity to study the effect of previously seen exam questions Issue 8                                                                                            51 Communication  It is possible that the students’ exam results are a reflection of them having studied the past paper, concluded that the same questions would not appear again and then taken a tactical decision not to revise these topics. However, the fact that they did not recognise the exam questions suggest that this is not the case as we would speculate that any student who had spent any time question spotting would have, in all likelihood, noticed that the same exam paper had been used two years running.   This accidental study has offered an insight into the usefulness of a common revision strategy. Students, lecturers and teachers alike think that reviewing past papers is an effective way to teach material and to familiarise the class with the format of the paper. The perception is that this translates into improved exam results. Prior research from a variety of disciplines appears to back up this belief1-3. However, these studies followed students’ performance after repeated exposure to practice questions. But this thorough approach to formative assessment is not always practiced, instead teachers and lecturers often may pay lip service to the strategy and, as we described did, spend a relatively short time familiarising students with the format of exam questions. However, the best way for students to become familiar with exam questions and their format is to practice them and then review the answers2c. But this requires students to engage in non-assessed work, which, as is often noted, can often be difficult4; a point most succinctly put by Rust “if work does not have marks attached many students will either not do it at all or only do it in a perfunctory way” 5. And in our experience this seems to be even more the case when the students know that the answers to the work will be provided in future lessons.   So the challenge is two-fold; change our (and others) assisted revision methods and encourage the students to take part in them.  Figure 2: Student responses to survey querying their perception of the exam paper and revision strategy. The survey was delivered to 30 of the 50 students in the cohort, all of whom completed and returned it. A clerical error provides a unique opportunity to study the effect of previously seen exam questions 52                                                                              New Directions Communication           We hope that the clerical error that has led to this study will serve to address these two challenges. The striking nature of the error can be used to illustrate to lecturers the discrepancy between perceived usefulness of model/past papers answers against their actual worth and so inspire them to provide engaging formative assignment. Then by presenting the story of our ‘mixed up’ exam papers to students as an amusing anecdote we may be able to highlight that simply listening to their lecturer work through an exam questions is not enough and they also need to engage in non-assessed work.  We now intend to implement more formative assessments as part of our own revision lectures and use this story to encourage student engage with them. We will, of course, follow the results of these changes closely with the intent of reporting the outcome.  We hope that by disseminating the results of this serendipitous study students and teachers may review the way they use practice exam questions to promote student revision.   References 1. Bol, L.; Hacker, D. J., A comparison of the effects of practice tests and traditional review on performance and calibration. Journal of Experimental Education 2001, 69 (2), 133-151. 2. (a) Fakcharoenphol, W.; Potter, E.; Stelzer, T., What students learn when studying physics practice exam problems. Physical Review Special Topics-Physics Education Research 2011, 7 (1); (b) Knaus, K. J.; Murphy, K. L.; Holme, T. A., Designing Chemistry Practice Exams for Enhanced Benefits An Instrument for Comparing Performance and Mental Effort Measures. Journal of Chemical Education 2009, 86 (7), 827-832; (c) Balch, W. R., Practice versus review exams and final exam Performance. Teaching of Psychology 1998, 25 (3), 181-185; (d) Kulik, J. A.; Kulik, C. L. C.; Bangert, R. L., Effects of Practice on Aptitude and Achievement-Test Scores. American Educational Research Journal 1984, 21 (2), 435-447. 3. Hacker, D. J.; Bol, L.; Horgan, D. D.; Rakow, E. A., Test prediction and performance in a classroom context. Journal of Educational Psychology 2000, 92 (1), 160-170. 4. (a) Weaver, M., Do students value feedback? Student perceptions of tutors’ written responses. Assessment & Evaluation in Higher Education 2006, 31 (3), 379–394; (b) Bloxham, S. a. B., P, Developing Effective Assessment in Higher Education. Open University Press, McGraw-Hill.: Maidenhead 2007. 5. Rust, C., The Impact of Assessment on Student Learning: How Can the Research Literature Practically Help to Inform the Development of Departmental Assessment Strategies and Learner Centred Assessment Practices? Active Learning in Higher Education 2002, 3 (2), 145-158.   Appendix 1:  Student survey. A clerical error provides a unique opportunity to study the effect of previously seen exam questions 

A clerical error provides a unique opportunity to study the effect of previously seen exam questions on exam results

https://journals.le.ac.uk/ojs1/index.php/new-directions/article/download/495/492

A clerical error provides a unique opportunity to study the effect of previously seen exam questions on exam results

Abstract

Similar works

Full text

Available Versions

University of Leicester Open Journals