Green, A

Mitchell, C

English

Leeds Beckett Repository

ALT JOURNAL NUMBER 7: WINTER 200918IntroductionThis paper aims to explore our experiences of implementing e-assessment in a sports marketing module at Leeds Met. It will highlight our successes and challenges, and through this, we hope, add to discussion and research in this area. This paper has been jointly written by the academic member of staff responsible for writing and delivering the content for the module and the technical expert who produced the actual e-assessment using the X-stream platform. We therefore hope that our collaborative activities will provide an insight from both an academic and a technical perspective.The UK education sector has recently experienced a shift from the traditional ‘pen and paper’-based assessment paradigm to a more dynamic interactive approach utilising new technology (JISC, 2006). In part, this transformation may have been brought about in a response to the Qualifications and Curriculum Authority (QCA)’s Blueprint for e-Assessment report. This provided targets for the implementation of technology in assessment and underlined the need to “embrace a technological future for assessment” (QCA, 2004). Similarly, the Joint Information Systems Committee (JISC) has further supported this technological aspiration with its Roadmap for e-Assessment (JISC, 2006). This Roadmap created a vision for 2014 and argued that a “pedagogically driven model for e-assessment is needed”. Most recently JISC concluded that e-assessment is more than just an alternative way of doing what we do already. Moreover, it is claimed that e-assessment can enhance the range of skills and knowledge being assessed and provide unprecedented diagnostic information for use by tutors and administrators (JISC, 2007). With this in mind we felt that a key pedagogical benefit of this e-assessment was that it was to add to the ‘rich diet’ of assessment already embedded within the course.In addition to these national agendas concerned with e-assessment, a number of scholars have explored the opportunities and challenges of initiating e-assessment. In particular, Bennett (2002) and Buzzetto-More and Alade (2006) have highlighted the advantages and disadvantages of using e-assessment. These are summarised below:Advantages•  Automated marking dramatically reduces the staff time spent reviewing submissions• Marking is consistent and unbiased•  Instant feedback can be given to students on completion of the assessment•  Questions can include rich-media content such as images/diagrams, animations, audio and video•  Statistical data about the assessment are immediately available for analysis by staff•  Once created, assessments and question-sets are reusable and transferable where appropriate•  Security – login authentication combined with the specific security settings around the assessment ensure that the assessment cannot be accessed until a designated user is in the appropriate location during the appropriate time slot•  This assessment method can be integrated into the institution’s existing Virtual Learning Environment, making it easy to establish the identity of each participant.disadvantages•  The time needed to design and build an effective assessment is ‘front-heavy’•  Many online assessment tools offer a limited range of question types such as multiple-choice, gap-fill, matching activities and calculation•  Open questions involving free text cannot be automatically marked•  Such reliance on technology can leave you susceptible to technical faults/failure•  Large assessments can be limited by the number of PCs available in one/several location/s•  Lack of staff confidence to engage with technology.Traditional forms of assessment often require staff (both academic and support) to focus their efforts predominantly at the post-assessment (i.e. marking) stage. In contrast, e-assessment requires concentrated effort during the initial preparation phase. This period can be intense and time-consuming and it is cited as the principal barrier to the development of institution-wide e-assessment. That said, those institutions that provided staff with dedicated IT support to assist with the development of such assessment were more likely to succeed in the transition to e-assessment (JISC, 2007).X-stream assessments: utilising new technology to assess sports marketing studentsAngela green and Colin MitchellALT JOURNAL NUMBER 7: WINTER 2009 19It has also been noted that while some academics are being innovative with the new technology, relatively few are using e-assessment at a summative level (JISC, 2007). MethodologyIt was decided that we would introduce an e-assessment to an undergraduate Level 3 Strategic Marketing module. A multiple choice question bank was created and constructed using the X-stream platform. We recognise that multiple choice tests are not ideal forms of assessment for Level 3 students. However, it was felt that because this was only a small component part of the overall module assessment, the benefits of being able to provide an individual element of assessment in this format to a large cohort far outweighed the negatives. The cohort (made up of five groups of students, n=101) was presented with the assessment at the same time on the same day. Following completion of the e-assessment, students were given the opportunity to click on the ‘my grade book’ feature to view their submissions. The students could see each of the questions with the correct answer and the answer they had given. They could view their overall grade for this part of the assessment and the contribution that it made to their final module grade. In addition students could see how well they had performed in relation to the overall cohort statistics. After viewing their performance on the module, one group of students (n=36) was asked to complete a short evaluative survey about this new form of e-assessment. The intention had been to question all 101 students who had completed the assessment but practical difficulties on the day made this impossible. As part of the evaluative tool a series of statements was presented to the students and they were asked to rate their agreement with the statements using a 5-point Likert scale. ResultsFigure 1Figure 2Figure 4Figure 6Figure 3Figure 5Online assessments are extremely useful tools for assessmentI prefer individual assessments to group assessmentsI like the fact that I get instant feedback for my workI like the idea of an individual piece of work because it allows me to obtain the grade I am worthNumber of studentsNumber of studentsNumber of studentsThe feedback is more meaningful to me because I get it immediately after submitting my answersThe fact I can compare my grade to others in the room is important to meALT JOURNAL NUMBER 7: WINTER 200920discussionFigure 1 shows that 66% of the students undertaking the e-assessment either agreed or strongly agreed with the statement that “online assessments are extremely useful tools for assessment”. Figure 2 shows that the majority of those questioned appreciated the fact that they were being given an individual assessment. The tutors have noted that increasing student numbers and reduced student:staff ratios have led to many large modules assessing through group work. Many students have raised concerns about this and have requested a move back towards individual assignments. The results presented in Figure 3 further support this argument with 65% preferring individual rather than group assessments. It is also interesting to note that this particular group of students had raised such concerns through their annual course review and therefore appreciated the move to individualised work.Similarly to the findings of Bennett (2002) and Buzzetto-More and Alade (2006), our students welcomed instant feedback for their assessment. Indeed, Figure 4 illustrates that 100% either “agreed” or “strongly agreed” that they liked instant feedback for their work. However, it was surprising to see in Figure 5 that only 36% felt that the feedback was more meaningful because it was given immediately following the completion of the assessment. We felt that the instant nature of this feedback which, in turn, led to academic discussion and debate within the exam room, was of pedagogical benefit to the students, so we expected this response to be higher. There are a number of possible reasons for this situation; perhaps students are not fully aware of the benefits of instant feedback because they experience it less frequently? It could also be that simply being given the correct answer to a question is not enough and more detailed and descriptive feedback is needed.In constructing the e-assessment we thought it would be valuable for students to compare their performance with the rest of the cohort. The results in Figure 6 show that students did not really want this service and, contrary to our expectation, they did not value this element.Within the broader context of developing this e-assessment we found, like JISC (2007), that this development was very front-heavy with much work done at the preparation stage. However, the benefits were seen with the automated marking which clearly saved time at the post-assessment stage. A further noticeable advantage of this e-assessment was the data management aspect. It was simple to transfer cohort grades from the X-stream system to other applications. A further benefit was the ability to see which questions were most difficult for the students, with the instant ability to view each question and find out what percentage had given each of the five alternative responses. This was useful for tutors, enabling them to see where students had difficulty and where changes to the delivery of teaching may be appropriate.There were a number of technical issues with the implementation of this assessment. First, it was difficult to book computing labs with 100+ PCs in one location. As a consequence students were working at different campuses, which made it impractical for the module tutor to be available to all students. Second, because so many students were submitting their assessments at the same time, the network infrastructure was placed under considerable stress. This led to a ‘bottleneck’ for data transfer and some students had to wait a considerable time while their assessment was submitted. Finally, it was recognised that students could access other internet sites while completing the test so we had to ensure invigilators were aware of this possibility and equipped to respond within University examination regulations. ALT JOURNAL NUMBER 7: WINTER 2009 21ConclusionThe results from this research provide an initial indication of our experiences of implementing e-assessment in a sports marketing module and, in particular, the ability to be assessed as an individual while in a large cohort. We recognise that while we have used e-assessment we have simply automated what could have been produced as a pen and paper exercise. In order to move this current educational paradigm forward there is a need to experiment with rich media content such as audio, animation and video clips. For example, we intend to incorporate television commercials in the forthcoming assessment. To make the feedback more meaningful for the students we plan to provide more detailed feedback for each of the questions presented. As part of this we will signpost additional sources for further information. Finally, in terms of internet security, to ensure students cannot access other sites while completing the assessment there should be a locking down of machines via the Internet Protocol number or by using products such as Respondus Lock Down Browser to temporarily disable the ability to access other sites.While our initial research idea is presented in this paper, we have now come to realise that we are actually at the beginning of a long-term collaborative research relationship and have many other avenues relating to assessment, learning, teaching and research that we want to follow. Indeed, since writing this paper we have developed our thinking further around the use of multiple choice questions at Level 3 and this will be published in a forthcoming edition of the INTI Journal. We have also recently initiated a pedagogical project that actively engages our sports marketing students in the creation and development of their own multiple choice questions. Furthermore, as part of this process we have extended our research methodology to include a combination of quantitative and qualitative approaches to data collection. Our modest aim in this paper has been superseded in many different ways and illustrates the value of bringing research into our teaching.ReferencesBennett, R.E. (2002) Inexorable and inevitable: The continuing story of technology and assessment. Journal of Technology, Learning and Assessment 1(1), 1-22. Available at: http://www.jtla.orgBuzzetto-More, N.A. and Alade, A.J. (2006) Best Practices in e-Assessment. Journal of Information Technology Education 5, 251-269.Joint Information Systems Committee (2006) Roadmap for e-Assessment. Milton Keynes: Open University.Joint Information Systems Committee (2007) Effective Practice with e-Assessment: An overview of technologies, policies and practice in further and higher education. Bristol: JISC.Qualifications and Curriculum Authority (2004) Blueprint for e-Assessment. Available at:  http://www.qca.org.uk/qca_11577.aspxAngela GreenSenior Lecturer, Sport DevelopmentColin MitchellLearning Technologist  Technology Enhanced Learning TeamCarnegie Faculty of Sport & Education 

Best Practices in e-Assessment.

Blueprint for e-Assessment. Available at: http://www.qca.org.uk/qca_11577.aspx Angela Green Senior Lecturer,

Effective Practice with e-Assessment: An overview of technologies, policies and practice in further and higher education.

Inexorable and inevitable: The continuing story of technology and assessment.

Roadmap for e-Assessment.

X-stream assessments: utilising new technology to assess sports marketing students

X-stream assessments: utilising new technology to assess sports marketing students

Abstract

Similar works

Full text

Available Versions

Leeds Beckett Repository